AIOps incident silencing

Reduce alert noise, prevent alert fatigue, and improve your team’s performance. We leverage machine learning to estimate how likely is an incident worth notifying you about. Silence incidents that are unlikely to be important with a dedicated escalation policy.

Which incidents get silenced?

We train a model for each team based on hundreds of signals from past incidents to estimate how likely an incident is worth notifying about.

The most important signals we use in incident silencing:

• Repetition: Monitors and alerts with many repeated incidents contribute to noisy alerting and make you less likely to notice real incidents.
• Incident length & No acknowledgment: Short incidents that auto-resolve before they are acknowledged can be addressed later and do not require the immediate attention of your on-call engineers.
• Ignored calls & Unopened emails: When you miss calls and unopened emails start cluttering your inbox, the important incidents can get lost in the noise.

Important incidents

We use over a hundred signals to recognize similar incidents to ensure we correctly recognize new important incidents to notify you about.

Examples of important incidents that won't be silenced:

• New error cause: Monitored site returning a 500 code after many silenced keyword errors.
• High alert value: Memory alert spiking to 95% after commonly getting silenced with a much lower allocated memory percentage.
• Completely new alert: Any alert that hasn’t been triggered before.
• Unusually long incident: Incident that usually gets silenced and auto-resolves in a few minutes, but this time, it's still ongoing after 15 minutes.

We use the length of an incident to differentiate between noisy and important incidents. Silenced incidents will be escalated with their original escalation if they take longer than usual to resolve.

Fine-tune your incident silencing

Rate the importance of incoming alerts and improve your silencing model with every incident. Give feedback in notification emails or when viewing an incident in Uptime.



Enable & configure incident silencing

Enable incident silencing for your team:

• Navigate to: Settings → Teams → Your team → Features
• Enable Incident silencing toggle

Adjust how confident the model needs to be to silence an incident:

• Higher percentage: More incidents will be silenced. Repetitive incidents will get silenced sooner.
• Lower percentage: Fewer incidents will be silenced. Incident silencing will be more conservative.

You can adjust the threshold later at any time to fine-tune the sensitivity of incident silencing.

Silenced incidents will be escalated using a dedicated escalation policy. Create a new escalation policy for silencing incidents:

• Silence calls: Create an escalation policy that doesn't call you but still notifies you using emails.
• Silence all notifications: Create an escalation policy with no steps to receive no notifications for silenced incidents.
• Custom time-based policy: Create an escalation policy that calls you during work hours and notifies you using emails at night.

Model training

Want to know more about how we train the silencing models?

• We train the silencing models every two hours.
• Your team needs at least five resolved incidents. Otherwise, the model won't be trained.
• It can take up to a few hours before you see the influence of your incident feedback.
• Incidents will be escalated as usual if there are not enough similar incidents yet for incident silencing.