Screening alerts before creating incidents

Not all alerts signify a critical incident. Many are informational or can be resolved without waking up an on-call engineer. To combat alert fatigue and ensure your team only focuses on what truly matters, you can set up a workflow to screen incoming alerts before deciding to declare a major incident.

This guide demonstrates a two-tiered approach using separate teams and escalation policies to create a buffer for alert triage.

The core of this setup involves two distinct teams, each with its own escalation policy:

  1. Alerts team: This team acts as a catch-all for every notification from your monitors and integrations. It uses a silent escalation policy that creates an incident in Better Stack but doesn't page anyone.
  2. Incidents team: This team is for confirmed, high-priority issues. It uses a standard escalation policy that pages the on-call engineer and triggers your full incident response process.

This setup allows your team to review a stream of non-urgent alerts and manually escalate only the critical ones into actionable incidents.

Step 1: Create the Alerts team

This team will be the first destination for all incoming alerts, acting as your screening layer.

  1. Go to Settings → Teams in your Better Stack account.
  2. Create a new team and name it Alerts.
  3. Assign team members that will be responsible for triaging alerts.

Step 2: Create a silent escalation policy

You'll need an escalation policy that logs an incident without any disruptive alerts.

  1. In the Alerts team, go to Escalation policies and click Create escalation policy.
  2. Name it New Alert Policy.
  3. Remove all default escalation steps by clicking the x on each step. The final policy should have no steps.
  4. Click Create escalation policy.

An escalation policy named "New Alert Policy" with no escalation steps.

Send alerts to a triage channel

You can add a single, non-intrusive notification to your silent policy, like a message to a dedicated Slack channel (e.g., #alerts-triage). This gives your team a central place to see incoming alerts without any noise.

Step 3: Create an escalation policy in your Incidents team

This team and its policy are for when an alert is confirmed to be a real incident. You can use your main team for this policy.

  1. In the team responsible for resolving the incidents, navigate to Escalation policies and click Create escalation policy.
  2. Name this policy something clear and actionable, like Declare an Incident or Page On-call.
  3. Configure the escalation steps for this policy to alert the on-call person for the Incidents team. Use steps that include calls, SMS, or critical push notifications.
  4. Click Create escalation policy.

An escalation policy named "Declare an Incident" with steps to call and page the on-call person.

Step 4: The screening and escalation workflow

With the setup complete, your team can now follow this simple workflow:

  1. In your Alerts, set escalation to Alerts team -> New Alert Policy. No one will be paged.
  2. A team member in the Alerts team reviews the incoming alerts on the Incidents page.
  3. When an alert requires immediate attention, open its incident detail page.
  4. Click the Escalate to button in the top-right corner.
  5. In the modal that appears, select the Declare an Incident escalation policy.
  6. Click Escalate.

The incident detail page with the "Escalate to" button highlighted.

This action triggers the Declare an Incident policy, paging the on-call engineer from the Incidents team and starting your incident response process.

Summary

By separating alerts from incidents, you empower your team to manage notifications effectively. This two-team setup reduces noise, prevents alert fatigue, and creates a clear, intentional process for declaring and responding to critical incidents.