Not all alerts signify a critical incident. Many are informational or can be resolved without waking up an on-call engineer. To combat alert fatigue and ensure your team only focuses on what truly matters, you can set up a workflow to screen incoming alerts before deciding to declare a major incident.
This guide demonstrates a two-tiered approach using separate teams and escalation policies to create a buffer for alert triage.
Recommended Setup
The core of this setup involves two distinct teams, each with its own escalation policy:
Alerts team: This team acts as a catch-all for every notification from your monitors and integrations. It uses a silent escalation policy that creates an incident in Better Stack but doesn't page anyone.
Incidents team: This team is for confirmed, high-priority issues. It uses a standard escalation policy that pages the on-call engineer and triggers your full incident response process.
This setup allows your team to review a stream of non-urgent alerts and manually escalate only the critical ones into actionable incidents.
Step 1: Create the Alerts team
This team will be the first destination for all incoming alerts, acting as your screening layer.
Assign team members that will be responsible for triaging alerts.
Step 2: Create a silent escalation policy
You'll need an escalation policy that logs an incident without any disruptive alerts.
In the Alerts team, go to Escalation policies and click Create escalation policy.
Name it New Alert Policy.
Remove all default escalation steps by clicking the x on each step. The final policy should have no steps.
Click Create escalation policy.
Send alerts to a triage channel
You can add a single, non-intrusive notification to your silent policy, like a message to a dedicated Slack channel (e.g., #alerts-triage). This gives your team a central place to see incoming alerts without any noise.
Step 3: Create an escalation policy in your Incidents team
This team and its policy are for when an alert is confirmed to be a real incident. You can use your main team for this policy.
In the team responsible for resolving the incidents, navigate to Escalation policies and click Create escalation policy.
Name this policy something clear and actionable, like Declare an Incident or Page On-call.
Configure the escalation steps for this policy to alert the on-call person for the Incidents team. Use steps that include calls, SMS, or critical push notifications.
Click Create escalation policy.
Step 4: The screening and escalation workflow
With the setup complete, your team can now follow this simple workflow:
In your Alerts, set escalation to Alerts team -> New Alert Policy. No one will be paged.
A team member in the Alerts team reviews the incoming alerts on the Incidents page.
When an alert requires immediate attention, open its incident detail page.
Click the Escalate to button in the top-right corner.
In the modal that appears, select the Declare an Incident escalation policy.
Click Escalate.
This action triggers the Declare an Incident policy, paging the on-call engineer from the Incidents team and starting your incident response process.
Summary
By separating alerts from incidents, you empower your team to manage notifications effectively. This two-team setup reduces noise, prevents alert fatigue, and creates a clear, intentional process for declaring and responding to critical incidents.