An alert rule often begins with a graph and a question: where should the line go? CPU at 80 percent? Queue depth at 500? Error rate at 2 percent? The answer can look precise while the operational decision remains undefined.

Start one step earlier. If this signal fires, what can a person do that should happen before the next normal review? If nobody can name the action, owner, and time limit, changing the threshold will not make it a good page. It may not need to be an alert at all.

Prometheus's alerting guidance recommends few alerts, a bias toward symptoms, and no pages where there is nothing to do. That is a useful operating constraint, not an argument against measurement. Keep the measurements. Be selective about which ones interrupt a human.

Write the response before the rule

An alert proposal should fit on a small decision card:

Question	What to record
What is happening?	The user-visible symptom, approaching risk, or failed safety mechanism
Who responds?	A team and a route that work at the hour the notification fires
What do they check first?	A current dashboard, request sample, job state, or other trustworthy evidence
What can they do?	A bounded mitigation, escalation, or decision with known authority
How quickly must they act?	The time available before material impact grows
What proves the response helped?	A recovery signal and a way to detect a harmful side effect

“Investigate” is usually too vague to count as the action. Investigation can be the first step, but the alert still needs a reason to interrupt someone now. If the only result is “look at this tomorrow,” route it to tomorrow's work.

The distinction matters especially with infrastructure metrics. High CPU can be a normal batch run, a harmless burst, or a symptom of a service that is dropping requests. It is excellent diagnostic data. Whether it deserves a page depends on the effect and the available intervention.

Page, ticket, or observe

Use three possible destinations rather than treating every interesting graph as a paging candidate.

Page when there is material current or imminent impact, a response must begin before the next staffed review, and the on-call person has a useful action or escalation path. A page is a request to spend someone's attention immediately. It should carry enough context to justify that cost.

Ticket when the signal identifies real work but waiting for the next staffed period does not materially worsen the outcome. A slow increase in capacity use may merit a ticket with a due date and owner. If the projection changes enough that the safe response window closes, a separate page condition may become appropriate.

Observe when the measurement helps diagnosis, trend analysis, or a later decision but does not yet establish a response. Put it on a dashboard, include it in an incident view, or analyze it during a planned review. Observation is not neglect. It is an explicit decision about how the data will be used.

The Google SRE Workbook's chapter on SLO alerting distinguishes faster, consequential error-budget consumption that warrants paging from slower consumption suitable for a ticket. Its example numbers illustrate a method; they are not thresholds to paste into an unrelated service. Set the urgency from your own service objective, traffic, and staffing.

A hypothetical queue-service alert review

Imagine an internal job queue with a promised completion window. The team proposes a page when queue depth exceeds 500. That may be useful, but 500 alone says little. The queue could usually drain 2,000 jobs in minutes, or 100 jobs could represent several hours of stalled work.

First, define the reader-facing symptom: jobs are completing later than the service's agreed window. Then inspect signals that represent it, such as age of the oldest eligible job, completion latency, and the share of jobs missing the window. Queue depth remains valuable diagnostic context.

Now ask what the responder can do. In this hypothetical service, the approved actions are to check worker health and job admission, pause a known low-priority producer if its owner has authorized that mitigation, and escalate to the queue-service owner when the cause is outside the on-call team's scope. A runbook names the supported checks and their stop conditions. The alert needs to point to that response and route to someone who can initiate it.

Suppose queue age is rising but remains comfortably inside the completion window, workers are healthy, and there is no approved intervention yet. That is a trend to observe or a capacity ticket. Suppose eligible jobs have begun missing the window and a supported mitigation can prevent further misses. That is a stronger paging case. If the only available action is an untested production change, the page still needs a safe escalation route; it does not authorize improvisation.

This example is deliberately qualitative. A real team must define the completion objective, measurement lag, response time, and approved actions before choosing numeric thresholds.

Tune the threshold after the destination is clear

Once the response is defined, a threshold becomes a timing tool. It should give the responder enough time to act without firing on every brief, harmless fluctuation.

Review these inputs together:

• Impact: Is the condition already hurting users, or is it a credible leading indicator with a short remaining response window?
• Duration: Would a short for period or a longer evaluation window filter transients without hiding a genuine incident?
• Traffic and sampling: Does a small denominator make percentages jump around? Could delayed data make the alert late?
• Ownership: Does the route reach the team that can act at night, on weekends, and during handoffs?
• Recovery: Does the condition clear when the service recovers, or will a responder need to guess whether a stale alarm is still meaningful?
• Failure of monitoring: If the data source disappears, how will the team distinguish healthy silence from broken telemetry?

The Prometheus guidance explicitly allows slack for small blips and recommends separate monitoring of the monitoring path. Those are both consequences of the same design goal: the notification should reflect a real decision, not merely a number crossing a line.

Avoid hiding a bad page by repeatedly relaxing its threshold. That can reduce noise while preserving the underlying ambiguity. Ask whether the response is still urgent and possible. If it is, improve the signal and timing. If it is not, change the destination.

Test the message, route, and outcome

An alert definition is incomplete until someone can use the fired notification. Review the rendered message, not just the query. It should identify the affected service and environment, explain the symptom in plain language, link to the right evidence, and identify the first supported action or escalation.

Test routing in a controlled way. Confirm that the intended team receives the notification, duplicates are grouped sensibly, maintenance does not create avoidable noise, and a missing owner does not send the page into a dead channel. Then review actual firings. For each one, record whether someone acted, whether the action mattered, and whether the notification arrived early enough.

This is where alert quality becomes a feedback loop. A false page may indicate a noisy signal, but it may also reveal an undefined objective, stale ownership, or a response nobody is permitted to take. A missed incident may show that the signal was too slow or too detached from user experience. Update the decision card and the rule together.

The companion article A Runbook Needs a Stop Condition covers the next problem: how an authorized responder knows when the procedure no longer applies. Alert design comes first because it decides whether to summon that responder at all.

The practical test

Before adding a page, ask the proposed owner to finish this sentence: “When this fires, I will check this evidence and take this supported action within this time, because waiting until the next review would make this impact worse.”

If that sentence cannot be completed, keep measuring and investigate the operating gap. Build a ticket if there is deferred work. Write the response and ownership before returning to the threshold. That is how an alert becomes a useful request for human judgment instead of an expensive way to display a graph.

For more practical engineering and operations writing, visit Slaptijack.