Stop guessing alarm thresholds: measure them, then replay an outage

Every alarm threshold I have ever seen argued about in a PR was a round number. 80%. 100ms. 60 connections. They feel principled. They are almost always somebody's first guess, frozen.

A guessed threshold fails in one of two directions, and both are expensive:

• Too tight — it flaps. People mute the channel. Now you have no alerting and a false sense that you do.
• Too loose — it never fires, and you find out during the postmortem that the alarm was technically present the whole time.

Here is the process I now use instead, from a recent engagement where I had to put 23 alarms on a system that had none. Identifiers are changed; the numbers are as measured.

Rule 1: no threshold without a measurement

Before writing a single alarm, I wrote the thing that measures what "normal" looks like. It is about 120 lines of shell and it earns its keep on day one.

summarise() { # label, namespace, dim-name, dim-value, metric, statistic
  local out
  out="$(aws cloudwatch get-metric-statistics \
    --namespace "$2" --metric-name "$5" --dimensions Name="$3",Value="$4" \
    --start-time "$FROM" --end-time "$TO" --period "$PERIOD" --statistics "$6" \
    --query "Datapoints[].$6" --output text 2>&1)" || true

  [[ -z "$out" ]] && { printf '  %-34s %s\n' "$1" "(no data)"; return; }

  echo "$out" | tr '\t' '\n' | sort -g | awk -v L="$1" '
    { a[NR] = $1 }
    END {
      i50 = int(NR * 0.50); if (i50 < 1) i50 = 1
      i95 = int(NR * 0.95); if (i95 < 1) i95 = 1
      printf "  %-34s min=%-12.4g p50=%-12.4g p95=%-12.4g max=%-12.4g n=%d\n", \
             L, a[1], a[i50], a[i95], a[NR], NR
    }'
}

summarise "CPUCreditBalance (Min)"     AWS/RDS DBInstanceIdentifier "$RDS_ID" CPUCreditBalance    Minimum
summarise "DatabaseConnections (Max)"  AWS/RDS DBInstanceIdentifier "$RDS_ID" DatabaseConnections Maximum
summarise "WriteLatency (Max)"         AWS/RDS DBInstanceIdentifier "$RDS_ID" WriteLatency        Maximum
summarise "EBSIOBalance% (Min)"        AWS/RDS DBInstanceIdentifier "$RDS_ID" "EBSIOBalance%"     Minimum

min / p50 / p95 / max per metric over two weeks. That is the entire output. The script then prints the currently declared thresholds next to it, so the comparison is one screen:

echo "Declared thresholds, for comparison:"
jq -r '.alarms[] | "  \(.suffix)\t\(.metric_name) \(.comparison_operator) \(.threshold)"' "$SPEC" \
  | column -t -s $'\t'

One gotcha that silently returns nothing

CloudWatch returns at most 1440 datapoints per call. Ask for 30 days at 5-minute resolution and you get an empty array, not an error. Pick the finest period that stays under the ceiling:

PERIOD=300
for candidate in 300 900 3600 21600; do
  PERIOD=$candidate
  [[ $(( SPAN_S / PERIOD )) -le 1400 ]] && break
done

Two thresholds the measurement corrected

Both of these were my own guesses, written before I ran the script, and both were wrong in a way that would have caused a specific, predictable problem.

Note that the correction went in opposite directions. There is no safe default bias — "be conservative" would have made the connections alarm worse and the latency alarm better. Only the data tells you which way to move.

The measured baseline then gets written into the spec file next to the threshold, so the next person to touch it knows what it was derived from:

{
  "suffix": "rds-conns",
  "threshold": 45,
  "baseline": "14d to 2026-07-29: p95 21, max 23. Pool ceiling is 60.",
  "comparison_operator": "GreaterThanThreshold"
}

An undocumented threshold is a threshold that will be nudged by someone chasing a flap at 3am, and never nudged back.

Rule 2: evaluation_periods is half the threshold

A threshold is two numbers, not one, and people only argue about the first. These two alarms are wildly different despite sharing a value:

{ "threshold": 85, "period": 300, "evaluation_periods": 1, "datapoints_to_alarm": 1 }
{ "threshold": 85, "period": 300, "evaluation_periods": 3, "datapoints_to_alarm": 3 }

The first fires on any five-minute spike. The second needs fifteen minutes of sustained pressure. For CPU on a burstable instance, the first is pure noise — the whole point of a burstable instance is that it spikes.

The rule I settled on:

• Fast signals (status checks, external availability) — short windows, datapoints_to_alarm below evaluation_periods so one flaky sample doesn't page but two do. 3 of 3 for a probe means a real outage waits 15 minutes; 2 of 3 cuts that to 10 while still absorbing a single blip.
• Slow, cumulative signals (credit balances, disk fill, connection counts) — full windows, 3 of 3. These never change fast, so a single-sample alarm is measuring jitter.
• Absence signals (no job has run) — the period is the threshold. More on that below.

Rule 3: replay a real incident before you trust it

This is the step almost nobody does, and it is the one that turns "we have alarms" into "we have alarms that work."

Pick an outage that already happened. Pull the metric history for that exact window. Ask: would this alarm have fired, and how long before the customer noticed?

The system I was working on had had a database host failure. Replaying the window against the proposed burst-credit alarm:

./baseline.sh --env production \
  --rds-id db-prod-old \
  --from 2026-07-29T04:00:00Z --to 2026-07-29T09:00:00Z

CPUCreditBalance (Min)   18.0 → 7.6 → 0.0

Credits crossed the proposed threshold of 75 more than two and a half hours before the host actually failed. That is not a marginal win; that is the difference between a scheduled intervention and an outage.

A second replay, against a different incident on the other environment, showed burst credits dipping to 36.6 during a webhook replay flood — comfortably under the threshold, so that alarm would have caught it too.

A subtle trap when replaying: CloudWatch keeps metrics under the resource identifier that was in use when they were emitted. After a restore-and-rename, the incident data still lives under the old identifier. If your replay returns "(no data)" for a window you know was eventful, that is usually why — hence the --rds-id override in the script above.

Replay is also how you find alarms that are useless without saying so. If an alarm would not have fired during any incident you have actually had, either the threshold is wrong or the metric is not measuring your failure mode. Both are worth knowing on day one rather than during the next incident.

Rule 4: some alarms should treat silence as failure

Most alarms should ignore missing data. A few must treat it as breaching, and choosing correctly is the difference between a useful alarm and a permanently yellow one.

{
  "suffix": "jobs-dead",
  "description": "No background job executed for 2 consecutive hours.",
  "metric_name": "JobsExecuted",
  "statistic": "Sum",
  "period": 3600,
  "evaluation_periods": 2,
  "comparison_operator": "LessThanThreshold",
  "threshold": 1,
  "treat_missing_data": "breaching"
}

Here silence is the failure being detected. If the worker fleet dies, no job executes, no metric datapoint is published, and an alarm set to missing would sit in INSUFFICIENT_DATA looking calm forever. breaching is correct.

But treat_missing_data: breaching has a nasty cold-start behaviour that will false-page you the moment you create the alarm — I wrote that one up separately in the alarm that pages on its first day.

Contrast with a latency alarm, where missing data means "no traffic", which is not an emergency:

{ "metric_name": "LatencyMs", "threshold": 2000, "treat_missing_data": "missing" }

Rule 5: re-measure after any resize

Thresholds are relative to hardware. The instant someone changes an instance class, every threshold derived from the old one is a guess again — and the change usually happens during an incident, when nobody is thinking about alarm files.

So the baseline script has a docstring that says exactly this, and the runbook says it again:

# Run this before changing any threshold, and after any instance resize
# or class change.

It is worth wiring into whatever your team actually reads. A comment in a script nobody runs is decoration.

The checklist

1. Measure min/p50/p95/max over at least two weeks before writing a number.
2. Write the measurement into the spec next to the threshold.
3. Set evaluation_periods deliberately — it is half the alarm.
4. Replay a real incident window and confirm the alarm fires with useful lead time.
5. Decide treat_missing_data per alarm; silence is sometimes the signal.
6. Re-measure after any resize.

Steps 1 and 4 are the ones that get skipped, and they are the two that decide whether the alarm is real.

Once your thresholds are defensible, the next question is what they are allowed to cost you. Put your numbers into the SLO & Error Budget Calculator and see how much downtime your target actually buys.