The alarm that pages on its first day — and the fix that broke reconciliation

I rolled out a new alert pipeline. Within a minute, two critical alarms fired and mentioned the channel. Both endpoints they watched were serving HTTP 200 in about 100 milliseconds. Nothing was wrong.

They self-resolved in roughly eight minutes.

The reason is written in the alarm's own state reason, which is the most useful error message AWS produces and one I had never actually read:

1 datapoint was received for 3 periods and 2 missing datapoints
were treated as [Breaching]

That is the whole bug, stated by the system that caused it. This post is what it means, the guard that fixes it, and the second bug that the guard introduced — because the follow-up is the more interesting lesson.

Identifiers are changed; this was a client engagement.

Why a healthy alarm fires the moment it exists

Some alarms must treat missing data as breaching. If you are alarming on absence — no background job ran, the external prober got no answer — then silence is the failure. treat_missing_data: missing on those alarms produces a permanently calm INSUFFICIENT_DATA state during a total outage, which is the exact opposite of what you want.

{
  "suffix": "probe-unavailable",
  "metric_name": "Available",
  "statistic": "Minimum",
  "period": 300,
  "evaluation_periods": 3,
  "datapoints_to_alarm": 2,
  "comparison_operator": "LessThanThreshold",
  "threshold": 1,
  "treat_missing_data": "breaching"
}

Correct. And also a landmine at creation time.

When CloudWatch creates that alarm it immediately evaluates its window. The window is three periods wide. The metric has existed for two minutes, so exactly one datapoint is in it. The other two periods are missing — and you told CloudWatch to treat missing as breaching. Two of three breaching, datapoints_to_alarm is 2, so: ALARM, at critical, with an @here.

Eight minutes later the window has filled with real datapoints and it flips to OK on its own.

This is not a bug in CloudWatch. It is the documented interaction of two settings that are individually correct. It only shows up on the very first evaluation, which means it is invisible in every environment where the alarm already exists — you only ever meet it during a rollout, which is precisely when a false critical page does the most damage to trust in a brand-new channel.

The guard: demand a full window before creating

The fix is to not create such an alarm until its metric has produced enough datapoints to fill an entire evaluation window.

# Has this metric produced at least `min_points` datapoints at its own resolution?
#
# Requiring a FULL evaluation window rather than a single datapoint is deliberate.
# Several of these alarms treat missing data as breaching, so an alarm created
# after one datapoint has arrived evaluates the other periods as breaching and
# fires immediately.
metric_has_data() { # namespace, metric-name, dimensions-json, period, min-points
  local ns="$1" name="$2" dims="$3" period="${4:-300}" min="${5:-1}" from to n lookback

  # Look back far enough to actually contain min_points windows, with headroom.
  lookback=$(( period * (min + 2) ))
  [[ $lookback -lt 10800 ]] && lookback=10800

  from="$(date -u -v-"${lookback}"S +%Y-%m-%dT%H:%M:%SZ 2>/dev/null \
       || date -u -d "${lookback} seconds ago" +%Y-%m-%dT%H:%M:%SZ)"
  to="$(date -u +%Y-%m-%dT%H:%M:%SZ)"

  n="$(aws cloudwatch get-metric-statistics --namespace "$ns" --metric-name "$name" \
    --dimensions "$dims" --start-time "$from" --end-time "$to" --period "$period" \
    --statistics Sum --query 'length(Datapoints)' --output text 2>/dev/null || echo 0)"

  [[ "$n" =~ ^[0-9]+$ ]] || n=0
  [[ "$n" -ge "$min" ]]
}

Two details in there that took a round of debugging each.

The lookback has to scale with the period. An hourly metric with two evaluation periods needs at least four hours of history to answer the question. A fixed three-hour lookback quietly reports "no data" forever for anything coarser than hourly.

date syntax differs between BSD and GNU. The -v form is macOS, the -d form is Linux. If your operator runs the script from a laptop and CI runs it from a container, you need both, and the || fallback is the least annoying way to get it.

Alarms that need the gate opt in from the spec file, so it is visible next to the alarm rather than buried in the script:

{
  "suffix": "jobs-dead",
  "namespace": "App/production",
  "metric_name": "JobsExecuted",
  "treat_missing_data": "breaching",
  "requires_metric_data": true
}

And the provisioner reports what it deferred, rather than silently doing less than you asked:

warn  held back until their data source produces datapoints:
        app-prod-jobs-dead
        app-prod-host-disk
      Run ./install-agent.sh --env production if you have not, wait ~10
      minutes, then re-run this script to create them.

An alarm you meant to create and did not is a gap. A gap you know about is a task.

The second bug: the guard ate reconciliation

Two days later I was wiring up one more alarm and noticed the provisioner printing this on every run:

--   app-staging-jobs-dead — App/staging/JobsExecuted has not filled
     3 evaluation periods yet

...while that alarm was already live, already OK, and had been for hours.

The metric is hourly. The gate wants three evaluation periods. So the gate takes three hours to satisfy, and every run inside that window skipped an alarm that already existed and was working perfectly.

Annoying. But the actual damage is one level deeper.

Why a cosmetic skip was a real defect

The whole design rests on a three-part contract:

• alarms.<env>.json is the source of truth.
• verify.sh surfaces divergence between the file and reality.
• provision.sh reconciles that divergence.

An alarm provision.sh refuses to touch cannot be reconciled. Someone nudges a threshold in the console; the drift report flags it; you run the provisioner to fix it; the provisioner skips it; both scripts report success. The console edit survives indefinitely and the tooling tells you everything is fine.

That is worse than having no drift detection, because a drift report that says "no drift" is load-bearing. People stop checking manually.

The fix

The gate's only job is to avoid creating an alarm into an immediate false page. It has nothing to say about an alarm that already exists — that one has already survived its cold start.

# Snapshot which alarms already exist. The data-source gate below must apply only
# to CREATION: an alarm that already exists has to be reconciled on every run, or
# a console edit to it would be skipped forever and there would be nothing for
# verify.sh's drift report to reconcile against.
EXISTING_ALARMS="$(aws cloudwatch describe-alarms --alarm-name-prefix "$ALARM_PREFIX" \
  --query 'MetricAlarms[].AlarmName' --output text 2>/dev/null | tr '\t' '\n')"

for i in $(seq 0 $((ALARM_COUNT - 1))); do
  # ...
  if [[ "$(get '.requires_metric_data // false')" == "true" ]] &&
     ! echo "$EXISTING_ALARMS" | grep -qx "$NAME"; then
    if ! metric_has_data "$(get '.namespace')" "$(get '.metric_name')" "$DIMS" \
                         "$(get '.period')" "$(get '.evaluation_periods')"; then
      skip "$NAME — metric has not filled its evaluation window yet; re-run shortly"
      continue
    fi
  fi

  aws cloudwatch put-metric-alarm --alarm-name "$NAME" ...
done

One added condition: && ! alarm_exists. Thirteen lines changed, including the comment explaining why.

Result:

That remaining withheld alarm is the guard doing its job: its metric source hadn't shipped to production yet, so creating it would page about a 404. Exactly the case the gate exists for.

The generalisable bit

I have now made this mistake in two very different codebases, so I think it is a category rather than an accident:

A guard that skips an operation also skips everything that operation was responsible for.

The guard was written to protect creation. It was placed on the whole loop. Everything else that loop did — reconciling thresholds, re-pointing actions, re-applying tags — went with it. The skip was visible in the output the entire time, printed on every run, and read as informational because the alarm it named was healthy.

Two habits that would have caught it earlier:

1. When you add a guard, name the single operation it protects, out loud, in the condition. requires_metric_data && ! exists reads as "don't create this yet". requires_metric_data alone reads as "don't do anything with this", which is what it did.
2. Distrust a skip whose subject is healthy. Skip lines are the least-read output any script produces. A skip that names a resource which is demonstrably fine is telling you the condition is measuring the wrong thing.

The drift checker needed the same fix, incidentally — it has to apply an identical rule, or it reports drift for alarms the provisioner is legitimately holding back:

# Mirrors provision.sh, including the "a full evaluation window, not one datapoint"
# rule — otherwise verify.sh would call an alarm drift while provision.sh is still
# legitimately holding it back.

Two scripts, one rule, stated twice. That duplication is a smell I chose to accept; the alternative was a shared library for two shell scripts that live in the same directory. Worth revisiting if a third appears.

Rolling out alerting into an environment that has none is its own small project. The Production Readiness Scorecard is a decent way to find out how many of these gaps you are carrying before you start.

The takeaway

treat_missing_data: breaching is correct for absence alarms and will false-page you on creation. Gate creation on a full evaluation window of real data — and gate creation only, or you will trade a one-time false page for a permanent, silent hole in your reconciliation.