Your health check is lying to you: liveness for workers, proxies and the edge

Almost every deployed app has a /up endpoint returning 200. Almost every one of them is used as though it means "the system is healthy". It doesn't. It means:

One web process, on one host, answered one HTTP request within the timeout.

Here are three real failures that a green /up sat through, all on the same system, inside three weeks. Identifiers are changed; this was a client engagement.

Each needs a different signal. None of them is a deeper /up.

Failure 1: the worker fleet is dead and the API is fine

This is the most common and the most expensive, because it is completely silent. Web and workers are separate processes. Kill the worker container and the API keeps returning 200 to everything, forever, while every asynchronous thing your product does stops.

The instinct is to alarm on "has a job run recently". That is the wrong signal, and I have the data to say so: on the system in question, one scheduled monitor job logged 5 executions in 60 minutes on production and 0 on staging during the same window — with a healthy, running worker on both. Job execution rate is a function of workload, not of health. An idle-but-perfectly-healthy fleet executes nothing.

The right signal is a heartbeat: something the worker refreshes because it is alive, not because it has work.

Most job runners already maintain one. This one keeps a row per running process with a refreshed timestamp, so the freshest timestamp across that table is the fleet's pulse:

module Health
  # Liveness probe for the background worker fleet.
  #
  # `/up` only proves the web server is answering. If the worker process dies, the
  # API keeps returning 200 while every background job silently stops, and nothing
  # notices until a customer asks why they got no email.
  class WorkersController < ActionController::API
    # Comfortably longer than the runner's own heartbeat interval, so a slow
    # refresh or a brief pause does not read as an outage.
    STALE_AFTER = 10.minutes

    def show
      last_seen, process_count = heartbeat

      if last_seen.nil?
        return respond(status: :service_unavailable, healthy: false,
                       reason: "no worker process has ever registered a heartbeat",
                       last_seen: nil, process_count: 0)
      end

      age = Time.current - last_seen
      if age > STALE_AFTER
        respond(status: :service_unavailable, healthy: false,
                reason: "newest heartbeat is #{age.to_i}s old (limit #{STALE_AFTER.to_i}s)",
                last_seen: last_seen, process_count: process_count)
      else
        respond(status: :ok, healthy: true, reason: nil,
                last_seen: last_seen, process_count: process_count)
      end
    rescue StandardError => e
      # A probe that 500s is indistinguishable from a probe that cannot reach the
      # app, and an unhandled exception here would also be reported to the error
      # tracker every five minutes. Report unhealthy and say why.
      Rails.logger.error("[health.workers] #{e.class}: #{e.message}")
      respond(status: :service_unavailable, healthy: false,
              reason: "heartbeat lookup failed: #{e.class}",
              last_seen: nil, process_count: 0)
    end

    private

    def heartbeat
      row = ActiveRecord::Base.connection.select_one(
        "SELECT MAX(updated_at) AS last_seen, COUNT(*) AS process_count FROM job_processes"
      )
      return [nil, 0] if row.blank?

      last_seen = row["last_seen"]
      last_seen = Time.zone.parse(last_seen.to_s) if last_seen.present? && !last_seen.is_a?(Time)
      [last_seen.presence, row["process_count"].to_i]
    end

    def respond(status:, healthy:, reason:, last_seen:, process_count:)
      render status: status, json: {
        healthy: healthy,
        reason: reason,
        last_heartbeat_at: last_seen&.utc&.iso8601,
        heartbeat_age_seconds: last_seen ? (Time.current - last_seen).to_i : nil,
        stale_after_seconds: STALE_AFTER.to_i,
        process_count: process_count
      }.compact
    end
  end
end

Four deliberate choices in there, each of which I have seen skipped:

• It inherits ActionController::API, not the app's base controller. The base controller mandates authentication, and a liveness endpoint that requires a valid session cannot be polled by an external prober. Nothing tenant-scoped is read and no record data is exposed — a timestamp and a count.
• It rescues everything and returns 503 with a reason. An unhandled exception in a health endpoint is a 500 that looks identical to an unreachable host, and it ships a duplicate to your error tracker every five minutes forever.
• It reports heartbeat_age_seconds and stale_after_seconds in the body. When the alarm fires, the first question is "how stale?" — answer it in the payload so nobody has to open a console.
• Raw SQL against the table, not the runner's model API. That API has moved between major versions of the library. The column has not.

The tripwire that needs no deploy

/up/workers requires shipping code. When the worker fleet is already dying and you have no release window, there is a coarser signal available immediately: most job runners log a line per completed job. Count those lines, alarm on zero:

{
  "suffix": "jobs-dead",
  "description": "No job execution logged for 2 consecutive hours.",
  "metric_name": "JobsExecuted",
  "statistic": "Sum",
  "period": 3600,
  "evaluation_periods": 2,
  "comparison_operator": "LessThanThreshold",
  "threshold": 1,
  "treat_missing_data": "breaching"
}

It is deliberately blunt — two hours of total silence, and it inherits the "job rate depends on workload" weakness above. But it costs one log metric filter, needs no application change, and it caught the gap while the real endpoint was still in review. Ship the blunt one first.

Failure 2: the origin is healthy and nobody can reach it

Every metric discussed so far is a cause signal, gathered from inside your infrastructure: CPU, memory, connections, heartbeats. All of them were green during a 35-minute outage where a CDN edge certificate stopped matching the origin hostname, and every user request failed at the edge.

Log-derived metrics cannot catch this either — a box that is not being reached is also not logging.

The only thing that catches it is a symptom signal from outside: something that makes a real request over the real hostname, through the real edge, on a schedule.

"""Probe the API from outside AWS's view of the app.

Deliberately runs OUTSIDE the VPC. It only needs public HTTPS and the CloudWatch
API, so keeping it out avoids a NAT gateway or interface endpoints just to reach
the metrics API.
"""

import json, os, ssl, time, urllib.error, urllib.request
import boto3

cloudwatch = boto3.client("cloudwatch")
NAMESPACE = os.environ.get("METRIC_NAMESPACE", "App/Probe")
TIMEOUT_S = float(os.environ.get("PROBE_TIMEOUT_S", "15"))
TARGETS = json.loads(os.environ["PROBE_TARGETS"])

# Verify certificates properly. A silently-downgraded TLS check would mean the
# prober keeps reporting healthy through exactly the certificate failures it is
# meant to catch.
SSL_CONTEXT = ssl.create_default_context()


def _get(url):
    """Return (status_code, elapsed_ms, error_or_none). Never raises."""
    request = urllib.request.Request(
        url, headers={"User-Agent": "endpoint-prober/1.0"}, method="GET"
    )
    started = time.monotonic()
    try:
        with urllib.request.urlopen(request, timeout=TIMEOUT_S, context=SSL_CONTEXT) as resp:
            resp.read(2048)  # drain enough to complete the round trip
            return resp.status, (time.monotonic() - started) * 1000.0, None
    except urllib.error.HTTPError as exc:
        # A 5xx or a 5xx-from-the-edge is a real answer, and its latency is meaningful.
        return exc.code, (time.monotonic() - started) * 1000.0, None
    except Exception as exc:  # timeout, DNS, TLS, connection reset
        return None, (time.monotonic() - started) * 1000.0, repr(exc)

The comment on SSL_CONTEXT is the one I would flag in review. Reaching for ssl._create_unverified_context() to make a prober "just work" turns it into a tool that reports healthy through the exact class of failure it was built to detect. If your prober needs verification disabled, you have found the outage.

The publish step, and the piece that keeps it from paging about unshipped code:

def handler(event, context):
    metrics = []
    for target in TARGETS:
        env = target["env"]
        dims = [{"Name": "Env", "Value": env}]

        status, elapsed_ms, error = _get(target["url"])
        available = 1 if status == 200 else 0

        metrics += [
            {"MetricName": "Available",  "Dimensions": dims, "Value": available, "Unit": "None"},
            {"MetricName": "LatencyMs",  "Dimensions": dims, "Value": round(elapsed_ms, 2),
             "Unit": "Milliseconds"},
        ]
        if status is not None:
            metrics.append({"MetricName": "StatusCode", "Dimensions": dims,
                            "Value": status, "Unit": "None"})
        if not available:
            # Printed so the reason is in the logs when the alarm fires.
            print(f"UNAVAILABLE env={env} url={target['url']} status={status} error={error}")

        workers_url = target.get("workers_url")
        if workers_url:
            w_status, w_elapsed, w_error = _get(workers_url)
            if w_status == 404:
                # Endpoint not deployed yet. Publish NOTHING, so the alarm stays
                # INSUFFICIENT_DATA instead of paging about a missing route.
                continue
            healthy = 1 if w_status == 200 else 0
            metrics.append({"MetricName": "WorkersHealthy", "Dimensions": dims,
                            "Value": healthy, "Unit": "None"})

    # PutMetricData caps at 1000 metrics per call; chunk anyway so adding targets
    # later cannot start silently dropping data.
    for i in range(0, len(metrics), 20):
        cloudwatch.put_metric_data(Namespace=NAMESPACE, MetricData=metrics[i:i + 20])

Publishing nothing on a 404 is the important bit. During a staged rollout, the worker endpoint exists in one environment and not the other. Publishing 0 for "unhealthy" in the environment where the route simply hasn't shipped yet produces a critical page about a deployment that hasn't happened. Absence of a metric is the honest representation of "I cannot tell yet", and the alarm sits in INSUFFICIENT_DATA until it can.

A prober is also the only check that survives the host being dead. Everything else in your monitoring stack that runs on the box goes silent at the same moment the box does.

Failure 3: slow, not down

The third signature is the nastiest: everything returns 200, and login takes 16 seconds. Every AWS status field reads healthy. /up is green. Users are gone.

The prober already measures this, because it records wall-clock latency for a real request over the public hostname:

{
  "suffix": "probe-latency",
  "description": "/up taking longer than 2000 ms from outside.",
  "metric_name": "LatencyMs",
  "statistic": "Average",
  "period": 300,
  "evaluation_periods": 3,
  "threshold": 2000,
  "comparison_operator": "GreaterThanThreshold",
  "treat_missing_data": "missing"
}

Healthy values on that system were 101 ms production and 126 ms staging, so 2000 ms is a twentyfold margin — deliberately loose. This alarm is for catastrophe, not for performance regression. Chasing a p95 regression with a synthetic prober hitting one trivial endpoint is a category error; that is what real-user monitoring is for.

Failure 4 (bonus): the log filter that never matched

The 5xx rate looked like the easy one. The proxy already emits structured JSON per request, so ship the 5xx lines and count them. I wrote the include regex, deployed it, and got zero datapoints.

The raw log event does not contain what the application printed. Docker's json-file driver nests the proxy's JSON inside a .log string field, so the text a filter actually sees is:

{"log":"{\"status\":503,\"path\":\"/api/x\"}\n","stream":"stdout","time":"..."}

The literal characters are status\":503, with a backslash, not "status":503. My regex matched the latter and therefore matched nothing, silently, forever. There is no error for "this filter matched no lines" — that is indistinguishable from "no 5xx happened", which is exactly what you hope to see.

Two follow-on traps in the same feature:

• Passing the filter pattern through jq fromjson parsed "status" as a JSON string and stripped the quotes that make it a valid filter term. CloudWatch rejected every pattern until the string was emitted verbatim.
• A naive :5 substring match happily matches a 200 response whose timestamp contains :5. The test set has to include that case.

The lesson that generalises past Docker: verify a log filter against a verbatim line pulled off the running host, not against what your application logs. Between the two sit a log driver, an agent, and an encoder, and each is entitled to reshape the bytes.

What to actually build

In the order I would do it again:

1. /up — you have it. Understand that it covers one process.
2. An external prober, every five minutes, over the public hostname. Cheapest signal, catches the failures nothing inside can see, survives the host dying.
3. A worker heartbeat endpoint, based on liveness, not activity. Wire it into the same prober.
4. A log-derived tripwire for the case where you cannot deploy right now.
5. Cause metrics (CPU, memory, connections) last. They explain outages. They do not detect them.

Most teams build that list in exactly the reverse order, because cause metrics are the ones the cloud console hands you for free.

If you are working out which of these gaps to close first, the Production Readiness Scorecard walks the same ground, and the Downtime Cost Calculator is a blunt way to justify the afternoon it takes.

The takeaway

Health checks answer the question they were written to answer, and teams read them as answering a much larger one. Write down what each of your checks actually proves, in one sentence. The gap between that sentence and "the system is healthy" is your monitoring backlog.