Postmortem: 487,000 webhooks in eleven hours

A partner platform's outbox replay went into a loop and delivered 487,195 events into a CRM's inbound webhook endpoint between 06:00 and 16:31 UTC. The receiver accepted every one of them, wrote a row for each, and queued a background job for each. The box sat above 90% CPU for eleven hours and came back only after a reboot.

Nobody was paged. The first anyone knew was a human noticing the machine was hot, a week later.

Two honest framing notes before the numbers, because they change how you should read this.

This happened on a shared pre-production environment, not on a customer-facing one. No end user lost data and no revenue-carrying request was ever slow. That is luck, not design — the producer was a real external system, the endpoint was the real endpoint, and the guards that failed are the same guards production runs. The flood picked the cheap door.

The interesting part is not the outage. It's that three separate protections existed and all three were structurally incapable of noticing. That part generalises to anyone accepting webhooks.

Names and identifiers are omitted — this was a client engagement — but the numbers are as measured in the system's own event store.

Timeline

Normal traffic for this integration is 15 to 350 events per day. The flood ran at roughly 50,000 per hour.

That seven-day gap between "it happened" and "anyone knew" is the finding, not a footnote. Everything below is downstream of it.

What the data said

The first useful query was a daily count. Do this before any theorising — the shape of a burst tells you what kind of bug you have.

 date       | count
------------+--------
 2026-07-21 | 487195   <-- eleven hours
 2026-07-20 |    240
 2026-07-19 |     39

A burst that arrives in one minute is a retry storm. A burst sustained for eleven hours at a steady rate is a loop — something upstream is generating work as fast as it can and not stopping.

The second query narrowed it to a single culprit:

 source | tenant | count  | distinct event ids | with replay marker
--------+--------+--------+--------------------+--------------------
      1 |      1 | 487195 |             487195 |             483307

Three facts, each of which mattered:

1. One source, one tenant. Not a platform-wide problem. One integration.
2. Every event id was unique. So the existing idempotency check — which deduplicates on event id — collapsed exactly zero of them.
3. 99.2% carried a replay marker. These were not new facts. They were re-deliveries of events the system had already processed.

And the event-type breakdown made the loop unmistakable: 452,342 of them were a single profile_updated event for a single tenant. One organisation's profile does not change four hundred thousand times in a day.

The confirming detail came from the producer's own control events. The integration contract includes replay_started / replay_completed markers so the receiver can log a backfill. That day recorded 1,638 replay_started events. A replay announces itself once. The producer had restarted its replay sixteen hundred times — roughly every twenty seconds, for eleven hours.

Why nothing stopped it

Three guards existed. All three were the wrong shape.

Idempotency deduplicates on event id, and a replay has a new one. This is correct by design — a replay is a deliberate re-delivery, so it gets a fresh identity and a pointer back to the original. The consequence is that the one mechanism specifically built to stop duplicate work is structurally blind to the most common source of duplicate work.

The rate limit was a DoS guard, not a flood guard. The edge throttle allowed 200 requests per second per source: 720,000 an hour. The flood peaked at fourteen per second. It ran at 7% of the limit and never came close to tripping it. A limit sized to stop a hostile actor saturating a NIC is not a limit that stops a well-behaved client being wrong very patiently.

There was no per-source budget at all. Every accepted event enqueued onto the same worker pool that serves interactive traffic — three threads, shared. At 40–80ms per job, 487,195 jobs is about eleven hours of work. The eleven hours of CPU was not a coincidence; it was the queue draining at exactly the rate it was being filled.

The general lesson: a rate limit that has never rejected anything is not evidence that traffic is healthy. It may just be set above every failure mode you actually have.

Root cause

The bug was in the producer. Something in its replay driver restarted the backfill instead of completing it.

The producer belongs to a customer. There is no ticket to file and no engineer to page — which is the entire point of the remediation. When the source of a flood is someone you cannot fix, the only durable answer is to stop consuming it. A receiver that assumes well-behaved producers has outsourced its availability to every integration it has ever signed.

The fix

Four changes, in the order a delivery meets them.

1. A token bucket, not a fixed window

The obvious first instinct is an hourly counter. It is also wrong, and worth explaining because the failure is subtle: a fixed window resets wholesale. A producer spends its entire allowance at 11:59 and its entire next allowance at 12:01 — two hours of budget inside two minutes, every hour, entirely "within limits". Damage is done by instantaneous rate, not by hourly total.

A token bucket refills continuously, which lets sustained rate and burst ceiling be two separate honest numbers:

• rate — events per hour, sustained
• burst — how many the bucket may hold at once, defaulted to a minute of the rate

Overshooting costs a token of debt, floored, so a producer recovers promptly once it backs off instead of staying locked out for hours.

The concurrency-safe SQL behind this is its own rabbit hole — the obvious implementation loses writes under exactly the load it exists to survive. That's a separate post.

2. Collapse replays that cannot change anything

A replay whose original already reached a terminal state cannot produce a different outcome by running again. So it collapses onto the original and returns 200 — the producer marks the delivery done and stops retrying.

Critically, replays of failed or unprocessed originals still process normally. That is what a replay mechanism is for, and breaking it to fix a flood would trade an outage for silent data loss.

3. A ceiling above the per-source one

Per-source budgets do not bound a customer account. Ten sources at 9,999/hour each is 100,000/hour with nothing over its limit. A tenant-wide bucket sits above the per-source ones and is checked first — it is the cheaper rejection, and the only one that can see a fan-out.

4. Separate the queues

Replays now enqueue on a bulk pool; live events keep the default pool. A backfill can be slow. It cannot be allowed to make today's webhooks slow.

The part that mattered most

None of the above would have helped on the day, because nobody knew it was happening for a week.

The original guard wrote a log line. That is not observability; it is a message in a bottle. A source could sit throttled for days and the first signal would be a user asking why their CRM was stale.

So the buckets carry throttled_at and throttled_count, the admin API returns them beside the configured limits, and an alert fires once per scope per interval — an eleven-hour flood pages you hourly, not half a million times. Rate-limiting your own alerting is not a nicety. An alert that fires 50,000 times is functionally identical to an alert that never fires.

What we deliberately did not do

We did not auto-disable the source. It is tempting: the producer is misbehaving, turn it off. But back-pressure via 429 is recoverable and self-clearing, whereas a disabled integration needs a human to notice and re-enable it. The failure mode of the aggressive option is a silent multi-day outage for a customer who did nothing wrong.

We did not deduplicate on payload content by default. It is available as an opt-in, and it ships off, because it is the one guard here that can lose data: a status going A→B, B→A, A→B is three legitimate events carrying two identical payloads. Collapsing them would silently drop a real transition. Opt-in, short-windowed, and documented as a trade — not a default.

We did not ask the producer to fix its loop. We would have liked to. But a remediation that depends on someone else's release cycle is not a remediation, and this particular someone is a customer, not a vendor.

Takeaways

1. Ask what your rate limit is actually for. A DoS ceiling and a flood ceiling are different numbers, usually by two orders of magnitude. Having one is not having the other.
2. Idempotency keyed on identity does not stop duplicated work when the protocol issues new identities for re-deliveries. Know which of your dedupe keys is stable across a replay.
3. A guard with no telemetry is not a guard. If you cannot see it engage, you will not know it is misconfigured until it is load-bearing.
4. You cannot fix your customers. Anything you consume from outside your trust boundary needs a ceiling, whether or not you believe the sender is well-behaved. Especially then.
5. Getting this in pre-production was free tuition. The same protocol, the same producer, and the same three blind guards were pointed at the real environment. Treat a non-critical environment falling over as the rehearsal it is, not as a low-priority ticket.

The investigation ran entirely over a session-manager shell on a box with no SSH and no direct database access — the commands are in Debugging a box you can only reach through SSM.