case_study
Webhook flood controls
Ingest hardening for a multi-tenant CRM (client engagement)
Backend engineer — incident investigation and remediation
- Ruby on Rails
- PostgreSQL
- GoodJob
- AWS SSM
- Docker
problem
The problem
An integration that normally delivers 15 to 350 events a day delivered 487,195 in a single working day. Every one was accepted, persisted, and queued onto the same worker pool that serves interactive traffic. The queue drained at exactly the rate it was filled, so the machine sat above 90% CPU for eleven hours and recovered only on reboot.
The uncomfortable part was not the flood. It was that three separate protections existed and none of them fired. Idempotency deduplicated on event ID, and a replay is issued a fresh ID by design — so it collapsed zero of the 483,307 replays. The rate limit was sized against denial-of-service, allowing 720,000 requests an hour; the flood ran at 7% of it. And nothing anywhere held a per-source or per-tenant budget.
Worse, nobody knew for a week. The one guard that could have spoken wrote a log line into an environment with no log aggregation. The root cause sat in the producer, which belongs to a customer — so there was no ticket to file and no engineer to page. The fix had to work without the other side changing anything.
approach
What I engineered
- Investigated end to end through AWS Session Manager on a box with no SSH, no bastion, and an IAM role without SendCommand — driving an interactive shell from a paced-stdin script, and reading every figure through the application's own database connection rather than handling credentials.
- Caught and corrected a wrong-datastore error mid-investigation: a stale local Postgres on the same host gave a coherent but 12%-low set of numbers. The tell was that its newest row predated the running process's uptime — a contradiction that is now a standing check before any figure is believed.
- Replaced the fixed-window instinct with a token bucket, because a fixed window resets wholesale and permits two full budgets across a boundary. Rate and burst became two separate honest numbers, with burst defaulting to a minute of the hourly rate and a floored debt so a producer recovers promptly instead of staying locked out for hours.
- Wrote the bucket as a single concurrency-safe UPDATE. The obvious CTE-based implementation is a read-modify-write in disguise and loses spends under exactly the concurrency it exists to survive; referencing the column inside its own SET expression makes Postgres re-evaluate against the winning row instead.
- Added a tenant-wide ceiling above the per-source budget, checked first — ten sources each under their individual limit still add up to a fan-out no per-source rule can see.
- Collapsed replays whose original had already reached a terminal state, while deliberately leaving replays of failed or unprocessed originals to run — breaking a replay mechanism to fix a flood trades an outage for silent data loss.
- Split the queues so backfills run on a bulk pool and live deliveries keep the default one, and returned 429 with a computed Retry-After rather than auto-disabling the integration: back-pressure is self-clearing, a disabled source needs a human.
- Made the guard visible — throttle counters and timestamps on every bucket, surfaced through the admin API, with alerting rate-limited to once per scope per interval. An alert that fires 50,000 times is functionally identical to one that never fires.
result
The result
A receiver that bounds what it will consume from outside its trust boundary, per source and per tenant, with throttle state observable instead of buried in a log line. The same protocol and the same producer point at every environment, so the class of failure is closed rather than the instance of it.
The incident landed on a shared pre-production environment, which is the only reason this is a postmortem and not an outage report. That was luck: the endpoint was real, the producer was real, and the three guards that failed are the ones production runs.
- 487,195
- events in 11 hours
- 99.2%
- were replays, each with a fresh ID
- 7%
- of the rate limit — it never tripped
Figures measured directly in the system's own event store during the July 2026 investigation. Client, product, and partner platform are unnamed under engagement confidentiality. No adoption or revenue figures are claimed.
work_with_me
Have a project like this?
Tell me the problem on a 20-minute call and I'll tell you how I'd approach it, the same way Webhook flood controls started.
Discuss your project