Testing webhooks locally, in both directions

API documentation written from source code is documentation nobody has run. It reads correctly — someone opened the controllers and wrote down what they implied — and it can still be wrong in exactly the places an integrator hits first.

The failure modes are boringly consistent. A body parameter the docs never name, so the first call anyone writes returns 422 whatever they guess. A subscribe example in a body shape the API rejects with 400. A signing secret the docs promise on create that the create response does not contain — so a receiver built from those docs cannot verify a single signature.

None of that shows up in a code review. It shows up the first time somebody runs it. This is how I run it.

Two directions, two failure modes

"Webhook integration" almost always means two independent channels, and people test one and ship both.

• Inbound — the other side POSTs signed events to you. Failures are loud (401, 400) except the two that matter, which are silent successes.
• Outbound — you POST signed events to them. Failures are invisible from the receiving side; they just… don't receive things.

They share almost no code. Proving one proves nothing about the other.

Set up so assertions are cheap

Before any webhook work, three things pay for themselves in the first hour.

A throwaway database and a spare port. Not your dev database. You are going to assert counts — "one delivery row", "no second deal" — and a colleague's background job writing to the same tables turns every count into a coin flip. A schema load plus seeds is a couple of minutes and everything after it is deterministic.

export DATABASE_URL="postgres://user:pass@localhost:5432/app_wh_test"
bin/rails db:create db:schema:load db:seed
bin/rails s -p 3100 -b 127.0.0.1

Jobs that run in-process. If deliveries are queued, either run the worker or use an in-process execution mode. Otherwise your "webhook didn't arrive" is really "nothing is draining the queue" and you will spend an hour on the wrong question.

A receiver you own. Not a hosted request bin — one you can assert against and make misbehave on purpose. Thirty lines, no dependencies:

import http from "node:http";

export function startReceiver(port, { respond = () => ({ code: 200 }) } = {}) {
  const received = [];
  const server = http.createServer((rq, rs) => {
    const chunks = [];
    rq.on("data", (c) => chunks.push(c));
    rq.on("end", () => {
      const raw = Buffer.concat(chunks).toString("utf8");
      // Keep the RAW body. Signatures are over bytes, not over your
      // re-serialization of the parsed object.
      received.push({ headers: rq.headers, raw, body: safeJson(raw) });
      const r = respond(received.at(-1));
      rs.writeHead(r.code, { "Content-Type": "application/json" });
      rs.end(r.body ?? "{}");
    });
  });
  return new Promise((res) => server.listen(port, "127.0.0.1",
    () => res({ server, received })));
}

That respond hook is the whole point. A hosted bin always answers 200; you need 500, 422, and 410 on demand to test the retry policy.

Direction 1 — receiving signed events

Sign the bytes you send, not the object you meant

Nearly every HMAC scheme in the wild signs a string built from a timestamp and the raw request body:

import crypto from "node:crypto";

export function sign(rawBody, secret, ts = Math.floor(Date.now() / 1000)) {
  const mac = crypto.createHmac("sha256", secret)
    .update(`${ts}.${rawBody}`)
    .digest("hex");
  return `t=${ts},v1=${mac}`;
}

const raw = JSON.stringify(payload);   // serialize ONCE
await fetch(url, {
  method: "POST",
  headers: { "Content-Type": "application/json", "X-Webhook-Signature": sign(raw, secret) },
  body: raw,                            // send the SAME string you signed
});

The bug that eats an afternoon: signing JSON.stringify(payload) and then passing payload to a client that serializes it again. Key order or spacing differs by one byte, the HMAC differs completely, and you get 401 with a correct implementation. If you script this in a GUI client with template variables, resolve the variables before signing — the server verifies the bytes that arrived, not your template.

The receiver must read the raw body before any JSON middleware touches it. In Rails that means request.raw_post; in Express, a raw-body parser mounted for that route only.

The rejection matrix is the deliverable

A receiver isn't "working" because the happy path returns 202. It's working when every rejection is the right rejection — that's what the sender builds its retry logic against. Drive each one and write down what you got:

Two details worth arguing about in review:

Verify the signature before spending the budget. If rate limiting happens first, an unauthenticated caller can burn a legitimate sender's quota and lock them out — the flood guard becomes the denial of service.

A stale timestamp needs its own answer. "Bad signature" and "your clock is five minutes off" send a caller to completely different code. Distinguish them.

The two silent successes

These are the dangerous responses, because they look like success in every log and dashboard:

• 200 duplicate — you've seen this event id before. Correct, and the right thing for the sender to treat as success.
• 200 ignored — accepted, and deliberately did nothing.

That second one is where integrations die quietly. Event routing is usually deny-by-default: an event type not on the allowlist, or with no published mapping, is accepted and dropped. A sender pushes ten thousand events, gets ten thousand 200s, and nothing appears in the product. Nobody is alerted, because nothing failed.

So: assert ignored explicitly in tests, put the flag in the response body, and say out loud in your docs that an empty allowlist means nothing happens.

Direction 2 — sending, and the guard that blocks you

Point a subscription at http://127.0.0.1:4000/hook, trigger an event, and nothing arrives. Your receiver logs nothing. The delivery row says:

Blocked outbound target (private_or_reserved_ip)

That is the SSRF guard, and it is right. Subscription URLs are user-controlled, so before connecting, the sender resolves the hostname and refuses private, loopback, link-local and reserved ranges — otherwise a customer points a webhook at 169.254.169.254 and reads cloud credentials back out of your delivery log. A good implementation also pins the connection to the IP it validated, so a short-TTL DNS rebind can't swap the target between check and connect.

Which means the guard blocks your laptop, by design. Three honest ways out:

1. A temporary, tightly-scoped local carve-out — an uncommitted, env-gated override that permits exactly 127.0.0.1 and nothing else. Real HTTP, real signatures, no data leaves the machine. Delete it after, and add a test that the guard still blocks 169.254.169.254.
2. A tunnel to a public hostname. Closest to production, but you are publishing test payloads to a third party — think before you do this with anything resembling customer data.
3. Test the delivery path with HTTP stubbed and verify signing separately. Weakest evidence; you never prove the socket, the headers, or TLS.

I use (1) and say so in the writeup. What you must not do is quietly disable the guard and forget — so gate it on an environment variable, keep it out of the commit, and re-verify the guard afterwards. An unexplained "we turned off SSRF protection to test" is how it ships.

Verify the signature independently

Do not verify your own signature with your own signing code — that only proves the function is deterministic. Reimplement the check in the receiver, from the spec, in another language if you can:

export function verify(header, rawBody, secret) {
  const parts = String(header).split(",");
  const t = parts.find((p) => p.startsWith("t="))?.slice(2);
  const sigs = parts.filter((p) => p.startsWith("v1=")).map((p) => p.slice(3));
  const expected = crypto.createHmac("sha256", secret)
    .update(`${t}.${rawBody}`).digest("hex");
  return sigs.some((s) => timingSafeEqual(s, expected));
}

Note sigs is a list. A sender in the middle of a secret rotation signs each delivery with both the old and new key, so the receiver keeps verifying while it picks the new one up out of band. If your rotation replaces the secret in one statement with no overlap window, every in-flight delivery fails verification and the integration is down until the receiver redeploys. Test the grace window: after rotating, one delivery should verify under both secrets; after a zero-grace cutover, only the new one.

If the sender also emits Standard Webhooks headers (webhook-id, webhook-timestamp, webhook-signature), verify that scheme too — it signs a different string (id.timestamp.body) and keys the HMAC on the base64-decoded secret, so a naive verifier silently fails on one of the two.

Assert delivery semantics, not just arrival

Make your receiver misbehave and read the delivery rows:

That third row is the one teams get wrong in both directions. Retrying a 422 six times over two hours to be told the same thing wastes the queue; not retrying a 503 loses the event. And a followed redirect re-opens the SSRF hole the guard just closed, because the second hop was never validated.

While you're there, check that a replay mints a new delivery id rather than re-arming the old one — otherwise the receiver's dedupe key stops meaning "one attempt" and they drop a legitimate re-send.

Prove every event type actually fires

This is the part people skip, and it's where I found real bugs.

List the event catalogue, subscribe to *, then trigger each event through the code path that owns it and record what arrived:

const seen = new Set(received.map((r) => r.headers["x-webhook-event"]));
for (const type of ALL_EVENT_TYPES) {
  record(type, seen.has(type) ? "received" : "NOT RECEIVED");
}

Some events have no HTTP surface — they're model callbacks, or a rescue branch deep in a service. Trigger them through the real path anyway (create the record; force the failure), never by calling the emitter directly. Calling the emitter proves the emitter works, which was never in doubt.

Doing exactly that surfaced a failure event that never fired at all. The service had two failure branches: one inside the run loop, which emitted properly, and one pre-flight bail-out — file missing, bad configuration — that marked the record failed and returned. No event, no audit entry. A subscriber watching job.started → job.completed | job.failed would wait forever, on the three failure modes most likely to happen.

You cannot find that by reading the happy path. You find it by iterating the catalogue and noticing one name never showed up.

Assert the async tail

Delivery is asynchronous, so "the request returned 200" is a third of the story.

The queue must drain. Measure it, don't assume it — sample the pending count until it hits zero and report how long that took. In a load run I watched a backlog peak at ~1,200 jobs and clear in twelve seconds; the same test with dead receivers behaved completely differently, because every delivery failed fast and scheduled a retry. Both are worth knowing.

Nothing may hold a transaction across the HTTP call. This is the classic webhook-sender bug: open a transaction, lock the delivery row, POST to a receiver that takes eight seconds, commit. Now you have an eight-second row lock per delivery, idle in transaction sessions, and vacuum blocked behind them. The safe shape is: claim with a single conditional UPDATE, POST outside any transaction, then update the row.

Watch for it directly while you generate load:

-- longest open transaction, in seconds
SELECT COALESCE(MAX(EXTRACT(EPOCH FROM (now() - xact_start)))::int, 0)
FROM pg_stat_activity WHERE datname = current_database();

-- anything blocked on a lock right now
SELECT COUNT(*) FROM pg_locks WHERE NOT granted;

If either number is non-trivial under load, you have found something more important than the webhook.

Traps that cost me hours

Your harness lies before the product does. I "found" a cross-tenant data leak that turned out to be a typo building the URL, so the request hit a list endpoint and returned my own tenant's records. Verify a shocking result twice before you write it down. A finding that would be a critical bug deserves a second, differently-shaped test.

Collection runners keep cookies. Sign in from a GUI client and it stores the session cookie; the next state-changing request now travels with a cookie, so CSRF protection engages and everything returns 403 — while the same request from a plain script works fine. Capture the CSRF token at sign-in and send it, or run token-only with no cookie jar.

Idempotency keys must be stable and event-derived. A fresh UUID per attempt defeats the entire mechanism. Key on your event id. Also: test the retry returns the original record with a duplicate flag rather than creating a second one — and re-run your suite twice, because a hardcoded key that worked on the first run returns 200 duplicate on the second and fails an assertion that expected 201.

"Registered" is not "configured". Registering an integration typically hands back an id and a secret and nothing else. Routing rules, allowlists and mappings are separate, and the defaults are usually deny-everything. Every "the webhook does nothing" report I've chased was configuration, not code.

Byte-for-byte comparisons of build output are a trap in CI, if you go on to automate any of this. Content hashes move whenever the toolchain moves. Compare the things that carry meaning — which routes exist, which operations are documented — not the bytes.

What "it works" should mean before you call it done

• Both directions driven over real HTTP, with signatures verified by an independent implementation.
• The full rejection matrix, including the two silent successes.
• Every event type in the catalogue observed on the wire, each triggered through its own code path.
• Retry classification, 410 handling, replay, and rotation grace, each proven by making the receiver misbehave.
• The queue drained; no transaction held across a network call.
• Every example in your documentation executed exactly as written. If a body shape appears in the docs, curl it. That is how you learn the envelope was wrong and the parameter had a different name.

Sizing the receiving end before you commit to someone else's volume? Put the event rate and your handler latency into the Throughput & Concurrency Calculator to see how many workers the tail actually needs — and if you publish the API, run the spec through the OpenAPI Spec Linter first.

Documentation written from the code instead of from a run is wrong in the places nobody thought to check, and those are the same places the first integrator lands. Every hour spent driving it end to end is an hour somebody else doesn't spend guessing — and the fastest way to find out whether your webhook integration works is to be, for an afternoon, the integrator who has to make it work.

If you want to see what the failure mode looks like at scale when the receiving side has no budget guard at all, the 487,000-webhook postmortem is the other half of this story.