Webhooks: Designing Them So They Don't Wake You Up at 3am

Webhooks look simple from the sending side: something happens, you POST a payload to a URL, done. From the receiving side, where we spend most of our time, they're one of the more common sources of 3am alerts — not because the concept is hard, but because most webhook implementations skip the parts that only matter once something goes wrong.

What breaks in practice

• No retry logic on the sending side, so a receiving server's brief downtime means a permanently lost event, with no way to know it ever happened.
• No idempotency handling on the receiving side, so a retried webhook — sent because the sender didn't get a timely acknowledgment — gets processed twice.
• No signature verification, leaving the endpoint open to anyone who discovers the URL and decides to send it fake events.
• Silent failures — a webhook handler that throws an error with nobody watching, so the failure is invisible until a customer notices the downstream effect.

What a reliable webhook setup actually looks like

On the receiving side: verify every payload's signature, acknowledge receipt immediately and process asynchronously, and design every handler to be safely re-run on the same event without duplicating its effect. This is the same rigor behind handling payment webhooks correctly, applied more generally.

A webhook you can't safely receive twice isn't actually reliable, no matter how well it works the first time.

This is the same category of unglamorous plumbing we described in API integrations — invisible when it's done right, and the first thing anyone notices when it isn't.