How to Inspect Webhooks for Agent Payment Integrations

TL;DR

When an agent payment integration breaks in production, the most likely cause is webhook handling. Either webhooks aren't being received, or they're being mis-handled, or signatures aren't being verified, or events are being processed out of order. This guide covers how to inspect them, debug them, and prevent the common failures.

Why are webhooks the #1 production issue?

Three reasons:

1. Hard to test exhaustively in sandbox. Webhooks fire on real-world async events (settlements take hours; chargebacks take days). Sandbox can simulate them, but the timing is artificial.

2. They expose your service's weakest assumptions. "Always idempotent." "Always orderable." "Always signed." None of these are guaranteed at the network layer; you need to enforce them.

3. They're discovered late. A webhook handler that silently fails for 24 hours doesn't break the user-facing flow immediately. By the time you notice (reconciliation drift, customer complaint), 24 hours of events are stale.

What does the Webhook Inspector show?

Five panels:

1. Event stream. Real-time list of webhooks fired to your endpoint. Sortable by time, event type, status (delivered / failed / retrying).

2. Payload viewer. Click any event to see the full JSON payload sent.

3. Response viewer. Your endpoint's response status code, headers, body. Useful for debugging "why did the webhook fail."

4. Retry history. Each delivery attempt. Timestamp, status, latency. Shows the full retry curve.

5. Replay button. Re-fire the same webhook to your endpoint right now. Useful when your endpoint was down + is back, or you've shipped a fix and want to test against the real event.

Five debugging techniques

1. Signature verification check

Most common failure mode: signature mismatch. Symptoms: your endpoint returns 401, every event fails to deliver.

Check:

``python

Correct

expected = hmac.new(WEBHOOK_SECRET.encode(), raw_body, hashlib.sha256).hexdigest()

if not hmac.compare_digest(provided_signature, expected):

return 401

Common mistake — hashing parsed JSON instead of raw body

Won't match because JSON re-serialization differs

`

2. Idempotency-key inspection

Webhooks can be delivered more than once (Shatale guarantees at-least-once delivery). Your handler must be idempotent.

Check:

  • You're storing processed event IDs in a database with a unique constraint.
  • Your handler returns 200 quickly when it sees a duplicate.
  • You're not double-applying ledger updates.

`python

def handle_event(event):

if event_already_processed(event["id"]):

return 200 # Idempotent acknowledgment

process_event(event)

mark_event_processed(event["id"])

return 200

`

3. Ordering analysis

Events can arrive out of order — auth.settled before auth.created, refund before original auth. Your handler must tolerate this.

Check:

  • You're using event timestamps (not delivery time) for ordering.
  • Out-of-order events are queued, not dropped.
  • Your business logic doesn't assume strict ordering (e.g. don't subtract from a balance based on a refund event before the original auth has been applied).

4. Payload diff

When a specific event type fails consistently, look at the actual payload vs your expected schema.

Common causes:

  • Schema added a new optional field that breaks your strict parser.
  • Data type changed (amount used to be int cents, now decimal? — verify in changelog).
  • Nested object added or removed.

The Webhook Inspector's payload viewer plus your handler's expected schema = a fast diff.

5. Replay against current code

After fixing a bug, replay specific failed events to verify the fix works against real production data:

  • Find the failed event in the inspector.
  • Click Replay.
  • Watch your endpoint's response.
  • Confirm the event is now processed correctly.
  • Move on to the next.

    • This is the highest-leverage technique — it lets you test fixes against real data without waiting for new events.

    What's the right webhook architecture for agent payments?

    Three design choices:

    1. Synchronous ack, async processing. Your endpoint should return 200 in <500ms. Heavy processing happens async in a worker queue.

    2. Idempotent processing with DB-level deduplication. Event ID + unique constraint = no duplicate processing.

    3. Order-tolerant business logic. Treat events as a stream that converges to correct state, not a strict sequence of operations.

    This architecture handles every webhook quirk: late delivery, duplicate delivery, out-of-order delivery, retry storms.

    What event types should I subscribe to?

    Critical for any agent product:

    Optional:

    FAQ

    How long does Shatale retry failed webhooks?

    24 hours with exponential backoff. After that, the endpoint is paused and an alert fires. Use the Inspector to replay the missed events once your endpoint is back.

    Can I use a different endpoint per event type?

    Yes — configure per-event endpoints in the Portal. Useful when different teams own different event handlers.

    What if I need to change my endpoint URL?

    Update in the Portal. Existing in-flight retries continue to the old URL until they expire (or until the next 24-hour window). New events go to the new URL.

    How do I test webhook delivery without firing real events?

    Sandbox + simulated event endpoints. Each event type has a corresponding /sandbox/simulate/{event_type}` endpoint that fires a test event to your configured webhook URL.

    What if my service is down during a critical event window?

    The Inspector lets you replay events after recovery. Plus: your downstream business logic should be designed to converge to correct state regardless of webhook delivery delays. Don't rely on webhooks for real-time state — pull from the API as backup.

    Related reading

    External references

    ---

    By Vlad K.. Last updated 2026-04-29.