Guides
Start typing to search…

Webhooks

Architect event-driven workflows

Introduction to DocuPanda Webhooks

Webhooks are how services notify each other of events. At their core they are just a POST request to a pre-determined endpoint. The endpoint can be whatever you want, and you can just add it in our webhook portal. You normally use one endpoint per service, and that endpoint listens to all of the event types. For example, if you receive webhooks from Acme Inc., you can structure your URL like: https://www.example.com/acme/webhooks/.

The way to indicate that a webhook has been processed is by returning a 2xx (status code 200-299) response to the webhook message within a reasonable time-frame (15s). It's also important to disable CSRF protection for this endpoint if the framework you use enables them by default.

Webhook Events

We have a variety of event types that you can subscribe to. For example, one such event is document.processed.success. As you may already know, when you upload a new document, it can take us seconds or possibly minutes to process the document fully. Instead of polling for a response, you could just listen for a document.processed.success event.

Here is a sample payload for document.processed.success:

{
  "documentId": "dcmt5678",
  "eventType": "document.processed.success",
  "jobId": "abcd1234"
}

Rarely something goes wrong, you may want to have a workflow that responds to a document failing to process. We So we have an equivalent event typedocument.processed.error. Here is a sample payload for document.processed.error:

{
  "documentId": "dcmt5678",
  "errorMessage": "Failed to process document. Reason: unreadable input.",
  "eventType": "document.processed.error",
  "jobId": "abcd1234"
}

We handle similar handling for every single

Adding an Endpoint

In order to start listening to messages sent through Svix, you will need to configure your endpoints.

Adding an endpoint is as simple as providing a URL that you control and a list of event types that you want to listen to.

Helpful Tip!

If you don't have a URL or your service isn't quite ready to start receiving events just yet, just press the with Svix Play button to have a unique URL generated for you.

You'll be able to view and inspect all operational webhooks sent to your Svix Play URL, making it effortless to get started.

If you don't specify any event types, by default, your endpoint will receive all events, regardless of type. This can be helpful for getting started and for testing, but we recommend changing this to a subset later on to avoid receiving unexpected messages.

Testing your Endpoint

Once you created an endpoint, you can test by sending a simulated event. This event will have a made-up documentId, but lets you make sure you understand the structure of the payload you're getting.

That's why we have a "Testing" tab for you to send example events to your endpoint.

After sending an example event, you can click into the message to view the message payload, all of the message attempts, and whether it succeeded or failed.

Webhook Signature Verification (optional)

Webhook signatures is your way to verify that webhook messages are sent by us. While not mandatory, verifying webhooks is a way to make sure no one can send you fake events, or overwhelm your service with irrelevant requests.

How to verify webhooks with Svix Libraries

We recommend using the webhook verification methods provided by svix.

import { Webhook } from "svix";

const secret = "whsec_MfKQ9r8GKYqrTwjUPD8ILPZIo2LaLaSw";

// Headers and payload come in from us, in the request that your endpoint receives
const headers = {...}
const payload = {..}

const wh = new Webhook(secret);
// Throws on error, returns the verified content on success
const payload = wh.verify(payload, headers);

For more instructions and examples of how to verify signatures, check out their webhook verification documentation.

Retries

We attempt to deliver each webhook message based on a retry schedule with exponential backoff. That means if your endpoint is unavailable for any reason, we retry rapidly at first, and then take increasingly longer pauses to give your service a chance to recover.

The schedule

Each message is attempted based on the following schedule, where each period is started following the failure of the preceding attempt:

  • Immediately
  • 5 seconds
  • 5 minutes
  • 30 minutes
  • 2 hours
  • 5 hours
  • 10 hours
  • 10 hours (in addition to the previous)

If an endpoint is removed or disabled delivery attempts to the endpoint will be disabled as well.

Running some quick matt on this schedule, you have just over 27 hours to process a webhook notification before we give up on it and mark it as permanently undelivered.

Manual retries

You can also use the application portal to manually retry each message at any time, or automatically retry ("Recover") all failed messages starting from a given date.

Troubleshooting Tips

There are some common reasons why your webhook endpoint is failing to verify the signature (if you choose to use verification):

Not using the raw payload body

This is the most common issue. When generating the signed content, we use the raw string body of the message payload.

If you convert JSON payloads into strings using methods like stringify, different implementations may produce different string representations of the JSON object, which can lead to discrepancies when verifying the signature. It's crucial to verify the payload exactly as it was sent, byte-for-byte or string-for-string, to ensure accurate verification.

Missing the secret key

Sometimes people simply use the wrong secret key. Make sure you grab your own secret key from the webhook dashboard - not it's NOT the same one as DocuPanda API key. using the wrong secret key. Remember that keys are unique to endpoints.

Sending the wrong response codes

When we receive a response with a 2xx status code, we interpret that as a successful delivery even if you indicate a failure in the response payload. Make sure to use the right response status codes so we know when message are supposed to succeed vs fail.

Responses timing out

We will consider any message that fails to send a response within {timeout duration} a failed message. If your endpoint is also processing complicated workflows, it may timeout and result in failed messages.

We suggest having your endpoint simply receive the message and add it to a queue to be processed asynchronously so you can respond promptly and avoiding getting timed out.

Failure Recovery

Re-enable a disabled endpoint

If all attempts to a specific endpoint fail for a period of 5 days, the endpoint will be disabled. To re-enable a disabled endpoint, go to the webhook dashboard, find the endpoint from the list and select "Enable Endpoint".

Recovering/Resending failed messages

Why Replay

  • If your service has downtime
  • If your endpoint was misconfigured

If you want to replay a single event, you can find the message from the UI and click the options menu next to any of the attempts.

resend message

From there, click "resend" to have the same message send to your endpoint again.

If you need to recover from a service outage and want to replay all the events since a given time, you can do so from the Endpoint page. On an endpoint's details page, click Options > Recover Failed Messages.

recover modal

From there, you can choose a time window to recover from.

For a more granular recovery - for example, if you know the exact timestamp that you want to recover from - you can click the options menu on any message from the endpoint page. From there, you can click "Replay..." and choose to "Replay all failed messages since this time."

IP Whitelist

In case your webhook receiving endpoint is behind a firewall or NAT, please reach out to us so we can whitelist the IP address of our webhook sending service.