Why am I receiving duplicate webhooks?

Web Scraper Cloud can send the same webhook notification more than once. This usually happens when the webhook endpoint is retried or when the same scraping job reaches another final state after using Continue.

Why duplicate webhooks happen

Your webhook endpoint must return an HTTP 2xx response within 10 seconds. Web Scraper Cloud retries the webhook if the endpoint takes too long or returns an HTTP status code of 300 or higher.

The first retry is sent after 5 seconds and the second after 10 seconds.

A new valid webhook can also be sent for the same scraping job if failed or empty URLs are processed again with Continue and the job reaches another final status.

How to prevent duplicate processing

Your webhook handler should be idempotent. Before starting downstream work, check whether the same scraping job and status have already been processed.

Use the scrapingjob_id  and status  from the webhook payload to identify the event.

Do not process large downloads inside the webhook request

Return the 2xx response first, then download and process the scraped data separately. For longer imports, queue the scrapingjob_id  and let a worker process the job after the webhook has been acknowledged.

See the Webhook documentation for more information.