What is the Web Scraper Cloud API rate limit?

The Web Scraper Cloud API allows up to 200 requests within a 15-minute period.

What happens when the limit is reached?

If the rate limit is exceeded, the API returns a 429 Too Many Requests  response. Wait for the current 15-minute period to reset before sending additional requests.

API responses include the following headers:

  • X-RateLimit-Limit  - the maximum number of requests allowed.
  • X-RateLimit-Remaining  - the number of requests remaining in the current period.
  • X-RateLimit-Reset  - when the current rate limit resets.

How to avoid reaching the rate limit

Use a webhook instead of repeatedly requesting the scraping-job status. Web Scraper Cloud can notify your endpoint when a job reaches a final status. You can then use the returned scrapingjob_id  to retrieve the completed data through the API.

If polling is required, increase the interval between status requests and stop polling after the job reaches a final status.

See the Web Scraper Cloud API documentation and Webhook documentation for more information.

Did this answer your question? Thanks for the feedback There was a problem submitting your feedback. Please try again later.

Still need help? Contact Us Contact Us