What do Request interval and Page load delay mean?
Request interval controls how quickly Web Scraper sends new page requests. Page load delay controls how long Web Scraper waits before extracting data from a loaded page.
Request interval
Request interval sets the minimum time between page requests made by the scraping job.
- Increase it to reduce the request rate.
- Reduce it to make the job run faster when the target website can handle the higher request rate.
Page load delay
Page load delay controls how long Web Scraper waits before data extraction begins.
Increase it when the website needs more time to render content or when JavaScript-generated values appear after the page initially loads.
How low can I set them?
Both settings have a minimum value of 500 ms. The default values are the recommended starting point.
If you lower either setting, run a Test scrape and confirm that the expected records and fields are still returned before using the faster settings for full jobs or schedules.
Which setting should I change?
- Change Request interval when you want to control how quickly new page requests are made.
- Change Page load delay when the page itself needs more or less time before extraction.
Reducing either value too far can result in incomplete or missing data if the website cannot respond or render quickly enough.
See the Run a Cloud scraping job documentation for more information.