My scraping job is running, although no results are being returned
In most cases, the reason for this is one of the following:
- The sitemap contains an 'Element Click Selector' or an 'Element Scroll Selector'. Due to how the scraper returns data, the initial records will only become available once the scraper has fully iterated through all of the available pages using the 'Element Click Selector' or has scrolled through all of the initial items by using the 'Element Scroll Selector'.
Please note that the Cloud Scraper will execute JavaScript for approximately 10 - 12 minutes continuously. If this time limit is exceeded, the scraper will automatically time out and restart the scraping process. If this is the case, please contact support.
-
If the sitemap has a page range defined within its start URL as well as if there is a 'Link Selector' within the '_root' of the sitemap. Example start URL containing a page range - https://webscraper.io/test-sites/e-commerce/static/computers/laptops?page=[1-20].
In this case, the scraper will iterate through all of the pages specified in the range first, to collect all of the available links, before moving to the next structure level of the sitemap. Once the scraper has reached the bottom level of the sitemap, records will start to be returned then.
-
The sitemap has a multi-level structure using ' Link Selectors' to navigate through the page. This is because the URLs are traversed in pseudo-random order, to ensure the most recent data is being scraped when crawling larger sites.
-
A combination of any of the listed reasons above.