What should I use as the start URL?
The start URL is the page where Web Scraper begins running your sitemap. Use the page where your first navigation or data extraction selector should run.
Choose the right start URL
A start URL is commonly:
- A category page
- A search results page
- A directory or listing page
- An individual detail page
For example, if you want to scrape products from a category and then follow each product link, use the category page as the start URL.
Use multiple start URLs when needed
You can add multiple start URLs when the same sitemap and selectors should run on several pages.
Avoid adding thousands of URLs manually if the website already provides pagination, navigation, a predictable URL pattern, or an XML sitemap.
Use URL ranges for predictable URLs
If only a number changes between pages, you can use a URL range. For example:
https://example.com/page/[1-3]
This generates:
https://example.com/page/1https://example.com/page/2https://example.com/page/3
Before scraping, check the first, middle, and last URL to confirm they all load the expected page structure.
See the Start URLs and URL ranges documentation for more information.