How do I scrape all pages, not just the first page?

If your sitemap only returns data from the first page of a listing, it usually needs pagination.

Add pagination to the sitemap

1. Create a Pagination selector.
2. Select the pagination control that moves to additional pages.
3. If the website has both numbered page links and a Next or arrow button, prefer the Next or arrow control where possible.
4. Keep the pagination type set to Auto unless the default behavior does not work correctly.
5. Place the listing wrapper and data selectors beneath the Pagination selector.

Check that later pages are discovered

Before running a full scrape, preview more than one pagination page. Confirm that the sitemap can reach the second and third pages and that the expected records are returned from them.

A typical sitemap structure looks like this:

_root
└── pagination
    └── items
        ├── name
        └── price

If the website does not use normal page links

A Load more button also uses a Pagination selector. If new items appear only when you scroll down the page, use an Element selector with scrolling enabled instead.

See the Pagination, load more and infinite scrolling documentation for the full setup options.