How do I scrape a website that requires login?

If the target data is only available after signing in, configure Website Log-In / State setup so Web Scraper Cloud can perform the required login steps before normal sitemap extraction begins.

Why local scraping may work without extra setup

Your browser may already have an active login session when you test the sitemap locally. A Cloud job runs separately and does not automatically use that browser session.

If login is not configured for Cloud, the job may receive a login page instead of the page containing the data.

Configure the login workflow

1. Enable Website Log-In / State setup for the sitemap.
2. Set Perform when not found selector to an element that proves the user is already signed in, such as an account label or another element visible only after login.
3. Add the actions required to complete the login, such as opening the login page, entering the username, entering the password, and clicking the sign-in button.
4. Use Element Preview to confirm that the trigger selector correctly detects the signed-in state.
5. Run a Cloud Test scrape and confirm that the protected page is reached before extraction begins.

Available Website Log-In / State setup actions

• Open URL: open a required page before continuing.
• Click: click a matching button, link, tab, or other element.
• Text Input: enter non-sensitive text.
• Password Input: enter a password.

Password values synced or imported into Cloud are encrypted. They cannot be exported and are blank in exported sitemap definitions.

If the login still fails

Open the Cloud job's Inspect view and check the returned page and failure reason. If a Website Log-In / State setup action cannot find a required input, button, link, or other element, the affected page is recorded as a Failed page.

A 401 Unauthorized response also indicates that the page requires authentication or that the current authenticated state was not accepted.

Use sign-in automation only where automated access is permitted by the website or where you have explicit permission.

See the Website State Setup documentation for more information.