How do I use Data quality control?

Data quality control lets you check completed Cloud jobs against thresholds you define, so you can catch incomplete or abnormal results even when the scraping job itself finishes successfully.

Available checks

Minimum record count: flags jobs that return fewer records than expected.
Maximum failed page %: flags jobs where too many pages failed because of access, CAPTCHA, timeout, or execution problems.
Maximum empty page %: flags jobs where too many processed pages returned no records.
Minimum % of fields that must be filled: flags datasets where too many expected fields are empty.

If any enabled criterion falls outside its configured threshold, Data quality control fails for that job.

How to configure it

Open the sitemap in Web Scraper Cloud.
Open the Data quality control tab.
Enable the checks that are relevant to the sitemap.
Set thresholds based on what a normal successful job should return.
Save the configuration.

If the sitemap already has a completed job, Fill with suggested values can use that previous result as a starting point. Allow for normal variation instead of treating a single run as an exact baseline.

Choose thresholds that match the sitemap

Different checks identify different problems. A low record count can reveal incomplete navigation, a high Failed page percentage can reveal access problems, and low field population can reveal a selector problem even when the record count looks normal.

Some Empty pages or missing fields may be expected. Account for those normal cases when setting thresholds.

If a quality check fails

Compare Records, Failed pages, Empty pages, and No value pages with a previous successful run.
Inspect affected Failed or Empty URLs and screenshots when available.
Preview the returned records and look for missing or inconsistent fields.
Check recent changes to selectors, Website Log-In / State setup, driver, proxy, Request interval, or Page load delay.

A Data quality failure does not change a successfully completed scraping job into a failed scraping job. It indicates that the returned dataset did not meet your configured quality thresholds.

See the Data quality control documentation for more information.