OpenParable
Skip to content
← Back to skills

Debug Crawler

Testing & Quality
debug-crawler

Debug Crawler is an Agent Skill for testing & quality published in the opensanctions/opensanctions repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Investigate a failing crawler and propose a fix, starting from a dataset name or an issues.json artifact URL. Covers pulling the diagnostic report, inspecting source data via Zyte, and common failure patterns including sources that are blocked, geo-blocked, 403/429-throttled, or behind a JavaScript challenge or anti-bot protection.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/opensanctions/opensanctions/blob/HEAD/.claude/skills/debug-crawler/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/debug-crawler, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

Debug a Failing Crawler

The user has provided a dataset name or issues.json artifact URL: $ARGUMENTS (In an artifact URL, the dataset name is the path segment after /artifacts/.)

Read zavod/docs as needed to understand how crawlers are normally written — the goal here is to fix the failing crawler in accordance with existing practices, not to refactor or standardise it.

Step 1: Get the diagnostic report

python -m contrib.maintenance.diagnose <dataset_name>

The report gives you the run verdict (failed since when, for how many runs), the dataset's resolved .yml and crawler paths, artifact links for the latest and last successful runs, the issues themselves (inlined, or grouped by pattern with the full issues.json linked), an assertions-vs-last-good-statistics drift table, and recent commits touching the dataset. Read the crawler's .yml and crawler.py from the paths it resolves, and note the row data on each issue — for source-value issues the keys are slugified column names, values are cell contents.

Step 2: Inspect the current source data

The source has likely changed. Use OPENSANCTIONS_ZYTE_API_KEY (already set in the environment) to fetch via Zyte when direct access times out or is blocked:

python3 -c "
import requests, os
from base64 import b64decode

ZYTE_API_KEY = os.environ['OPENSANCTIONS_ZYTE_API_KEY']
url = '<the Source data URL from the diagnostic report>'

resp = requests.post(
    'https://api.zyte.com/v1/extract',
    auth=(ZYTE_API_KEY, ''),
    json={'url': url, 'httpResponseBody': True, 'httpResponseHeaders': True},
    timeout=60
)
resp.raise_for_status()
content = b64decode(resp.json()['httpResponseBody'])
# then parse content as appropriate for the source format
"

Add 'geolocation': 'US' (or the relevant country code) to the Zyte request when the source geo-restricts access — and add the matching geolocation= argument to the fetch_resource / fetch_html call in the crawler.

If the fix is to move the crawler onto Zyte (the source is now blocked, geo-blocked, throttled, or behind a JavaScript challenge), see zavod/docs/best_practices/http_operations.md for choosing the right helper (fetch_html for browser rendering, fetch_text / fetch_json / fetch_resource otherwise) and remember to set ci_test: false on the dataset.

Step 3: Diagnose

Compare what the source actually contains against what the crawler expects.

Common failures

SymptomCauseFix
Expected field/column not foundSource renamed or restructured columnsUpdate the crawler to match the new structure
First page parses fine, later pages failPer-page header handling no longer matches sourceAdjust header-reading logic to match current source
403 / empty response from ZyteSource geo-restricts contentAdd geolocation= to the fetch call
Assertion on entity count failsSource grew or shrankVerify the count is real — the report's assertion table shows the drift vs the last successful run; check the linked delta.json for what changed. Update assertions: bounds if changes can be explained by e.g. sanctions expiring, but never widen the envelope to fit a collapsed count (that's a broken crawl, not drift).
Unexpected keys in audit_dataNew columns added to sourcePop and handle (or explicitly ignore) the new fields

Step 4: Fix and verify

After making code changes, delete the cached source file so the fresh copy is fetched:

rm -f data/datasets/<dataset_name>/source.*

zavod crawl datasets/<path>/<dataset_name>.yml

Check data/datasets/<dataset_name>/issues.log for remaining warnings. Then export and confirm the delta is plausible:

zavod export datasets/<path>/<dataset_name>.yml

A healthy run shows:

  • No errors in the crawl log
  • Delta (added/deleted/modified) consistent with elapsed time since the last run
  • Entity counts within the assertions: bounds in the .yml