HTTP Response Handling
Researchhttp-response-handlingHTTP Response Handling is an Agent Skill for research published in the HKUDS/OpenSpace repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.
Handle websites requiring JavaScript by using curl with browser headers and validating file types.
How to use this skill
Bring this guide into your coding agent with a prompt tailored to the tool you use.
- Open your project in Codex.
- Copy and paste the prompt below.
- Review files and risks before approving any action.
Inspect the Agent Skill at https://github.com/HKUDS/OpenSpace/blob/HEAD/gdpval_bench/skills/http-response-handling/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/http-response-handling, but wait for my approval before writing files or running scripts.
Copying this prompt does not install or run the skill. Review third-party files before use.
HTTP Response Handling for JavaScript-Dependent Sites
When to Use This Skill
Use this technique when you need to fetch content from websites that:
- Render content dynamically with JavaScript
- Return placeholder HTML when accessed by non-browser clients
- Deliver different content based on User-Agent headers
Core Technique
Step 1: Fetch Content with Browser-Like Headers
Use curl with a realistic User-Agent header to mimic a real browser:
curl -L -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" -o output.html "https://example.com"
Key flags:
-L— Follow redirects-A— Set User-Agent header to mimic a real browser-o— Save output to file for inspection
Step 2: Detect Placeholder HTML Responses
After fetching, check if you received a placeholder response instead of actual content:
# Check file size (placeholder responses are often very small)
wc -c output.html
# Check for common placeholder indicators
grep -i "javascript" output.html | head -5
grep -i "loading" output.html | head -5
grep -i "noscript" output.html | head -5
Signs of a placeholder response:
- File size is suspiciously small (<5KB for content pages)
- Contains大量 JavaScript but minimal actual content
- Has "loading", "spinner", or "noscript" tags
- Missing expected text/data from the page
Step 3: Validate File Type Before Parsing
Before attempting format-specific parsing, validate the file type:
# Check the file type
file output.html
# Check the actual content type (if you have the headers)
curl -I -A "Mozilla/5.0 ..." "https://example.com" | grep -i content-type
# Inspect first few lines
head -50 output.html
Common checks:
- HTML files: Should start with
<!DOCTYPEor<html - JSON files: Should start with
{or[ - PDF files: Should start with
%PDF - Empty/error pages: May contain error messages or generic HTML
Step 4: Handle Different Scenarios
If you got valid HTML content:
# Proceed with HTML parsing or extraction
grep -oP '(?<=<title>).*?(?=</title>)' output.html
If you got a placeholder/JS-dependent response:
- Option A: Use a headless browser (Playwright, Selenium)
- Option B: Look for an API endpoint that returns JSON directly
- Option C: Check if the site has a mobile/API version with simpler responses
If you got an unexpected file type:
# Check what was actually returned
file output.html
# Adjust your approach based on actual content
case $(file -b --mime-type output.html) in
"text/html")
# Parse as HTML
;;
"application/json")
# Parse as JSON
;;
"application/pdf")
# Handle as PDF
;;
*)
echo "Unexpected file type: $(file -b --mime-type output.html)"
;;
esac
Quick Reference Script
#!/bin/bash
# fetch-with-validation.sh
URL="$1"
OUTPUT="${2:-output.html}"
# Fetch with browser headers
curl -L -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" -o "$OUTPUT" "$URL"
# Get file info
SIZE=$(wc -c < "$OUTPUT")
TYPE=$(file -b --mime-type "$OUTPUT")
echo "Downloaded: $OUTPUT"
echo "Size: $SIZE bytes"
echo "Type: $TYPE"
# Warn about potential issues
if [ "$SIZE" -lt 1000 ]; then
echo "WARNING: File is very small - may be a placeholder response"
fi
if [ "$TYPE" = "text/html" ]; then
if grep -qi "javascript\|loading\|spinner" "$OUTPUT"; then
echo "WARNING: Content may be JavaScript-dependent"
fi
fi
Common Pitfalls
- Not checking file type: Assuming HTML when you got JSON or an error page
- Using default curl User-Agent: Many sites block or serve minimal content to bots
- Ignoring file size: Very small files are often error pages or placeholders
- Not following redirects: Some sites redirect based on User-Agent