url-scraping

URL Scraping

Overview

Price Buddy extracts product data (title, price, image, availability) from retailer URLs. The pipeline is: Store lookup -> HTTP/API fetch -> Strategy-based extraction -> Price storage. Each store defines its own scrape strategy specifying how to extract each field from the page HTML.

Core Architecture

Scraping Pipeline

1. A URL is submitted (via UI, command, or scheduled job)
2. ScrapeUrl resolves the Store by matching the URL's domain against Store.domains
3. The appropriate scraper service (HTTP or API) fetches the page HTML
4. Each field (title, price, image, availability) is extracted using the store's scrape strategy
5. Extracted price is normalized to a float and stored as a Price record
6. The parent Product.price_cache is updated with denormalized data from all URLs

Key Classes

Enums

Store Configuration

Domain Matching

Stores have a domains JSON column containing an array of {"domain": "example.com"} objects. The domainFilter scope matches a URL's host against these. Auto-created stores register both example.com and www.example.com.

Scrape Strategy

The scrape_strategy JSON column defines extraction rules per field:

[
    'title' => ['type' => 'schema_org', 'value' => null],
    'price' => ['type' => 'selector', 'value' => '.price-current'],
    'image' => ['type' => 'selector', 'value' => 'meta[property="og:image"]|content'],
    'availability' => ['type' => 'selector', 'value' => '.stock-status'],
]

Each field entry has:

• type - One of ScraperStrategyType values
• value - The selector/xpath/regex/json-path (null for schema_org)
• prepend / append (optional) - Strings to prepend/append to extracted value

Strategy Types

schema_org - Extracts from JSON-LD @type: Product. No value needed. Looks for offers.lowPrice, offers.price, or offers.priceSpecification.price for price. Best option when available.

selector - CSS selector with special syntax:

• .price - Get text content of element
• .selector|attribute - Get attribute value (pipe delimiter). Example: meta[property="og:image"]|content
• !.selector - Get raw HTML (exclamation prefix). Example: !.product-description

xpath - XPath expression. Use @attribute for attributes, text() for text.

regex - Regular expression. Capture group () extracts the value. Example: ~"price":\s?"(.*?)"~

json - JSON dot-notation path for JSON responses.

Store Settings

The settings JSON column holds:

• scraper_service - http (curl, fast) or api (browser-based, JS rendering)
• scraper_service_settings - Newline-separated key=value pairs passed as scraper options
• test_url - A URL for testing the store's scrape configuration
• locale_settings.locale / locale_settings.currency - For price formatting

Cookies

The cookies text column stores cookies to send with requests (useful for stores that require session/consent cookies).

Availability Match Config

The scrape_strategy.availability.match config maps scraped availability values to StockStatus cases:

'availability' => [
    'type' => 'selector',
    'value' => '.stock-label',
    'match' => [
        'default' => 'in_stock',
        'out_of_stock' => ['type' => 'match', 'value' => 'Out of Stock'],
        'pre_order' => ['type' => 'regex', 'value' => 'pre.?order'],
    ],
],

Match types: match (exact string comparison) or regex (pattern match). If no match config exists, any non-empty availability value maps to OutOfStock.

Scraper Services

HTTP (ScraperService::Http) - Fast curl-based requests via jez500/web-scraper-for-laravel. Default for most stores.

API (ScraperService::Api) - Browser-based requests through a scraper API service (configured via SCRAPER_BASE_URL env, default http://scraper:3000). Use for JavaScript-rendered pages. Slower and more resource-intensive.

Auto Store Creation

AutoCreateStore automatically detects scrape settings when adding a URL for a new domain. It tries extraction methods in priority order:

1. Schema.org JSON-LD
2. CSS selectors from config/price_buddy.php auto_create_store_strategies
3. Regex patterns from the same config

The config file at config/price_buddy.php contains the default selectors and regex patterns tried during auto-detection. Common selectors include meta[property="og:title"]|content, .price, [itemProp="price"]|content, etc.

Retry & Caching

• ScrapeUrl::scrape() retries up to max_attempts_to_scrape times (default 3, configurable in AppSettings)
• On retry, use_cache is set to false to bypass stale cached responses
• If scrape returns false (no store found), retries stop immediately
• Cache TTL is controlled by AppSettings::scrape_cache_ttl (default 720 minutes)
• Sleep between scrapes is configurable via AppSettings::sleep_seconds_between_scrape (default 10 seconds)

Price Storage

• Prices are normalized to floats via CurrencyHelper::toFloat() with locale/currency awareness
• price_factor on Url supports bulk items (e.g., price_factor=6 for a 6-pack gives unit_price)
• Each scrape creates a new Price record (historical tracking)
• Product.price_cache is a denormalized JSON column updated after each price change via Product::updatePriceCache()

Jobs & Scheduling

• FetchAll command (lando artisan app:fetch-all) triggers PriceFetcherService::updateAllPrices()
• Products are chunked (default 10) and dispatched as UpdateAllPricesJob batches
• Each product dispatches UpdateProductPricesJob which calls Product::updatePrices()
• Schedule is controlled by AppSettings::scrape_schedule cron expression (default 0 6 * * *)
• Job timeout is 20 minutes (PriceFetcherService::JOB_TIMEOUT)

Debugging Scraping Issues

Common Problems

"No store found for URL" - The URL's domain doesn't match any store's domains array. Check domain spelling, www vs non-www variants. Use Store::query()->domainFilter($host)->first() to test.

"Missing title/price when scraping" - The scrape strategy selectors don't match the page HTML. Common causes:

• Page uses JavaScript rendering but store uses HTTP scraper (switch to API)
• CSS class names changed on the retailer's site
• Page returns different HTML to bots (anti-scraping)
• Cached response is stale (retry bypasses cache automatically)

Price extraction returns null - The selector matches but CurrencyHelper::toFloat() can't parse the value. Check for currency symbols, thousands separators, or non-numeric text in the matched content.

Availability incorrectly showing out of stock - Check the match config in the store's scrape strategy. Without a match config, any non-empty availability value maps to OutOfStock.

Debugging Techniques

1. Check logs - Scraping errors log to the db channel with URL context. Check the activity log in the admin UI.
2. Test scrape in tinker - ScrapeUrl::new('https://example.com/product')->scrape() returns the full result array including body, errors, and extracted fields.
3. Test store resolution - ScrapeUrl::new('https://example.com/product')->getStore() returns the matched store or null.
4. Test auto-detection - AutoCreateStore::new('https://example.com/product')->strategyParse() shows what would be auto-detected.
5. Inspect raw HTML - The scrape result includes body with the fetched HTML. Check if selectors match the actual page content.
6. Test selector parsing - ScrapeUrl::parseSelector('.selector|attr') returns the parsed array to verify selector syntax.
7. Use the store test URL - Each store has a test_url setting specifically for testing its scrape configuration in the Filament admin UI.

Additional Resources

• references/scrape-strategy-examples.md - Detailed examples of scrape strategies for common retailer patterns