wayback-indirect-asset-recovery

Wayback Indirect Asset Recovery

Problem

When archiving content from defunct services, direct asset URLs (images, videos, avatars) often return 404/503 from Wayback Machine even when the pages referencing them were archived. The assets themselves may not have been crawled, but their URLs exist in archived HTML.

Context / Trigger Conditions

• Direct Wayback URL for asset returns 404 or 503
• You have a profile/page URL but not the asset URL
• Recovering media from defunct services (Vine, Tumblr, defunct startups)
• Wayback has the page but not the embedded assets
• CDN-hosted assets that weren't in Wayback's crawl scope

Solution

Step 1: Fetch the Archived Page

import requests
import re

session = requests.Session()
session.headers['User-Agent'] = 'YourCrawler/0.1 (archival research)'

# Fetch archived profile/page
page_url = f"https://web.archive.org/web/20170110/https://example.com/user/{username}"
resp = session.get(page_url, timeout=20, allow_redirects=True)

Step 2: Extract Asset URLs from HTML

# Try multiple patterns - pages structure varies
patterns = [
    # Open Graph image
    (r'og:image["\s]+content="([^"]+)"', 'og:image'),
    # JSON data in page
    (r'"avatarUrl"\s*:\s*"([^"]+)"', 'JSON avatarUrl'),
    (r'"imageUrl"\s*:\s*"([^"]+)"', 'JSON imageUrl'),
    # CDN URLs
    (r'(https?://[^"\s]+cdn\.[^"\s]+\.(jpg|png|mp4))', 'CDN URL'),
    # S3 URLs
    (r'(https?://[^"\s]+\.s3\.amazonaws\.com/[^"\s]+)', 'S3 URL'),
]

for pattern, name in patterns:
    match = re.search(pattern, html)
    if match:
        asset_url = match.group(1).replace('&', '&')
        break

Step 3: Handle Wayback-Wrapped URLs

URLs extracted from archived pages are often already Wayback URLs:

# If URL is already wrapped, convert im_ to id_ for raw content
if 'web.archive.org/web/' in asset_url:
    # Replace /web/TIMESTAMP/ or /web/TIMESTAMPim_/ with /web/TIMESTAMPid_/
    download_url = re.sub(r'/web/(\d+)(im_)?/', r'/web/\1id_/', asset_url)
else:
    # Wrap raw URL in Wayback
    download_url = f"https://web.archive.org/web/{timestamp}id_/{asset_url}"

Step 4: Fallback Methods if Wayback Fails

If Wayback returns 404/503 for the asset, try alternatives:

# Extract original CDN URL
cdn_match = re.search(r'(https?://[^/]+cdn\.[^"\s]+)', asset_url)
if cdn_match:
    original_url = cdn_match.group(1)

    # Method 1: Try live CDN via IP (if DNS is dead but servers live)
    # See: dead-cdn-dns-bypass skill

    # Method 2: Try different Wayback timestamps
    for ts in ['20170110', '20160601', '20150601']:
        alt_url = f"https://web.archive.org/web/{ts}id_/{original_url}"
        resp = session.get(alt_url, timeout=30)
        if resp.status_code == 200:
            break

    # Method 3: Check if asset exists on successor platform
    # (e.g., user migrated to Twitter/TikTok with same username)

Verification

1. Check HTTP 200 response from final download URL
2. Validate content-type matches expected type
3. Verify file magic bytes (JPEG: \xff\xd8\xff, PNG: \x89PNG)
4. Confirm file size > minimum threshold

Example: Recovering Vine Avatars

def recover_avatar_from_profile(vanity_url: str, user_id: str, session) -> bool:
    """Recover avatar by crawling archived Vine profile."""
    timestamps = ['20170110', '20161201', '20160601', '20150601']

    for ts in timestamps:
        # Fetch archived profile page
        profile_url = f"https://web.archive.org/web/{ts}/https://vine.co/{vanity_url}"
        resp = session.get(profile_url, timeout=20)

        if resp.status_code != 200:
            continue

        # Extract avatar URL from page
        match = re.search(r'og:image["\s]+content="([^"]+)"', resp.text)
        if not match:
            match = re.search(r'"avatarUrl"\s*:\s*"([^"]+)"', resp.text)

        if match:
            avatar_url = match.group(1).replace('&', '&')

            # Convert to raw content URL
            if 'web.archive.org/web/' in avatar_url:
                download_url = re.sub(r'/web/(\d+)(im_)?/', r'/web/\1id_/', avatar_url)
            else:
                download_url = f"https://web.archive.org/web/{ts}id_/{avatar_url}"

            # Download
            img_resp = session.get(download_url, timeout=30)
            if img_resp.status_code == 200 and len(img_resp.content) > 100:
                # Validate and save
                if img_resp.content[:3] == b'\xff\xd8\xff':  # JPEG
                    save_avatar(user_id, img_resp.content, 'jpg')
                    return True

    return False

Result: Recovered avatars for 95% of top Vine creators (including Logan Paul, Nash Grier, Lele Pons) whose direct avatar URLs weren't archived.

Notes

• Wayback im_ vs id_ modifiers: im_ returns image with Wayback toolbar, id_ returns raw bytes
• Rate limiting: Respect Wayback's servers - add 1-2 second delays between requests
• Multiple timestamps: Try several archive dates - some may have the asset, others may not
• HTML entity decoding: Always decode & to & in extracted URLs
• Redirect handling: Use allow_redirects=True - Wayback often redirects to nearest snapshot

Related Skills

• dead-cdn-dns-bypass - For when CDN DNS is dead but servers still respond
• wayback-api-archive-recovery - For discovering what was archived via CDX API
• wayback-machine-raw-content-id-modifier - For fetching raw content without Wayback wrapper