OpenParable
Skip to content
← Back to skills

R2 Glacier Migration

DevOps & Security
r2-glacier-migration

R2 Glacier Migration is an Agent Skill for devops & security published in the pollinations/pollinations repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Monitor and manage R2 to AWS Glacier Deep Archive migration. Use when checking transfer status, resuming transfers, or managing the archive migration.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/pollinations/pollinations/blob/HEAD/.claude/skills/r2-glacier-migration/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/r2-glacier-migration, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

R2 → AWS Glacier Deep Archive Migration

Migrate ~42.6 TB from Cloudflare R2 to AWS S3 Glacier Deep Archive.

Cost Savings

  • R2: ~$638/month → Glacier: ~$42/month
  • Savings: ~$596/month

Buckets

BucketSizeObjectsStatus
pollinations-text16.96 TB1.86B🔄 In progress
pollinations-images25.64 TB221M⏳ Pending

Quick Commands

Check Status

# Is transfer running?
ssh ninon "screen -ls"

# View recent logs
ssh ninon "tail -50 ~/r2-glacier/transfer-text.log"

# Check checkpoint
ssh ninon "cat /tmp/r2-glacier-pollinations-text-checkpoint.json | jq ."

Attach to Screen (Interactive)

ssh -t ninon "screen -r r2-text"
# Detach: Ctrl+A, D

Resume if Crashed

ssh ninon "screen -dmS r2-text bash -c '~/r2-glacier/run-text-transfer.sh; exec bash'"

File Locations

LocationPathPurpose
Skillscripts/r2-to-glacier-streaming.jsMain transfer script
Skillscripts/package.jsonDependencies
ninon~/r2-glacier/Deployed script + deps
ninon~/r2-glacier/run-text-transfer.shWrapper with credentials
ninon~/r2-glacier/transfer-text.logTransfer log
ninon/tmp/r2-glacier-*-checkpoint.jsonResume checkpoint

Script Usage

# Test (small batch)
node r2-to-glacier-streaming.js -b pollinations-text --batch-size 100 --max-batches 1

# Full transfer
node r2-to-glacier-streaming.js -b pollinations-text --batch-size 50000 --concurrency 30

# Resume after interruption
node r2-to-glacier-streaming.js -b pollinations-text --resume

# Dry run (count only)
node r2-to-glacier-streaming.js -b pollinations-text --dry-run

AWS Setup

  • Bucket: s3://pollinations-archive
  • Region: us-east-1
  • Storage Class: DEEP_ARCHIVE

Verify Archives

aws s3 ls s3://pollinations-archive/ --recursive --human-readable | head -20

Important Notes

  • ⚠️ 180-day minimum on Glacier Deep Archive - don't delete R2 until verified!
  • Some objects fail with "Header overflow" (very large responses) - acceptable loss
  • Text bucket (1.86B objects) takes several days
  • R2 egress is free, AWS ingress is free

Start Images Bucket (After Text Complete)

  1. Update ~/r2-glacier/run-text-transfer.sh on ninon:

    # Change bucket name
node r2-to-glacier-streaming.js -b pollinations-images --batch-size 50000 --concurrency 30

  2. Start new screen:

    ssh ninon "screen -dmS r2-images bash -c '~/r2-glacier/run-images-transfer.sh; exec bash'"

Tracking

  • Issue: #5860
  • PR: #5861