Daft Distributed Scaling
DevOps & Securitydaft-distributed-scalingDaft Distributed Scaling is an Agent Skill for devops & security published in the Eventual-Inc/Daft repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.
Scale Daft workflows to distributed Ray clusters. Invoke when optimizing performance or handling large data.
How to use this skill
Bring this guide into your coding agent with a prompt tailored to the tool you use.
- Open your project in Codex.
- Copy and paste the prompt below.
- Review files and risks before approving any action.
Inspect the Agent Skill at https://github.com/Eventual-Inc/Daft/blob/HEAD/.claude/skills/daft-distributed-scaling/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/daft-distributed-scaling, but wait for my approval before writing files or running scripts.
Copying this prompt does not install or run the skill. Review third-party files before use.
Daft Distributed Scaling
Scale single-node workflows to distributed execution.
Core Strategies
| Strategy | API | Use Case | Pros/Cons |
|---|---|---|---|
| Shuffle | repartition(N) | Light data (e.g. file paths), Joins | Global balance. High memory usage (materializes data). |
| Streaming | into_batches(N) | Heavy data (images, tensors) | Low memory (streaming). High scheduling overhead if batches too small. |
Quick Recipes
1. Light Data: Repartitioning
Best for distributing file paths before heavy reads.
# Create enough partitions to saturate workers
df = daft.read_parquet("s3://metadata").repartition(100)
df = df.with_column("data", read_heavy_data(df["path"]))
2. Heavy Data: Streaming Batches
Best for processing large partitions without OOM.
# Stream 1GB partition in 64-row chunks to control memory
df = df.read_parquet("heavy_data").into_batches(64)
df = df.with_column("embed", model.predict(df["img"]))
Advanced Tuning
Target: Keep all actors busy without OOM or scheduling bottlenecks.
Formula 1: Repartitioning (Light Data / Paths)
Calculate the Max Partition Count to ensure each task has enough data to feed local actors.
- Min Rows Per Partition =
Batch Size * (Total Concurrency / Nodes) - Max Partitions =
Total Rows / Min Rows Per Partition
Example:
- 1M rows, 4 nodes, 16 total concurrency, Batch Size 64.
- Min Rows:
64 * (16/4) = 256. - Max Partitions:
1,000,000 / 256 ≈ 3906. - Recommendation: Use ~1000 partitions to run multiple batches per task.
df = df.repartition(1000) # Balanced fan-out
Formula 2: Streaming (Heavy Data / Images)
Avoid creating tiny partitions. Use into_batches to stream data within larger partitions.
Strategy: Keep partitions large (e.g. 1GB+), use into_batches(Batch Size) to control memory.
# Stream batches to control memory usage per actor
df = df.into_batches(64).with_column("preds", model(max_concurrency=16).predict(df["img"]))