OpenParable
Skip to content
← Back to skills

Dataset Manager

Testing & Quality
dataset-manager

Dataset Manager is an Agent Skill for testing & quality published in the sirius-db/sirius repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Use this skill to generate benchmark datasets (TPC-H, TPC-DS, etc.). Trigger when the user needs test data at a specific scale factor for benchmarking or testing. Supports parquet and duckdb output formats.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/sirius-db/sirius/blob/HEAD/.claude/skills/dataset-manager/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/dataset-manager, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

Dataset Generator

Generate benchmark datasets by running the corresponding shell script under test/<benchmark>_performance/.

Gather Parameters

Parse $ARGUMENTS for:

  • Benchmark: name from the registry below (first positional arg)
  • Scale factor: integer (second positional arg)
  • Format: --format duckdb|parquet (optional — each benchmark has a default)
  • Cluster (TPC-H duckdb only): --cluster / --cluster-keys <spec> (optional — physically sorts tables so row-group pruning works; errors with --format parquet)
  • Output path: --output <path> (optional — scripts have sensible defaults)

If any required parameter (benchmark, scale factor) is missing, ask the user.

Benchmark Registry

Each entry follows the same structure: script location, command template, supported formats, defaults, and prerequisites.

TPC-H

FieldValue
Scripttest/tpch_performance/generate_tpch_data.sh
Default formatparquet
Formatsparquet (tpchgen-rs), duckdb (DuckDB dbgen())
Default output (parquet)test_datasets/tpch_parquet_sf<SF>
Default output (duckdb)test_datasets/tpch_sf<SF>.duckdb
Default output (duckdb + --cluster)test_datasets/tpch_sf<SF>_sorted.duckdb
PrerequisitesParquet: pixi env (rust, python, pyarrow). DuckDB: build/release/duckdb
cd test/tpch_performance && pixi run bash generate_tpch_data.sh <SF> --format <FORMAT> [--cluster] [--cluster-keys <spec>] [--output <path>]

Notes:

  • If the parquet output directory already exists, the script skips generation
  • --cluster (duckdb only) physically sorts tables at load so each row group's per-column min/max becomes selective — this is what makes Sirius's native-scan row-group pruning skip work on date-filtered queries. It writes a distinct tpch_sf<SF>_sorted.duckdb so it can coexist with the unsorted dataset.
  • Default cluster keys: lineitem:l_shipdate,orders:o_orderdate. Override with --cluster-keys "table:col,table:col,..." (implies --cluster).
  • --cluster errors if combined with --format parquet (tpchgen-rs writes decimals as FIXED_LEN_BYTE_ARRAY, which disables Sirius row-group pruning for the whole file).

TPC-DS

FieldValue
Scripttest/tpcds_performance/generate_tpcds_data.sh
Default formatduckdb
Formatsduckdb, parquet
Default output (duckdb)test_datasets/tpcds_sf<SF>.duckdb
Default output (parquet)test_datasets/tpcds_parquet_sf<SF>
Prerequisitesbuild/release/duckdb
cd test/tpcds_performance && bash generate_tpcds_data.sh <SF> --format <FORMAT> [--output <path>]

Notes:

  • Also extracts TPC-DS query files to test/tpcds_performance/queries/q{1..99}.sql

Prerequisites

For any benchmark that requires the DuckDB binary, check before running:

test -x build/release/duckdb

If missing, tell the user to build first: CMAKE_BUILD_PARALLEL_LEVEL=$(nproc) make

Report Results

  • Benchmark name
  • Output path
  • Format used
  • Whether generation was skipped (output already existed) or completed