OpenParable
Skip to content
← Back to skills

Datachain Jobs

DevOps & Security
datachain-jobs

Datachain Jobs is an Agent Skill for devops & security published in the datachain-ai/datachain repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Use when asked about Studio job analytics — compute hours, user spend, failure rates, cost estimation, cluster usage. Generates and maintains dc-knowledge/jobs/index.md.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/datachain-ai/datachain/blob/HEAD/src/datachain/skill/jobs/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/datachain-jobs, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

You are now loaded with the datachain-jobs skill. Maintain a jobs analytics file at dc-knowledge/jobs/index.md. Follow the 3-step flow below exactly.

Step 1 — Check Staleness

python3 {skill_dir}/scripts/jobs.py --plan
  • If "studio_available": false → report the error message and stop.
  • If "up_to_date": true → skip to Step 3.
  • If "up_to_date": false → continue to Step 2.

Step 2 — Fetch & Write

python3 {skill_dir}/scripts/jobs.py --fetch [--days N] [--limit N] [--enrich]
  • Use --days N from the user's request if stated (e.g. "last 7 days" → --days 7). Default: --days 30.
  • Add --enrich only when the question requires duration, workers, or cluster data AND enriched: false in an existing index — tell the user it makes one API call per terminal job.
  • If the script fails → report the error and stop.

Write dc-knowledge/jobs/index.md using EXACTLY this format:

---
generated: <generated from script output>
days_covered: <days_covered>
total_jobs: <filtered_count>
failed_count: <failed_count>
complete_count: <complete_count>
running_count: <running_count>
other_count: <other_count>
enriched: <true|false>
duration_note: "Wall-clock duration (submit→finish). Null when enriched=false or job still running."
truncated: <true|false>
---

## Clusters

| Name | Cloud | Max Workers | Default |
|------|-------|-------------|---------|
| <name> | <cloud_provider> | <max_workers> | <yes if is_default else no> |

## Jobs

| Date | ID | Name | Status | User | Workers | Duration | Cluster | Python |
|------|----|------|--------|------|---------|----------|---------|--------|
| <created_display> | <id> | <name> | <status> | <created_by> | <workers> | <duration_str or —> | <cluster_name or —> | <python_version or —> |

Section rules:

  • Omit ## Clusters if the clusters array is empty.
  • Duration cell: duration_str value (e.g. "9000s") when known, — when null.
  • Workers: always a number (workers field, defaults to 1).
  • Cluster, Python: use — when null.
  • Date column: created_display (YYYY-MM-DD HH:MM UTC).
  • Rows: newest-first (already sorted by script).
  • If truncated: true, add after the table: _(Results truncated at <limit> jobs. Use --limit N for more.)_

Step 3 — Answer

Read dc-knowledge/jobs/index.md and answer the user's question.

Duration arithmetic

Duration cells contain plain seconds strings like "9000s". Parse the integer before s, sum, then convert:

  • Example: filter rows for user "alice" in the last 7 days, sum all Duration values → total seconds → divide by 3600 for hours.
  • If all Duration cells are — (enriched: false) → say: "Duration data requires enrichment. Re-fetch with: python3 {skill_dir}/scripts/jobs.py --fetch --enrich" and offer to do so.

Failure rate

  • Overall: failed_count / total_jobs * 100 from frontmatter.
  • Per user or per day: count rows matching Status = failed in the table.

Price estimation

When the user asks for cost:

  1. If hourly rate unknown → ask: "What is the instance hourly rate in $/hr? (e.g. 3.20 for $3.20/hr)"
  2. If Workers column is all — → ask: "How many workers per job?" or compute single-worker cost and note it.
  3. Compute per job: duration_seconds / 3600 × rate × workers. Group by user/day/cluster as requested.
  4. Present as a table: User | Compute-hours | Est. cost (@$X/hr × N workers)

Per-cluster / per-user analytics

Filter the Jobs table by the Cluster or User column. Aggregate (Ns) Duration values for totals.