OpenParable
Skip to content
← Back to skills

Etl Pipeline Builder

Development
etl-pipeline-builder

Etl Pipeline Builder is an Agent Skill for development published in the a5c-ai/babysitter repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Build and manage ETL pipelines for data migration with transformation, CDC, and monitoring

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/a5c-ai/babysitter/blob/HEAD/library/specializations/code-migration-modernization/skills/etl-pipeline-builder/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/etl-pipeline-builder, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

ETL Pipeline Builder Skill

Builds and manages ETL (Extract, Transform, Load) pipelines for data migration, supporting incremental loads, CDC, and comprehensive monitoring.

Purpose

Enable data pipeline creation for:

  • Source-to-target mapping
  • Transformation definition
  • Incremental load setup
  • CDC configuration
  • Pipeline monitoring

Capabilities

1. Source-to-Target Mapping

  • Define column mappings
  • Handle schema differences
  • Configure data type conversions
  • Manage derived columns

2. Transformation Definition

  • Data type transformations
  • Value mappings
  • Aggregations
  • Lookups and enrichments

3. Incremental Load Setup

  • Define watermarks
  • Configure incremental columns
  • Handle deletes
  • Manage merge logic

4. CDC Configuration

  • Log-based CDC
  • Trigger-based CDC
  • Timestamp-based CDC
  • Full load comparison

5. Error Handling

  • Define retry policies
  • Configure dead letter queues
  • Handle data quality issues
  • Implement alerting

6. Pipeline Monitoring

  • Track pipeline metrics
  • Monitor data volumes
  • Alert on failures
  • Generate SLA reports

Tool Integrations

ToolTypeIntegration Method
Apache AirflowOrchestrationPython
dbtTransformationCLI
AirbyteData integrationAPI
FivetranSaaS ETLAPI
AWS DMSCloud migrationCLI
DebeziumCDCConfig

Output Schema

{
  "pipelineId": "string",
  "timestamp": "ISO8601",
  "pipeline": {
    "name": "string",
    "source": {},
    "target": {},
    "mappings": [],
    "transformations": [],
    "schedule": "string"
  },
  "artifacts": {
    "dagFile": "string",
    "configFile": "string",
    "sqlFiles": []
  },
  "deployment": {
    "status": "string",
    "url": "string"
  }
}

Integration with Migration Processes

  • database-schema-migration: Data movement
  • cloud-migration: Cloud data pipelines
  • data-format-migration: Format transformation

Related Skills

  • data-migration-validator: Validation
  • schema-comparator: Schema mapping

Related Agents

  • database-migration-orchestrator: Pipeline orchestration
  • data-architect-agent: Pipeline design