Etl Pipeline Builder
Developmentetl-pipeline-builderEtl Pipeline Builder is an Agent Skill for development published in the a5c-ai/babysitter repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.
Build and manage ETL pipelines for data migration with transformation, CDC, and monitoring
QUICK START
How to use this skill
Bring this guide into your coding agent with a prompt tailored to the tool you use.
- Open your project in Codex.
- Copy and paste the prompt below.
- Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/a5c-ai/babysitter/blob/HEAD/library/specializations/code-migration-modernization/skills/etl-pipeline-builder/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/etl-pipeline-builder, but wait for my approval before writing files or running scripts.
Copying this prompt does not install or run the skill. Review third-party files before use.
ETL Pipeline Builder Skill
Builds and manages ETL (Extract, Transform, Load) pipelines for data migration, supporting incremental loads, CDC, and comprehensive monitoring.
Purpose
Enable data pipeline creation for:
- Source-to-target mapping
- Transformation definition
- Incremental load setup
- CDC configuration
- Pipeline monitoring
Capabilities
1. Source-to-Target Mapping
- Define column mappings
- Handle schema differences
- Configure data type conversions
- Manage derived columns
2. Transformation Definition
- Data type transformations
- Value mappings
- Aggregations
- Lookups and enrichments
3. Incremental Load Setup
- Define watermarks
- Configure incremental columns
- Handle deletes
- Manage merge logic
4. CDC Configuration
- Log-based CDC
- Trigger-based CDC
- Timestamp-based CDC
- Full load comparison
5. Error Handling
- Define retry policies
- Configure dead letter queues
- Handle data quality issues
- Implement alerting
6. Pipeline Monitoring
- Track pipeline metrics
- Monitor data volumes
- Alert on failures
- Generate SLA reports
Tool Integrations
| Tool | Type | Integration Method |
|---|---|---|
| Apache Airflow | Orchestration | Python |
| dbt | Transformation | CLI |
| Airbyte | Data integration | API |
| Fivetran | SaaS ETL | API |
| AWS DMS | Cloud migration | CLI |
| Debezium | CDC | Config |
Output Schema
{
"pipelineId": "string",
"timestamp": "ISO8601",
"pipeline": {
"name": "string",
"source": {},
"target": {},
"mappings": [],
"transformations": [],
"schedule": "string"
},
"artifacts": {
"dagFile": "string",
"configFile": "string",
"sqlFiles": []
},
"deployment": {
"status": "string",
"url": "string"
}
}
Integration with Migration Processes
- database-schema-migration: Data movement
- cloud-migration: Cloud data pipelines
- data-format-migration: Format transformation
Related Skills
data-migration-validator: Validationschema-comparator: Schema mapping
Related Agents
database-migration-orchestrator: Pipeline orchestrationdata-architect-agent: Pipeline design