Add Ingestr Source
Developmentadd-ingestr-sourceAdd Ingestr Source is an Agent Skill for development published in the bruin-data/bruin repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.
Add Bruin CLI support for a new ingestr source. Use when a task asks to implement a new ingestr connection/source type, wire that source into Bruin ingestr assets, update available source tables, add ingestion docs or example assets, regenerate connection schema expectations, or address review feedback for a newly added ingestr source.
How to use this skill
Bring this guide into your coding agent with a prompt tailored to the tool you use.
- Open your project in Codex.
- Copy and paste the prompt below.
- Review files and risks before approving any action.
Inspect the Agent Skill at https://github.com/bruin-data/bruin/blob/HEAD/.agents/skills/add-ingestr-source/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/add-ingestr-source, but wait for my approval before writing files or running scripts.
Copying this prompt does not install or run the skill. Review third-party files before use.
Add Ingestr Source
Workflow
- Read the ingestr docs first: https://getbruin.com/docs/ingestr/. Then read the upstream source-specific docs or PR. Capture the source URI scheme, required URI parameters, optional URI parameters, supported
source_tablevalues, incremental behavior, primary keys, and any file/table hints. - Inspect similar Bruin sources before editing. Prefer an existing package with the same shape: API/table source, OAuth/API-key source, database-like source, or file/path source.
- Implement the connection config and URI builder.
- Wire the connection into config loading, connection manager registration, and ingestr asset source discovery.
- Add source-table registry entries, docs, example assets, tests, and regenerated expectations.
- Run the checks required by the repo instructions before committing.
Implementation Checklist
Add or update these files for a new source named <source>:
pkg/<source>/config.go: theConfigstruct andGetIngestrURI().pkg/<source>/db.go(orclient.go, matching the package you copy from): theClientandNewClient.pkg/config/connections.go: add<Source>Connectionwith YAML/JSON/mapstructure tags andGetName(). Tag every credential/secret field — anything that must not be printed in logs (API keys, tokens, passwords, client secrets, access keys, etc.) — withsensitive:"true"so it is redacted. For fields that hold a path to a credential file (e.g. a service-account JSON or private-key file), usesensitive_file:"true"instead —sensitive:"true"would mask only the path, whilesensitive_file:"true"redacts the file's contents.pkg/config/manager.go: add the connection slice toConnections, then updateAddConnection,DeleteConnection, andMergeFrom.pkg/connection/connection.go: import the package, add aManagermap, addAdd<Source>ConnectionFromConfig, store it inavailableConnectionsandAllConnectionDetails, and process it inNewManagerFromConfigWithContext.pkg/pipeline/pipeline.go: add adefaultMappingentry"<source>": "<source>-default".pkg/ingestr/sources.go: addSourceTablesRegistry["<source>"].docs/ingestion/<source>.md: document.bruin.ymlconfig, ingestr asset YAML, source tables, options, incremental behavior, and example assets.docs/.vitepress/config.mjs: add the docs sidebar entry.pkg/config/manager_test.go: cover the new connection inAddConnection/DeleteConnection/MergeFrom.pkg/config/testdata/simple.yml: add a sample connection block.pkg/config/testdata/simple_win.yml: add the same block (keep in sync withsimple.yml).integration-tests/expectations/expected_connections_schema.json: regenerate after the connection schema changes.
Add every required and optional parameter from the source URI — do not stop at the common ones. Take the authoritative list from the ingestr repo (bruin-data/ingestr) source code — the source's URI/DSN parser — not just the docs, and match each param's name and type exactly.
For sources without fixed tables, add representative source_table formats instead of pretending the source has enumerated tables. File/path sources usually need examples for exact paths, globs, and format hints.
URI Builder Pattern
Build ingestr URIs with net/url.Values; do not concatenate unescaped query parameters.
Validate required fields in deterministic order using a slice, not a map. This keeps user-facing errors stable and makes tests precise.
type requiredField struct {
key string
value string
}
requiredFields := []requiredField{
{"tenant_id", c.TenantID},
{"client_id", c.ClientID},
}
for _, field := range requiredFields {
field.value = strings.TrimSpace(field.value)
if field.value == "" {
return "", fmt.Errorf("%s: %s must be provided", sourceName, field.key)
}
params.Set(field.key, field.value)
}
Use pointer fields for optional numeric values where zero is meaningful, such as max_files: 0. Reject negative limits before adding them to the URI.
Tests
Add focused tests for:
- URI encoding, optional parameters, zero-valued pointer options, missing required fields, and invalid limits in
pkg/<source>. Add<Source>ConnectionFromConfigregistering the client and connection details.Config.AddConnection,Config.DeleteConnection, andConnections.MergeFrom.GetSourceTables("<source>")when source-table examples are added.
Regenerate the connection schema expectation after adding a connection type:
go run -tags="no_duckdb_arrow" . internal connections | jq . > integration-tests/expectations/expected_connections_schema.json
Inspect the generated diff. Ordering-only movement can happen, but keep only intentional changes.
Docs
Base docs on ingestr's source documentation, not guesses. Include:
.bruin.ymlconnection config with required and optional fields.- One minimal ingestr asset.
- Several realistic example assets when
source_tablesyntax has variants. - A table of available fixed tables, or available
source_tableformats for path/query-based sources. - Primary keys, metadata columns, and whether extraction is incremental or full-refresh.
Keep examples executable-looking but generic. Use the same destination pattern as nearby ingestion docs unless the source needs something different.
Version Caveat
Check the pinned ingestr version before bumping it, usually in pkg/python/uv.go. If the upstream source exists only in an unreleased ingestr PR or commit, wire Bruin support but do not bump to a nonexistent release. Mention the release dependency in the final note or PR when relevant.
Validation
During iteration, run focused package tests first. Before finishing app-code changes, follow the repo instructions exactly: run make format and make test.
For docs-only follow-ups, use the smallest relevant validation such as git diff --check, Markdown inspection, or docs build if the repo has a fast docs check.