braindb-custom-profile

Authoring a BrainDB custom profile

A custom profile lets you (a) shape how the wiki maintainer/writer name and structure pages and (b) feed BrainDB a custom ingestion source — with zero effect on default behaviour when the profile is not active. One env switch turns a profile on; one self-contained folder holds everything it needs.

The committed custom-profiles/hackernews/ folder is a complete, keyless worked example to copy from. custom-profiles/README.md is the short contract; this is the deep how-to.

The model

• One switch: CUSTOM_PROFILE in the repo-root .env names the active profile(s), comma-separated. Unset ⇒ every prompt is byte-identical to the baked-in default and the ingestor supervisor sleeps.
• One folder: custom-profiles/<name>/ holds the profile's prompt fragments, an optional ingestor.py, and an optional .env.

Folder contract

custom-profiles/<name>/
├── wiki_maintainer.add.md      # appended to the maintainer prompt
├── wiki_maintainer.replace.md  # OR replaces it entirely (advanced)
├── wiki_writer.add.md          # appended to the writer prompt
├── wiki_writer.replace.md      # OR replaces it entirely (advanced)
├── ingestor.py                 # optional: a standalone source feeder
├── .env                        # optional: the ingestor's own config/secrets
└── README.md

Targets are wiki_maintainer and wiki_writer. <target>.add.md is appended to the base prompt; <target>.replace.md replaces it. With several active profiles, add fragments are concatenated in CUSTOM_PROFILE order. All files are optional.

What each prompt shapes

• wiki_maintainer decides, per orphan entity: skip / create (with a proposed_name) / attach / consolidate. Shape naming and dedup here (e.g. "name companies by official name; same ticker = same company").
• wiki_writer authors the page body (free markdown). Shape structure here — the sections, a labelled profile block, a dated chronicle.

Rules that keep it safe

1. Prefer add over replace. add keeps the base prompt — and all the machine contract it carries — intact and only layers your guidance on top. replace makes you re-own that contract.
2. Never break the machine contract (the writer body is parsed for these):
   • [[ref:UUID]] inline citations — how a wiki links to its evidence;
   • <!-- section:NAME --> markers — the section system (arbitrary NAMEs are allowed, so you may add e.g. profile, background, current-developments);
   • the <!-- wiki:meta canonical_name=… keywords=a;b --> header — the only source of a page's keywords.
3. Cite or (unknown) — never invent. A structured field gets a value and its [[ref:UUID]] only when a source supports it; otherwise write (unknown).
4. Make the clustering entity salient in the dropped file. Facts cluster into a wiki by the keyword they're tagged with, so put the entity (ticker, company, project) prominently in the file (e.g. a **Tickers:** MSFT header line) so the extractor tags facts with it.

The ingestor (optional)

A profile may ship ingestor.py: a standalone, long-running loop that feeds BrainDB. The committed profile_runner sidecar launches each active profile's ingestor.py as an isolated subprocess (restart-on-exit), and sleeps when no profile is active — so a broken ingestor can never affect the api/watcher/wiki.

The simplest, least-coupled ingestor just writes files into data/sources/ and lets the existing watcher do ingestion + fact-extraction — no DB or agent calls. Pattern:

• load config from a sibling .env (a tiny parser into os.environ, or python-dotenv);
• poll your source; for each new item (track seen ids in a .state/ file), write one <prefix>-<id>.md into data/sources/, with the entity made salient;
• prune old files under data/sources/ingested/ (the verbatim text is stored in the DB at ingest, so the file is only a carrier);
• sleep and repeat.

Resolve data/sources/ from the script's own location so it works in the container and standalone: Path(__file__).resolve().parents[1] / "data" / "sources". Load secrets from the sibling .env so no profile-specific variable ever appears in the public docker-compose.yml. Keep files small (title + a few fields).

Activate and verify

1. (if the profile has secrets) cp custom-profiles/<name>/.env.example custom-profiles/<name>/.env and fill it.
2. In the repo-root .env: CUSTOM_PROFILE=<name> (or a,b for several).
3. docker compose up -d — the api picks up the prompt shaping; profile_runner launches the ingestor.
4. Watch it flow:

   ls data/sources/                                   # files dropped by the ingestor
   curl -s "http://localhost:8000/api/v1/entities?entity_type=wiki&limit=20"
   curl -s -X POST http://localhost:8000/api/v1/memory/context \
     -H "Content-Type: application/json" -d '{"queries":["<an entity name>"]}'
   

To deactivate: remove the CUSTOM_PROFILE line and docker compose up -d — defaults restored.

Worked example

custom-profiles/hackernews/ is a complete, keyless profile (Hacker News → tech-entity wikis). Copy its ingestor.py + wiki_*.add.md as a starting point, then run it with just CUSTOM_PROFILE=hackernews — no key required.

custom-profiles/gdrive/ is a second example: a folder follower that ingests a Google Drive folder incrementally — only new files and, on edits, only the changed sections (a .state manifest + per-doc snapshots, each delta self-describing what part it is and where it belongs). Copy it when your source is a watched folder of documents that change over time rather than a feed.

A note on privacy: keep real profiles that carry a domain prompt or an API key gitignored (see .gitignore); only generic teaching profiles like example/ and hackernews/ are committed.