vdjdb-extract

/extract — VDJdb Data Extraction Skill

Purpose

Extract T-cell receptor antigen-specificity records from arbitrary source files and write a single TSV in VDJdb chunk format. Every value written to the output must be verified back against the original source to prevent hallucinations. This skill feeds directly into /format and /proofread.

Invocation

/extract [path-to-folder-or-file]

The input may be a folder or individual file(s) containing any mix of:

• Supplementary Excel/CSV/TSV tables
• PDF manuscripts or supplementary PDFs
• Plain-text files (FASTA, exported tables)
• 10X Genomics filtered_contig_annotations.csv / clonotype files
• AIRR-format TSV files (productive, v_call, j_call, cdr3_aa columns)
• Adaptive Biotech ImmunoSEQ exports (TSV; gene columns use TCRB/TCRA prefix with zero-padded numbers like TCRBV06-05*01 — flag all gene names for conversion in /format)
• Jupyter notebooks or scripts used by the authors

If the user limits scope (e.g., "only beta chains", "skip MHC data"), respect that limit and note it in the extraction log.

Excel-specific pitfalls (apply when source is .xlsx/.xls)

1. Embedded sub-header rows: Submitters often repeat column headers mid-table to mark new donors or groups. These appear as rows where gene columns contain TCRα, TCRβ, TRAV, TRBV, CDR3α, CDR3β, CDR3 (literal text). Filter by checking both CDR3 and gene columns — some sub-header rows have blank CDR3 cells and TCRα/TCRβ only in gene columns; the simpler "check CDR3 column for header text" filter will miss them.

2. Allele suffix with functionality code: Some cells contain the gene name formatted as TRAV16*01 F (allele + space + IMGT functionality code). Standard regex \*\d+\s*$ fails because F follows the space. Use re.sub(r'\*.*

   #x27;, '', v).strip() to strip everything from * onwards.

3. Excel formula artifacts: Cell merging or formula errors can produce values like TRAJ3+D107:D1082 (gene name + cell reference). Strip everything after + to recover the gene: val.split('+')[0].strip().

4. TRBJ/TRBD column swap: Submitters sometimes place TRBJ before TRBD in their table despite the column header saying the opposite. Always verify by gene name prefix (e.g., TRBJ2-7*01 starting with TRBJ → it is a J gene regardless of which column it's in). Apply swap correction when prefix contradicts column header.

5. Non-standard characters in CDR3: Excel auto-correct, copy-paste artefacts, or annotation notations can introduce characters like #, X, * in CDR3 fields. Exclude rows containing non-20-AA characters; log the exclusion.

6. Frequency as Excel formula: Cells like =I4/26*100 appear as literal strings if the workbook is loaded without data_only=True. Always use data_only=True in openpyxl to get cached computed values.

---

⚠️ Absolute Requirements (Non-Negotiable)

CDR3 sequences

• Must contain only standard amino acids: ARNDCQEGHILKMFPSTWYV
• Canonical form: starts with C, ends with F or W
• Minimum length: 4 residues

Handling non-standard CDR3s:

chunks_with_unconventional_aa/ is only for non-standard amino acids (beyond the 20 canonical). Non-canonical start/end residues stay in chunks/.

Epitope sequences

• Must be standard amino acids only
• If authors describe a chemical modification, a non-peptide antigen, or a long peptide pool: flag prominently in the log and ask before including

References — strict enforcement

Only these formats are acceptable in reference.id:

NEVER invent or guess a PMID. If uncertain, leave blank and ask the user. DOI and preprint URLs must be quoted exactly from the source.

Hallucination prevention (mandatory for every extracted value)

After extracting any amino acid sequence, gene name, species name, MHC allele, or reference ID:

1. Run a grep or direct text search in the original source file to confirm the exact string is present
2. Log the verification result (found / not found / found with minor variant)
3. If the value cannot be confirmed in the source: mark as [UNVERIFIED] and do not include it without explicit user approval

---

Step-by-Step Workflow

Step 1 — Survey the source folder

1. List all files and identify types (PDF, XLS, TSV, CSV, FASTQ, etc.)
2. Note which files are likely to contain: TCR sequences, antigen/epitope data, MHC/HLA data, methods, references
3. Share the inventory with the user before proceeding if the folder contains more than 3 files or has an unclear structure

Step 2 — Build a cross-reference graph

Source data is often spread across multiple files. Build an explicit graph:

1. Identify all ID columns in each file: barcode, clone ID, sample ID, donor ID, clonotype ID, barcode, well ID, etc.
2. Determine which IDs appear in multiple files and can be used to join records
3. Perform the join; log the join keys used

Ambiguities (one-to-many links, missing join keys, contradictory values between files) must be logged immediately. Do not silently pick one option.

Step 3 — Extract TCR complex fields

For each record, extract the following. Leave blank if absent — never use a placeholder string (NA, N/A, null, nan, -, .).

At least one of cdr3.alpha or cdr3.beta must be non-blank per row. Both mhc.a and mhc.b must be filled if MHC data exists.

Cross-reference epitopes against patches/antigen_epitope_species_gene.dict — if the epitope is already known, use the dict's gene and species values.

Step 4 — Extract method fields

These fields directly affect the VDJdb confidence score (0–3) computed by py_src/ScoreFactory.py. Extract them carefully from the methods section.

If the paper uses a method not in the above lists, do not force it into an existing category. Log it as a "Novel method" candidate for extending the VDJdb specification.

Score shortcut: If a PDB structure ID is available, record it in meta.structure.id — this grants score 3 automatically, bypassing all other scoring logic.

Step 5 — Extract metadata fields

Fill as many of the 12 meta columns as the source supports. Leave others blank.

Step 6 — Assemble the output TSV

Canonical column order (must match exactly):

chunk.id
cdr3.alpha  v.alpha  j.alpha  cdr3.beta  v.beta  d.beta  j.beta
species  mhc.a  mhc.b  mhc.class  antigen.epitope  antigen.gene  antigen.species
reference.id
method.identification  method.frequency  method.singlecell  method.sequencing  method.verification
meta.study.id  meta.cell.subset  meta.subset.frequency  meta.subject.cohort  meta.subject.id
meta.replica.id  meta.clone.id  meta.epitope.id  meta.tissue
meta.donor.MHC  meta.donor.MHC.method  meta.structure.id
comment

Formatting rules:

• Tab-separated, UTF-8, Unix line endings (LF)
• chunk.id: sequential integers starting from 1
• Blank fields: truly empty (no quotes, no NA, no -)
• No trailing whitespace; no quoted fields (TSV, not CSV)
• comment is optional — include column only if at least one row has a comment

Step 7 — Write the extraction log

Write <output_basename>_extraction_log.txt containing:

1. Source inventory: all files found, their types, and assigned roles
2. Cross-reference graph: which ID columns were used to link which files, and join cardinality
3. Ambiguities: every unclear/ambiguous case and the decision made (or flagged for user)
4. Verification results: for each field, confirm or report failure to verify in source
5. Unverified values: any value included at user direction despite not being directly verifiable
6. Novel methods: identification/verification methods not in the current VDJdb vocabulary
7. Excluded records: rows dropped and the reason (failed canonical check, missing required fields, etc.)
8. Scope restrictions: any user-imposed limits on what was extracted

---

Validation Reference Files

---

Output

• Primary output: <PMID_xxxxxxx>_unformatted.txt — raw VDJdb TSV (gene names not yet IMGT-normalised)
• Log: <PMID_xxxxxxx>_extraction_log.txt — full provenance and ambiguity record

Suggest the output filename based on PMID if available (e.g., PMID_40713946_unformatted.txt), otherwise use the folder name or ask the user.

---

Next Steps

1. Run /format on the output to normalise gene names, species, and MHC alleles to IMGT standard
2. Run /proofread to validate against QC scripts in py_src/