add-grammar

Adding a New Tree-sitter Grammar to diffsitter

Follow this checklist to add support for a new language $ARGUMENTS. If the user did not provide a language name, ask for one before proceeding.

Prerequisites

Read these files first:

• build.rs -- grammars() function and GrammarCompileInfo struct
• src/parse.rs -- FILE_EXTS map and language resolution

Step-by-Step Checklist

Step 1: Add the grammar as a git submodule

git submodule add https://github.com/tree-sitter/tree-sitter-$ARGUMENTS grammars/tree-sitter-$ARGUMENTS
git submodule update --init --recursive

Verify the submodule has a src/ directory containing at minimum parser.c. Check for a scanner.c or scanner.cc file -- this determines whether you need C or C++ compilation.

Step 2: Strip Rust bindings from the grammar repo

Tree-sitter grammar repos typically include Rust bindings that would interfere with diffsitter's custom build process. Remove them:

cd grammars/tree-sitter-$ARGUMENTS
rm -rf bindings/rust Cargo.toml build.rs
cd ../..

This is critical -- if Cargo.toml or build.rs remain in the grammar directory, Cargo may try to compile the grammar as a separate crate and conflict with diffsitter's build script.

Step 3: Add GrammarCompileInfo to build.rs

In build.rs, add a new entry to the grammars() function's vector. The struct has these fields:

GrammarCompileInfo {
    /// The language name -- must match what you use in FILE_EXTS and the
    /// tree_sitter_$ARGUMENTS() constructor function name
    display_name: "$ARGUMENTS",
    /// Path to the grammar root (contains src/ directory)
    path: PathBuf::from("grammars/tree-sitter-$ARGUMENTS"),
    /// C source files in src/ to compile
    c_sources: vec!["parser.c"],           // always include parser.c
    /// C++ source files in src/ to compile (empty if no C++ scanner)
    cpp_sources: vec![],
    ..Default::default()
}

Determining sources:

• parser.c is always present and always compiled as C.
• If src/scanner.c exists, add "scanner.c" to c_sources.
• If src/scanner.cc exists, add "scanner.cc" to cpp_sources. Do NOT put .cc files in c_sources.
• Some grammars have no scanner at all (e.g., json, go, java, c).

Examples from the codebase:

C-only (parser + C scanner):

GrammarCompileInfo {
    display_name: "rust",
    path: PathBuf::from("grammars/tree-sitter-rust"),
    c_sources: vec!["parser.c", "scanner.c"],
    ..Default::default()
}

C + C++ scanner:

GrammarCompileInfo {
    display_name: "ruby",
    path: PathBuf::from("grammars/tree-sitter-ruby"),
    c_sources: vec!["parser.c"],
    cpp_sources: vec!["scanner.cc"],
    ..GrammarCompileInfo::default()
}

Parser only (no scanner):

GrammarCompileInfo {
    display_name: "json",
    path: PathBuf::from("grammars/tree-sitter-json"),
    c_sources: vec!["parser.c"],
    ..Default::default()
}

Step 4: Add file extension mappings to src/parse.rs

Add entries to the FILE_EXTS static phf_map! in src/parse.rs. The key is the file extension (without dot), the value is the display_name from Step 3:

static FILE_EXTS: phf::Map<&'static str, &'static str> = phf_map! {
    // ... existing entries ...
    "$EXT" => "$ARGUMENTS",
};

Add all common extensions for the language. For example, C++ has "cc", "cpp", "hpp", and "tpp".

Step 5: Build and verify

cargo build

The build script will:

1. Compile the grammar's C/C++ sources via the cc crate
2. Generate an unsafe extern "C" { pub fn tree_sitter_$ARGUMENTS() -> Language; } declaration
3. Add the language to the generated LANGUAGES phf_map

If the build fails, check:

• Are the source file paths correct? The build script prepends {path}/src/ to each filename.
• Does the grammar's src/ directory exist? Run ls grammars/tree-sitter-$ARGUMENTS/src/.
• Are git submodules initialized? Run git submodule update --init --recursive.

Step 6: Add test data (optional but recommended)

If you want to add integration or snapshot tests, add test input files under resources/ and write tests that parse them.

Run the full test suite:

cargo test --all

The static_load_parsers test in src/parse.rs will automatically test that the new grammar can be loaded by tree-sitter, since it iterates over all entries in the LANGUAGES map.

Common Pitfalls

Monorepo grammars

Some tree-sitter grammars contain multiple languages in one repository. In this case, the path field must point to the subdirectory containing the src/ folder, not the repo root.

Examples from the codebase:

• TypeScript: The tree-sitter-typescript repo has typescript/ and tsx/ subdirectories, each with their own src/:

  GrammarCompileInfo {
      display_name: "typescript",
      path: PathBuf::from("grammars/tree-sitter-typescript/typescript"),
      // ...
  }
  GrammarCompileInfo {
      display_name: "tsx",
      path: PathBuf::from("grammars/tree-sitter-typescript/tsx"),
      // ...
  }
  

• OCaml: tree-sitter-ocaml has grammars/ocaml/ subdirectory:

  path: PathBuf::from("grammars/tree-sitter-ocaml/grammars/ocaml"),
  

• PHP: tree-sitter-php has a php/ subdirectory:

  path: PathBuf::from("grammars/tree-sitter-php/php"),
  

• Markdown: tree-sitter-markdown has a tree-sitter-markdown/ subdirectory:

  path: PathBuf::from("grammars/tree-sitter-markdown/tree-sitter-markdown"),
  

C++ scanner pitfalls

If the grammar has a C++ scanner (scanner.cc), it must go in cpp_sources, not c_sources. The build script compiles C and C++ sources with different compilers (cc vs c++) and links them into separate static libraries.

Non-standard include paths

If the grammar's headers are not in src/ (the default include path), use the include_paths field:

GrammarCompileInfo {
    display_name: "some_lang",
    path: PathBuf::from("grammars/tree-sitter-some-lang"),
    c_sources: vec!["parser.c"],
    include_paths: Some(vec![
        PathBuf::from("grammars/tree-sitter-some-lang/include"),
    ]),
    ..Default::default()
}

display_name must match the constructor function

The display_name is used to generate the FFI symbol name tree_sitter_{display_name}. This must match the actual symbol exported by the compiled grammar. For languages with hyphens, use underscores in the display name (e.g., c_sharp for tree-sitter-c-sharp).

Grammar submodule coordination

Grammar submodule updates are coordinated via the diffsitter-grammars repository using nvchecker. For ongoing maintenance, consider adding the grammar there too.

ABI compatibility

The grammar must have a compatible tree-sitter ABI version. The ts_language_abi_checked function in src/parse.rs verifies this at runtime, checking that the grammar's ABI version falls within MIN_COMPATIBLE_LANGUAGE_VERSION..=LANGUAGE_VERSION. If you see an AbiOutOfRange error, the grammar may need to be rebuilt with a compatible tree-sitter version.