Languages

Tree-sitter tokenization, supported languages, per-language calibration for monorepos, and test-integrity coverage per language.

argot is language-agnostic by design. All language-specific logic — import extraction, callee extraction, prose masking, sampleable-range enumeration — is encapsulated in LanguageAdapter implementations. Nothing in the scorer hardcodes a framework, a library, or a corpus.

Tree-sitter tokenization

Hunks are tokenized with tree-sitter, an incremental, error-tolerant parser. Two properties matter here:

• It works on partial, syntactically invalid fragments — essential, because a diff hunk is almost always a mid-block slice that doesn’t parse on its own.
• It gives a single uniform interface for every supported language, so adding a language is a matter of writing an adapter, not a new pipeline.

Supported out of the box

Python, TypeScript, JavaScript, Go, Rust, Java, C#, C, C++, Ruby, PHP, and Pascal — twelve languages, each with its own tree-sitter adapter, each benchmarked on a real open-source corpus:

Each adapter carries the full language-specific surface — import extraction, callee extraction, prose masking, definition/binding resolution, and test-shape extraction for the integrity rules.

TypeScript and JavaScript are separate adapters and separate models: .ts/.tsx and .js/.jsx are written differently, so argot learns each voice on its own (and treats a TypeScript repo’s transpiled .js output as generated, not authored). We publish the leak-free per-corpus numbers — catch rate and false alarms, with commit-level confidence intervals — on the benchmarks page. Nothing is hidden.

More languages are adapter-shaped work — the model and pipeline don’t change.

Per-language calibration

A mixed Python + TypeScript monorepo would, with a single threshold, calibrate against a joint distribution dominated by whichever language has broader token diversity. argot instead emits one threshold per language present in the repo, and check dispatches each hunk by file extension.

So a TypeScript hunk is judged against the repo’s TypeScript voice, and a Python hunk against its Python voice — no cross-language bleed. This is automatic; there’s nothing to configure.

What gets excluded

The token model only makes sense over code, so a few things are kept out of both training and scoring:

• Data-dominant files — modules that are ≥80% top-level array/object literals (locale tables, fixtures, generated lookups). The same structural predicate runs at fit and check time.
• Comments and docstrings — blanked before scoring, so prose doesn’t inflate the surprise signal.
• Test files and conventional directories — skipped by the built-in argot:recommended set, which you can extend or replace with argot.toml’s [exclude]. See Configure.

Test-integrity coverage

The integrity rule group reads tests themselves, so it has its own per-language layer: an adapter per language extracts test cases, assertion sites, skip/disable markers, and expected literals from each ecosystem’s own conventions (pytest/unittest, it/test blocks, #[test], @Test, xUnit attributes, gtest/Catch2 macros, FPCUnit/DUnit procedure Test*, and more), and the three integrity rules run across all 12 supported languages. Two caveats, published rather than hidden: C has no universal test framework, so coverage is harness-visible only (the curl/redis-style suites it was validated on); plain Go can’t express walker-visible tautology or comparison-widening events — a t.Error* call’s truth lives in the surrounding if-guard, not the call itself — recorded as not-applicable for that language rather than silently skipped.