Benchmarks

argot flags what a linter structurally can't: code that is valid, typed and lint-clean but doesn't fit this repo — the kind of thing an AI agent drags in. It has five distinct detectors: four get their own scorecard below, each with its own catch metric and its own false-alarm metric; the fifth catches an AI agent gaming a test to force a failing suite green, benchmarked separately (154/164 (93.9%) — detector-specific fixture catch; controls and accepted-history results are separate; accepted-history controls are reported separately below).

What the headline metric covers. All these detectors fire as ordinary rules — error severity by default (the check fails; test-weakened and the migration-awaresuperseded ship warn), each one configurable to warn oroff in [rules]. The headline catch/over-fire numbers argot is judged on cover detectors 1 & 4; detectors 2 & 3 are measured separately below, and the test-integrity group has its own separate benchmark — none of the three fold into the headline. Every catch number is held-out; every false-alarm number is a leak-free temporal holdout (fit at an old commit, replay only unseen commits). Higher catch, lower false-alarm.

1 · Foreign-pattern detection · the gate

The base voice model, learned entirely from your git history. It fires on two gated classes of 0-usage foreign pattern: a foreign import / dep(a package the repo never imports — the import stage catches it by construction; a foreign concurrency lib is just a dep and folds in here) and aforeign API / unfamiliar callee (a call into a foreign library that isn't an explicit import — the harder call-receiver stage). Catch isvisible (620/637 (97.3%) — detector-specific fixture recall; not a product-wide accuracy claim, the foreign symbol is in the diff) vs masked (a name colliding with one of your own, or a dynamic import() — a documented statistical limit). Numbers are per-corpus, fed from CI so they can't drift from what ships.

Commit-level bootstrap 95% CIs. Numbers published before July 2026 were measured train-on-test and were materially optimistic — seeissue #92. Fed from CI, so the numbers can't drift from what ships.

Public claim provenance: foreign 36 corpora / 12 languages / visible foreign-import and foreign-api fixture symbols after excluded-host purge · d1007f50; generated_at=2026-07-28; integrity 23 corpora / 12 languages; authored test-gaming fixtures, production fit→check · 1b0de32d; just integrity-verify, 2026-07-28. The combined brief has no approved aggregate public claim.

catch by difficulty across 756 graded fixtures / 34 corpora: easy 378/383 (99%) · medium 242/254 (95%) · masked 25/119 (21%) — name collisions & dynamic imports, documented limit

run d1007f50 · 2026-07-28 · ⚠️ marks corpora under the 300-hunk sample bar (excluded from the worst-FP tiles) · false-alarm = over-fire on the repo's own code; novel-pattern detections shown muted

Catch is split by whether the foreign symbol is visible in the diff (easy/medium — an explicit import, FQN call, or distinct API name — the pattern an agent introduces in the open) or masked (hard — a foreign call whose name collides with one of the repo's own, an attested root namespace, or a dynamic import()). Masked foreign is the documented statistical limit of a name-based guardrail. In-vocabulary misuse of a library the repo already uses is a separate, deliberately-ungated class (below), not counted here.

Read the two columns honestly. Foreign import / depvisible catch is near-100% on nearly every repo — an explicit foreign import is caught by construction, so it is the easy, reliable half (foreign concurrency libs are folded in here: they are deps, caught the same way, and gave no distinct signal as their own column). The half that actually varies isforeign API (70–100%): catching a foreign callee that isn't an explicit import is the harder job. Each cell's second line (“N/M hard”) is the masked tier — where the real difficulty hides (import/dep ~22%, API ~20% on the hard cases) — surfaced so no column reads as a clean 100% while its hard cases quietly miss.

Novel-pattern catch aggregates the two gated foreign classes (foreign import / dep — incl. concurrency libs — and foreign API) — this is the metric argot gates on.False alarm is over-fire: a fire on the repo's own existing code (0 new symbols) — the true false alarm, held near zero.Detections (muted, “+x%”) are fires on a symbol that is 0-usage at fit — a real new dependency or API argot correctly flags for review — so they never count against the score. Reinvention and misplacement (below) are measured separately and are not folded into these numbers. ⚠️ marks corpora under the 300-hunk sample bar.

2 · Reinvention · rule redundant

A local code-embedding index — built with a 15.6 MB model compiled into the binary, nothing to download — finds a new function that reimplements one the repo already has.Catch is high — parallel agents write faithful reimplementations of a repo's own functions (renamed, restructured); planted as new code, argot flags them redundant at ≥ 72% on every corpus (median 94%). False-fire is calibrated away, not hidden. Real repos hold real parallel code, and a naïve embedding sense fires on all of it (5–14% of real-commit hunks). argot filters the shapes that aren't reinventions — thin wrappers, interface / family methods (a linter'son_send across 271 cops), dense sibling clusters, co-located protocol-variant families (curl's cf_h3_*next to cf_h2_*) — and atfit it measures itself against your own history: a mini-replay of the repo's recently added functions estimates its false-fire rate, and repos that practice systematic parallel implementation (saleor's checkout/order mirrors) automatically get a stricter mode. On the leak-free clean-commit replay this lands at amedian 0.4%/hunk, ≤ 2% on 30 of 31 corpora — jellyfin the lone exception at 4.5% (its C# tree at the pinned commit both over-fires and under-catches; the twin-rate guard holds every other repo).redundant is measured separately — never folded into the headline catch/over-fire numbers: argot names the nearest existing function and lets you decide. (Like every rule it fails the check by default; set it to warn oroff in [rules], or mute the hit.)

Catch = recall over planted faithful reimplementations (caught/total). False-fire / hunk = share of replayed-commit hunks that fired redundant (fires / hunks scanned); the muted line is the judged-true rate after three independent reviewers removed the genuine reinventions (majority vote, default false-alarm). Shown for every corpus; window 150, leak-free holdout. Measured separately — never folded into the headline catch/over-fire metric.

3 · Misplacement · rule misplaced

The same index judges where a function belongs: if its nearest peers nearly all live in a different package than the one it's filed under, it's probably misplaced. The "package" isn't a fixed directory depth — atfit, argot walks each repo's tree to its real package granularity (descending through containers likesrc/ or src/Illuminate/),merges packages that are semantically entangled (a header-only library's src/ and include/are one thing, not two), and calibrates the vote against itself: a transplant simulation on the repo's own functions picks the thresholds, and when no configuration reaches usable recall the sense is disabled for that repo— a flat single-package library gets silence, not noise (marked “—” below).Catch = transplant recall: a real function re-filed into a foreign but existing package fires misplaced at 85–99% on every judging repo (median 96%)— up from a median of 66% (and literally 0% on composer, whose whole tree sat in one fixed-depth area) before the self-calibration. False-fire = the leak-free clean-commit holdout: ≤ 1.2% of hunks on every repo. The former outliers are gone the honest way — fmt (20%: a header-only layout with no separable architecture) now abstains, and junit5 (5.5%: near-identicaljunit-platform-* siblings) merges them and lands at 0.

Catch = transplant recall (n = functions sampled); “—” marks a repo where the fit-time self-calibration disabled placement — flat single-package or semantically-fused layouts (bat, commander, express, ink, rich, fmt) with no separable architecture to judge; their false-fire is 0 by construction (the sense abstains). False-fire / hunk = replayed-commit hunks that firedmisplaced (fires / hunks scanned), window 150. Measured separately — never folded into the headline catch/over-fire metric.

4 · Architecture · gated

argot builds a directed graph of the repo's own internal module dependencies at fit time — which layer is allowed to import which — and learns the layering the repo actually follows. At check time it flags a module-dependency edgethat breaks it: a reversed dependency (a low-level module reaching back up into a caller), a cycle the edge closes, or an edge out of a sink layer (a models/ file importing views/, a utility reaching into the app layer). It's the relational analog of Foreign-pattern detection above — Foreign catches a foreign import; this catches a foreign edge between the repo's own modules, the shape an agent produces when it wires code together without noticing how the repo is layered. Pure-Rust, no added runtime dependency, and it gates at the same mildest (unusual) tier as Foreign above.

Real recall = planted foreign layering-edge fixtures caught / total (caught/total). Control-FP = planted clean, in-layering edges that should not fire, fires / total —0/148 across every corpus. Over-fire = false-alarm rate on the repo's own replayed history (leak-free holdout, same protocol as Foreign above). Fed from CI so the numbers can't drift from what ships; gated at the mildest (unusual) tier.