Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/kryptic-sh/langr

Fast, low-memory single-purpose language detector using an LLM subword vocab and token→language posteriors
https://github.com/kryptic-sh/langr

Last synced: 13 days ago
JSON representation

Fast, low-memory single-purpose language detector using an LLM subword vocab and token→language posteriors

Awesome Lists containing this project

README 

          
# langr



[![CI](https://github.com/kryptic-sh/langr/actions/workflows/ci.yml/badge.svg)](https://github.com/kryptic-sh/langr/actions/workflows/ci.yml)



Fast, low-memory, single-purpose **language detector** for Rust.



`langr` borrows an LLM's subword vocabulary (a HuggingFace `tokenizer.json`,

e.g. Qwen3's Apache-2.0 byte-level BPE) to turn text into token ids, then

aggregates a trained `token -> P(language)` table into a language mixture. No

neural net runs at inference time — just tokenize and sum.



- **Fast** — tokenize + a handful of adds per token. Microsecond range.

- **Low memory** — the model stores only the top-K languages per token, a few MB

  instead of a dense `vocab x languages` matrix.

- **Mixture-aware** — returns the full language breakdown with shares, so "80%

  French, 20% English" falls out directly.



## How it works



Two artifacts, both built offline, loaded at runtime:



1. **Tokenizer** — a pretrained subword vocab. `text -> [token ids]`.

2. **Model** — `token_id -> top-K {language: P(lang | token)}` plus a

   discriminative weight per token.



Detection is one pass:



```text

tokens = tokenize(text)

for t in tokens:

    for (lang, p) in model[t]:      # precomputed top-K

        score[lang] += weight[t] * p

normalize score by sum of weights

-> { fr: 0.80, en: 0.20, ... }

```



Averaging the per-token posteriors yields the mixture. Function-subwords shared

across languages get a low weight (from posterior entropy), so they add little

noise.



## Output schema



```json

{

  "language": "fr",

  "confidence": 0.8,

  "margin": 0.6,

  "is_multilingual": true,

  "languages": [

    { "lang": "fr", "score": 0.8 },

    { "lang": "en", "score": 0.2 }

  ],

  "scored_tokens": 10

}

```



## Quick start



```sh

# 1. fetch a permissive tokenizer vocab (Qwen3, Apache-2.0)

./scripts/fetch-tokenizer.sh



# 2. train a model from the bundled sample corpus (4 languages)

cargo run --release --bin langr-train -- \

  --corpus sample-corpus --tokenizer tokenizer.json --out model.bin



# 3. detect

cargo run --release --bin langr-detect -- \

  --tokenizer tokenizer.json --model model.bin \

  "Bonjour le monde, this is a small test."

```



The bundled `sample-corpus/` is only a smoke test. For real accuracy, train on a

proper multilingual corpus — see below.



## Pretrained model



**langr-v1** — 84 languages (ISO 639-3), Tatoeba + OpenSubtitles, ~3.4 MB.

Neutral-set accuracy **92.6%** (FLORES-200), and `confidence ≥ 0.30 ⇒ ~95%`

precision. It is a derived artifact (not committed); rebuild it exactly or fetch

the release matching `models/manifest.json` (which pins the tokenizer by

SHA-256). Full details, eval, calibration, and licensing in

[`MODEL_CARD.md`](MODEL_CARD.md).



## Training corpus



Layout: one subdirectory per language code, holding UTF-8 text files (any

extension), read line by line.



```text

corpus/

  eng/  news.txt  wiki.txt

  fra/  news.txt

  jpn/  wiki.txt

```



### Build one with `langr-corpus`



The optional `langr-corpus` bin (behind the `corpus` feature) downloads and

prepares a corpus, labeled with uniform 3-char ISO 639-3 codes. Downloads run in

parallel with retries and report failures — no silent truncation. Two sources:



```sh

# Tatoeba: clean per-language sentences (bz2 TSV), discovered from the index

cargo run --release --features corpus --bin langr-corpus -- \

  --source tatoeba --out corpus --test-out test --jobs 10



# OpenSubtitles: conversational subtitles (gzip, OPUS), the informal register.

# Test files are namespaced per source, so this won't clobber Tatoeba's split.

cargo run --release --features corpus --bin langr-corpus -- \

  --source opensubtitles --out corpus --test-out test --jobs 8



# CC-100: CommonCrawl monolingual text (xz); raw web crawl — clean before use.

cargo run --release --features corpus --bin langr-corpus -- \

  --source cc100 --out corpus --no-test --max-sentences 30000 --jobs 10

```



### Mixing registers (formal + informal)



To stay reliable on both proper grammar and casual/slang text, blend a clean

formal source (Tatoeba, Wikipedia) with a clean informal one (OpenSubtitles),

balanced to a similar size per language, and **test on both registers**. Adding

OpenSubtitles to a Tatoeba model, measured on held-out sets of each register:



| Model                          | Formal (Tatoeba) | Informal (subtitles) |

| ------------------------------ | ---------------- | -------------------- |

| Tatoeba only                   | 87.5%            | 66.1%                |

| **Tatoeba + OpenSubtitles**    | 86.4%            | **75.1%**            |

| Tatoeba + OpenSubtitles + Wiki | 83.6%            | 72.3%                |



Informal accuracy jumps ~9 points for a ~1-point formal cost — because subtitles

are _clean_ informal register. **Tatoeba + OpenSubtitles is the recommended

blend.** Two sources that did _not_ help:



- **OPUS-Wikipedia (`wikipedia`)** regressed both registers: its v1.0 mono set

  covers only ~20 languages (so they swamp the balance), and encyclopedic text

  is dense with foreign proper nouns (place/person names) that read as

  cross-lingual noise. For broad clean formal data, use full Wikipedia dumps

  with real markup+entity stripping, not this corpus.

- **Raw CC-100** — see below.



Tatoeba discovers every language from its index (~430), keeps those with at

least `--min` sentences, caps at `--max-sentences`, and splits `--train` into

`corpus//tatoeba.txt` with the remainder into `test/tatoeba/.txt`

(test files are namespaced by source). Eval a register with

`--test-dir test/tatoeba` or `--test-dir test/opensubtitles`. Pass

`--langs eng,fra,jpn` to fetch a specific set.



Coverage is uneven: ~219 languages clear the Tatoeba threshold, but only ~85

have enough data (≥5k sentences) to be production-grade — the rest are a thin

long tail.



> **Data quality beats volume.** Naively adding raw CC-100 to a Tatoeba-trained

> model _lowered_ held-out accuracy (87.5% → 80% on short sentences): web crawl

> carries boilerplate, embedded English, and script/variant collisions (e.g.

> Latin-script Serbian colliding with Croatian). To benefit from web data,

> language-filter each line, dedup/strip boilerplate, normalize script, and

> evaluate on a neutral set like FLORES-200 — not a naive dump.



### Other sources (permissive licenses)



- **Leipzig Corpora Collection** — per-language sentence packs, 250+ langs.

- **Wikipedia dumps** — good tail-language coverage.

- **OSCAR** — bulk CommonCrawl text (same cleaning caveat as CC-100).



Validate on held-out **FLORES-200**. Match your training domain to your input

domain (e.g. informal/social text) for best accuracy.



## Library use



```rust

use langr::Detector;



let detector = Detector::load("tokenizer.json", "model.bin")?;

let result = detector.detect("Bonjour le monde, this is a test.")?;

println!("{} ({:.0}%)", result.language, result.confidence * 100.0);

# Ok::<(), anyhow::Error>(())

```



## Performance



Detection is ~93% tokenization and ~7% arithmetic, so the aggregation is not the

bottleneck — parallelism and letting the compiler vectorize the tokenizer are.



- **`detect_batch`** runs inputs across all cores. On a 32-core box it reaches

  **~1.28M detections/s** (vs ~108k/s single-threaded — 11.9×).

- **Training** parallelizes tokenization across languages: on the same box a

  43-language corpus (~1.5M sentences) trains in **1.45s vs 13.7s**

  single-thread (9.4×).

- **`target-cpu=native`** (in `.cargo/config.toml`) lets LLVM emit AVX2/AVX512

  (or NEON) across `langr` and its dependencies, including the tokenizer. Remove

  that file for portable binaries.

- **`with_max_input_bytes(n)`** caps how much of each input is tokenized.

  Detection is ~93–99% tokenization and cost scales with length, so on long text

  a prefix bounds latency: on 1.3 KB paragraphs, capping to 512 B is ~3.3×

  faster with the dominant language unchanged (longer docs win more). It biases

  the mixture toward the prefix, so leave it off if you need faithful

  full-document breakdowns.



Detection also uses the tokenizer's offset-free `encode_fast` path and a

thread-local accumulator, so the hot path allocates only the result.



Measured with 43 languages held out from Tatoeba: **94% top-1 accuracy** on

short single sentences (higher on longer text), 3.3 MB model, ~9.3 µs/detect

single-threaded.



Reproduce with the bundled examples:



```sh

cargo run --release --example eval  -- -t tokenizer.json -m model.bin -d test

cargo run --release --example bench -- -t tokenizer.json -m model.bin -i test/eng.txt

```



## Licensing & data



`langr` ships **code only**. It bundles no vocab, no corpus, and no trained

model — you fetch a tokenizer and bring your own training data. This keeps the

repository cleanly MIT with no third-party data-license entanglement.



- **Code + Rust dependencies** — all permissive (MIT / Apache-2.0 / BSD / Zlib /

  Unlicense / BSL / Unicode). No copyleft. The source is MIT.

- **Tokenizer vocab** — a _runtime input_ you fetch, not a bundled asset.

  Qwen3's is Apache-2.0. Used with MIT code that is fine; just don't commit or

  relicense it (the repo `.gitignore`s `tokenizer.json`).

- **Training corpora** — bring your own. Note many common corpora are **not**

  freely redistributable: Leipzig is CC BY-NC (non-commercial), Wikipedia and

  FLORES-200 are CC BY-SA (share-alike), CC-100 / OSCAR carry CommonCrawl

  third-party copyright. Tatoeba (CC BY) and UDHR (public domain) are the

  redistributable ones. The repo `.gitignore`s `/corpus/`.

- **Trained `model.bin`** — a derivative statistical table. Whether it counts as

  a derivative of the corpus is legally unsettled; the repo `.gitignore`s

  `*.bin`. If you publish a pretrained model, do it as a **separate release

  artifact** (not in this MIT source tree), trained only on permissive data,

  with its own notice.



### Third-party notices



Binaries statically compile the dependencies. Generate the bundled-license

notice file before distributing binaries:



```sh

cargo install cargo-about   # once

./scripts/gen-notices.sh    # writes THIRD-PARTY-LICENSES.md

```



The accepted-license allowlist lives in `about.toml`; `cargo about` fails if a

dependency ever introduces a license not on it, so copyleft can't slip in

unnoticed.



## License



MIT — see [`LICENSE`](LICENSE).