{"id":51563917,"url":"https://github.com/kryptic-sh/langr","last_synced_at":"2026-07-10T13:02:28.382Z","repository":{"id":368593134,"uuid":"1285855536","full_name":"kryptic-sh/langr","owner":"kryptic-sh","description":"Fast, low-memory single-purpose language detector using an LLM subword vocab and token→language posteriors","archived":false,"fork":false,"pushed_at":"2026-07-01T10:17:16.000Z","size":108,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-01T10:26:35.787Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kryptic-sh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":["mxaddict"],"ko_fi":"mxaddict"}},"created_at":"2026-07-01T07:54:32.000Z","updated_at":"2026-07-01T10:17:19.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/kryptic-sh/langr","commit_stats":null,"previous_names":["kryptic-sh/langr"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/kryptic-sh/langr","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kryptic-sh%2Flangr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kryptic-sh%2Flangr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kryptic-sh%2Flangr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kryptic-sh%2Flangr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kryptic-sh","download_url":"https://codeload.github.com/kryptic-sh/langr/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kryptic-sh%2Flangr/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35331955,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-10T02:00:06.465Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-10T13:02:22.974Z","updated_at":"2026-07-10T13:02:28.376Z","avatar_url":"https://github.com/kryptic-sh.png","language":"Rust","funding_links":["https://github.com/sponsors/mxaddict","https://ko-fi.com/mxaddict"],"categories":[],"sub_categories":[],"readme":"# langr\n\n[![CI](https://github.com/kryptic-sh/langr/actions/workflows/ci.yml/badge.svg)](https://github.com/kryptic-sh/langr/actions/workflows/ci.yml)\n\nFast, low-memory, single-purpose **language detector** for Rust.\n\n`langr` borrows an LLM's subword vocabulary (a HuggingFace `tokenizer.json`,\ne.g. Qwen3's Apache-2.0 byte-level BPE) to turn text into token ids, then\naggregates a trained `token -\u003e P(language)` table into a language mixture. No\nneural net runs at inference time — just tokenize and sum.\n\n- **Fast** — tokenize + a handful of adds per token. Microsecond range.\n- **Low memory** — the model stores only the top-K languages per token, a few MB\n  instead of a dense `vocab x languages` matrix.\n- **Mixture-aware** — returns the full language breakdown with shares, so \"80%\n  French, 20% English\" falls out directly.\n\n## How it works\n\nTwo artifacts, both built offline, loaded at runtime:\n\n1. **Tokenizer** — a pretrained subword vocab. `text -\u003e [token ids]`.\n2. **Model** — `token_id -\u003e top-K {language: P(lang | token)}` plus a\n   discriminative weight per token.\n\nDetection is one pass:\n\n```text\ntokens = tokenize(text)\nfor t in tokens:\n    for (lang, p) in model[t]:      # precomputed top-K\n        score[lang] += weight[t] * p\nnormalize score by sum of weights\n-\u003e { fr: 0.80, en: 0.20, ... }\n```\n\nAveraging the per-token posteriors yields the mixture. Function-subwords shared\nacross languages get a low weight (from posterior entropy), so they add little\nnoise.\n\n## Output schema\n\n```json\n{\n  \"language\": \"fr\",\n  \"confidence\": 0.8,\n  \"margin\": 0.6,\n  \"is_multilingual\": true,\n  \"languages\": [\n    { \"lang\": \"fr\", \"score\": 0.8 },\n    { \"lang\": \"en\", \"score\": 0.2 }\n  ],\n  \"scored_tokens\": 10\n}\n```\n\n## Quick start\n\n```sh\n# 1. fetch a permissive tokenizer vocab (Qwen3, Apache-2.0)\n./scripts/fetch-tokenizer.sh\n\n# 2. train a model from the bundled sample corpus (4 languages)\ncargo run --release --bin langr-train -- \\\n  --corpus sample-corpus --tokenizer tokenizer.json --out model.bin\n\n# 3. detect\ncargo run --release --bin langr-detect -- \\\n  --tokenizer tokenizer.json --model model.bin \\\n  \"Bonjour le monde, this is a small test.\"\n```\n\nThe bundled `sample-corpus/` is only a smoke test. For real accuracy, train on a\nproper multilingual corpus — see below.\n\n## Pretrained model\n\n**langr-v1** — 84 languages (ISO 639-3), Tatoeba + OpenSubtitles, ~3.4 MB.\nNeutral-set accuracy **92.6%** (FLORES-200), and `confidence ≥ 0.30 ⇒ ~95%`\nprecision. It is a derived artifact (not committed); rebuild it exactly or fetch\nthe release matching `models/manifest.json` (which pins the tokenizer by\nSHA-256). Full details, eval, calibration, and licensing in\n[`MODEL_CARD.md`](MODEL_CARD.md).\n\n## Training corpus\n\nLayout: one subdirectory per language code, holding UTF-8 text files (any\nextension), read line by line.\n\n```text\ncorpus/\n  eng/  news.txt  wiki.txt\n  fra/  news.txt\n  jpn/  wiki.txt\n```\n\n### Build one with `langr-corpus`\n\nThe optional `langr-corpus` bin (behind the `corpus` feature) downloads and\nprepares a corpus, labeled with uniform 3-char ISO 639-3 codes. Downloads run in\nparallel with retries and report failures — no silent truncation. Two sources:\n\n```sh\n# Tatoeba: clean per-language sentences (bz2 TSV), discovered from the index\ncargo run --release --features corpus --bin langr-corpus -- \\\n  --source tatoeba --out corpus --test-out test --jobs 10\n\n# OpenSubtitles: conversational subtitles (gzip, OPUS), the informal register.\n# Test files are namespaced per source, so this won't clobber Tatoeba's split.\ncargo run --release --features corpus --bin langr-corpus -- \\\n  --source opensubtitles --out corpus --test-out test --jobs 8\n\n# CC-100: CommonCrawl monolingual text (xz); raw web crawl — clean before use.\ncargo run --release --features corpus --bin langr-corpus -- \\\n  --source cc100 --out corpus --no-test --max-sentences 30000 --jobs 10\n```\n\n### Mixing registers (formal + informal)\n\nTo stay reliable on both proper grammar and casual/slang text, blend a clean\nformal source (Tatoeba, Wikipedia) with a clean informal one (OpenSubtitles),\nbalanced to a similar size per language, and **test on both registers**. Adding\nOpenSubtitles to a Tatoeba model, measured on held-out sets of each register:\n\n| Model                          | Formal (Tatoeba) | Informal (subtitles) |\n| ------------------------------ | ---------------- | -------------------- |\n| Tatoeba only                   | 87.5%            | 66.1%                |\n| **Tatoeba + OpenSubtitles**    | 86.4%            | **75.1%**            |\n| Tatoeba + OpenSubtitles + Wiki | 83.6%            | 72.3%                |\n\nInformal accuracy jumps ~9 points for a ~1-point formal cost — because subtitles\nare _clean_ informal register. **Tatoeba + OpenSubtitles is the recommended\nblend.** Two sources that did _not_ help:\n\n- **OPUS-Wikipedia (`wikipedia`)** regressed both registers: its v1.0 mono set\n  covers only ~20 languages (so they swamp the balance), and encyclopedic text\n  is dense with foreign proper nouns (place/person names) that read as\n  cross-lingual noise. For broad clean formal data, use full Wikipedia dumps\n  with real markup+entity stripping, not this corpus.\n- **Raw CC-100** — see below.\n\nTatoeba discovers every language from its index (~430), keeps those with at\nleast `--min` sentences, caps at `--max-sentences`, and splits `--train` into\n`corpus/\u003ccode\u003e/tatoeba.txt` with the remainder into `test/tatoeba/\u003ccode\u003e.txt`\n(test files are namespaced by source). Eval a register with\n`--test-dir test/tatoeba` or `--test-dir test/opensubtitles`. Pass\n`--langs eng,fra,jpn` to fetch a specific set.\n\nCoverage is uneven: ~219 languages clear the Tatoeba threshold, but only ~85\nhave enough data (≥5k sentences) to be production-grade — the rest are a thin\nlong tail.\n\n\u003e **Data quality beats volume.** Naively adding raw CC-100 to a Tatoeba-trained\n\u003e model _lowered_ held-out accuracy (87.5% → 80% on short sentences): web crawl\n\u003e carries boilerplate, embedded English, and script/variant collisions (e.g.\n\u003e Latin-script Serbian colliding with Croatian). To benefit from web data,\n\u003e language-filter each line, dedup/strip boilerplate, normalize script, and\n\u003e evaluate on a neutral set like FLORES-200 — not a naive dump.\n\n### Other sources (permissive licenses)\n\n- **Leipzig Corpora Collection** — per-language sentence packs, 250+ langs.\n- **Wikipedia dumps** — good tail-language coverage.\n- **OSCAR** — bulk CommonCrawl text (same cleaning caveat as CC-100).\n\nValidate on held-out **FLORES-200**. Match your training domain to your input\ndomain (e.g. informal/social text) for best accuracy.\n\n## Library use\n\n```rust\nuse langr::Detector;\n\nlet detector = Detector::load(\"tokenizer.json\", \"model.bin\")?;\nlet result = detector.detect(\"Bonjour le monde, this is a test.\")?;\nprintln!(\"{} ({:.0}%)\", result.language, result.confidence * 100.0);\n# Ok::\u003c(), anyhow::Error\u003e(())\n```\n\n## Performance\n\nDetection is ~93% tokenization and ~7% arithmetic, so the aggregation is not the\nbottleneck — parallelism and letting the compiler vectorize the tokenizer are.\n\n- **`detect_batch`** runs inputs across all cores. On a 32-core box it reaches\n  **~1.28M detections/s** (vs ~108k/s single-threaded — 11.9×).\n- **Training** parallelizes tokenization across languages: on the same box a\n  43-language corpus (~1.5M sentences) trains in **1.45s vs 13.7s**\n  single-thread (9.4×).\n- **`target-cpu=native`** (in `.cargo/config.toml`) lets LLVM emit AVX2/AVX512\n  (or NEON) across `langr` and its dependencies, including the tokenizer. Remove\n  that file for portable binaries.\n- **`with_max_input_bytes(n)`** caps how much of each input is tokenized.\n  Detection is ~93–99% tokenization and cost scales with length, so on long text\n  a prefix bounds latency: on 1.3 KB paragraphs, capping to 512 B is ~3.3×\n  faster with the dominant language unchanged (longer docs win more). It biases\n  the mixture toward the prefix, so leave it off if you need faithful\n  full-document breakdowns.\n\nDetection also uses the tokenizer's offset-free `encode_fast` path and a\nthread-local accumulator, so the hot path allocates only the result.\n\nMeasured with 43 languages held out from Tatoeba: **94% top-1 accuracy** on\nshort single sentences (higher on longer text), 3.3 MB model, ~9.3 µs/detect\nsingle-threaded.\n\nReproduce with the bundled examples:\n\n```sh\ncargo run --release --example eval  -- -t tokenizer.json -m model.bin -d test\ncargo run --release --example bench -- -t tokenizer.json -m model.bin -i test/eng.txt\n```\n\n## Licensing \u0026 data\n\n`langr` ships **code only**. It bundles no vocab, no corpus, and no trained\nmodel — you fetch a tokenizer and bring your own training data. This keeps the\nrepository cleanly MIT with no third-party data-license entanglement.\n\n- **Code + Rust dependencies** — all permissive (MIT / Apache-2.0 / BSD / Zlib /\n  Unlicense / BSL / Unicode). No copyleft. The source is MIT.\n- **Tokenizer vocab** — a _runtime input_ you fetch, not a bundled asset.\n  Qwen3's is Apache-2.0. Used with MIT code that is fine; just don't commit or\n  relicense it (the repo `.gitignore`s `tokenizer.json`).\n- **Training corpora** — bring your own. Note many common corpora are **not**\n  freely redistributable: Leipzig is CC BY-NC (non-commercial), Wikipedia and\n  FLORES-200 are CC BY-SA (share-alike), CC-100 / OSCAR carry CommonCrawl\n  third-party copyright. Tatoeba (CC BY) and UDHR (public domain) are the\n  redistributable ones. The repo `.gitignore`s `/corpus/`.\n- **Trained `model.bin`** — a derivative statistical table. Whether it counts as\n  a derivative of the corpus is legally unsettled; the repo `.gitignore`s\n  `*.bin`. If you publish a pretrained model, do it as a **separate release\n  artifact** (not in this MIT source tree), trained only on permissive data,\n  with its own notice.\n\n### Third-party notices\n\nBinaries statically compile the dependencies. Generate the bundled-license\nnotice file before distributing binaries:\n\n```sh\ncargo install cargo-about   # once\n./scripts/gen-notices.sh    # writes THIRD-PARTY-LICENSES.md\n```\n\nThe accepted-license allowlist lives in `about.toml`; `cargo about` fails if a\ndependency ever introduces a license not on it, so copyleft can't slip in\nunnoticed.\n\n## License\n\nMIT — see [`LICENSE`](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkryptic-sh%2Flangr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkryptic-sh%2Flangr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkryptic-sh%2Flangr/lists"}