{"id":51546156,"url":"https://github.com/nitotm/eldc","last_synced_at":"2026-07-09T18:30:52.567Z","repository":{"id":364763309,"uuid":"1256260767","full_name":"nitotm/eldc","owner":"nitotm","description":"The fastest high-accuracy natural language detector. Written in C.","archived":false,"fork":false,"pushed_at":"2026-07-05T13:28:22.000Z","size":43984,"stargazers_count":8,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-07-05T15:12:58.127Z","etag":null,"topics":["c","language","language-classification","language-detection","language-detector","language-identification","languages-classifier","natural-language","natural-language-processing","nlp"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nitotm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":"nitotm","ko_fi":"nitotm","custom":"https://linktr.ee/nitotm"}},"created_at":"2026-06-01T15:57:35.000Z","updated_at":"2026-07-05T13:28:26.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/nitotm/eldc","commit_stats":null,"previous_names":["nitotm/eldc"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/nitotm/eldc","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nitotm%2Feldc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nitotm%2Feldc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nitotm%2Feldc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nitotm%2Feldc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nitotm","download_url":"https://codeload.github.com/nitotm/eldc/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nitotm%2Feldc/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35309827,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-09T02:00:07.329Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["c","language","language-classification","language-detection","language-detector","language-identification","languages-classifier","natural-language","natural-language-processing","nlp"],"created_at":"2026-07-09T18:30:52.067Z","updated_at":"2026-07-09T18:30:52.557Z","avatar_url":"https://github.com/nitotm.png","language":"C","funding_links":["https://github.com/sponsors/nitotm","https://ko-fi.com/nitotm","https://linktr.ee/nitotm"],"categories":[],"sub_categories":[],"readme":"# Efficient Language Detector - C\n\n\u003cdiv align=\"center\"\u003e\n\n[![license](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://www.apache.org/licenses/LICENSE-2.0)\n[![supported languages](https://img.shields.io/badge/supported%20languages-60-brightgreen.svg)](#languages)\n\t\n\u003c/div\u003e\n\n**What is a language detector?**  \nIt is a tool that identifies which language a text is written in. For example, `detect(\"Hola\")` returns \"es\" for Spanish.\n\n---\n\nEfficient language detector, written in C, is the fastest high-accuracy natural language detector.  \nELDC can be compiled into a library, a command line executable, or easily installed as a python package.  \n\nELD is also available in [PHP](https://github.com/nitotm/efficient-language-detector) (v3), [Javascript](https://github.com/nitotm/efficient-language-detector-js) (v2), and pure Python (v1 outdated). ELD-C (or ELDC) is v3.\n\n\u003e Making \"the fastest\" or most accurate language identification tool can be trivial using unlimited resources, but doing both things while being memory constrained, is what ELD-C has the edge on. It's 2x faster than **Google's CLD2** (previously the fastest decent detector for the last 10 years), and 6x faster than **Facebook's Fasttext**. It's also more accurate, based on the benchmarks below, than **Lingua** (referred to as the *most accurate*), and 100x faster for good measure.\n\n1. [ELDC Python package](#eldc-python-package)\n2. [ELDC Library](#eldc-library)\n3. [Command line executable](#command-line-executable)\n4. [Benchmarks](#benchmarks)\n5. [Languages](#languages)\n6. [More info](#more-info)\n\n## ELDC Python package\n\nBy default, `pip` will install a pre-compiled binary package for your system. If you need to build from source, you will need a C compiler (GCC, Clang, or MSVC) and Python headers, then use command `pip install --no-binary eldc eldc` or `pip install .` to build from local files.\n\n### Installation\n\n```bash\n$ pip install eldc\n```\n\n### How to use?\n\nFull demo at `examples/demo_eldc_package.py`\n```python\nimport eldc\n\neldc.init()\n\n# Optionally, we can make an isolated configuration instance\neldc_instance = eldc.instance()\n\n# We can set a language filter, global setter\neldc.set_languages([\"en\", \"es\", \"fr\"])  # also accepts string \"en, es, fr\"\n# returns a list of the set languages ['en', 'es', 'fr']\neldc.set_languages([])  # reset all\n# Instance setter, works the same way\neldc_instance.set_languages([\"en\", \"es\", \"fr\"])\n\n# ISO 639-2/T output, (default iso639-1)\n# eldc.set_scheme(\"iso639-2t\")  global setter\n# eldc_instance.set_scheme(\"iso639-2t\")  instance setter\n\n# Simple detect, returns a string with language code, or \"und\" for undetermined\neldc.detect(\"Bonjour le monde\")  # 'fr'\neldc_instance.detect('Bonjour')  # 'fr'\n# Use detect_mt() for multi-threaded parallel Python threads\n\n# for detect_details() we can choose up to how many scores we want\neldc.set_scores(2) # default 3, max 20. Global setter\neldc_instance.set_scores(2) # Instance setter\n\nr = eldc.detect_details(\"Hola mundo\")  # multi-threaded capable by default\nprint(r.language)  # (str)  'es'\nprint(r.scores)    # (dict) {'es': 0.80, 'pt': 0.57} Scores are between 0 and 1\nprint(r.reliable)  # (bool) True or False\nr = eldc_instance.detect_details(\"Hola mundo\")  # Instance example\n\n# eldc.LANGUAGES and eldc.LANGUAGES_ISO2T return list of all available languages\n```\n\n## ELDC Library\n\nCompile a library for Linux `.so`, Windows `.dll` or Darwin (macOS) `.dylib`. To be used with your preferred programming language.  \nI included demo examples at `examples/` folder for: Java, TypeScript/Node/Js, Go, Rust, .NET/C#, PHP, Ruby, and Python. (Not 100% validated yet)\n\n### Installation\n\nDownload repostory or clone.\n```bash\ngit clone https://github.com/nitotm/eldc.git\ncd eldc/src/eldc\n```\n\n* Linux\n```bash\ngcc -O3 -shared -fPIC -DELD_BUILD_DLL -o libeldc.so  eldc_lib.c -lm\n```\n* Windows MinGW-w64\n```bash\ngcc -O3 -shared -DELD_BUILD_DLL -o eldc.dll eldc_lib.c -lm -Wl,--out-implib,libeldc.a\n```\n* macOS (Darwin)\n```bash\ngcc -O3 -shared -fPIC -DELD_BUILD_DLL -o libeldc.dylib eldc_lib.c -lm\n```\n\n* Windows. Use Developer Command Prompt.\n```bash\ncl /O2 /LD /DELD_BUILD_DLL eldc_lib.c /Fe:eldc.dll\n```\n\n### How to use?\n\nFind complete demos at the root folder of this repository, for each programming language `examples/`: `demo_eldc_lib.py`, `demo_eldc_lib.ts`, `demo_eldc_lib.go`, etc.  \nHere is a simple demo in PHP, as it is quite readable.\n```php\n$ffi = FFI::cdef('\n    typedef struct { const char *language; float score; } EldcScoreItem;\n    typedef struct {\n        const char   *language;\n        int           reliable;\n        int           n_scores;\n        EldcScoreItem scores[20];\n    } EldcDetectResult;\n\n    struct EldcConfig;\n    typedef struct EldcConfig EldcConfig;\n\n    void        eldc_init(void);\n    void        eldc_close(void);\n    const char *eldc_detect(const char *text);\n    void        eldc_detect_details(const char *text, EldcDetectResult *result);\n    const char *eldc_set_languages(const char *codes);\n    void        eldc_set_scheme(const char *scheme);\n    void        eldc_set_scores(int n);\n\n    EldcConfig *eldc_config_create(void);\n    void        eldc_config_free(EldcConfig *cfg);\n    const char *eldc_detect_cfg(EldcConfig *cfg, const char *text);\n    void        eldc_detect_details_cfg(EldcConfig *cfg, const char *text, EldcDetectResult *result);\n    const char *eldc_set_languages_cfg(EldcConfig *cfg, const char *codes);\n    void        eldc_set_scheme_cfg(EldcConfig *cfg, const char *scheme);\n    void        eldc_set_scores_cfg(EldcConfig *cfg, int n);\n', './libeldc.so');  // Windows: 'eldc.dll', macOS: './libeldc.dylib'\n\n$ffi-\u003eeldc_init();\n\n// Optionally, we can make an isolated configuration instance\n$i_config = $ffi-\u003eeldc_config_create();\n\n$ffi-\u003eeldc_detect(\"Bonjour le monde\");        // \"fr\"\n$ffi-\u003eeldc_detect_cfg($i_config, \"Bonjour\");  // \"fr\", same behavior\n\n// detect_details() to retrieve full data\n$r = $ffi-\u003enew(\"EldcDetectResult\");\n$ffi-\u003eeldc_detect_details(\"Bonjour le monde\", FFI::addr($r));\n$r-\u003elanguage;  // string: \"fr\"\n$r-\u003ereliable;  // int: 1 (0 for false, 1 for true)\n$r-\u003en_scores;  // int: 3 (default, up to)\n$r-\u003escores[0]-\u003elanguage;  // string: \"fr\"\n$r-\u003escores[0]-\u003escore;  // float: 0.9016\n// Isolated configuration version example, works the same way\n$ffi-\u003eeldc_detect_details_cfg($i_config, \"Bonjour\", FFI::addr($r));\n\n// Return up to X scores. Default 3, max 20. Global setter.\n$ffi-\u003eeldc_set_scores(2);  \n$r2 = $ffi-\u003enew(\"EldcDetectResult\"); \n$ffi-\u003eeldc_detect_details(\"Bonjour le monde\", FFI::addr($r2));\n$r2-\u003en_scores;  // int: 2\n// Isolated instance setter\n$ffi-\u003eeldc_set_scores_cfg($i_config, 2);\n\n// Set a language subset, returns validated languages\n$ffi-\u003eeldc_set_languages(\"en,fr,de\");  // string: \"en,fr,de\" Global setter\n$ffi-\u003eeldc_detect(\"Hola mundo, bonito dia\");  // string: \"fr\"\n$ffi-\u003eeldc_set_languages(\"\");  // reset\n$ffi-\u003eeldc_set_languages_cfg($i_config, \"en,fr,es\");  // instance setter\n\n$ffi-\u003eeldc_set_scheme(\"iso639-2t\");  // Default \"iso639-1\". Global setter\n$ffi-\u003eeldc_set_scheme_cfg($i_config, \"iso639-2t\");  // instance setter\n$ffi-\u003eeldc_detect(\"Hola mundo, bonito dia\");  // string: \"spa\"\n\n// Cleanup\n$ffi-\u003eeldc_config_free($i_config);\n$ffi-\u003eeldc_close();  // Unloads eldc resources, globally\n```\n\n## Command line executable\n\nThere are 2 versions, standard `eld.c` and multi thread `eld_mt.c`.\nYou might use this executable just to try it, but its input file processing is the fastest ELD-C implementation suitable for production and heavy workloads.\n\n### Installation\n\nDownload repostory or clone.\n```bash\ngit clone https://github.com/nitotm/eldc.git\ncd eldc/src/eldc\n```\n\n* Linux or macOS\n```bash\ngcc -O3 -march=native -o eldc eld.c -lm\n# Or multi thread executable\ngcc -O3 -march=native -o eldc_mt eld_mt.c -lm -lpthread\n```\n\n* Windows MinGW-w64\n```bash\ngcc -O3 -o eldc.exe eld.c -lm -static\n# Or multi thread executable\ngcc -O3 -o eldc_mt.exe eld_mt.c -lm -lpthread -static\n```\n\n* Windows. Use Developer Command Prompt.\n```bash\ncl /O2 eld.c /Fe:eldc.exe\n```\n\n### How to use?\n\nIf we input text (after flags), it will make a single detect; if not, it will read from stdin; one result per line.  \nIf we use `--scores` or `--reliable`, it will return JSON, if not, a simple unquoted string with language code or `und` for undetected.\n```bash\n-h, --help              This message\n    --list-languages    Print all supported codes and exit\n-v, --verbose           Loading info, timing, throughput\n-l, --languages CODES   Restrict to a subset, e.g. -l \"es,en,de,fr\"\n                        Accepts ISO 639-1 or ISO 639-2/T codes.\n-s, --scores [N]        Output compact JSON with top-N normalised [0,1] scores\n                        N must be 1..20; omit N to get all 20.\n                        Example: {\"language\":\"en\",\"scores\":{\"en\":0.9234,...}}\n-r, --reliable          Add \"reliable\" boolean to JSON output\n    --scheme NAME       iso639-1 (default) | iso639-2t\n```\nFor eld_mt.c, we also have the flag `-t, --threads` to limit threads `-t 4`  \n\nExamples: (on Windows use `eldc.exe`)\n```bash\n./eldc \"Bonjour le monde\"\n./eldc -l \"es,en,fr,de\" --scheme iso639-2t \"Hola mundo\"\n./eldc --scores --reliable \"Hello world\"\n./eldc \u003c corpus.txt \u003e results.txt\n./eldc --verbose\n```\n\n## Benchmarks\n\n**Contenders**\n\n| URL                                                      | Version      | Core Language  |\n|:---------------------------------------------------------|:-------------|:---------------|\n| https://github.com/nitotm/eldc/                          | 0.4.0        | C              |\n| https://github.com/pemistahl/lingua-py                   | 2.0.2        | Rust           |\n| https://github.com/facebookresearch/fastText             | 0.9.2        | C++            |\n| https://github.com/CLD2Owners/cld2                       | Aug 21, 2015 | C++            |\n| https://github.com/google/cld3                           | Aug 28, 2020 | C++            |\n| https://github.com/wooorm/franc                          | 7.2.0        | Javascript     |\n\n\nBenchmarks:\n* **Tatoeba**: *18MB*, short sentences from Tatoeba, 50 languages supported by all contenders, up to 10k lines each.\n\u003e * For Tatoeba, I limited all detectors to the 50 languages subset, making the comparison as fair as possible.\n\u003e * Also, Tatoeba is not part of **ELD** training dataset (nor tuning), but it is for **fasttext**\n* **ELD Test**: *10MB*, sentences from the 60 languages supported by ELD, 1000 lines each. Extracted from the 60GB of ELD training data.\n* **Sentences**: *8MB*, sentences from *Lingua* benchmark, minus unsupported languages and Yoruba which had broken characters.\n* **Word pairs** and **Single words**, ~*1MB*, also from Lingua, same 53 languages.\n\nOther notes:\n* **ELDC pyc** is eldc python package.\n* I added **ELDC \\\u003cfile\u003e** bench to show full potential without a wrapper, *ELDC \\\u003cfile\u003e* bench times include: file read, detect \u0026 save results. `./eldc \u003c eld_test.txt \u003e results.txt -v`\n* **ELDC \\\u003cfile\u003e -t 4** stands for: command line with multi thread (4 threads), `./eldc_mt \u003c eld_test.txt \u003e results.txt -v -t 4`\n\n\u003c!--- Time table\n|                       | Tatoeba-50   | ELD test     | Sentences    | Word pairs   | Single words |\n|:----------------------|:------------:|:------------:|:------------:|:------------:|:------------:|\n| **ELDC \u003cfile\u003e**       |     0.43\"    |      0.19\"   |      0.16\"   |     0.05\"    |     0.03\"    |\n| **ELDC \u003cfile\u003e -t 4**  |     0.18\"    |      0.08\"   |      0.06\"   |     0.02\"    |     0.01\"    |\n| **ELDC pyc**          |     0.62\"    |      0.22\"   |      0.17\"   |     0.06\"    |     0.04\"    |\n| **Lingua**            |    98\"       |     27\"      |     24\"      |     8.2\"     |     5.9\"     |\n| **fasttext-subset**   |    12\"       |      2.7\"    |      2.3\"    |     1.2\"     |     1.1\"     |\n| **fasttext-all**      |     --       |      2.4\"    |      2.0\"    |     0.91\"    |     0.73\"    |\n| **CLD2**              |     3.5\"     |      0.71\"   |      0.59\"   |     0.35\"    |     0.32\"    |\n| **Lingua-low**        |    37\"       |     13\"      |     11\"      |     3.0\"     |     2.3\"     |\n| **CLD3**              |    30\"       |      9\"      |      7.4\"    |     2.8\"     |     2.2\"     |\n| **franc**             |    43\"       |     10\"      |      9\"      |     4.1\"     |     3.2\"     |\n--\u003e\nTime execution benchmark:\n\u003cimg alt=\"timetable\" width=\"800\" src=\"https://raw.githubusercontent.com/nitotm/eldc/main/benchmark/time_table.svg\"\u003e\nAccuracy:\n\u003c!-- Accuracy table\n|                     | Tatoeba-50 | ELD test     | Sentences    | Word pairs   | Single words |\n|:--------------------|:----------:|:------------:|:------------:|:------------:|:------------:|\n| **Nito-ELDC**       |   98.7%    | 99.8%        | 99.4%        | 94.7%        | 83.4%        |\n| **Lingua**          |   96.1%    | 99.2%        | 98.7%        | 93.4%        | 80.7%        |\n| **fasttext-subset** |   94.1%    | 98.0%        | 97.9%        | 83.1%        | 67.8%        |\n| **fasttext-all**    |     --     | 97.4%        | 97.6%        | 81.5%        | 65.7%        |\n| **CLD2** *          |   92.1% *  | 98.1%        | 97.4%        | 85.6%        | 70.7%        |\n| **Lingua-low**      |   89.3     | 97.3%        | 96.3%        | 84.1%        | 68.6%        |\n| **CLD3** *          |   88.2% *  | 95.7%        | 96.4%        | 70.9%        | 52.7%        |\n| **franc**           |   76.9%    | 93.8%        | 92.3%        | 67.0%        | 53.8%        |\n--\u003e\n\u003cimg alt=\"accuracy table\" width=\"800\" src=\"https://raw.githubusercontent.com/nitotm/eldc/main/benchmark/accuracy_table.svg\"\u003e  \n\n* **Lingua** participates with 54 languages, **Franc** with 58.  \n* **fasttext** does not have a built-in subset option, so to show its accuracy and speed potential I made two benchmarks, fasttext-all not being limited by any subset at any test  \n* \u003csup style=\"color:#08e\"\u003e*\u003c/sup\u003e Google's **CLD2** \u0026 **CLD3** also lacks subset option, and it's difficult to make a subset even with its option `bestEffort = True`, as usually returns only one language, so it has a comparative disadvantage.\n* Time is normalized: (total lines * time) / processed lines\n\nELD-C comes out as the fastest detector. For reference, with the command line executable an i7-4770 can process files at over 1M lines per second (1GB/15sec.) with only 1 thread.  \n\nI also included a multithreaded version, that can process files almost as fast as the I/O can support, with multithread an i7-4770 jumps to 4M lines per second or 1GB of text in 5 seconds (read 21M lines + classify + store results). In short, it's unnecessarily fast.  \n\nThis feat would be meaningless if it weren't for the fact that it could also be one of the most accurate detectors; which it is for this benchmark. Accuracy is more benchmark dependent, but it is clearly among the most accurate detectors.\n\n## Languages\n\n* These are the 60 supported languages for *Nito-ELDC*.\n\n\u003e Amharic, Arabic, Azerbaijani (Latin), Belarusian, Bulgarian, Bengali, Catalan, Czech, Danish, German, Greek, English, Spanish, Estonian, Basque, Persian, Finnish, French, Gujarati, Hebrew, Hindi, Croatian, Hungarian, Armenian, Icelandic, Italian, Japanese, Georgian, Kannada, Korean, Kurdish (Arabic), Lao, Lithuanian, Latvian, Malayalam, Marathi, Malay (Latin), Dutch, Norwegian, Oriya, Punjabi, Polish, Portuguese, Romanian, Russian, Slovak, Slovene, Albanian, Serbian (Cyrillic), Swedish, Tamil, Telugu, Thai, Tagalog, Turkish, Ukrainian, Urdu, Vietnamese, Yoruba, Chinese\n\n* These are the *ISO 639-1 codes* that include the 60 languages. Plus `'und'` for undetermined  \nIt is the default ELD language scheme. `--scheme iso639-1`\n\n\u003e am, ar, az, be, bg, bn, ca, cs, da, de, el, en, es, et, eu, fa, fi, fr, gu, he, hi, hr, hu, hy, is, it, ja, ka, kn, ko, ku, lo, lt, lv, ml, mr, ms, nl, no, or, pa, pl, pt, ro, ru, sk, sl, sq, sr, sv, ta, te, th, tl, tr, uk, ur, vi, yo, zh\n\n\n* *ISO 639-2/T* codes (which are also valid *639-3*) `--scheme iso639-2t`.\n\n\u003e amh, ara, aze, bel, bul, ben, cat, ces, dan, deu, ell, eng, spa, est, eus, fas, fin, fra, guj, heb, hin, hrv, hun, hye, isl, ita, jpn, kat, kan, kor, kur, lao, lit, lav, mal, mar, msa, nld, nor, ori, pan, pol, por, ron, rus, slk, slv, sqi, srp, swe, tam, tel, tha, tgl, tur, ukr, urd, vie, yor, zho\n  \n***\n\n### More info\n* ELD-C executable is 24MB, with a similar runtime memory use.\n* ELD-C only reads first 1000 bytes of the input string (benchmarks are fair, with all lines under), but could be modded, if you feel an increased `--limit` flag/option is necessary, open a discussion.\n* Unlike other versions of ELD, ELD-C only comes with the 'large' database size, as that is the optimal one, but other sizes could be added.\n* Next improvement could be a better training data set, my own \"small\" 60GB of data are not as clean as I wish, `fineweb-2` looks good.\n\n#### Donations and suggestions\n\nIf you wish to donate for open source improvements, hire me for private modifications, request alternative dataset training, or contact me, please use the following link: https://linktr.ee/nitotm","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnitotm%2Feldc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnitotm%2Feldc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnitotm%2Feldc/lists"}