{"id":13585229,"url":"https://github.com/pemistahl/lingua-py","last_synced_at":"2025-05-14T19:02:35.383Z","repository":{"id":37732198,"uuid":"385555242","full_name":"pemistahl/lingua-py","owner":"pemistahl","description":"The most accurate natural language detection library for Python, suitable for short text and mixed-language text","archived":false,"fork":false,"pushed_at":"2025-03-20T18:44:10.000Z","size":300910,"stargazers_count":1322,"open_issues_count":26,"forks_count":45,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-04-13T13:57:29.639Z","etag":null,"topics":["language-classification","language-detection","language-identification","language-recognition","natural-language-processing","nlp","python-library"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pemistahl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-07-13T09:52:34.000Z","updated_at":"2025-04-12T11:30:46.000Z","dependencies_parsed_at":"2023-02-16T01:16:08.798Z","dependency_job_id":"c18a767c-c01e-4f40-b73c-dd5ba7ebfe46","html_url":"https://github.com/pemistahl/lingua-py","commit_stats":{"total_commits":174,"total_committers":6,"mean_commits":29.0,"dds":"0.37356321839080464","last_synced_commit":"b5fa518ec8326af314fba85df087ae8b6bf1e70f"},"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pemistahl%2Flingua-py","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pemistahl%2Flingua-py/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pemistahl%2Flingua-py/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pemistahl%2Flingua-py/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pemistahl","download_url":"https://codeload.github.com/pemistahl/lingua-py/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248724587,"owners_count":21151559,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["language-classification","language-detection","language-identification","language-recognition","natural-language-processing","nlp","python-library"],"created_at":"2024-08-01T15:04:49.031Z","updated_at":"2025-04-13T13:57:38.381Z","avatar_url":"https://github.com/pemistahl.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n![lingua](https://raw.githubusercontent.com/pemistahl/lingua-py/main/images/logo.png)\n\n[![build status](https://github.com/pemistahl/lingua-rs/actions/workflows/python-build.yml/badge.svg)](https://github.com/pemistahl/lingua-rs/actions/workflows/python-build.yml)\n[![codecov](https://codecov.io/gh/pemistahl/lingua-rs/branch/main/graph/badge.svg)](https://codecov.io/gh/pemistahl/lingua-rs)\n[![supported languages](https://img.shields.io/badge/supported%20languages-75-green.svg)](#3-which-languages-are-supported)\n![supported Python versions](https://img.shields.io/badge/Python-%3E%3D%203.10-blue)\n[![pypi](https://img.shields.io/badge/PYPI-v2.1.0-blue)](https://pypi.org/project/lingua-language-detector)\n[![license](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://www.apache.org/licenses/LICENSE-2.0)\n\u003c/div\u003e\n\n\u003cbr\u003e\n\n## 1. What does this library do?\n\nIts task is simple: It tells you which language some text is written in.\nThis is very useful as a preprocessing step for linguistic data\nin natural language processing applications such as text classification and\nspell checking. Other use cases, for instance, might include routing e-mails\nto the right geographically located customer service department, based on the\ne-mails' languages.\n\n## 2. Why does this library exist?\n\nLanguage detection is often done as part of large machine learning frameworks\nor natural language processing applications. In cases where you don't need\nthe full-fledged functionality of those systems or don't want to learn the\nropes of those, a small flexible library comes in handy.\n\nPython is widely used in natural language processing, so there are a couple\nof comprehensive open source libraries for this task, such as Google's\n[*CLD 2*](https://github.com/CLD2Owners/cld2) and\n[*CLD 3*](https://github.com/google/cld3),\n[*Langid*](https://github.com/saffsd/langid.py),\n[*Simplemma*](https://github.com/adbar/simplemma) and\n[*Langdetect*](https://github.com/Mimino666/langdetect).\nUnfortunately, except for the last one they have two major drawbacks:\n\n1. Detection only works with quite lengthy text fragments. For very short\n   text snippets such as Twitter messages, they do not provide adequate results.\n2. The more languages take part in the decision process, the less accurate are\n   the detection results.\n\n*Lingua* aims at eliminating these problems. She nearly does not need any\nconfiguration and yields pretty accurate results on both long and short text,\neven on single words and phrases. She draws on both rule-based and statistical\nNaive Bayes methods but does not use neural networks or any dictionaries of words.\nShe does not need a connection to any external API or service either.\nOnce the library has been downloaded, it can be used completely offline.\n\n## 3. A short history of this library\n\nThis library started as a pure Python implementation. Python's quick prototyping\ncapabilities made an important contribution to its improvements. Unfortunately,\nthere was always a tradeoff between performance and memory consumption. At first,\n*Lingua's* language models were stored in dictionaries during runtime. This led\nto quick performance at the cost of large memory consumption (more than 3 GB).\nBecause of that, the language models were then stored in NumPy arrays instead of\ndictionaries. Memory consumption reduced to approximately 800 MB but CPU\nperformance dropped significantly. Both approaches were not satisfying.\n\nStarting from version 2.0.0, the pure Python implementation was replaced with\ncompiled Python bindings to the native\n[Rust implementation](https://github.com/pemistahl/lingua-rs) of *Lingua*.\nThis decision has led to both quick performance and a small memory\nfootprint of less than 1 GB. The pure Python implementation is still available\nin a [separate branch](https://github.com/pemistahl/lingua-py/tree/pure-python-impl)\nin this repository and will be kept up-to-date in subsequent 1.* releases.\nThere are environments that do not support native Python extensions such as\n[Juno](https://juno.sh/), so a pure Python implementation is still useful.\nBoth 1.* and 2.* versions will remain available on the Python package index (PyPI).\n\n## 4. Which languages are supported?\n\nCompared to other language detection libraries, *Lingua's* focus is on\n*quality over quantity*, that is, getting detection right for a small set of\nlanguages first before adding new ones. Currently, the following 75 languages\nare supported:\n\n- A\n    - Afrikaans\n    - Albanian\n    - Arabic\n    - Armenian\n    - Azerbaijani\n- B\n    - Basque\n    - Belarusian\n    - Bengali\n    - Norwegian Bokmal\n    - Bosnian\n    - Bulgarian\n- C\n    - Catalan\n    - Chinese\n    - Croatian\n    - Czech\n- D\n    - Danish\n    - Dutch\n- E\n    - English\n    - Esperanto\n    - Estonian\n- F\n    - Finnish\n    - French\n- G\n    - Ganda\n    - Georgian\n    - German\n    - Greek\n    - Gujarati\n- H\n    - Hebrew\n    - Hindi\n    - Hungarian\n- I\n    - Icelandic\n    - Indonesian\n    - Irish\n    - Italian\n- J\n    - Japanese\n- K\n    - Kazakh\n    - Korean\n- L\n    - Latin\n    - Latvian\n    - Lithuanian\n- M\n    - Macedonian\n    - Malay\n    - Maori\n    - Marathi\n    - Mongolian\n- N\n    - Norwegian Nynorsk\n- P\n    - Persian\n    - Polish\n    - Portuguese\n    - Punjabi\n- R\n    - Romanian\n    - Russian\n- S\n    - Serbian\n    - Shona\n    - Slovak\n    - Slovene\n    - Somali\n    - Sotho\n    - Spanish\n    - Swahili\n    - Swedish\n- T\n    - Tagalog\n    - Tamil\n    - Telugu\n    - Thai\n    - Tsonga\n    - Tswana\n    - Turkish\n- U\n    - Ukrainian\n    - Urdu\n- V\n    - Vietnamese\n- W\n    - Welsh\n- X\n    - Xhosa\n- Y\n    - Yoruba\n- Z\n    - Zulu\n\n## 5. How accurate is it?\n\n*Lingua* is able to report accuracy statistics for some bundled test data\navailable for each supported language. The test data for each language is split\ninto three parts:\n\n1. a list of single words with a minimum length of 5 characters\n2. a list of word pairs with a minimum length of 10 characters\n3. a list of complete grammatical sentences of various lengths\n\nBoth the language models and the test data have been created from separate\ndocuments of the [Wortschatz corpora](https://wortschatz.uni-leipzig.de)\noffered by Leipzig University, Germany. Data crawled from various news websites\nhave been used for training, each corpus comprising one million sentences.\nFor testing, corpora made of arbitrarily chosen websites have been used, each\ncomprising ten thousand sentences. From each test corpus, a random unsorted\nsubset of 1000 single words, 1000 word pairs and 1000 sentences has been\nextracted, respectively.\n\nGiven the generated test data, I have compared the detection results of\n*Lingua*, *Langdetect*, *Langid*, *Simplemma*, *CLD 2* and *CLD 3*\nrunning over the data of *Lingua's* supported 75 languages. Languages that are\nnot supported by the other detectors are simply ignored for them during the\ndetection process.\n\nEach of the following sections contains three plots. The bar plot shows the detailed accuracy\nresults for each supported language. The box plots illustrate the distributions of the\naccuracy values for each classifier. The boxes themselves represent the areas which the\nmiddle 50 % of data lie within. Within the colored boxes, the horizontal lines mark the\nmedian of the distributions.\n\n### 5.1 Single word detection\n\n\u003cbr/\u003e\n\n\u003cimg src=\"https://raw.githubusercontent.com/pemistahl/lingua-py/main/images/plots/boxplot-single-words.png\" alt=\"Single Word Detection Performance\" /\u003e\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n\u003cimg src=\"https://raw.githubusercontent.com/pemistahl/lingua-py/main/images/plots/boxplot-single-language-mode-single-words.png\" alt=\"Single Word Detection Performance\" /\u003e\n\n\u003cbr/\u003e\n\n\u003cdetails\u003e\n    \u003csummary\u003eBar plot\u003c/summary\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/pemistahl/lingua-py/main/images/plots/barplot-single-words.png\" alt=\"Single Word Detection Performance\" /\u003e\n\u003c/details\u003e\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n### 5.2 Word pair detection\n\n\u003cbr/\u003e\n\n\u003cimg src=\"https://raw.githubusercontent.com/pemistahl/lingua-py/main/images/plots/boxplot-word-pairs.png\" alt=\"Word Pair Detection Performance\" /\u003e\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n\u003cimg src=\"https://raw.githubusercontent.com/pemistahl/lingua-py/main/images/plots/boxplot-single-language-mode-word-pairs.png\" alt=\"Word Pair Detection Performance\" /\u003e\n\n\u003cbr/\u003e\n\n\u003cdetails\u003e\n    \u003csummary\u003eBar plot\u003c/summary\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/pemistahl/lingua-py/main/images/plots/barplot-word-pairs.png\" alt=\"Word Pair Detection Performance\" /\u003e\n\u003c/details\u003e\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n### 5.3 Sentence detection\n\n\u003cbr/\u003e\n\n\u003cimg src=\"https://raw.githubusercontent.com/pemistahl/lingua-py/main/images/plots/boxplot-sentences.png\" alt=\"Sentence Detection Performance\" /\u003e\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n\u003cimg src=\"https://raw.githubusercontent.com/pemistahl/lingua-py/main/images/plots/boxplot-single-language-mode-sentences.png\" alt=\"Sentence Detection Performance\" /\u003e\n\n\u003cbr/\u003e\n\n\u003cdetails\u003e\n    \u003csummary\u003eBar plot\u003c/summary\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/pemistahl/lingua-py/main/images/plots/barplot-sentences.png\" alt=\"Sentence Detection Performance\" /\u003e\n\u003c/details\u003e\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n### 5.4 Average detection\n\n\u003cbr/\u003e\n\n\u003cimg src=\"https://raw.githubusercontent.com/pemistahl/lingua-py/main/images/plots/boxplot-average.png\" alt=\"Average Detection Performance\" /\u003e\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n\u003cimg src=\"https://raw.githubusercontent.com/pemistahl/lingua-py/main/images/plots/boxplot-single-language-mode-average.png\" alt=\"Average Detection Performance\" /\u003e\n\n\u003cbr/\u003e\n\n\u003cdetails\u003e\n    \u003csummary\u003eBar plot\u003c/summary\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/pemistahl/lingua-py/main/images/plots/barplot-average.png\" alt=\"Average Detection Performance\" /\u003e\n\u003c/details\u003e\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n### 5.5 Mean, median and standard deviation\n\nThe tables found [here](https://github.com/pemistahl/lingua-py/tree/main/tables)\nshow detailed statistics for each language and classifier including mean, median and standard deviation.\n\n## 6. How fast is it?\n\nThe accuracy reporter script measures the time each language detector needs\nto classify 3000 input texts for each of the supported 75 languages. The results\nbelow have been produced on an iMac 3.6 Ghz 8-Core Intel Core i9 with 40 GB RAM.\n\nLingua in [multi-threaded mode](https://github.com/pemistahl/lingua-py#117-single-threaded-versus-multi-threaded-language-detection)\nis one of the fastest algorithms in this comparison. CLD 2 and 3\nare similarly fast as they have been implemented in C or C++. Pure Python libraries\nsuch as Simplemma, Langid or Langdetect a significantly slower.\n\n| Detector                                     |             Time |\n|----------------------------------------------|-----------------:|\n| Lingua (low accuracy mode, multi-threaded)   |         3.00 sec |\n| Lingua (high accuracy mode, multi-threaded)  |         7.97 sec |\n| CLD 2                                        |         8.65 sec |\n| CLD 3                                        |        16.77 sec |\n| Lingua (low accuracy mode, single-threaded)  |        20.46 sec |\n| Lingua (high accuracy mode, single-threaded) |        51.88 sec |\n| Simplemma                                    |  2 min 36.44 sec |\n| Langid                                       |  3 min 50.40 sec |\n| Langdetect                                   | 10 min 43.96 sec |\n\n## 7. Why is it better than other libraries?\n\nEvery language detector uses a probabilistic\n[n-gram](https://en.wikipedia.org/wiki/N-gram) model trained on the character\ndistribution in some training corpus. Most libraries only use n-grams of size 3\n(trigrams) which is satisfactory for detecting the language of longer text\nfragments consisting of multiple sentences. For short phrases or single words,\nhowever, trigrams are not enough. The shorter the input text is, the less\nn-grams are available. The probabilities estimated from such few n-grams are not\nreliable. This is why *Lingua* makes use of n-grams of sizes 1 up to 5 which\nresults in much more accurate prediction of the correct language.\n\nA second important difference is that *Lingua* does not only use such a\nstatistical model, but also a rule-based engine. This engine first determines\nthe alphabet of the input text and searches for characters which are unique\nin one or more languages. If exactly one language can be reliably chosen this\nway, the statistical model is not necessary anymore. In any case, the\nrule-based engine filters out languages that do not satisfy the conditions of\nthe input text. Only then, in a second step, the probabilistic n-gram model is\ntaken into consideration. This makes sense because loading less language models\nmeans less memory consumption and better runtime performance.\n\nIn general, it is always a good idea to restrict the set of languages to be\nconsidered in the classification process using the respective api methods.\nIf you know beforehand that certain languages are never to occur in an input\ntext, do not let those take part in the classification process. The filtering\nmechanism of the rule-based engine is quite good, however, filtering based on\nyour own knowledge of the input text is always preferable.\n\n## 8. Test report generation\n\nIf you want to reproduce the accuracy results above, you can generate the test\nreports yourself for all classifiers and languages by installing\n[Poetry](https://python-poetry.org) and executing:\n\n    poetry install --no-root --only script\n    poetry run python3 scripts/accuracy_reporter.py\n\nAccuracy reports for only a subset of classifiers and / or languages can be created by\npassing command line arguments:\n\n    poetry run python3 scripts/accuracy_reporter.py --detectors cld2 lingua-high-accuracy --languages bulgarian german\n\nFor each detector and language, a test report file is then written into\n[`/accuracy-reports`](https://github.com/pemistahl/lingua-py/tree/main/accuracy-reports).\nAs an example, here is the current output of the *Lingua* German report:\n\n```\n##### German #####\n\n\u003e\u003e\u003e Accuracy on average: 89.27%\n\n\u003e\u003e Detection of 1000 single words (average length: 9 chars)\nAccuracy: 74.20%\nErroneously classified as Dutch: 2.30%, Danish: 2.20%, English: 2.20%, Latin: 1.80%, Bokmal: 1.60%, Italian: 1.30%, Basque: 1.20%, Esperanto: 1.20%, French: 1.20%, Swedish: 0.90%, Afrikaans: 0.70%, Finnish: 0.60%, Nynorsk: 0.60%, Portuguese: 0.60%, Yoruba: 0.60%, Sotho: 0.50%, Tsonga: 0.50%, Welsh: 0.50%, Estonian: 0.40%, Irish: 0.40%, Polish: 0.40%, Spanish: 0.40%, Tswana: 0.40%, Albanian: 0.30%, Icelandic: 0.30%, Tagalog: 0.30%, Bosnian: 0.20%, Catalan: 0.20%, Croatian: 0.20%, Indonesian: 0.20%, Lithuanian: 0.20%, Romanian: 0.20%, Swahili: 0.20%, Zulu: 0.20%, Latvian: 0.10%, Malay: 0.10%, Maori: 0.10%, Slovak: 0.10%, Slovene: 0.10%, Somali: 0.10%, Turkish: 0.10%, Xhosa: 0.10%\n\n\u003e\u003e Detection of 1000 word pairs (average length: 18 chars)\nAccuracy: 93.90%\nErroneously classified as Dutch: 0.90%, Latin: 0.90%, English: 0.70%, Swedish: 0.60%, Danish: 0.50%, French: 0.40%, Bokmal: 0.30%, Irish: 0.20%, Tagalog: 0.20%, Tsonga: 0.20%, Afrikaans: 0.10%, Esperanto: 0.10%, Estonian: 0.10%, Finnish: 0.10%, Italian: 0.10%, Maori: 0.10%, Nynorsk: 0.10%, Somali: 0.10%, Swahili: 0.10%, Turkish: 0.10%, Welsh: 0.10%, Zulu: 0.10%\n\n\u003e\u003e Detection of 1000 sentences (average length: 111 chars)\nAccuracy: 99.70%\nErroneously classified as Dutch: 0.20%, Latin: 0.10%\n```\n\n## 9. How to add it to your project?\n\n*Lingua* is available in the [Python Package Index](https://pypi.org/project/lingua-language-detector)\nand can be installed with:\n\n    pip install lingua-language-detector\n\n## 10. How to build?\n\n*Lingua* requires Python \u003e= 3.10.\nFirst create a virtualenv and install the Python wheel for your platform with `pip`.\n\n```\ngit clone https://github.com/pemistahl/lingua-py.git\ncd lingua-py\npython3 -m venv .venv\nsource .venv/bin/activate\npip install lingua-language-detector\n```\n\nIn the scripts directory, there are Python scripts for writing accuracy reports,\ndrawing plots and writing accuracy values in an HTML table. The dependencies\nfor these scripts are managed by [Poetry](https://python-poetry.org) which\nyou need to install if you have not done so yet. In order to install the script\ndependencies in your virtualenv, run\n\n    poetry install --no-root --only script\n\nThe project makes uses of type annotations which allow for static type checking with\n[Mypy](http://mypy-lang.org). Run the following commands for checking the types:\n\n    poetry install --no-root --only dev\n    poetry run mypy\n\nThe Python source code is formatted with [Black](https://github.com/psf/black):\n\n    poetry run black .\n\n## 11. How to use?\n\n### 11.1 Basic usage\n\n```python\n\u003e\u003e\u003e from lingua import Language, LanguageDetectorBuilder\n\u003e\u003e\u003e languages = [Language.ENGLISH, Language.FRENCH, Language.GERMAN, Language.SPANISH]\n\u003e\u003e\u003e detector = LanguageDetectorBuilder.from_languages(*languages).build()\n\u003e\u003e\u003e language = detector.detect_language_of(\"languages are awesome\")\n\u003e\u003e\u003e language\nLanguage.ENGLISH\n\u003e\u003e\u003e language.iso_code_639_1\nIsoCode639_1.EN\n\u003e\u003e\u003e language.iso_code_639_1.name\n'EN'\n\u003e\u003e\u003e language.iso_code_639_3\nIsoCode639_3.ENG\n\u003e\u003e\u003e language.iso_code_639_3.name\n'ENG'\n```\n\nThe entire library is thread-safe, i.e. you can use a single `LanguageDetector` instance and\nits methods in multiple threads. Multiple instances of `LanguageDetector` share thread-safe\naccess to the language models, so every language model is loaded into memory just once, no\nmatter how many instances of `LanguageDetector` have been created.\n\n### 11.2 Minimum relative distance\n\nBy default, *Lingua* returns the most likely language for a given input text.\nHowever, there are certain words that are spelled the same in more than one\nlanguage. The word *prologue*, for instance, is both a valid English and French\nword. *Lingua* would output either English or French which might be wrong in\nthe given context. For cases like that, it is possible to specify a minimum\nrelative distance that the logarithmized and summed up probabilities for\neach possible language have to satisfy. It can be stated in the following way:\n\n```python\n\u003e\u003e\u003e from lingua import Language, LanguageDetectorBuilder\n\u003e\u003e\u003e languages = [Language.ENGLISH, Language.FRENCH, Language.GERMAN, Language.SPANISH]\n\u003e\u003e\u003e detector = LanguageDetectorBuilder.from_languages(*languages)\\\n.with_minimum_relative_distance(0.9)\\\n.build()\n\u003e\u003e\u003e print(detector.detect_language_of(\"languages are awesome\"))\nNone\n```\n\nBe aware that the distance between the language probabilities is dependent on\nthe length of the input text. The longer the input text, the larger the\ndistance between the languages. So if you want to classify very short text\nphrases, do not set the minimum relative distance too high. Otherwise, `None`\nwill be returned most of the time as in the example above. This is the return\nvalue for cases where language detection is not reliably possible.\n\n### 11.3 Confidence values\n\nKnowing about the most likely language is nice but how reliable is the computed\nlikelihood? And how less likely are the other examined languages in comparison\nto the most likely one? These questions can be answered as well:\n\n```python\n\u003e\u003e\u003e from lingua import Language, LanguageDetectorBuilder\n\u003e\u003e\u003e languages = [Language.ENGLISH, Language.FRENCH, Language.GERMAN, Language.SPANISH]\n\u003e\u003e\u003e detector = LanguageDetectorBuilder.from_languages(*languages).build()\n\u003e\u003e\u003e confidence_values = detector.compute_language_confidence_values(\"languages are awesome\")\n\u003e\u003e\u003e for confidence in confidence_values:\n...     print(f\"{confidence.language.name}: {confidence.value:.2f}\")\nENGLISH: 0.93\nFRENCH: 0.04\nGERMAN: 0.02\nSPANISH: 0.01\n```\n\nIn the example above, a list is returned containing those languages which the\ncalling instance of LanguageDetector has been built from, sorted by\ntheir confidence value in descending order. Each value is a probability between\n0.0 and 1.0. The probabilities of all languages will sum to 1.0.\nIf the language is unambiguously identified by the rule engine, the value 1.0\nwill always be returned for this language. The other languages will receive a\nvalue of 0.0.\n\nThere is also a method for returning the confidence value for one specific\nlanguage only:\n\n```python\n\u003e\u003e\u003e from lingua import Language, LanguageDetectorBuilder\n\u003e\u003e\u003e languages = [Language.ENGLISH, Language.FRENCH, Language.GERMAN, Language.SPANISH]\n\u003e\u003e\u003e detector = LanguageDetectorBuilder.from_languages(*languages).build()\n\u003e\u003e\u003e confidence_value = detector.compute_language_confidence(\"languages are awesome\", Language.FRENCH)\n\u003e\u003e\u003e print(f\"{confidence_value:.2f}\")\n0.04\n```\n\nThe value that this method computes is a number between 0.0 and 1.0. If the\nlanguage is unambiguously identified by the rule engine, the value 1.0 will\nalways be returned. If the given language is not supported by this detector\ninstance, the value 0.0 will always be returned.\n\n### 11.4 Eager loading versus lazy loading\n\nBy default, *Lingua* uses lazy-loading to load only those language models on\ndemand which are considered relevant by the rule-based filter engine. For web\nservices, for instance, it is rather beneficial to preload all language models\ninto memory to avoid unexpected latency while waiting for the service response.\nIf you want to enable the eager-loading mode, you can do it like this:\n\n```python\nLanguageDetectorBuilder.from_all_languages().with_preloaded_language_models().build()\n```\n\nMultiple instances of `LanguageDetector` share the same language models in\nmemory which are accessed asynchronously by the instances.\n\n### 11.5 Low accuracy mode versus high accuracy mode\n\n*Lingua's* high detection accuracy comes at the cost of being noticeably slower\nthan other language detectors. The large language models also consume significant\namounts of memory. These requirements might not be feasible for systems running low\non resources. If you want to classify mostly long texts or need to save resources,\nyou can enable a *low accuracy mode* that loads only a small subset of the language\nmodels into memory:\n\n```python\nLanguageDetectorBuilder.from_all_languages().with_low_accuracy_mode().build()\n```\n\nThe downside of this approach is that detection accuracy for short texts consisting\nof less than 120 characters will drop significantly. However, detection accuracy for\ntexts which are longer than 120 characters will remain mostly unaffected.\n\nIn high accuracy mode (the default), the language detector consumes approximately\n1 GB of memory if all language models are loaded. In low accuracy mode, memory\nconsumption is reduced to approximately 100 MB.\n\nAn alternative for a smaller memory footprint and faster performance is to reduce the set\nof languages when building the language detector. In most cases, it is not advisable to\nbuild the detector from all supported languages. When you have knowledge about\nthe texts you want to classify you can almost always rule out certain languages as impossible\nor unlikely to occur.\n\n### 11.6 Single-language mode\n\nIf you build a `LanguageDetector` from one language only it will operate in single-language mode.\nThis means the detector will try to find out whether a given text has been written in the given language or not.\nIf not, then `None` will be returned, otherwise the given language.\n\nIn single-language mode, the detector decides based on a set of unique and most common n-grams which\nhave been collected beforehand for every supported language. It turns out that unique and most common\nn-grams help to improve accuracy in low accuracy mode, so they are used for that mode as well. In high\naccuracy mode, however, they do not make a significant difference, that's why they are left out.\n\n### 11.7 Detection of multiple languages in mixed-language texts\n\nIn contrast to most other language detectors, *Lingua* is able to detect multiple languages\nin mixed-language texts. This feature can yield quite reasonable results but it is still\nin an experimental state and therefore the detection result is highly dependent on the input\ntext. It works best in high-accuracy mode with multiple long words for each language.\nThe shorter the phrases and their words are, the less accurate are the results. Reducing the\nset of languages when building the language detector can also improve accuracy for this task\nif the languages occurring in the text are equal to the languages supported by the respective\nlanguage detector instance.\n\n```python\n\u003e\u003e\u003e from lingua import Language, LanguageDetectorBuilder\n\u003e\u003e\u003e languages = [Language.ENGLISH, Language.FRENCH, Language.GERMAN]\n\u003e\u003e\u003e detector = LanguageDetectorBuilder.from_languages(*languages).build()\n\u003e\u003e\u003e sentence = \"Parlez-vous français? \" + \\\n...            \"Ich spreche Französisch nur ein bisschen. \" + \\\n...            \"A little bit is better than nothing.\"\n\u003e\u003e\u003e for result in detector.detect_multiple_languages_of(sentence):\n...     print(f\"{result.language.name}: '{sentence[result.start_index:result.end_index]}'\")\nFRENCH: 'Parlez-vous français? '\nGERMAN: 'Ich spreche Französisch nur ein bisschen. '\nENGLISH: 'A little bit is better than nothing.'\n```\n\nIn the example above, a list of\n[`DetectionResult`](https://github.com/pemistahl/lingua-py/blob/pure-python-impl/lingua/detector.py#L148)\nis returned. Each entry in the list describes a contiguous single-language text section,\nproviding start and end indices of the respective substring.\n\n### 11.8 Single-threaded versus multi-threaded language detection\n\nThe `LanguageDetector` methods explained above all operate in a single thread.\nIf you want to classify a very large set of texts, you will probably want to\nuse all available CPU cores efficiently in multiple threads for maximum performance.\n\nEvery single-threaded method has a multi-threaded equivalent that accepts a list of texts\nand returns a list of results.\n\n| Single-threaded                      | Multi-threaded                                   |\n|--------------------------------------|--------------------------------------------------|\n| `detect_language_of`                 | `detect_languages_in_parallel_of`                |\n| `detect_multiple_languages_of`       | `detect_multiple_languages_in_parallel_of`       |\n| `compute_language_confidence_values` | `compute_language_confidence_values_in_parallel` |\n| `compute_language_confidence`        | `compute_language_confidence_in_parallel`        |\n\n### 11.9 Methods to build the LanguageDetector\n\nThere might be classification tasks where you know beforehand that your\nlanguage data is definitely not written in Latin, for instance. The detection\naccuracy can become better in such cases if you exclude certain languages from\nthe decision process or just explicitly include relevant languages:\n\n```python\nfrom lingua import LanguageDetectorBuilder, Language, IsoCode639_1, IsoCode639_3\n\n# Include all languages available in the library.\nLanguageDetectorBuilder.from_all_languages()\n\n# Include only languages that are not yet extinct (= currently excludes Latin).\nLanguageDetectorBuilder.from_all_spoken_languages()\n\n# Include only languages written with Cyrillic script.\nLanguageDetectorBuilder.from_all_languages_with_cyrillic_script()\n\n# Exclude only the Spanish language from the decision algorithm.\nLanguageDetectorBuilder.from_all_languages_without(Language.SPANISH)\n\n# Only decide between English and German.\nLanguageDetectorBuilder.from_languages(Language.ENGLISH, Language.GERMAN)\n\n# Select languages by ISO 639-1 code.\nLanguageDetectorBuilder.from_iso_codes_639_1(IsoCode639_1.EN, IsoCode639_1.DE)\n\n# Select languages by ISO 639-3 code.\nLanguageDetectorBuilder.from_iso_codes_639_3(IsoCode639_3.ENG, IsoCode639_3.DEU)\n```\n\n### 11.10 Differences to native Python enums\n\nAs version \u003e= 2.0 has been implemented in Rust with Python bindings implemented\nwith [PyO3](https://pyo3.rs), there are some limitations with regard to enums.\n[PyO3 does not yet support meta classes](https://github.com/PyO3/pyo3/issues/906),\nthat's why Lingua's enums do not exactly behave like native Python enums.\n\nIf you want to iterate through all members of the `Language` enum, for instance,\nyou can do it like this:\n\n```python\n# This won't work\n# for language in Language:\n#    print(language)\n\n# But this will work\nfor language in sorted(Language.all()):\n    print(language)\n```\n\nPyO3 enums are not subscriptable. If you want to get an enum member dynamically,\nyou can do it like this:\n\n```python\n# This won't work\n# assert Language[\"GERMAN\"] == Language.GERMAN\n\n# But this will work\nassert Language.from_str(\"GERMAN\") == Language.GERMAN\nassert Language.from_str(\"german\") == Language.GERMAN\nassert Language.from_str(\"GeRmAn\") == Language.GERMAN\n```\n\n## 12. What's next for version 2.2.0?\n\nTake a look at the [planned issues](https://github.com/pemistahl/lingua-py/milestone/10).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpemistahl%2Flingua-py","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpemistahl%2Flingua-py","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpemistahl%2Flingua-py/lists"}