{"id":13826997,"url":"https://github.com/m3hrdadfi/zabanshenas","last_synced_at":"2025-07-09T02:33:18.774Z","repository":{"id":48321984,"uuid":"338100455","full_name":"m3hrdadfi/zabanshenas","owner":"m3hrdadfi","description":"Zabanshenas is a solution for identifying the most likely language of a piece of written text. Demo (👇 )","archived":false,"fork":false,"pushed_at":"2021-08-02T14:04:49.000Z","size":87975,"stargazers_count":17,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-08-05T09:15:52.585Z","etag":null,"topics":["attention-is-all-you-need","detected-languages","langdetect","language-detector","languages","langugage-recognition","pytorch-implementation","roberta","transformer","transformers"],"latest_commit_sha":null,"homepage":"https://huggingface.co/spaces/m3hrdadfi/zabanshenas","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/m3hrdadfi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-02-11T17:28:36.000Z","updated_at":"2024-05-31T18:43:05.000Z","dependencies_parsed_at":"2022-08-20T08:11:13.972Z","dependency_job_id":null,"html_url":"https://github.com/m3hrdadfi/zabanshenas","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m3hrdadfi%2Fzabanshenas","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m3hrdadfi%2Fzabanshenas/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m3hrdadfi%2Fzabanshenas/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m3hrdadfi%2Fzabanshenas/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/m3hrdadfi","download_url":"https://codeload.github.com/m3hrdadfi/zabanshenas/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225481163,"owners_count":17481162,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention-is-all-you-need","detected-languages","langdetect","language-detector","languages","langugage-recognition","pytorch-implementation","roberta","transformer","transformers"],"created_at":"2024-08-04T09:01:48.160Z","updated_at":"2024-11-20T06:30:43.974Z","avatar_url":"https://github.com/m3hrdadfi.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003eZabanshenas 🕵\u003c/h1\u003e\n\u003cp align=\"center\"\u003e\u003ca href=\"https://doi.org/10.5281/zenodo.5029022\"\u003e\u003cimg src=\"https://zenodo.org/badge/DOI/10.5281/zenodo.5029022.svg\" alt=\"DOI\"\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cbr/\u003e\u003cbr/\u003e\n\nA Transformer-based solution for identifying the most likely language of a written document/text. \n**Zabanshenas** is a Persian word that has two meanings:\n\n- A person who studies linguistics.\n- A way to identify the type of written language.\n\n\u003cbr/\u003e\n\n\n## Introduction \n\nIn this repository, I will use another perspective in creating a language detection model using [Transformers](https://arxiv.org/abs/1706.03762). Nowadays, Transformers have played a massive role in Natural Language Processing fields. In short, Transformers uses an attention mechanism to boost the speed and extract a high level of information (abstraction).\n\nThere are plenty of ways, solutions, and packages to find the language of a written piece of text or document. All of them have their pros and cons. Some able to detect faster and support as many languages as possible. However, in this case, I intend to use Transformers to understand similar groups of languages and cover 235 languages thanks to [WiLI-2018](https://arxiv.org/abs/1801.07779) and the Transformer architecture.\n\nThis model can detect a written language in three different stages: paragraph, sentence, and subset of text between three and four tokens.\n\n## WilI-2018 (Cleaned version)\n\nIt is a benchmark for monolingual written natural language identification (high covering of a specific language). It contains 229,095 paragraphs that cover 235 languages. Language distribution includes 122 Indo-European languages, 22 Austronesian languages, 17 Turkic languages, 14 Uralic languages, 11 Niger-Congo languages, 10 Sino-Tibetan languages, 9 Afro-Asiatic languages, 6 constructed languages, and 24 languages of smaller families. It also consists of groups of similar languages and dialects:\n- Arabic, Egyptian Arabic\n- English, Old English, Scots\n- Standard Chinese, Min Nan Chinese, Hakka Chinese, Literary Chinese, Wu Chinese\n- German, Bavarian, Low German, Palatine German, Ripuarisch, Alemannic German, Pennsylvania German\n- Belarusian, Belarusian (Taraschkewiza)\n- Kurdish, Central Kurdish\n- Indonesian, Minangkabau, Banyumasan, Banjar, Sundanese, Javanese\n- Languages are spoken in India:\n - Maithili, Bhojpuri\n - Bengali, Bishnupriya\n - Konkani, Marathi\n- Russian, Komi-Permyak\n- Persian, Gilaki, Mazanderani\n\nOn the other hand, this dataset addresses low-resources languages, as shown in Fig 1: \n- Chechen\n- Haitian Creole\n- Newari\n- Pampanga \n\n\u003cdetails\u003e\n\u003csummary\u003eMore info:\u003c/summary\u003e\n\n\u003cp align=\"center\"\u003e\n \u003cimg src=\"reports/samples_per_lang.png\" alt=\"\"\u003e\n \u003cbr\u003e\n \u003cem\u003eFig 1: The distribution of samples per language.\u003c/em\u003e\n\u003c/p\u003e\n\nAs well as minor textual snippet languages (word level, character level), as shown in Fig 2, 3:\n\n- Word Level\n - Literary Chinese\n - Japanese\n - Standard Chinese\n - Wu Chinese\n - Chechen\n- Character Level:\n - Chechen\n - Haitian Creole\n - Newari\n - Minangkabau\n - Pampanga\n\n\u003cp align=\"center\"\u003e\n \u003cimg src=\"reports/words_per_lang.png\" alt=\"\"\u003e\n \u003cbr\u003e\n \u003cem\u003eFig 2: The distribution of words per language.\u003c/em\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n \u003cimg src=\"reports/chars_per_lang.png\" alt=\"\"\u003e\n \u003cbr\u003e\n \u003cem\u003eFig 3: The distribution of characters per language.\u003c/em\u003e\n\u003c/p\u003e\n\n\nInstead of using a word or character level of tokenization, I used subword tokenization [BPE](https://arxiv.org/abs/1909.03341) with max sequence length of 512. It allows the Transformer to have a rich vocabulary size while learning meaningful context-independent representations. The distribution of vocabulary in WiLI-2018 is shown in Fig 4.\n\n\u003cp align=\"center\"\u003e\n \u003cimg src=\"reports/tokens_per_lang.png\" alt=\"\"\u003e\n \u003cbr\u003e\n \u003cem\u003eFig 4: The distribution of BPE-tokens per language.\u003c/em\u003e\n\u003c/p\u003e\n\n\u003c/details\u003e\n\n## Model (Architecture)\n[RoBERTa](https://arxiv.org/abs/1907.11692) as a candidate model is used for this specific task with the following procedure. \nFirstly, the model is trained on the WILI-2018 corpus and then fine-tuned as a SequenceClassification task to detect independent and similar languages and dialects. \nThe data is divided into three segments, 70% of the whole data (train + test) as paragraph choices, 15% tokenized into sentences, and what remains, split into the subset of three or five tokens per record to achieve better results.\n\n**In total, the dataset consists of a 1M training set, 11K validation set, and 11K testing set.**\n\n## Evaluation\n\nPrecision, recall, F1 scores for each language and level are presented in the following tables.\n\n\u003cdetails\u003e\n\u003csummary\u003eParagraph level\u003c/summary\u003e\n\n| language | precision | recall | f1-score |\n|:--------------------------------------:|:---------:|:--------:|:--------:|\n| Achinese (ace) | 1.000000 | 0.982143 | 0.990991 |\n| Afrikaans (afr) | 1.000000 | 1.000000 | 1.000000 |\n| Alemannic German (als) | 1.000000 | 0.946429 | 0.972477 |\n| Amharic (amh) | 1.000000 | 0.982143 | 0.990991 |\n| Old English (ang) | 0.981818 | 0.964286 | 0.972973 |\n| Arabic (ara) | 0.846154 | 0.982143 | 0.909091 |\n| Aragonese (arg) | 1.000000 | 1.000000 | 1.000000 |\n| Egyptian Arabic (arz) | 0.979592 | 0.857143 | 0.914286 |\n| Assamese (asm) | 0.981818 | 0.964286 | 0.972973 |\n| Asturian (ast) | 0.964912 | 0.982143 | 0.973451 |\n| Avar (ava) | 0.941176 | 0.905660 | 0.923077 |\n| Aymara (aym) | 0.964912 | 0.982143 | 0.973451 |\n| South Azerbaijani (azb) | 0.965517 | 1.000000 | 0.982456 |\n| Azerbaijani (aze) | 1.000000 | 1.000000 | 1.000000 |\n| Bashkir (bak) | 1.000000 | 0.978261 | 0.989011 |\n| Bavarian (bar) | 0.843750 | 0.964286 | 0.900000 |\n| Central Bikol (bcl) | 1.000000 | 0.982143 | 0.990991 |\n| Belarusian (Taraschkewiza) (be-tarask) | 1.000000 | 0.875000 | 0.933333 |\n| Belarusian (bel) | 0.870968 | 0.964286 | 0.915254 |\n| Bengali (ben) | 0.982143 | 0.982143 | 0.982143 |\n| Bhojpuri (bho) | 1.000000 | 0.928571 | 0.962963 |\n| Banjar (bjn) | 0.981132 | 0.945455 | 0.962963 |\n| Tibetan (bod) | 1.000000 | 0.982143 | 0.990991 |\n| Bosnian (bos) | 0.552632 | 0.375000 | 0.446809 |\n| Bishnupriya (bpy) | 1.000000 | 0.982143 | 0.990991 |\n| Breton (bre) | 1.000000 | 0.964286 | 0.981818 |\n| Bulgarian (bul) | 1.000000 | 0.964286 | 0.981818 |\n| Buryat (bxr) | 0.946429 | 0.946429 | 0.946429 |\n| Catalan (cat) | 0.982143 | 0.982143 | 0.982143 |\n| Chavacano (cbk) | 0.914894 | 0.767857 | 0.834951 |\n| Min Dong (cdo) | 1.000000 | 0.982143 | 0.990991 |\n| Cebuano (ceb) | 1.000000 | 1.000000 | 1.000000 |\n| Czech (ces) | 1.000000 | 1.000000 | 1.000000 |\n| Chechen (che) | 1.000000 | 1.000000 | 1.000000 |\n| Cherokee (chr) | 1.000000 | 0.963636 | 0.981481 |\n| Chuvash (chv) | 0.938776 | 0.958333 | 0.948454 |\n| Central Kurdish (ckb) | 1.000000 | 1.000000 | 1.000000 |\n| Cornish (cor) | 1.000000 | 1.000000 | 1.000000 |\n| Corsican (cos) | 1.000000 | 0.982143 | 0.990991 |\n| Crimean Tatar (crh) | 1.000000 | 0.946429 | 0.972477 |\n| Kashubian (csb) | 1.000000 | 0.963636 | 0.981481 |\n| Welsh (cym) | 1.000000 | 1.000000 | 1.000000 |\n| Danish (dan) | 1.000000 | 1.000000 | 1.000000 |\n| German (deu) | 0.828125 | 0.946429 | 0.883333 |\n| Dimli (diq) | 0.964912 | 0.982143 | 0.973451 |\n| Dhivehi (div) | 1.000000 | 1.000000 | 1.000000 |\n| Lower Sorbian (dsb) | 1.000000 | 0.982143 | 0.990991 |\n| Doteli (dty) | 0.940000 | 0.854545 | 0.895238 |\n| Emilian (egl) | 1.000000 | 0.928571 | 0.962963 |\n| Modern Greek (ell) | 1.000000 | 1.000000 | 1.000000 |\n| English (eng) | 0.588889 | 0.946429 | 0.726027 |\n| Esperanto (epo) | 1.000000 | 0.982143 | 0.990991 |\n| Estonian (est) | 0.963636 | 0.946429 | 0.954955 |\n| Basque (eus) | 1.000000 | 0.982143 | 0.990991 |\n| Extremaduran (ext) | 0.982143 | 0.982143 | 0.982143 |\n| Faroese (fao) | 1.000000 | 1.000000 | 1.000000 |\n| Persian (fas) | 0.948276 | 0.982143 | 0.964912 |\n| Finnish (fin) | 1.000000 | 1.000000 | 1.000000 |\n| French (fra) | 0.710145 | 0.875000 | 0.784000 |\n| Arpitan (frp) | 1.000000 | 0.946429 | 0.972477 |\n| Western Frisian (fry) | 0.982143 | 0.982143 | 0.982143 |\n| Friulian (fur) | 1.000000 | 0.982143 | 0.990991 |\n| Gagauz (gag) | 0.981132 | 0.945455 | 0.962963 |\n| Scottish Gaelic (gla) | 0.982143 | 0.982143 | 0.982143 |\n| Irish (gle) | 0.949153 | 1.000000 | 0.973913 |\n| Galician (glg) | 1.000000 | 1.000000 | 1.000000 |\n| Gilaki (glk) | 0.981132 | 0.945455 | 0.962963 |\n| Manx (glv) | 1.000000 | 1.000000 | 1.000000 |\n| Guarani (grn) | 1.000000 | 0.964286 | 0.981818 |\n| Gujarati (guj) | 1.000000 | 0.982143 | 0.990991 |\n| Hakka Chinese (hak) | 0.981818 | 0.964286 | 0.972973 |\n| Haitian Creole (hat) | 1.000000 | 1.000000 | 1.000000 |\n| Hausa (hau) | 1.000000 | 0.945455 | 0.971963 |\n| Serbo-Croatian (hbs) | 0.448276 | 0.464286 | 0.456140 |\n| Hebrew (heb) | 1.000000 | 0.982143 | 0.990991 |\n| Fiji Hindi (hif) | 0.890909 | 0.890909 | 0.890909 |\n| Hindi (hin) | 0.981481 | 0.946429 | 0.963636 |\n| Croatian (hrv) | 0.500000 | 0.636364 | 0.560000 |\n| Upper Sorbian (hsb) | 0.955556 | 1.000000 | 0.977273 |\n| Hungarian (hun) | 1.000000 | 1.000000 | 1.000000 |\n| Armenian (hye) | 1.000000 | 0.981818 | 0.990826 |\n| Igbo (ibo) | 0.918033 | 1.000000 | 0.957265 |\n| Ido (ido) | 1.000000 | 1.000000 | 1.000000 |\n| Interlingue (ile) | 1.000000 | 0.962264 | 0.980769 |\n| Iloko (ilo) | 0.947368 | 0.964286 | 0.955752 |\n| Interlingua (ina) | 1.000000 | 1.000000 | 1.000000 |\n| Indonesian (ind) | 0.761905 | 0.872727 | 0.813559 |\n| Icelandic (isl) | 1.000000 | 1.000000 | 1.000000 |\n| Italian (ita) | 0.861538 | 1.000000 | 0.925620 |\n| Jamaican Patois (jam) | 1.000000 | 0.946429 | 0.972477 |\n| Javanese (jav) | 0.964912 | 0.982143 | 0.973451 |\n| Lojban (jbo) | 1.000000 | 1.000000 | 1.000000 |\n| Japanese (jpn) | 1.000000 | 1.000000 | 1.000000 |\n| Karakalpak (kaa) | 0.965517 | 1.000000 | 0.982456 |\n| Kabyle (kab) | 1.000000 | 0.964286 | 0.981818 |\n| Kannada (kan) | 0.982143 | 0.982143 | 0.982143 |\n| Georgian (kat) | 1.000000 | 0.964286 | 0.981818 |\n| Kazakh (kaz) | 0.980769 | 0.980769 | 0.980769 |\n| Kabardian (kbd) | 1.000000 | 0.982143 | 0.990991 |\n| Central Khmer (khm) | 0.960784 | 0.875000 | 0.915888 |\n| Kinyarwanda (kin) | 0.981132 | 0.928571 | 0.954128 |\n| Kirghiz (kir) | 1.000000 | 1.000000 | 1.000000 |\n| Komi-Permyak (koi) | 0.962264 | 0.910714 | 0.935780 |\n| Konkani (kok) | 0.964286 | 0.981818 | 0.972973 |\n| Komi (kom) | 1.000000 | 0.962264 | 0.980769 |\n| Korean (kor) | 1.000000 | 1.000000 | 1.000000 |\n| Karachay-Balkar (krc) | 1.000000 | 0.982143 | 0.990991 |\n| Ripuarisch (ksh) | 1.000000 | 0.964286 | 0.981818 |\n| Kurdish (kur) | 1.000000 | 0.964286 | 0.981818 |\n| Ladino (lad) | 1.000000 | 1.000000 | 1.000000 |\n| Lao (lao) | 0.961538 | 0.909091 | 0.934579 |\n| Latin (lat) | 0.877193 | 0.943396 | 0.909091 |\n| Latvian (lav) | 0.963636 | 0.946429 | 0.954955 |\n| Lezghian (lez) | 1.000000 | 0.964286 | 0.981818 |\n| Ligurian (lij) | 1.000000 | 0.964286 | 0.981818 |\n| Limburgan (lim) | 0.938776 | 1.000000 | 0.968421 |\n| Lingala (lin) | 0.980769 | 0.927273 | 0.953271 |\n| Lithuanian (lit) | 0.982456 | 1.000000 | 0.991150 |\n| Lombard (lmo) | 1.000000 | 1.000000 | 1.000000 |\n| Northern Luri (lrc) | 1.000000 | 0.928571 | 0.962963 |\n| Latgalian (ltg) | 1.000000 | 0.982143 | 0.990991 |\n| Luxembourgish (ltz) | 0.949153 | 1.000000 | 0.973913 |\n| Luganda (lug) | 1.000000 | 1.000000 | 1.000000 |\n| Literary Chinese (lzh) | 1.000000 | 1.000000 | 1.000000 |\n| Maithili (mai) | 0.931034 | 0.964286 | 0.947368 |\n| Malayalam (mal) | 1.000000 | 0.982143 | 0.990991 |\n| Banyumasan (map-bms) | 0.977778 | 0.785714 | 0.871287 |\n| Marathi (mar) | 0.949153 | 1.000000 | 0.973913 |\n| Moksha (mdf) | 0.980000 | 0.890909 | 0.933333 |\n| Eastern Mari (mhr) | 0.981818 | 0.964286 | 0.972973 |\n| Minangkabau (min) | 1.000000 | 1.000000 | 1.000000 |\n| Macedonian (mkd) | 1.000000 | 0.981818 | 0.990826 |\n| Malagasy (mlg) | 0.981132 | 1.000000 | 0.990476 |\n| Maltese (mlt) | 0.982456 | 1.000000 | 0.991150 |\n| Min Nan Chinese (nan) | 1.000000 | 1.000000 | 1.000000 |\n| Mongolian (mon) | 1.000000 | 0.981818 | 0.990826 |\n| Maori (mri) | 1.000000 | 1.000000 | 1.000000 |\n| Western Mari (mrj) | 0.982456 | 1.000000 | 0.991150 |\n| Malay (msa) | 0.862069 | 0.892857 | 0.877193 |\n| Mirandese (mwl) | 1.000000 | 0.982143 | 0.990991 |\n| Burmese (mya) | 1.000000 | 1.000000 | 1.000000 |\n| Erzya (myv) | 0.818182 | 0.964286 | 0.885246 |\n| Mazanderani (mzn) | 0.981481 | 1.000000 | 0.990654 |\n| Neapolitan (nap) | 1.000000 | 0.981818 | 0.990826 |\n| Navajo (nav) | 1.000000 | 1.000000 | 1.000000 |\n| Classical Nahuatl (nci) | 0.981481 | 0.946429 | 0.963636 |\n| Low German (nds) | 0.982143 | 0.982143 | 0.982143 |\n| West Low German (nds-nl) | 1.000000 | 1.000000 | 1.000000 |\n| Nepali (macrolanguage) (nep) | 0.881356 | 0.928571 | 0.904348 |\n| Newari (new) | 1.000000 | 0.909091 | 0.952381 |\n| Dutch (nld) | 0.982143 | 0.982143 | 0.982143 |\n| Norwegian Nynorsk (nno) | 1.000000 | 1.000000 | 1.000000 |\n| Bokmål (nob) | 1.000000 | 1.000000 | 1.000000 |\n| Narom (nrm) | 0.981818 | 0.964286 | 0.972973 |\n| Northern Sotho (nso) | 1.000000 | 1.000000 | 1.000000 |\n| Occitan (oci) | 0.903846 | 0.839286 | 0.870370 |\n| Livvi-Karelian (olo) | 0.982456 | 1.000000 | 0.991150 |\n| Oriya (ori) | 0.964912 | 0.982143 | 0.973451 |\n| Oromo (orm) | 0.982143 | 0.982143 | 0.982143 |\n| Ossetian (oss) | 0.982143 | 1.000000 | 0.990991 |\n| Pangasinan (pag) | 0.980000 | 0.875000 | 0.924528 |\n| Pampanga (pam) | 0.928571 | 0.896552 | 0.912281 |\n| Panjabi (pan) | 1.000000 | 1.000000 | 1.000000 |\n| Papiamento (pap) | 1.000000 | 0.964286 | 0.981818 |\n| Picard (pcd) | 0.849057 | 0.849057 | 0.849057 |\n| Pennsylvania German (pdc) | 0.854839 | 0.946429 | 0.898305 |\n| Palatine German (pfl) | 0.946429 | 0.946429 | 0.946429 |\n| Western Panjabi (pnb) | 0.981132 | 0.962963 | 0.971963 |\n| Polish (pol) | 0.933333 | 1.000000 | 0.965517 |\n| Portuguese (por) | 0.774648 | 0.982143 | 0.866142 |\n| Pushto (pus) | 1.000000 | 0.910714 | 0.953271 |\n| Quechua (que) | 0.962963 | 0.928571 | 0.945455 |\n| Tarantino dialect (roa-tara) | 1.000000 | 0.964286 | 0.981818 |\n| Romansh (roh) | 1.000000 | 0.928571 | 0.962963 |\n| Romanian (ron) | 0.965517 | 1.000000 | 0.982456 |\n| Rusyn (rue) | 0.946429 | 0.946429 | 0.946429 |\n| Aromanian (rup) | 0.962963 | 0.928571 | 0.945455 |\n| Russian (rus) | 0.859375 | 0.982143 | 0.916667 |\n| Yakut (sah) | 1.000000 | 0.982143 | 0.990991 |\n| Sanskrit (san) | 0.982143 | 0.982143 | 0.982143 |\n| Sicilian (scn) | 1.000000 | 1.000000 | 1.000000 |\n| Scots (sco) | 0.982143 | 0.982143 | 0.982143 |\n| Samogitian (sgs) | 1.000000 | 0.982143 | 0.990991 |\n| Sinhala (sin) | 0.964912 | 0.982143 | 0.973451 |\n| Slovak (slk) | 1.000000 | 0.982143 | 0.990991 |\n| Slovene (slv) | 1.000000 | 0.981818 | 0.990826 |\n| Northern Sami (sme) | 0.962264 | 0.962264 | 0.962264 |\n| Shona (sna) | 0.933333 | 1.000000 | 0.965517 |\n| Sindhi (snd) | 1.000000 | 1.000000 | 1.000000 |\n| Somali (som) | 0.948276 | 1.000000 | 0.973451 |\n| Spanish (spa) | 0.739130 | 0.910714 | 0.816000 |\n| Albanian (sqi) | 0.982143 | 0.982143 | 0.982143 |\n| Sardinian (srd) | 1.000000 | 0.982143 | 0.990991 |\n| Sranan (srn) | 1.000000 | 1.000000 | 1.000000 |\n| Serbian (srp) | 1.000000 | 0.946429 | 0.972477 |\n| Saterfriesisch (stq) | 1.000000 | 0.964286 | 0.981818 |\n| Sundanese (sun) | 1.000000 | 0.977273 | 0.988506 |\n| Swahili (macrolanguage) (swa) | 1.000000 | 1.000000 | 1.000000 |\n| Swedish (swe) | 1.000000 | 1.000000 | 1.000000 |\n| Silesian (szl) | 1.000000 | 0.981481 | 0.990654 |\n| Tamil (tam) | 0.982143 | 1.000000 | 0.990991 |\n| Tatar (tat) | 1.000000 | 1.000000 | 1.000000 |\n| Tulu (tcy) | 0.982456 | 1.000000 | 0.991150 |\n| Telugu (tel) | 1.000000 | 0.920000 | 0.958333 |\n| Tetum (tet) | 1.000000 | 0.964286 | 0.981818 |\n| Tajik (tgk) | 1.000000 | 1.000000 | 1.000000 |\n| Tagalog (tgl) | 1.000000 | 1.000000 | 1.000000 |\n| Thai (tha) | 0.932203 | 0.982143 | 0.956522 |\n| Tongan (ton) | 1.000000 | 0.964286 | 0.981818 |\n| Tswana (tsn) | 1.000000 | 1.000000 | 1.000000 |\n| Turkmen (tuk) | 1.000000 | 0.982143 | 0.990991 |\n| Turkish (tur) | 0.901639 | 0.982143 | 0.940171 |\n| Tuvan (tyv) | 1.000000 | 0.964286 | 0.981818 |\n| Udmurt (udm) | 1.000000 | 0.982143 | 0.990991 |\n| Uighur (uig) | 1.000000 | 0.982143 | 0.990991 |\n| Ukrainian (ukr) | 0.963636 | 0.946429 | 0.954955 |\n| Urdu (urd) | 1.000000 | 0.982143 | 0.990991 |\n| Uzbek (uzb) | 1.000000 | 1.000000 | 1.000000 |\n| Venetian (vec) | 1.000000 | 0.982143 | 0.990991 |\n| Veps (vep) | 0.982456 | 1.000000 | 0.991150 |\n| Vietnamese (vie) | 0.964912 | 0.982143 | 0.973451 |\n| Vlaams (vls) | 1.000000 | 0.982143 | 0.990991 |\n| Volapük (vol) | 1.000000 | 1.000000 | 1.000000 |\n| Võro (vro) | 0.964286 | 0.964286 | 0.964286 |\n| Waray (war) | 1.000000 | 0.982143 | 0.990991 |\n| Walloon (wln) | 1.000000 | 1.000000 | 1.000000 |\n| Wolof (wol) | 0.981481 | 0.963636 | 0.972477 |\n| Wu Chinese (wuu) | 0.981481 | 0.946429 | 0.963636 |\n| Xhosa (xho) | 1.000000 | 0.964286 | 0.981818 |\n| Mingrelian (xmf) | 1.000000 | 0.964286 | 0.981818 |\n| Yiddish (yid) | 1.000000 | 1.000000 | 1.000000 |\n| Yoruba (yor) | 0.964912 | 0.982143 | 0.973451 |\n| Zeeuws (zea) | 1.000000 | 0.982143 | 0.990991 |\n| Cantonese (zh-yue) | 0.981481 | 0.946429 | 0.963636 |\n| Standard Chinese (zho) | 0.932203 | 0.982143 | 0.956522 |\n| accuracy | 0.963055 | 0.963055 | 0.963055 |\n| macro avg | 0.966424 | 0.963216 | 0.963891 |\n| weighted avg | 0.966040 | 0.963055 | 0.963606 |\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eSentence level\u003c/summary\u003e\n\n| language | precision | recall | f1-score |\n|:--------------------------------------:|:---------:|:--------:|:--------:|\n| Achinese (ace) | 0.754545 | 0.873684 | 0.809756 |\n| Afrikaans (afr) | 0.708955 | 0.940594 | 0.808511 |\n| Alemannic German (als) | 0.870130 | 0.752809 | 0.807229 |\n| Amharic (amh) | 1.000000 | 0.820000 | 0.901099 |\n| Old English (ang) | 0.966667 | 0.906250 | 0.935484 |\n| Arabic (ara) | 0.907692 | 0.967213 | 0.936508 |\n| Aragonese (arg) | 0.921569 | 0.959184 | 0.940000 |\n| Egyptian Arabic (arz) | 0.964286 | 0.843750 | 0.900000 |\n| Assamese (asm) | 0.964286 | 0.870968 | 0.915254 |\n| Asturian (ast) | 0.880000 | 0.795181 | 0.835443 |\n| Avar (ava) | 0.864198 | 0.843373 | 0.853659 |\n| Aymara (aym) | 1.000000 | 0.901961 | 0.948454 |\n| South Azerbaijani (azb) | 0.979381 | 0.989583 | 0.984456 |\n| Azerbaijani (aze) | 0.989899 | 0.960784 | 0.975124 |\n| Bashkir (bak) | 0.837209 | 0.857143 | 0.847059 |\n| Bavarian (bar) | 0.741935 | 0.766667 | 0.754098 |\n| Central Bikol (bcl) | 0.962963 | 0.928571 | 0.945455 |\n| Belarusian (Taraschkewiza) (be-tarask) | 0.857143 | 0.733333 | 0.790419 |\n| Belarusian (bel) | 0.775510 | 0.752475 | 0.763819 |\n| Bengali (ben) | 0.861111 | 0.911765 | 0.885714 |\n| Bhojpuri (bho) | 0.965517 | 0.933333 | 0.949153 |\n| Banjar (bjn) | 0.891566 | 0.880952 | 0.886228 |\n| Tibetan (bod) | 1.000000 | 1.000000 | 1.000000 |\n| Bosnian (bos) | 0.375000 | 0.323077 | 0.347107 |\n| Bishnupriya (bpy) | 0.986301 | 1.000000 | 0.993103 |\n| Breton (bre) | 0.951613 | 0.893939 | 0.921875 |\n| Bulgarian (bul) | 0.945055 | 0.877551 | 0.910053 |\n| Buryat (bxr) | 0.955556 | 0.843137 | 0.895833 |\n| Catalan (cat) | 0.692308 | 0.750000 | 0.720000 |\n| Chavacano (cbk) | 0.842857 | 0.641304 | 0.728395 |\n| Min Dong (cdo) | 0.972973 | 1.000000 | 0.986301 |\n| Cebuano (ceb) | 0.981308 | 0.954545 | 0.967742 |\n| Czech (ces) | 0.944444 | 0.915385 | 0.929687 |\n| Chechen (che) | 0.875000 | 0.700000 | 0.777778 |\n| Cherokee (chr) | 1.000000 | 0.970588 | 0.985075 |\n| Chuvash (chv) | 0.875000 | 0.836957 | 0.855556 |\n| Central Kurdish (ckb) | 1.000000 | 0.983051 | 0.991453 |\n| Cornish (cor) | 0.979592 | 0.969697 | 0.974619 |\n| Corsican (cos) | 0.986842 | 0.925926 | 0.955414 |\n| Crimean Tatar (crh) | 0.958333 | 0.907895 | 0.932432 |\n| Kashubian (csb) | 0.920354 | 0.904348 | 0.912281 |\n| Welsh (cym) | 0.971014 | 0.943662 | 0.957143 |\n| Danish (dan) | 0.865169 | 0.777778 | 0.819149 |\n| German (deu) | 0.721311 | 0.822430 | 0.768559 |\n| Dimli (diq) | 0.915966 | 0.923729 | 0.919831 |\n| Dhivehi (div) | 1.000000 | 0.991228 | 0.995595 |\n| Lower Sorbian (dsb) | 0.898876 | 0.879121 | 0.888889 |\n| Doteli (dty) | 0.821429 | 0.638889 | 0.718750 |\n| Emilian (egl) | 0.988095 | 0.922222 | 0.954023 |\n| Modern Greek (ell) | 0.988636 | 0.966667 | 0.977528 |\n| English (eng) | 0.522727 | 0.784091 | 0.627273 |\n| Esperanto (epo) | 0.963855 | 0.930233 | 0.946746 |\n| Estonian (est) | 0.922222 | 0.873684 | 0.897297 |\n| Basque (eus) | 1.000000 | 0.941176 | 0.969697 |\n| Extremaduran (ext) | 0.925373 | 0.885714 | 0.905109 |\n| Faroese (fao) | 0.855072 | 0.887218 | 0.870849 |\n| Persian (fas) | 0.879630 | 0.979381 | 0.926829 |\n| Finnish (fin) | 0.952830 | 0.943925 | 0.948357 |\n| French (fra) | 0.676768 | 0.943662 | 0.788235 |\n| Arpitan (frp) | 0.867925 | 0.807018 | 0.836364 |\n| Western Frisian (fry) | 0.956989 | 0.890000 | 0.922280 |\n| Friulian (fur) | 1.000000 | 0.857143 | 0.923077 |\n| Gagauz (gag) | 0.939024 | 0.802083 | 0.865169 |\n| Scottish Gaelic (gla) | 1.000000 | 0.879121 | 0.935673 |\n| Irish (gle) | 0.989247 | 0.958333 | 0.973545 |\n| Galician (glg) | 0.910256 | 0.922078 | 0.916129 |\n| Gilaki (glk) | 0.964706 | 0.872340 | 0.916201 |\n| Manx (glv) | 1.000000 | 0.965517 | 0.982456 |\n| Guarani (grn) | 0.983333 | 1.000000 | 0.991597 |\n| Gujarati (guj) | 1.000000 | 0.991525 | 0.995745 |\n| Hakka Chinese (hak) | 0.955224 | 0.955224 | 0.955224 |\n| Haitian Creole (hat) | 0.833333 | 0.666667 | 0.740741 |\n| Hausa (hau) | 0.936709 | 0.913580 | 0.925000 |\n| Serbo-Croatian (hbs) | 0.452830 | 0.410256 | 0.430493 |\n| Hebrew (heb) | 0.988235 | 0.976744 | 0.982456 |\n| Fiji Hindi (hif) | 0.936709 | 0.840909 | 0.886228 |\n| Hindi (hin) | 0.965517 | 0.756757 | 0.848485 |\n| Croatian (hrv) | 0.443820 | 0.537415 | 0.486154 |\n| Upper Sorbian (hsb) | 0.951613 | 0.830986 | 0.887218 |\n| Hungarian (hun) | 0.854701 | 0.909091 | 0.881057 |\n| Armenian (hye) | 1.000000 | 0.816327 | 0.898876 |\n| Igbo (ibo) | 0.974359 | 0.926829 | 0.950000 |\n| Ido (ido) | 0.975000 | 0.987342 | 0.981132 |\n| Interlingue (ile) | 0.880597 | 0.921875 | 0.900763 |\n| Iloko (ilo) | 0.882353 | 0.821918 | 0.851064 |\n| Interlingua (ina) | 0.952381 | 0.895522 | 0.923077 |\n| Indonesian (ind) | 0.606383 | 0.695122 | 0.647727 |\n| Icelandic (isl) | 0.978261 | 0.882353 | 0.927835 |\n| Italian (ita) | 0.910448 | 0.910448 | 0.910448 |\n| Jamaican Patois (jam) | 0.988764 | 0.967033 | 0.977778 |\n| Javanese (jav) | 0.903614 | 0.862069 | 0.882353 |\n| Lojban (jbo) | 0.943878 | 0.929648 | 0.936709 |\n| Japanese (jpn) | 1.000000 | 0.764706 | 0.866667 |\n| Karakalpak (kaa) | 0.940171 | 0.901639 | 0.920502 |\n| Kabyle (kab) | 0.985294 | 0.837500 | 0.905405 |\n| Kannada (kan) | 0.975806 | 0.975806 | 0.975806 |\n| Georgian (kat) | 0.953704 | 0.903509 | 0.927928 |\n| Kazakh (kaz) | 0.934579 | 0.877193 | 0.904977 |\n| Kabardian (kbd) | 0.987952 | 0.953488 | 0.970414 |\n| Central Khmer (khm) | 0.928571 | 0.829787 | 0.876404 |\n| Kinyarwanda (kin) | 0.953125 | 0.938462 | 0.945736 |\n| Kirghiz (kir) | 0.927632 | 0.881250 | 0.903846 |\n| Komi-Permyak (koi) | 0.750000 | 0.776786 | 0.763158 |\n| Konkani (kok) | 0.893491 | 0.872832 | 0.883041 |\n| Komi (kom) | 0.734177 | 0.690476 | 0.711656 |\n| Korean (kor) | 0.989899 | 0.989899 | 0.989899 |\n| Karachay-Balkar (krc) | 0.928571 | 0.917647 | 0.923077 |\n| Ripuarisch (ksh) | 0.915789 | 0.896907 | 0.906250 |\n| Kurdish (kur) | 0.977528 | 0.935484 | 0.956044 |\n| Ladino (lad) | 0.985075 | 0.904110 | 0.942857 |\n| Lao (lao) | 0.896552 | 0.812500 | 0.852459 |\n| Latin (lat) | 0.741935 | 0.831325 | 0.784091 |\n| Latvian (lav) | 0.710526 | 0.878049 | 0.785455 |\n| Lezghian (lez) | 0.975309 | 0.877778 | 0.923977 |\n| Ligurian (lij) | 0.951807 | 0.897727 | 0.923977 |\n| Limburgan (lim) | 0.909091 | 0.921053 | 0.915033 |\n| Lingala (lin) | 0.942857 | 0.814815 | 0.874172 |\n| Lithuanian (lit) | 0.892857 | 0.925926 | 0.909091 |\n| Lombard (lmo) | 0.766234 | 0.951613 | 0.848921 |\n| Northern Luri (lrc) | 0.972222 | 0.875000 | 0.921053 |\n| Latgalian (ltg) | 0.895349 | 0.865169 | 0.880000 |\n| Luxembourgish (ltz) | 0.882353 | 0.750000 | 0.810811 |\n| Luganda (lug) | 0.946429 | 0.883333 | 0.913793 |\n| Literary Chinese (lzh) | 1.000000 | 1.000000 | 1.000000 |\n| Maithili (mai) | 0.893617 | 0.823529 | 0.857143 |\n| Malayalam (mal) | 1.000000 | 0.975000 | 0.987342 |\n| Banyumasan (map-bms) | 0.924242 | 0.772152 | 0.841379 |\n| Marathi (mar) | 0.874126 | 0.919118 | 0.896057 |\n| Moksha (mdf) | 0.771242 | 0.830986 | 0.800000 |\n| Eastern Mari (mhr) | 0.820000 | 0.860140 | 0.839590 |\n| Minangkabau (min) | 0.973684 | 0.973684 | 0.973684 |\n| Macedonian (mkd) | 0.895652 | 0.953704 | 0.923767 |\n| Malagasy (mlg) | 1.000000 | 0.966102 | 0.982759 |\n| Maltese (mlt) | 0.987952 | 0.964706 | 0.976190 |\n| Min Nan Chinese (nan) | 0.975000 | 1.000000 | 0.987342 |\n| Mongolian (mon) | 0.954545 | 0.933333 | 0.943820 |\n| Maori (mri) | 0.985294 | 1.000000 | 0.992593 |\n| Western Mari (mrj) | 0.966292 | 0.914894 | 0.939891 |\n| Malay (msa) | 0.770270 | 0.695122 | 0.730769 |\n| Mirandese (mwl) | 0.970588 | 0.891892 | 0.929577 |\n| Burmese (mya) | 1.000000 | 0.964286 | 0.981818 |\n| Erzya (myv) | 0.535714 | 0.681818 | 0.600000 |\n| Mazanderani (mzn) | 0.968750 | 0.898551 | 0.932331 |\n| Neapolitan (nap) | 0.892308 | 0.865672 | 0.878788 |\n| Navajo (nav) | 0.984375 | 0.984375 | 0.984375 |\n| Classical Nahuatl (nci) | 0.901408 | 0.761905 | 0.825806 |\n| Low German (nds) | 0.896226 | 0.913462 | 0.904762 |\n| West Low German (nds-nl) | 0.873563 | 0.835165 | 0.853933 |\n| Nepali (macrolanguage) (nep) | 0.704545 | 0.861111 | 0.775000 |\n| Newari (new) | 0.920000 | 0.741935 | 0.821429 |\n| Dutch (nld) | 0.925926 | 0.872093 | 0.898204 |\n| Norwegian Nynorsk (nno) | 0.847059 | 0.808989 | 0.827586 |\n| Bokmål (nob) | 0.861386 | 0.852941 | 0.857143 |\n| Narom (nrm) | 0.966667 | 0.983051 | 0.974790 |\n| Northern Sotho (nso) | 0.897436 | 0.921053 | 0.909091 |\n| Occitan (oci) | 0.958333 | 0.696970 | 0.807018 |\n| Livvi-Karelian (olo) | 0.967742 | 0.937500 | 0.952381 |\n| Oriya (ori) | 0.933333 | 1.000000 | 0.965517 |\n| Oromo (orm) | 0.977528 | 0.915789 | 0.945652 |\n| Ossetian (oss) | 0.958333 | 0.841463 | 0.896104 |\n| Pangasinan (pag) | 0.847328 | 0.909836 | 0.877470 |\n| Pampanga (pam) | 0.969697 | 0.780488 | 0.864865 |\n| Panjabi (pan) | 1.000000 | 1.000000 | 1.000000 |\n| Papiamento (pap) | 0.876190 | 0.920000 | 0.897561 |\n| Picard (pcd) | 0.707317 | 0.568627 | 0.630435 |\n| Pennsylvania German (pdc) | 0.827273 | 0.827273 | 0.827273 |\n| Palatine German (pfl) | 0.882353 | 0.914634 | 0.898204 |\n| Western Panjabi (pnb) | 0.964286 | 0.931034 | 0.947368 |\n| Polish (pol) | 0.859813 | 0.910891 | 0.884615 |\n| Portuguese (por) | 0.535714 | 0.833333 | 0.652174 |\n| Pushto (pus) | 0.989362 | 0.902913 | 0.944162 |\n| Quechua (que) | 0.979167 | 0.903846 | 0.940000 |\n| Tarantino dialect (roa-tara) | 0.964912 | 0.901639 | 0.932203 |\n| Romansh (roh) | 0.914894 | 0.895833 | 0.905263 |\n| Romanian (ron) | 0.880597 | 0.880597 | 0.880597 |\n| Rusyn (rue) | 0.932584 | 0.805825 | 0.864583 |\n| Aromanian (rup) | 0.783333 | 0.758065 | 0.770492 |\n| Russian (rus) | 0.517986 | 0.765957 | 0.618026 |\n| Yakut (sah) | 0.954023 | 0.922222 | 0.937853 |\n| Sanskrit (san) | 0.866667 | 0.951220 | 0.906977 |\n| Sicilian (scn) | 0.984375 | 0.940299 | 0.961832 |\n| Scots (sco) | 0.851351 | 0.900000 | 0.875000 |\n| Samogitian (sgs) | 0.977011 | 0.876289 | 0.923913 |\n| Sinhala (sin) | 0.406154 | 0.985075 | 0.575163 |\n| Slovak (slk) | 0.956989 | 0.872549 | 0.912821 |\n| Slovene (slv) | 0.907216 | 0.854369 | 0.880000 |\n| Northern Sami (sme) | 0.949367 | 0.892857 | 0.920245 |\n| Shona (sna) | 0.936508 | 0.855072 | 0.893939 |\n| Sindhi (snd) | 0.984962 | 0.992424 | 0.988679 |\n| Somali (som) | 0.949153 | 0.848485 | 0.896000 |\n| Spanish (spa) | 0.584158 | 0.746835 | 0.655556 |\n| Albanian (sqi) | 0.988095 | 0.912088 | 0.948571 |\n| Sardinian (srd) | 0.957746 | 0.931507 | 0.944444 |\n| Sranan (srn) | 0.985714 | 0.945205 | 0.965035 |\n| Serbian (srp) | 0.950980 | 0.889908 | 0.919431 |\n| Saterfriesisch (stq) | 0.962500 | 0.875000 | 0.916667 |\n| Sundanese (sun) | 0.778846 | 0.910112 | 0.839378 |\n| Swahili (macrolanguage) (swa) | 0.915493 | 0.878378 | 0.896552 |\n| Swedish (swe) | 0.989247 | 0.958333 | 0.973545 |\n| Silesian (szl) | 0.944444 | 0.904255 | 0.923913 |\n| Tamil (tam) | 0.990000 | 0.970588 | 0.980198 |\n| Tatar (tat) | 0.942029 | 0.902778 | 0.921986 |\n| Tulu (tcy) | 0.980519 | 0.967949 | 0.974194 |\n| Telugu (tel) | 0.965986 | 0.965986 | 0.965986 |\n| Tetum (tet) | 0.898734 | 0.855422 | 0.876543 |\n| Tajik (tgk) | 0.974684 | 0.939024 | 0.956522 |\n| Tagalog (tgl) | 0.965909 | 0.934066 | 0.949721 |\n| Thai (tha) | 0.923077 | 0.882353 | 0.902256 |\n| Tongan (ton) | 0.970149 | 0.890411 | 0.928571 |\n| Tswana (tsn) | 0.888889 | 0.926316 | 0.907216 |\n| Turkmen (tuk) | 0.968000 | 0.889706 | 0.927203 |\n| Turkish (tur) | 0.871287 | 0.926316 | 0.897959 |\n| Tuvan (tyv) | 0.948454 | 0.859813 | 0.901961 |\n| Udmurt (udm) | 0.989362 | 0.894231 | 0.939394 |\n| Uighur (uig) | 1.000000 | 0.953333 | 0.976109 |\n| Ukrainian (ukr) | 0.893617 | 0.875000 | 0.884211 |\n| Urdu (urd) | 1.000000 | 1.000000 | 1.000000 |\n| Uzbek (uzb) | 0.636042 | 0.886700 | 0.740741 |\n| Venetian (vec) | 1.000000 | 0.941176 | 0.969697 |\n| Veps (vep) | 0.858586 | 0.965909 | 0.909091 |\n| Vietnamese (vie) | 1.000000 | 0.940476 | 0.969325 |\n| Vlaams (vls) | 0.885714 | 0.898551 | 0.892086 |\n| Volapük (vol) | 0.975309 | 0.975309 | 0.975309 |\n| Võro (vro) | 0.855670 | 0.864583 | 0.860104 |\n| Waray (war) | 0.972222 | 0.909091 | 0.939597 |\n| Walloon (wln) | 0.742138 | 0.893939 | 0.810997 |\n| Wolof (wol) | 0.882979 | 0.954023 | 0.917127 |\n| Wu Chinese (wuu) | 0.961538 | 0.833333 | 0.892857 |\n| Xhosa (xho) | 0.934066 | 0.867347 | 0.899471 |\n| Mingrelian (xmf) | 0.958333 | 0.929293 | 0.943590 |\n| Yiddish (yid) | 0.984375 | 0.875000 | 0.926471 |\n| Yoruba (yor) | 0.868421 | 0.857143 | 0.862745 |\n| Zeeuws (zea) | 0.879518 | 0.793478 | 0.834286 |\n| Cantonese (zh-yue) | 0.896552 | 0.812500 | 0.852459 |\n| Standard Chinese (zho) | 0.906250 | 0.935484 | 0.920635 |\n| accuracy | 0.881051 | 0.881051 | 0.881051 |\n| macro avg | 0.903245 | 0.880618 | 0.888996 |\n| weighted avg | 0.894174 | 0.881051 | 0.884520 |\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eToken level\u003c/summary\u003e\n\n| language | precision | recall | f1-score |\n|:--------------------------------------:|:---------:|:--------:|:--------:|\n| Achinese (ace) | 0.873846 | 0.827988 | 0.850299 |\n| Afrikaans (afr) | 0.638060 | 0.732334 | 0.681954 |\n| Alemannic German (als) | 0.673780 | 0.547030 | 0.603825 |\n| Amharic (amh) | 0.997743 | 0.954644 | 0.975717 |\n| Old English (ang) | 0.840816 | 0.693603 | 0.760148 |\n| Arabic (ara) | 0.768737 | 0.840749 | 0.803132 |\n| Aragonese (arg) | 0.493671 | 0.505181 | 0.499360 |\n| Egyptian Arabic (arz) | 0.823529 | 0.741935 | 0.780606 |\n| Assamese (asm) | 0.948454 | 0.893204 | 0.920000 |\n| Asturian (ast) | 0.490000 | 0.508299 | 0.498982 |\n| Avar (ava) | 0.813636 | 0.655678 | 0.726166 |\n| Aymara (aym) | 0.795833 | 0.779592 | 0.787629 |\n| South Azerbaijani (azb) | 0.832836 | 0.863777 | 0.848024 |\n| Azerbaijani (aze) | 0.867470 | 0.800000 | 0.832370 |\n| Bashkir (bak) | 0.851852 | 0.750000 | 0.797688 |\n| Bavarian (bar) | 0.560897 | 0.522388 | 0.540958 |\n| Central Bikol (bcl) | 0.708229 | 0.668235 | 0.687651 |\n| Belarusian (Taraschkewiza) (be-tarask) | 0.615635 | 0.526462 | 0.567568 |\n| Belarusian (bel) | 0.539952 | 0.597855 | 0.567430 |\n| Bengali (ben) | 0.830275 | 0.885086 | 0.856805 |\n| Bhojpuri (bho) | 0.723118 | 0.691517 | 0.706965 |\n| Banjar (bjn) | 0.619586 | 0.726269 | 0.668699 |\n| Tibetan (bod) | 0.999537 | 0.991728 | 0.995617 |\n| Bosnian (bos) | 0.330849 | 0.403636 | 0.363636 |\n| Bishnupriya (bpy) | 0.941634 | 0.949020 | 0.945312 |\n| Breton (bre) | 0.772222 | 0.745308 | 0.758527 |\n| Bulgarian (bul) | 0.771505 | 0.706897 | 0.737789 |\n| Buryat (bxr) | 0.741935 | 0.753149 | 0.747500 |\n| Catalan (cat) | 0.528716 | 0.610136 | 0.566516 |\n| Chavacano (cbk) | 0.409449 | 0.312625 | 0.354545 |\n| Min Dong (cdo) | 0.951264 | 0.936057 | 0.943599 |\n| Cebuano (ceb) | 0.888298 | 0.876640 | 0.882431 |\n| Czech (ces) | 0.806045 | 0.758294 | 0.781441 |\n| Chechen (che) | 0.857143 | 0.600000 | 0.705882 |\n| Cherokee (chr) | 0.997840 | 0.952577 | 0.974684 |\n| Chuvash (chv) | 0.874346 | 0.776744 | 0.822660 |\n| Central Kurdish (ckb) | 0.984848 | 0.953545 | 0.968944 |\n| Cornish (cor) | 0.747596 | 0.807792 | 0.776529 |\n| Corsican (cos) | 0.673913 | 0.708571 | 0.690808 |\n| Crimean Tatar (crh) | 0.498801 | 0.700337 | 0.582633 |\n| Kashubian (csb) | 0.797059 | 0.794721 | 0.795888 |\n| Welsh (cym) | 0.829609 | 0.841360 | 0.835443 |\n| Danish (dan) | 0.649789 | 0.622222 | 0.635707 |\n| German (deu) | 0.559406 | 0.763514 | 0.645714 |\n| Dimli (diq) | 0.835580 | 0.763547 | 0.797941 |\n| Dhivehi (div) | 1.000000 | 0.980645 | 0.990228 |\n| Lower Sorbian (dsb) | 0.740484 | 0.694805 | 0.716918 |\n| Doteli (dty) | 0.616314 | 0.527132 | 0.568245 |\n| Emilian (egl) | 0.822993 | 0.769625 | 0.795414 |\n| Modern Greek (ell) | 0.972043 | 0.963753 | 0.967880 |\n| English (eng) | 0.260492 | 0.724346 | 0.383183 |\n| Esperanto (epo) | 0.766764 | 0.716621 | 0.740845 |\n| Estonian (est) | 0.698885 | 0.673835 | 0.686131 |\n| Basque (eus) | 0.882716 | 0.841176 | 0.861446 |\n| Extremaduran (ext) | 0.570605 | 0.511628 | 0.539510 |\n| Faroese (fao) | 0.773987 | 0.784017 | 0.778970 |\n| Persian (fas) | 0.709836 | 0.809346 | 0.756332 |\n| Finnish (fin) | 0.866261 | 0.796089 | 0.829694 |\n| French (fra) | 0.496263 | 0.700422 | 0.580927 |\n| Arpitan (frp) | 0.663366 | 0.584302 | 0.621329 |\n| Western Frisian (fry) | 0.750000 | 0.756148 | 0.753061 |\n| Friulian (fur) | 0.713555 | 0.675545 | 0.694030 |\n| Gagauz (gag) | 0.728125 | 0.677326 | 0.701807 |\n| Scottish Gaelic (gla) | 0.831601 | 0.817996 | 0.824742 |\n| Irish (gle) | 0.868852 | 0.801296 | 0.833708 |\n| Galician (glg) | 0.469816 | 0.454315 | 0.461935 |\n| Gilaki (glk) | 0.703883 | 0.687204 | 0.695444 |\n| Manx (glv) | 0.873047 | 0.886905 | 0.879921 |\n| Guarani (grn) | 0.848580 | 0.793510 | 0.820122 |\n| Gujarati (guj) | 0.995643 | 0.926978 | 0.960084 |\n| Hakka Chinese (hak) | 0.898403 | 0.904971 | 0.901675 |\n| Haitian Creole (hat) | 0.719298 | 0.518987 | 0.602941 |\n| Hausa (hau) | 0.815353 | 0.829114 | 0.822176 |\n| Serbo-Croatian (hbs) | 0.343465 | 0.244589 | 0.285714 |\n| Hebrew (heb) | 0.891304 | 0.933941 | 0.912125 |\n| Fiji Hindi (hif) | 0.662577 | 0.664615 | 0.663594 |\n| Hindi (hin) | 0.782301 | 0.778169 | 0.780229 |\n| Croatian (hrv) | 0.360308 | 0.374000 | 0.367026 |\n| Upper Sorbian (hsb) | 0.745763 | 0.611111 | 0.671756 |\n| Hungarian (hun) | 0.876812 | 0.846154 | 0.861210 |\n| Armenian (hye) | 0.988201 | 0.917808 | 0.951705 |\n| Igbo (ibo) | 0.825397 | 0.696429 | 0.755448 |\n| Ido (ido) | 0.760479 | 0.814103 | 0.786378 |\n| Interlingue (ile) | 0.701299 | 0.580645 | 0.635294 |\n| Iloko (ilo) | 0.688356 | 0.844538 | 0.758491 |\n| Interlingua (ina) | 0.577889 | 0.588235 | 0.583016 |\n| Indonesian (ind) | 0.415879 | 0.514019 | 0.459770 |\n| Icelandic (isl) | 0.855263 | 0.790754 | 0.821745 |\n| Italian (ita) | 0.474576 | 0.561247 | 0.514286 |\n| Jamaican Patois (jam) | 0.826087 | 0.791667 | 0.808511 |\n| Javanese (jav) | 0.670130 | 0.658163 | 0.664093 |\n| Lojban (jbo) | 0.896861 | 0.917431 | 0.907029 |\n| Japanese (jpn) | 0.931373 | 0.848214 | 0.887850 |\n| Karakalpak (kaa) | 0.790393 | 0.827744 | 0.808637 |\n| Kabyle (kab) | 0.828571 | 0.759162 | 0.792350 |\n| Kannada (kan) | 0.879357 | 0.847545 | 0.863158 |\n| Georgian (kat) | 0.916399 | 0.907643 | 0.912000 |\n| Kazakh (kaz) | 0.900901 | 0.819672 | 0.858369 |\n| Kabardian (kbd) | 0.923345 | 0.892256 | 0.907534 |\n| Central Khmer (khm) | 0.976667 | 0.816156 | 0.889226 |\n| Kinyarwanda (kin) | 0.824324 | 0.726190 | 0.772152 |\n| Kirghiz (kir) | 0.674766 | 0.779698 | 0.723447 |\n| Komi-Permyak (koi) | 0.652830 | 0.633700 | 0.643123 |\n| Konkani (kok) | 0.778865 | 0.728938 | 0.753075 |\n| Komi (kom) | 0.737374 | 0.572549 | 0.644592 |\n| Korean (kor) | 0.984615 | 0.967603 | 0.976035 |\n| Karachay-Balkar (krc) | 0.869416 | 0.857627 | 0.863481 |\n| Ripuarisch (ksh) | 0.709859 | 0.649485 | 0.678331 |\n| Kurdish (kur) | 0.883777 | 0.862884 | 0.873206 |\n| Ladino (lad) | 0.660920 | 0.576441 | 0.615797 |\n| Lao (lao) | 0.986175 | 0.918455 | 0.951111 |\n| Latin (lat) | 0.581250 | 0.636986 | 0.607843 |\n| Latvian (lav) | 0.824513 | 0.797844 | 0.810959 |\n| Lezghian (lez) | 0.898955 | 0.793846 | 0.843137 |\n| Ligurian (lij) | 0.662903 | 0.677100 | 0.669927 |\n| Limburgan (lim) | 0.615385 | 0.581818 | 0.598131 |\n| Lingala (lin) | 0.836207 | 0.763780 | 0.798354 |\n| Lithuanian (lit) | 0.756329 | 0.804714 | 0.779772 |\n| Lombard (lmo) | 0.556818 | 0.536986 | 0.546722 |\n| Northern Luri (lrc) | 0.838574 | 0.753296 | 0.793651 |\n| Latgalian (ltg) | 0.759531 | 0.755102 | 0.757310 |\n| Luxembourgish (ltz) | 0.645062 | 0.614706 | 0.629518 |\n| Luganda (lug) | 0.787535 | 0.805797 | 0.796562 |\n| Literary Chinese (lzh) | 0.921951 | 0.949749 | 0.935644 |\n| Maithili (mai) | 0.777778 | 0.761658 | 0.769634 |\n| Malayalam (mal) | 0.993377 | 0.949367 | 0.970874 |\n| Banyumasan (map-bms) | 0.531429 | 0.453659 | 0.489474 |\n| Marathi (mar) | 0.748744 | 0.818681 | 0.782152 |\n| Moksha (mdf) | 0.728745 | 0.800000 | 0.762712 |\n| Eastern Mari (mhr) | 0.790323 | 0.760870 | 0.775316 |\n| Minangkabau (min) | 0.953271 | 0.886957 | 0.918919 |\n| Macedonian (mkd) | 0.816399 | 0.849722 | 0.832727 |\n| Malagasy (mlg) | 0.925187 | 0.918317 | 0.921739 |\n| Maltese (mlt) | 0.869421 | 0.890017 | 0.879599 |\n| Min Nan Chinese (nan) | 0.743707 | 0.820707 | 0.780312 |\n| Mongolian (mon) | 0.852194 | 0.838636 | 0.845361 |\n| Maori (mri) | 0.934726 | 0.937173 | 0.935948 |\n| Western Mari (mrj) | 0.818792 | 0.827119 | 0.822934 |\n| Malay (msa) | 0.508065 | 0.376119 | 0.432247 |\n| Mirandese (mwl) | 0.650407 | 0.685225 | 0.667362 |\n| Burmese (mya) | 0.995968 | 0.972441 | 0.984064 |\n| Erzya (myv) | 0.475783 | 0.503012 | 0.489019 |\n| Mazanderani (mzn) | 0.775362 | 0.701639 | 0.736661 |\n| Neapolitan (nap) | 0.628993 | 0.595349 | 0.611708 |\n| Navajo (nav) | 0.955882 | 0.937500 | 0.946602 |\n| Classical Nahuatl (nci) | 0.679758 | 0.589005 | 0.631136 |\n| Low German (nds) | 0.669789 | 0.690821 | 0.680143 |\n| West Low German (nds-nl) | 0.513889 | 0.504545 | 0.509174 |\n| Nepali (macrolanguage) (nep) | 0.640476 | 0.649758 | 0.645084 |\n| Newari (new) | 0.928571 | 0.745902 | 0.827273 |\n| Dutch (nld) | 0.553763 | 0.553763 | 0.553763 |\n| Norwegian Nynorsk (nno) | 0.569277 | 0.519231 | 0.543103 |\n| Bokmål (nob) | 0.519856 | 0.562500 | 0.540338 |\n| Narom (nrm) | 0.691275 | 0.605882 | 0.645768 |\n| Northern Sotho (nso) | 0.950276 | 0.815166 | 0.877551 |\n| Occitan (oci) | 0.483444 | 0.366834 | 0.417143 |\n| Livvi-Karelian (olo) | 0.816850 | 0.790780 | 0.803604 |\n| Oriya (ori) | 0.981481 | 0.963636 | 0.972477 |\n| Oromo (orm) | 0.885714 | 0.829218 | 0.856536 |\n| Ossetian (oss) | 0.822006 | 0.855219 | 0.838284 |\n| Pangasinan (pag) | 0.842105 | 0.715655 | 0.773748 |\n| Pampanga (pam) | 0.770000 | 0.435028 | 0.555957 |\n| Panjabi (pan) | 0.996154 | 0.984791 | 0.990440 |\n| Papiamento (pap) | 0.674672 | 0.661670 | 0.668108 |\n| Picard (pcd) | 0.407895 | 0.356322 | 0.380368 |\n| Pennsylvania German (pdc) | 0.487047 | 0.509485 | 0.498013 |\n| Palatine German (pfl) | 0.614173 | 0.570732 | 0.591656 |\n| Western Panjabi (pnb) | 0.926267 | 0.887417 | 0.906426 |\n| Polish (pol) | 0.797059 | 0.734417 | 0.764457 |\n| Portuguese (por) | 0.500914 | 0.586724 | 0.540434 |\n| Pushto (pus) | 0.941489 | 0.898477 | 0.919481 |\n| Quechua (que) | 0.854167 | 0.797665 | 0.824950 |\n| Tarantino dialect (roa-tara) | 0.669794 | 0.724138 | 0.695906 |\n| Romansh (roh) | 0.745527 | 0.760649 | 0.753012 |\n| Romanian (ron) | 0.805486 | 0.769048 | 0.786845 |\n| Rusyn (rue) | 0.718543 | 0.645833 | 0.680251 |\n| Aromanian (rup) | 0.288482 | 0.730245 | 0.413580 |\n| Russian (rus) | 0.530120 | 0.690583 | 0.599805 |\n| Yakut (sah) | 0.853521 | 0.865714 | 0.859574 |\n| Sanskrit (san) | 0.931343 | 0.896552 | 0.913616 |\n| Sicilian (scn) | 0.734139 | 0.618321 | 0.671271 |\n| Scots (sco) | 0.571429 | 0.540816 | 0.555701 |\n| Samogitian (sgs) | 0.829167 | 0.748120 | 0.786561 |\n| Sinhala (sin) | 0.909474 | 0.935065 | 0.922092 |\n| Slovak (slk) | 0.738235 | 0.665782 | 0.700139 |\n| Slovene (slv) | 0.671123 | 0.662269 | 0.666667 |\n| Northern Sami (sme) | 0.800676 | 0.825784 | 0.813036 |\n| Shona (sna) | 0.761702 | 0.724696 | 0.742739 |\n| Sindhi (snd) | 0.950172 | 0.946918 | 0.948542 |\n| Somali (som) | 0.849462 | 0.802030 | 0.825065 |\n| Spanish (spa) | 0.325234 | 0.413302 | 0.364017 |\n| Albanian (sqi) | 0.875899 | 0.832479 | 0.853637 |\n| Sardinian (srd) | 0.750000 | 0.711061 | 0.730012 |\n| Sranan (srn) | 0.888889 | 0.771084 | 0.825806 |\n| Serbian (srp) | 0.824561 | 0.814356 | 0.819427 |\n| Saterfriesisch (stq) | 0.790087 | 0.734417 | 0.761236 |\n| Sundanese (sun) | 0.764192 | 0.631769 | 0.691700 |\n| Swahili (macrolanguage) (swa) | 0.763496 | 0.796247 | 0.779528 |\n| Swedish (swe) | 0.838284 | 0.723647 | 0.776758 |\n| Silesian (szl) | 0.819788 | 0.750809 | 0.783784 |\n| Tamil (tam) | 0.985765 | 0.955172 | 0.970228 |\n| Tatar (tat) | 0.469780 | 0.795349 | 0.590674 |\n| Tulu (tcy) | 0.893300 | 0.873786 | 0.883436 |\n| Telugu (tel) | 1.000000 | 0.913690 | 0.954899 |\n| Tetum (tet) | 0.765116 | 0.744344 | 0.754587 |\n| Tajik (tgk) | 0.828418 | 0.813158 | 0.820717 |\n| Tagalog (tgl) | 0.751468 | 0.757396 | 0.754420 |\n| Thai (tha) | 0.933884 | 0.807143 | 0.865900 |\n| Tongan (ton) | 0.920245 | 0.923077 | 0.921659 |\n| Tswana (tsn) | 0.873397 | 0.889070 | 0.881164 |\n| Turkmen (tuk) | 0.898438 | 0.837887 | 0.867107 |\n| Turkish (tur) | 0.666667 | 0.716981 | 0.690909 |\n| Tuvan (tyv) | 0.857143 | 0.805063 | 0.830287 |\n| Udmurt (udm) | 0.865517 | 0.756024 | 0.807074 |\n| Uighur (uig) | 0.991597 | 0.967213 | 0.979253 |\n| Ukrainian (ukr) | 0.771341 | 0.702778 | 0.735465 |\n| Urdu (urd) | 0.877647 | 0.855505 | 0.866434 |\n| Uzbek (uzb) | 0.655652 | 0.797040 | 0.719466 |\n| Venetian (vec) | 0.611111 | 0.527233 | 0.566082 |\n| Veps (vep) | 0.672862 | 0.688213 | 0.680451 |\n| Vietnamese (vie) | 0.932406 | 0.914230 | 0.923228 |\n| Vlaams (vls) | 0.594427 | 0.501305 | 0.543909 |\n| Volapük (vol) | 0.765625 | 0.942308 | 0.844828 |\n| Võro (vro) | 0.797203 | 0.740260 | 0.767677 |\n| Waray (war) | 0.930876 | 0.930876 | 0.930876 |\n| Walloon (wln) | 0.636804 | 0.693931 | 0.664141 |\n| Wolof (wol) | 0.864220 | 0.845601 | 0.854809 |\n| Wu Chinese (wuu) | 0.848921 | 0.830986 | 0.839858 |\n| Xhosa (xho) | 0.837398 | 0.759214 | 0.796392 |\n| Mingrelian (xmf) | 0.943396 | 0.874126 | 0.907441 |\n| Yiddish (yid) | 0.955729 | 0.897311 | 0.925599 |\n| Yoruba (yor) | 0.812010 | 0.719907 | 0.763190 |\n| Zeeuws (zea) | 0.617737 | 0.550409 | 0.582133 |\n| Cantonese (zh-yue) | 0.859649 | 0.649007 | 0.739623 |\n| Standard Chinese (zho) | 0.845528 | 0.781955 | 0.812500 |\n| accuracy | 0.749527 | 0.749527 | 0.749527 |\n| macro avg | 0.762866 | 0.742101 | 0.749261 |\n| weighted avg | 0.762006 | 0.749527 | 0.752910 |\n\n\u003c/details\u003e\n\nAs can be seen, the model outperforms on groups of similar or dialects languages. \nFor instance, the f1 scores for the Persian language and similar languages like Gilaki, Northern Luri, Central Kurdish, Kurdish, and Mazanderani are 92%, 91%, 92%, 99%, 94%, and 93%, respectively. \n\n# How to Install\nWe recommend Python 3.7 or higher, PyTorch 1.6.0 or higher. The code does not work with Python 2.7.\n\n```bash\npip install git+https://github.com/m3hrdadfi/zabanshenas.git\n```\n\n## How to Use\n\nYou can use this code snippet to identify the most likely language of a written document. You just have to say: ZABANSHENAS (detector) -\u003e BESHNAS (detect) 😎. \n\n*Sounds interesting, doesn't it?*\n\n```python\nfrom zabanshenas.zabanshenas import Zabanshenas\n\nzabanshenas = Zabanshenas()\ntext = \"زیر لکه‌های زمان احساسات محو میشن تو یکی دیگه شدی و من هنوز اینجام\"\n# Beneath the strains of time, the feelings disappear, you are someone else, I'm still right here!\n\nr = zabanshenas.detect(text, return_all_scores=False)\nprint(r)\n```\n\nOutput:\n```text\n[\n {\n \"language\": \"Persian\",\n \"code\": \"fas\",\n \"score\": 0.6105580925941467\n }\n]\n```\n\nOr you can find out all the candidates' scores using the following snippet.\n\n```python\nfrom zabanshenas.zabanshenas import Zabanshenas\n\nzabanshenas = Zabanshenas()\ntext = \"زیر لکه‌های زمان احساسات محو میشن تو یکی دیگه شدی و من هنوز اینجام\"\n# Beneath the strains of time, the feelings disappear, you are someone else, I'm still right here!\n\nr = zabanshenas.detect(text, return_all_scores=True)\nprint(r)\n```\nOutput:\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eSee all the 235 candidates\u003c/summary\u003e\n\n```text\n[\n [\n {\n \"language\": \"Persian\",\n \"code\": \"fas\",\n \"score\": 0.6105580925941467\n },\n {\n \"language\": \"Gilaki\",\n \"code\": \"glk\",\n \"score\": 0.29982829093933105\n },\n {\n \"language\": \"Northern Luri\",\n \"code\": \"lrc\",\n \"score\": 0.04840774089097977\n },\n {\n \"language\": \"Mazanderani\",\n \"code\": \"mzn\",\n \"score\": 0.030142733827233315\n },\n {\n \"language\": \"South Azerbaijani\",\n \"code\": \"azb\",\n \"score\": 0.005220199003815651\n },\n {\n \"language\": \"Urdu\",\n \"code\": \"urd\",\n \"score\": 0.0019745035097002983\n },\n {\n \"language\": \"Pushto\",\n \"code\": \"pus\",\n \"score\": 0.0015690263826400042\n },\n {\n \"language\": \"Western Panjabi\",\n \"code\": \"pnb\",\n \"score\": 0.0005721596535295248\n },\n {\n \"language\": \"Central Kurdish\",\n \"code\": \"ckb\",\n \"score\": 0.00025537016335874796\n },\n {\n \"language\": \"Sindhi\",\n \"code\": \"snd\",\n \"score\": 0.0001820324978325516\n },\n {\n \"language\": \"Egyptian Arabic\",\n \"code\": \"arz\",\n \"score\": 0.0001247940381290391\n },\n {\n \"language\": \"Arabic\",\n \"code\": \"ara\",\n \"score\": 7.754910620860755e-05\n },\n {\n \"language\": \"Korean\",\n \"code\": \"kor\",\n \"score\": 5.718228203477338e-05\n },\n {\n \"language\": \"Fiji Hindi\",\n \"code\": \"hif\",\n \"score\": 3.5903740354115143e-05\n },\n {\n \"language\": \"Uighur\",\n \"code\": \"uig\",\n \"score\": 3.5565532016335055e-05\n },\n {\n \"language\": \"Maori\",\n \"code\": \"mri\",\n \"score\": 2.1078320060041733e-05\n },\n {\n \"language\": \"Literary Chinese\",\n \"code\": \"lzh\",\n \"score\": 2.09943773370469e-05\n },\n {\n \"language\": \"Navajo\",\n \"code\": \"nav\",\n \"score\": 1.8877935872296803e-05\n },\n {\n \"language\": \"Mongolian\",\n \"code\": \"mon\",\n \"score\": 1.783044899639208e-05\n },\n {\n \"language\": \"Basque\",\n \"code\": \"eus\",\n \"score\": 1.2980432074982673e-05\n },\n {\n \"language\": \"Moksha\",\n \"code\": \"mdf\",\n \"score\": 1.2325609532126691e-05\n },\n {\n \"language\": \"Tongan\",\n \"code\": \"ton\",\n \"score\": 1.1610675755946431e-05\n },\n {\n \"language\": \"Min Dong\",\n \"code\": \"cdo\",\n \"score\": 1.1508132956805639e-05\n },\n {\n \"language\": \"Sinhala\",\n \"code\": \"sin\",\n \"score\": 1.0617596672091167e-05\n },\n {\n \"language\": \"Venetian\",\n \"code\": \"vec\",\n \"score\": 1.0375520105299074e-05\n },\n {\n \"language\": \"Western Mari\",\n \"code\": \"mrj\",\n \"score\": 1.0316403859178536e-05\n },\n {\n \"language\": \"Malayalam\",\n \"code\": \"mal\",\n \"score\": 1.0265099263051525e-05\n },\n {\n \"language\": \"Interlingua\",\n \"code\": \"ina\",\n \"score\": 1.0040446795755997e-05\n },\n {\n \"language\": \"Tatar\",\n \"code\": \"tat\",\n \"score\": 9.836200661084149e-06\n },\n {\n \"language\": \"Cantonese\",\n \"code\": \"zh-yue\",\n \"score\": 9.80662207439309e-06\n },\n {\n \"language\": \"Wu Chinese\",\n \"code\": \"wuu\",\n \"score\": 9.661145668360405e-06\n },\n {\n \"language\": \"Igbo\",\n \"code\": \"ibo\",\n \"score\": 9.207592484017368e-06\n },\n {\n \"language\": \"Waray\",\n \"code\": \"war\",\n \"score\": 8.970115231932141e-06\n },\n {\n \"language\": \"Yiddish\",\n \"code\": \"yid\",\n \"score\": 8.926748705562204e-06\n },\n {\n \"language\": \"Udmurt\",\n \"code\": \"udm\",\n \"score\": 8.702583727426827e-06\n },\n {\n \"language\": \"Dhivehi\",\n \"code\": \"div\",\n \"score\": 8.36203707876848e-06\n },\n {\n \"language\": \"Newari\",\n \"code\": \"new\",\n \"score\": 8.140945283230394e-06\n },\n {\n \"language\": \"Karachay-Balkar\",\n \"code\": \"krc\",\n \"score\": 8.123539373627864e-06\n },\n {\n \"language\": \"Lojban\",\n \"code\": \"jbo\",\n \"score\": 8.114019692584407e-06\n },\n {\n \"language\": \"Sanskrit\",\n \"code\": \"san\",\n \"score\": 8.087784408417065e-06\n },\n {\n \"language\": \"Luganda\",\n \"code\": \"lug\",\n \"score\": 8.023569534998387e-06\n },\n {\n \"language\": \"Maithili\",\n \"code\": \"mai\",\n \"score\": 7.723083399469033e-06\n },\n {\n \"language\": \"Kirghiz\",\n \"code\": \"kir\",\n \"score\": 7.715119863860309e-06\n },\n {\n \"language\": \"Standard Chinese\",\n \"code\": \"zho\",\n \"score\": 7.5126054071006365e-06\n },\n {\n \"language\": \"Amharic\",\n \"code\": \"amh\",\n \"score\": 7.451813871739432e-06\n },\n {\n \"language\": \"Chechen\",\n \"code\": \"che\",\n \"score\": 7.444541097356705e-06\n },\n {\n \"language\": \"Gujarati\",\n \"code\": \"guj\",\n \"score\": 7.395997727144277e-06\n },\n {\n \"language\": \"Tibetan\",\n \"code\": \"bod\",\n \"score\": 7.390805421891855e-06\n },\n {\n \"language\": \"Komi\",\n \"code\": \"kom\",\n \"score\": 7.373077551164897e-06\n },\n {\n \"language\": \"Lao\",\n \"code\": \"lao\",\n \"score\": 7.351867679972202e-06\n },\n {\n \"language\": \"Wolof\",\n \"code\": \"wol\",\n \"score\": 7.305452982109273e-06\n },\n {\n \"language\": \"Silesian\",\n \"code\": \"szl\",\n \"score\": 7.301976893359097e-06\n },\n {\n \"language\": \"Northern Sotho\",\n \"code\": \"nso\",\n \"score\": 7.2927336987049785e-06\n },\n {\n \"language\": \"Armenian\",\n \"code\": \"hye\",\n \"score\": 7.243447726068553e-06\n },\n {\n \"language\": \"Arpitan\",\n \"code\": \"frp\",\n \"score\": 7.137540251278551e-06\n },\n {\n \"language\": \"Bishnupriya\",\n \"code\": \"bpy\",\n \"score\": 7.062033091642661e-06\n },\n {\n \"language\": \"Azerbaijani\",\n \"code\": \"aze\",\n \"score\": 6.906778253323864e-06\n },\n {\n \"language\": \"Tajik\",\n \"code\": \"tgk\",\n \"score\": 6.730050699843559e-06\n },\n {\n \"language\": \"Old English \",\n \"code\": \"ang\",\n \"score\": 6.6442084971640725e-06\n },\n {\n \"language\": \"Marathi\",\n \"code\": \"mar\",\n \"score\": 6.63194168737391e-06\n },\n {\n \"language\": \"Kurdish\",\n \"code\": \"kur\",\n \"score\": 6.615779057028703e-06\n },\n {\n \"language\": \"Lithuanian\",\n \"code\": \"lit\",\n \"score\": 6.561998816323467e-06\n },\n {\n \"language\": \"Russian\",\n \"code\": \"rus\",\n \"score\": 6.4370215113740414e-06\n },\n {\n \"language\": \"Tulu\",\n \"code\": \"tcy\",\n \"score\": 6.370255960064242e-06\n },\n {\n \"language\": \"Extremaduran\",\n \"code\": \"ext\",\n \"score\": 6.3398160818906035e-06\n },\n {\n \"language\": \"Aymara\",\n \"code\": \"aym\",\n \"score\": 6.288398708420573e-06\n },\n {\n \"language\": \"Lower Sorbian\",\n \"code\": \"dsb\",\n \"score\": 6.209619641595054e-06\n },\n {\n \"language\": \"Classical Nahuatl\",\n \"code\": \"nci\",\n \"score\": 5.954705557087436e-06\n },\n {\n \"language\": \"Polish\",\n \"code\": \"pol\",\n \"score\": 5.952156243438367e-06\n },\n {\n \"language\": \"Cebuano\",\n \"code\": \"ceb\",\n \"score\": 5.911888820264721e-06\n },\n {\n \"language\": \"Hakka Chinese\",\n \"code\": \"hak\",\n \"score\": 5.756284735980444e-06\n },\n {\n \"language\": \"Georgian\",\n \"code\": \"kat\",\n \"score\": 5.656391749653267e-06\n },\n {\n \"language\": \"Mingrelian\",\n \"code\": \"xmf\",\n \"score\": 5.57373004994588e-06\n },\n {\n \"language\": \"Telugu\",\n \"code\": \"tel\",\n \"score\": 5.5334053286060225e-06\n },\n {\n \"language\": \"Doteli\",\n \"code\": \"dty\",\n \"score\": 5.510717073775595e-06\n },\n {\n \"language\": \"Portuguese\",\n \"code\": \"por\",\n \"score\": 5.50901131646242e-06\n },\n {\n \"language\": \"Komi-Permyak\",\n \"code\": \"koi\",\n \"score\": 5.447328476293478e-06\n },\n {\n \"language\": \"Eastern Mari\",\n \"code\": \"mhr\",\n \"score\": 5.414771294454113e-06\n },\n {\n \"language\": \"Lezghian\",\n \"code\": \"lez\",\n \"score\": 5.2741329454875086e-06\n },\n {\n \"language\": \"Nepali (macrolanguage)\",\n \"code\": \"nep\",\n \"score\": 5.273408532957546e-06\n },\n {\n \"language\": \"Samogitian\",\n \"code\": \"sgs\",\n \"score\": 5.207636149862083e-06\n },\n {\n \"language\": \"Bhojpuri\",\n \"code\": \"bho\",\n \"score\": 5.19551804245566e-06\n },\n {\n \"language\": \"Occitan\",\n \"code\": \"oci\",\n \"score\": 5.172901182959322e-06\n },\n {\n \"language\": \"Western Frisian\",\n \"code\": \"fry\",\n \"score\": 5.066170615464216e-06\n },\n {\n \"language\": \"Vlaams\",\n \"code\": \"vls\",\n \"score\": 5.014707312511746e-06\n },\n {\n \"language\": \"Japanese\",\n \"code\": \"jpn\",\n \"score\": 4.986791282135528e-06\n },\n {\n \"language\": \"V\\u00f5ro\",\n \"code\": \"vro\",\n \"score\": 4.9785726332629565e-06\n },\n {\n \"language\": \"Rusyn\",\n \"code\": \"rue\",\n \"score\": 4.937043286190601e-06\n },\n {\n \"language\": \"Hindi\",\n \"code\": \"hin\",\n \"score\": 4.9325194595439825e-06\n },\n {\n \"language\": \"Sicilian\",\n \"code\": \"scn\",\n \"score\": 4.8434171731059905e-06\n },\n {\n \"language\": \"Somali\",\n \"code\": \"som\",\n \"score\": 4.722482117358595e-06\n },\n {\n \"language\": \"Galician\",\n \"code\": \"glg\",\n \"score\": 4.664954758482054e-06\n },\n {\n \"language\": \"Kazakh\",\n \"code\": \"kaz\",\n \"score\": 4.485120825847844e-06\n },\n {\n \"language\": \"Kannada\",\n \"code\": \"kan\",\n \"score\": 4.438274572748924e-06\n },\n {\n \"language\": \"Oromo\",\n \"code\": \"orm\",\n \"score\": 4.422903202794259e-06\n },\n {\n \"language\": \"Albanian\",\n \"code\": \"sqi\",\n \"score\": 4.410150268085999e-06\n },\n {\n \"language\": \"Minangkabau\",\n \"code\": \"min\",\n \"score\": 4.407007509144023e-06\n },\n {\n \"language\": \"Finnish\",\n \"code\": \"fin\",\n \"score\": 4.374884611024754e-06\n },\n {\n \"language\": \"Ossetian\",\n \"code\": \"oss\",\n \"score\": 4.322507265897002e-06\n },\n {\n \"language\": \"Volap\\u00fck\",\n \"code\": \"vol\",\n \"score\": 4.30220188718522e-06\n },\n {\n \"language\": \"Min Nan Chinese\",\n \"code\": \"nan\",\n \"score\": 4.2357942220405675e-06\n },\n {\n \"language\": \"Bashkir\",\n \"code\": \"bak\",\n \"score\": 4.212616204313235e-06\n },\n {\n \"language\": \"Ligurian\",\n \"code\": \"lij\",\n \"score\": 4.1821313061518595e-06\n },\n {\n \"language\": \"Welsh\",\n \"code\": \"cym\",\n \"score\": 4.174029982095817e-06\n },\n {\n \"language\": \"Slovene\",\n \"code\": \"slv\",\n \"score\": 4.172954504610971e-06\n },\n {\n \"language\": \"Dimli\",\n \"code\": \"diq\",\n \"score\": 4.078176516486565e-06\n },\n {\n \"language\": \"Chuvash\",\n \"code\": \"chv\",\n \"score\": 4.048466053063748e-06\n },\n {\n \"language\": \"Panjabi\",\n \"code\": \"pan\",\n \"score\": 3.940522674383828e-06\n },\n {\n \"language\": \"Cornish\",\n \"code\": \"cor\",\n \"score\": 3.940297119697789e-06\n },\n {\n \"language\": \"West Low German\",\n \"code\": \"nds-nl\",\n \"score\": 3.926987574232044e-06\n },\n {\n \"language\": \"Cherokee\",\n \"code\": \"chr\",\n \"score\": 3.9112833292165305e-06\n },\n {\n \"language\": \"Ido\",\n \"code\": \"ido\",\n \"score\": 3.892145286954474e-06\n },\n {\n \"language\": \"Friulian\",\n \"code\": \"fur\",\n \"score\": 3.869370175380027e-06\n },\n {\n \"language\": \"Ukrainian\",\n \"code\": \"ukr\",\n \"score\": 3.7814761526533403e-06\n },\n {\n \"language\": \"Vietnamese\",\n \"code\": \"vie\",\n \"score\": 3.7795757634739857e-06\n },\n {\n \"language\": \"Emilian\",\n \"code\": \"egl\",\n \"score\": 3.7286854421836324e-06\n },\n {\n \"language\": \"Hungarian\",\n \"code\": \"hun\",\n \"score\": 3.706084498844575e-06\n },\n {\n \"language\": \"Haitian Creole\",\n \"code\": \"hat\",\n \"score\": 3.6860656109638512e-06\n },\n {\n \"language\": \"Jamaican Patois\",\n \"code\": \"jam\",\n \"score\": 3.6750652725459076e-06\n },\n {\n \"language\": \"Turkmen\",\n \"code\": \"tuk\",\n \"score\": 3.6414037367649144e-06\n },\n {\n \"language\": \"Gagauz\",\n \"code\": \"gag\",\n \"score\": 3.6310443647380453e-06\n },\n {\n \"language\": \"Yakut\",\n \"code\": \"sah\",\n \"score\": 3.611620968513307e-06\n },\n {\n \"language\": \"Breton\",\n \"code\": \"bre\",\n \"score\": 3.5204120649723336e-06\n },\n {\n \"language\": \"Afrikaans\",\n \"code\": \"afr\",\n \"score\": 3.5164177916158224e-06\n },\n {\n \"language\": \"Assamese\",\n \"code\": \"asm\",\n \"score\": 3.5076063795713708e-06\n },\n {\n \"language\": \"Crimean Tatar\",\n \"code\": \"crh\",\n \"score\": 3.4974791560671292e-06\n },\n {\n \"language\": \"Tswana\",\n \"code\": \"tsn\",\n \"score\": 3.4639840578165604e-06\n },\n {\n \"language\": \"Malagasy\",\n \"code\": \"mlg\",\n \"score\": 3.4424308523739455e-06\n },\n {\n \"language\": \"Tamil\",\n \"code\": \"tam\",\n \"score\": 3.433554866205668e-06\n },\n {\n \"language\": \"Belarusian (Taraschkewiza)\",\n \"code\": \"be-tarask\",\n \"score\": 3.4065565159835387e-06\n },\n {\n \"language\": \"Scottish Gaelic\",\n \"code\": \"gla\",\n \"score\": 3.383374632903724e-06\n },\n {\n \"language\": \"Latin\",\n \"code\": \"lat\",\n \"score\": 3.299320724181598e-06\n },\n {\n \"language\": \"Chavacano\",\n \"code\": \"cbk\",\n \"score\": 3.277132236689795e-06\n },\n {\n \"language\": \"Tarantino dialect\",\n \"code\": \"roa-tara\",\n \"score\": 3.2704483601264656e-06\n },\n {\n \"language\": \"Modern Greek\",\n \"code\": \"ell\",\n \"score\": 3.2669522624928504e-06\n },\n {\n \"language\": \"Ladino\",\n \"code\": \"lad\",\n \"score\": 3.1890219815977616e-06\n },\n {\n \"language\": \"Latgalian\",\n \"code\": \"ltg\",\n \"score\": 3.1830948046263075e-06\n },\n {\n \"language\": \"Pampanga\",\n \"code\": \"pam\",\n \"score\": 3.1460281206818763e-06\n },\n {\n \"language\": \"Tagalog\",\n \"code\": \"tgl\",\n \"score\": 3.100457433902193e-06\n },\n {\n \"language\": \"Hebrew\",\n \"code\": \"heb\",\n \"score\": 3.0715009415871464e-06\n },\n {\n \"language\": \"Serbo-Croatian\",\n \"code\": \"hbs\",\n \"score\": 3.050950908800587e-06\n },\n {\n \"language\": \"Achinese\",\n \"code\": \"ace\",\n \"score\": 3.0138855890982086e-06\n },\n {\n \"language\": \"Italian\",\n \"code\": \"ita\",\n \"score\": 3.003329993589432e-06\n },\n {\n \"language\": \"English\",\n \"code\": \"eng\",\n \"score\": 2.97778979074792e-06\n },\n {\n \"language\": \"Burmese\",\n \"code\": \"mya\",\n \"score\": 2.9546490623033606e-06\n },\n {\n \"language\": \"Spanish\",\n \"code\": \"spa\",\n \"score\": 2.9272057417983888e-06\n },\n {\n \"language\": \"Papiamento\",\n \"code\": \"pap\",\n \"score\": 2.8780641514458694e-06\n },\n {\n \"language\": \"Sardinian\",\n \"code\": \"srd\",\n \"score\": 2.866505383281037e-06\n },\n {\n \"language\": \"Esperanto\",\n \"code\": \"epo\",\n \"score\": 2.848199301297427e-06\n },\n {\n \"language\": \"Serbian\",\n \"code\": \"srp\",\n \"score\": 2.7479175059852423e-06\n },\n {\n \"language\": \"Zeeuws\",\n \"code\": \"zea\",\n \"score\": 2.7430314730736427e-06\n },\n {\n \"language\": \"Czech\",\n \"code\": \"ces\",\n \"score\": 2.7409500944486354e-06\n },\n {\n \"language\": \"Bengali\",\n \"code\": \"ben\",\n \"score\": 2.6958239232044434e-06\n },\n {\n \"language\": \"Erzya\",\n \"code\": \"myv\",\n \"score\": 2.6273187359038275e-06\n },\n {\n \"language\": \"Croatian\",\n \"code\": \"hrv\",\n \"score\": 2.6178654479735997e-06\n },\n {\n \"language\": \"Buryat\",\n \"code\": \"bxr\",\n \"score\": 2.60430465459649e-06\n },\n {\n \"language\": \"Swahili (macrolanguage)\",\n \"code\": \"swa\",\n \"score\": 2.6016373340098653e-06\n },\n {\n \"language\": \"Pangasinan\",\n \"code\": \"pag\",\n \"score\": 2.60037768384791e-06\n },\n {\n \"language\": \"Xhosa\",\n \"code\": \"xho\",\n \"score\": 2.580123918960453e-06\n },\n {\n \"language\": \"Bosnian\",\n \"code\": \"bos\",\n \"score\": 2.5763115445442963e-06\n },\n {\n \"language\": \"Low German\",\n \"code\": \"nds\",\n \"score\": 2.5743340756889665e-06\n },\n {\n \"language\": \"Kinyarwanda\",\n \"code\": \"kin\",\n \"score\": 2.568235458966228e-06\n },\n {\n \"language\": \"Aromanian\",\n \"code\": \"rup\",\n \"score\": 2.520287125662435e-06\n },\n {\n \"language\": \"Aragonese\",\n \"code\": \"arg\",\n \"score\": 2.4836215288814856e-06\n },\n {\n \"language\": \"Tetum\",\n \"code\": \"tet\",\n \"score\": 2.396502168267034e-06\n },\n {\n \"language\": \"Quechua\",\n \"code\": \"que\",\n \"score\": 2.3799134396540467e-06\n },\n {\n \"language\": \"Livvi-Karelian\",\n \"code\": \"olo\",\n \"score\": 2.3709426386631094e-06\n },\n {\n \"language\": \"Kashubian\",\n \"code\": \"csb\",\n \"score\": 2.358733354412834e-06\n },\n {\n \"language\": \"Avar\",\n \"code\": \"ava\",\n \"score\": 2.330698407604359e-06\n },\n {\n \"language\": \"Hausa\",\n \"code\": \"hau\",\n \"score\": 2.286114295202424e-06\n },\n {\n \"language\": \"Ripuarisch\",\n \"code\": \"ksh\",\n \"score\": 2.254129412904149e-06\n },\n {\n \"language\": \"Bulgarian\",\n \"code\": \"bul\",\n \"score\": 2.2492179141408997e-06\n },\n {\n \"language\": \"Oriya\",\n \"code\": \"ori\",\n \"score\": 2.1661755909008207e-06\n },\n {\n \"language\": \"Interlingue\",\n \"code\": \"ile\",\n \"score\": 2.059975486190524e-06\n },\n {\n \"language\": \"Guarani\",\n \"code\": \"grn\",\n \"score\": 2.024690957114217e-06\n },\n {\n \"language\": \"Banjar\",\n \"code\": \"bjn\",\n \"score\": 2.0237362150510307e-06\n },\n {\n \"language\": \"Thai\",\n \"code\": \"tha\",\n \"score\": 2.01868806470884e-06\n },\n {\n \"language\": \"Dutch\",\n \"code\": \"nld\",\n \"score\": 1.9297158360132016e-06\n },\n {\n \"language\": \"Kabyle\",\n \"code\": \"kab\",\n \"score\": 1.9132662600895856e-06\n },\n {\n \"language\": \"Palatine German\",\n \"code\": \"pfl\",\n \"score\": 1.9122355752188014e-06\n },\n {\n \"language\": \"Javanese\",\n \"code\": \"jav\",\n \"score\": 1.8900879013017402e-06\n },\n {\n \"language\": \"Banyumasan\",\n \"code\": \"map-bms\",\n \"score\": 1.8552185565567925e-06\n },\n {\n \"language\": \"Faroese\",\n \"code\": \"fao\",\n \"score\": 1.8414674514133367e-06\n },\n {\n \"language\": \"Scots\",\n \"code\": \"sco\",\n \"score\": 1.818199393710529e-06\n },\n {\n \"language\": \"Central Khmer\",\n \"code\": \"khm\",\n \"score\": 1.7993022538576042e-06\n },\n {\n \"language\": \"Slovak\",\n \"code\": \"slk\",\n \"score\": 1.7988603531193803e-06\n },\n {\n \"language\": \"Belarusian\",\n \"code\": \"bel\",\n \"score\": 1.782583581189101e-06\n },\n {\n \"language\": \"Swedish\",\n \"code\": \"swe\",\n \"score\": 1.7702136574371252e-06\n },\n {\n \"language\": \"Saterfriesisch\",\n \"code\": \"stq\",\n \"score\": 1.7663436437942437e-06\n },\n {\n \"language\": \"Latvian\",\n \"code\": \"lav\",\n \"score\": 1.7178032294395962e-06\n },\n {\n \"language\": \"Konkani\",\n \"code\": \"kok\",\n \"score\": 1.690383783170546e-06\n },\n {\n \"language\": \"Tuvan\",\n \"code\": \"tyv\",\n \"score\": 1.672853159107035e-06\n },\n {\n \"language\": \"Walloon\",\n \"code\": \"wln\",\n \"score\": 1.6722132158975e-06\n },\n {\n \"language\": \"Sranan\",\n \"code\": \"srn\",\n \"score\": 1.646132773203135e-06\n },\n {\n \"language\": \"Picard\",\n \"code\": \"pcd\",\n \"score\": 1.6385885146519286e-06\n },\n {\n \"language\": \"Limburgan\",\n \"code\": \"lim\",\n \"score\": 1.6372666777897393e-06\n },\n {\n \"language\": \"French\",\n \"code\": \"fra\",\n \"score\": 1.6239549722740776e-06\n },\n {\n \"language\": \"Icelandic\",\n \"code\": \"isl\",\n \"score\": 1.5904075780781568e-06\n },\n {\n \"language\": \"Irish\",\n \"code\": \"gle\",\n \"score\": 1.5750525790281245e-06\n },\n {\n \"language\": \"Corsican\",\n \"code\": \"cos\",\n \"score\": 1.570832523611898e-06\n },\n {\n \"language\": \"Alemannic German\",\n \"code\": \"als\",\n \"score\": 1.5651218063794659e-06\n },\n {\n \"language\": \"German\",\n \"code\": \"deu\",\n \"score\": 1.5594737305946182e-06\n },\n {\n \"language\": \"Upper Sorbian\",\n \"code\": \"hsb\",\n \"score\": 1.5125158370210556e-06\n },\n {\n \"language\": \"Romanian\",\n \"code\": \"ron\",\n \"score\": 1.5119784393391456e-06\n },\n {\n \"language\": \"Manx\",\n \"code\": \"glv\",\n \"score\": 1.5035052456369158e-06\n },\n {\n \"language\": \"Lingala\",\n \"code\": \"lin\",\n \"score\": 1.493238073635439e-06\n },\n {\n \"language\": \"Malay\",\n \"code\": \"msa\",\n \"score\": 1.4067626352698426e-06\n },\n {\n \"language\": \"Maltese\",\n \"code\": \"mlt\",\n \"score\": 1.370485165352875e-06\n },\n {\n \"language\": \"Luxembourgish\",\n \"code\": \"ltz\",\n \"score\": 1.3397349221122568e-06\n },\n {\n \"language\": \"Estonian\",\n \"code\": \"est\",\n \"score\": 1.3280839539220324e-06\n },\n {\n \"language\": \"Kabardian\",\n \"code\": \"kbd\",\n \"score\": 1.3062604011793155e-06\n },\n {\n \"language\": \"Macedonian\",\n \"code\": \"mkd\",\n \"score\": 1.2802570381609257e-06\n },\n {\n \"language\": \"Pennsylvania German\",\n \"code\": \"pdc\",\n \"score\": 1.2550040082714986e-06\n },\n {\n \"language\": \"Sundanese\",\n \"code\": \"sun\",\n \"score\": 1.1068191270169336e-06\n },\n {\n \"language\": \"Iloko\",\n \"code\": \"ilo\",\n \"score\": 1.0791690101541462e-06\n },\n {\n \"language\": \"Karakalpak\",\n \"code\": \"kaa\",\n \"score\": 1.0603262126096524e-06\n },\n {\n \"language\": \"Norwegian Nynorsk\",\n \"code\": \"nno\",\n \"score\": 1.0554679192864569e-06\n },\n {\n \"language\": \"Yoruba\",\n \"code\": \"yor\",\n \"score\": 1.046297711582156e-06\n },\n {\n \"language\": \"Neapolitan\",\n \"code\": \"nap\",\n \"score\": 1.0279602520313347e-06\n },\n {\n \"language\": \"Danish\",\n \"code\": \"dan\",\n \"score\": 1.0038916116172913e-06\n },\n {\n \"language\": \"Indonesian\",\n \"code\": \"ind\",\n \"score\": 9.83746303973021e-07\n },\n {\n \"language\": \"Mirandese\",\n \"code\": \"mwl\",\n \"score\": 8.806521236692788e-07\n },\n {\n \"language\": \"Catalan\",\n \"code\": \"cat\",\n \"score\": 8.687447348165733e-07\n },\n {\n \"language\": \"Turkish\",\n \"code\": \"tur\",\n \"score\": 8.384120064874878e-07\n },\n {\n \"language\": \"Veps\",\n \"code\": \"vep\",\n \"score\": 7.812500371073838e-07\n },\n {\n \"language\": \"Bokm\\u00e5l\",\n \"code\": \"nob\",\n \"score\": 7.427178161378833e-07\n },\n {\n \"language\": \"Shona\",\n \"code\": \"sna\",\n \"score\": 6.660703775196453e-07\n },\n {\n \"language\": \"Bavarian\",\n \"code\": \"bar\",\n \"score\": 6.222485353646334e-07\n },\n {\n \"language\": \"Uzbek\",\n \"code\": \"uzb\",\n \"score\": 6.021850822435226e-07\n },\n {\n \"language\": \"Central Bikol\",\n \"code\": \"bcl\",\n \"score\": 5.77034370508045e-07\n },\n {\n \"language\": \"Asturian\",\n \"code\": \"ast\",\n \"score\": 5.743918336520437e-07\n },\n {\n \"language\": \"Lombard\",\n \"code\": \"lmo\",\n \"score\": 4.6301857992148143e-07\n },\n {\n \"language\": \"Romansh\",\n \"code\": \"roh\",\n \"score\": 4.5534079617937095e-07\n },\n {\n \"language\": \"Narom\",\n \"code\": \"nrm\",\n \"score\": 3.6611126574825903e-07\n },\n {\n \"language\": \"Northern Sami\",\n \"code\": \"sme\",\n \"score\": 1.0723972820869676e-07\n }\n ]\n]\n```\n\n\u003c/details\u003e\n\n## Citation\n\nPlease cite this repository in publications as the following:\n\n```bibtex\n@misc{ZabanShenas,\n author = {Mehrdad Farahani},\n title = {Zabanshenas is a solution for identifying the most likely language of a piece of written text},\n month = feb,\n year = 2021,\n publisher = {Zenodo},\n version = {v2.0.1},\n doi = {10.5281/zenodo.5029022},\n url = {https://doi.org/10.5281/zenodo.5029022}\n}\n```\n\n\n## License\n[Apache License 2.0](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm3hrdadfi%2Fzabanshenas","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fm3hrdadfi%2Fzabanshenas","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm3hrdadfi%2Fzabanshenas/lists"}