{"id":22864945,"url":"https://github.com/mohsenim/language-identification","last_synced_at":"2025-07-05T04:04:32.814Z","repository":{"id":203926923,"uuid":"161049969","full_name":"mohsenim/Language-Identification","owner":"mohsenim","description":"Language Identification using  Näive Bayes","archived":false,"fork":false,"pushed_at":"2018-12-09T15:12:14.000Z","size":182,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-05T12:08:56.507Z","etag":null,"topics":["language-identification","language-identifier","naive-bayes","text-analysis"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mohsenim.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-12-09T14:52:16.000Z","updated_at":"2024-02-20T11:16:31.000Z","dependencies_parsed_at":null,"dependency_job_id":"33ed01dc-f1b6-4836-9697-7190ff269233","html_url":"https://github.com/mohsenim/Language-Identification","commit_stats":null,"previous_names":["mohsenim/language-identification"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mohsenim/Language-Identification","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mohsenim%2FLanguage-Identification","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mohsenim%2FLanguage-Identification/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mohsenim%2FLanguage-Identification/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mohsenim%2FLanguage-Identification/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mohsenim","download_url":"https://codeload.github.com/mohsenim/Language-Identification/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mohsenim%2FLanguage-Identification/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260009855,"owners_count":22945548,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["language-identification","language-identifier","naive-bayes","text-analysis"],"created_at":"2024-12-13T11:32:07.953Z","updated_at":"2025-06-15T16:38:37.752Z","avatar_url":"https://github.com/mohsenim.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Language Identification using Character N-grams\n\nApplied for a visiting position, I was asked to implement a method for language\nidentification of a document. Though simple, it works well. So, I put it here.\n\nMy approach for identifying the language of a text is to utilize character\nn-grams. Fortunately, it is easy to collect training data for this task. For each\nlanguage we only need a number of documents. Once we have training texts\nwe can use them and extract features. In my method, features are character\nn-grams. N-grams are extracted for each language and their frequencies are\ncalculated. There are several machine learning methods which can be applied\nfor language identification. I exploit Näive Bayes, a simple yet very effective\nmethod. Näive Bayes uses the probability of n-grams to identify the language\nof a new text.\n\n### Model of Languages\n\nFor language identification a model of languages should be created. In the\ncode, a list of text files, each containing training texts for a language, is read at the begining.\nThe name of the files indicates languages. To add a language, one can simply\nadd another text file and put some texts in it. These files are used to create the\nlanguage identification model. For each language, a list of character n-grams\nand their frequencies are extracted and saved into a file. The default maximum\nlength of n-grams is set to 3.\n\n### Langauge Identification\n\nIn the language identification phase, files containing character n-grams and\ntheir frequencies are read into a Map. As the total frequency of n-grams are\nrequired to calculate probabilities, the total frequencies are also saved into a\nMap. To identify the language of an input text, first it is converted to a list\nof n-grams. Afterwards, p(Language|T ext) is computed using Näive Bayes.\nThe prior probabilities of languages are set equal to each other as there is no\ninformation about the domain the code may be used in. I also apply Laplace\nmethod to smooth p(n-gram|Language). The language with the maximum\nprobability p(Language|T ext) is considered as the language of the given text.\n\n### Parameters\n\nParameters of the code are the maximum length of n-grams, the path to\nthe language text files, and the path to the language n-grams files. The default\nvalue of the parameters is set in a property file called properties.prop. The\ntest code performs on four languages: English, French, Persian and Arabic.\n\n### Licence\nMIT\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmohsenim%2Flanguage-identification","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmohsenim%2Flanguage-identification","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmohsenim%2Flanguage-identification/lists"}