{"id":15014179,"url":"https://github.com/davebulaval/spacy-language-detection","last_synced_at":"2025-10-06T08:31:06.266Z","repository":{"id":57469522,"uuid":"404357888","full_name":"davebulaval/spacy-language-detection","owner":"davebulaval","description":"Fully customizable language detection for spaCy pipeline","archived":false,"fork":true,"pushed_at":"2021-09-08T16:28:11.000Z","size":45,"stargazers_count":8,"open_issues_count":3,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-11-16T17:21:26.161Z","etag":null,"topics":["language-detection","nlp","spacy","spacy-extension"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"Abhijit-2592/spacy-langdetect","license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/davebulaval.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-09-08T13:27:48.000Z","updated_at":"2024-08-30T22:43:37.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/davebulaval/spacy-language-detection","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davebulaval%2Fspacy-language-detection","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davebulaval%2Fspacy-language-detection/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davebulaval%2Fspacy-language-detection/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davebulaval%2Fspacy-language-detection/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/davebulaval","download_url":"https://codeload.github.com/davebulaval/spacy-language-detection/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235515424,"owners_count":19002481,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["language-detection","nlp","spacy","spacy-extension"],"created_at":"2024-09-24T19:45:17.873Z","updated_at":"2025-10-06T08:31:00.978Z","avatar_url":"https://github.com/davebulaval.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Here is spacy_language_detection\n\nSpacy_language_detection is a fully customizable language detection for [spaCy](https://github.com/explosion/spaCy)\npipeline forked from\n[spacy-langdetect](https://github.com/Abhijit-2592/spacy-langdetect) in order to fix the seed problem (see [this issue](https://github.com/Abhijit-2592/spacy-langdetect/issues/3)) and to update it with spaCy 3.0.\n\nUse spacy_language_detection to\n\n- Detect the language of a document,\n- Detect the language of the sentences of a document.\n\n## Installation\n\n`pip install spacy-language-detection`\n\n## Basic Usage\n\nOut of the box, under the hood, it uses [langdetect](https://github.com/Mimino666/langdetect) to detect languages on\nspaCy's Doc and Span objects.\n\nHere is how to use it for spaCy 3.0\nsee [here](https://github.com/davebulaval/spacy-language-detection/blob/master/examples/detect_text_language_spacy2.py)\nfor an example with spaCy 2.0.\n\n```python\nimport spacy\nfrom spacy.language import Language\n\nfrom spacy_language_detection import LanguageDetector\n\n\ndef get_lang_detector(nlp, name):\n    return LanguageDetector(seed=42)  # We use the seed 42\n\n\nnlp_model = spacy.load(\"en_core_web_sm\")\nLanguage.factory(\"language_detector\", func=get_lang_detector)\nnlp_model.add_pipe('language_detector', last=True)\n\n# Document level language detection\njob_title = \"Senior NLP Research Engineer\"\ndoc = nlp_model(job_title)\nlanguage = doc._.language\nprint(language)\n\n# Sentence level language detection\ntext = \"This is English text. Er lebt mit seinen Eltern und seiner Schwester in Berlin. Yo me divierto todos los días en el parque. Je m'appelle Angélica Summer, j'ai 12 ans et je suis canadienne.\"\ndoc = nlp_model(text)\nfor i, sent in enumerate(doc.sents):\n    print(sent, sent._.language)\n```\n\n## Using your own language detector\n\nSuppose you are not happy with the accuracy of the out-of-the-box language detector, or you have your own language\ndetector, which you want to use with a spaCy pipeline. How do you do it? That's where the `language_detection_function`\nargument comes in. The function takes in a spaCy Doc or Span object and can return any Python object which is stored\nin `doc._.language` and `span._.language`. For example, let's say you want to\nuse [googletrans](https://pypi.org/project/googletrans/) as your language detection module:\n\n```python\nimport spacy\nfrom spacy.tokens import Doc, Span\nfrom spacy_language_detection import LanguageDetector\n# install using pip install googletrans\nfrom googletrans import Translator\n\nnlp = spacy.load(\"en\")\n\n\ndef custom_detection_function(spacy_object):\n    # Custom detection function should take a spaCy Doc or a Span\n    assert isinstance(spacy_object, Doc) or isinstance(\n        spacy_object, Span), \"spacy_object must be a spacy Doc or Span object but it is a {}\".format(type(spacy_object))\n    detection = Translator().detect(spacy_object.text)\n    return {'language': detection.lang, 'score': detection.confidence}\n\n\ndef get_lang_detector(nlp, name):\n    return LanguageDetector(language_detection_function=custom_detection_function, seed=42)  # We use the seed 42\n\n\nnlp_model = spacy.load(\"en_core_web_sm\")\nLanguage.factory(\"language_detector\", func=get_lang_detector)\nnlp_model.add_pipe('language_detector', last=True)\n\ntext = \"This is English text. Er lebt mit seinen Eltern und seiner Schwester in Berlin. Yo me divierto todos los días en el parque. Je m'appelle Angélica Summer, j'ai 12 ans et je suis canadienne.\"\n\n# Document level language detection\ndoc = nlp_model(text)\nlanguage = doc._.language\nprint(language)\n\n# Sentence level language detection\ntext = \"This is English text. Er lebt mit seinen Eltern und seiner Schwester in Berlin. Yo me divierto todos los días en el parque. Je m'appelle Angélica Summer, j'ai 12 ans et je suis canadienne.\"\ndoc = nlp_model(text)\nfor i, sent in enumerate(doc.sents):\n    print(sent, sent._.language)\n```\n\nSimilarly, you can also use [pycld2](https://pypi.org/project/pycld2/) and other language detectors with spaCy.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavebulaval%2Fspacy-language-detection","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdavebulaval%2Fspacy-language-detection","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavebulaval%2Fspacy-language-detection/lists"}