{"id":15661168,"url":"https://github.com/joshdevins/demo-es-lang-ident","last_synced_at":"2025-07-14T10:07:55.195Z","repository":{"id":137340357,"uuid":"233814462","full_name":"joshdevins/demo-es-lang-ident","owner":"joshdevins","description":"Demo: Elasticsearch Language Identification","archived":false,"fork":false,"pushed_at":"2020-01-23T13:42:59.000Z","size":45,"stargazers_count":19,"open_issues_count":0,"forks_count":8,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-05T21:35:49.516Z","etag":null,"topics":["demo","elasticsearch","language-identification","nlp","search"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/joshdevins.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-01-14T10:19:30.000Z","updated_at":"2024-11-26T09:26:31.000Z","dependencies_parsed_at":null,"dependency_job_id":"a228a3f1-c9ad-4c94-8331-f0435cb34290","html_url":"https://github.com/joshdevins/demo-es-lang-ident","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/joshdevins/demo-es-lang-ident","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joshdevins%2Fdemo-es-lang-ident","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joshdevins%2Fdemo-es-lang-ident/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joshdevins%2Fdemo-es-lang-ident/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joshdevins%2Fdemo-es-lang-ident/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/joshdevins","download_url":"https://codeload.github.com/joshdevins/demo-es-lang-ident/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joshdevins%2Fdemo-es-lang-ident/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265277586,"owners_count":23739416,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["demo","elasticsearch","language-identification","nlp","search"],"created_at":"2024-10-03T13:26:13.825Z","updated_at":"2025-07-14T10:07:55.172Z","avatar_url":"https://github.com/joshdevins.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Demo: Elasticsearch Language Identification\n\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n\nA demo of the Elasticsearch language identification for search use-cases, using the [WiLI-2018 corpus](https://arxiv.org/abs/1801.07779) (or a corpus of your choosing).\n\n# Setup\n\n## Prerequisites\n\nTo run the demo, aside from Elasticsearch, you'll need:\n\n1. [Python 3.x](https://www.python.org/downloads/)\n1. Linux/macOS tools: `make` and `wget`\n\n## Elasticsearch\n\nSee commands below for details.\n\n1. Download and install [Elasticsearch](https://www.elastic.co/downloads/elasticsearch), v7.6 or higher, with the Basic license (default)\n1. Install analysis plugins: [ICU](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu.html) `icu`, [Japanese](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-kuromoji.html) `kuromoji`, [Korean](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-nori.html) `nori`, [Chinese](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-smartcn.html) `smartcN`  \n1. Install a [German decompounder dictionary](https://github.com/uschindler/german-decompounder)\n\n### Installation Commands\n\nRun the following commands from the base of your Elasticsearch installation.\n\nAnalysis plugins:\n\n```bash\nbin/elasticsearch-plugin install analysis-icu\nbin/elasticsearch-plugin install analysis-kuromoji\nbin/elasticsearch-plugin install analysis-nori\nbin/elasticsearch-plugin install analysis-smartcn\n```\n\nGerman decompounder dictionaries:\n\n```bash\nmkdir -p config/analysis/de\ncd config/analysis/de\nwget https://raw.githubusercontent.com/uschindler/german-decompounder/master/de_DR.xml\nwget https://raw.githubusercontent.com/uschindler/german-decompounder/master/dictionary-de.txt\n```\n\n## Environment\n\nSet up the demo environment using `make all`. This will create a Python virtual environment with the required dependencies and download datasets for the demo.\n\nFor more details, have a look at the targets in the `Makefile`.\n\n# Demo\n\nThe demo contains two scripts: `bin/index` for indexing and `bin/search` for searching. Use the `--help` option to see instructions and available options for each script.\n\n## Indexing\n\nTo index documents for the demo, run `bin/index` either with all documents (default) or you can choose to index a subset of the documents. For the purposes of the following examples, we index just the first 10k documents: `bin/index --max 10000`\n\n## Search Examples\n\n### German Decompounding\n\nDue to the way the German language compounds words into larger words, special analysis is required to break them up into constituent parts for searching. Try searching for the term \"jahr\" (meaning \"year\") and you will see the results of per-language analysis.\n\n```\n# only matching exactly on the term \"jahr\"\nbin/search --strategy default jahr\n\n# matches: \"jahr\", \"jahre\", \"jahren\", \"jahrhunderts\", etc.\nbin/search --strategy per-field jahr\n```\n\n### Common Term\n\nSome domains use English or Latin terminology. Trying search for the term \"computer\".\n\n```\n# only matching exactly on the term \"computer\", but multiple languages are in the results\nbin/search --strategy default computer\n\n# matches compound German words as well: \"Computersicherheit\" (computer security)\nbin/search --strategy per-field computer\n```\n\n### Non-Latin Scripts\n\nOf course it's easy to see how Latin scripts can work even when just using the default or ICU analyzers. However for non-Latin scripts such as CJK (Chinese/Japanese/Korean), we really don't get any good results. Let's try searching for some common Japanese terms and see how our per-language analysis helps.\n\n```\n# standard analyzer gets poor precision and returns irrelevant/non-matching results with \"network\"/\"internet\": \"网络\"\nbin/search --strategy default 网络\n\n# ICU and language-specific analysis gets things right, but note the different scores\nbin/search --strategy icu 网络\nbin/search --strategy per-field 网络\n```\n\n### Per-Field or Per-Index\n\nLet's compare the scores of per-field and per-index strategies. Note how sometimes the order is different even if the scores are almost exactly the same. In some cases, this can impact search relevance metrics such as nDCG and precision@k. Choose your strategy accordingly!\n\n```\n# English tokens: order unchanged\nbin/search --strategy per-field networking\nbin/search --strategy per-index networking\n\n# Mixed-use tokens: order differs (last item)\nbin/search --strategy per-field university\nbin/search --strategy per-index university\n```\n\n# License\n\n```\nCopyright 2020 Josh Devins, Elastic NV\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n   http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoshdevins%2Fdemo-es-lang-ident","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoshdevins%2Fdemo-es-lang-ident","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoshdevins%2Fdemo-es-lang-ident/lists"}