{"id":13766254,"url":"https://github.com/aparo/opensearch-ingest-langdetect","last_synced_at":"2026-03-08T09:38:02.760Z","repository":{"id":66571731,"uuid":"427768469","full_name":"aparo/opensearch-ingest-langdetect","owner":"aparo","description":"Ingest processor doing language detection for fields. Port of https://github.com/spinscale/elasticsearch-ingest-langdetect","archived":false,"fork":false,"pushed_at":"2025-05-18T13:04:22.000Z","size":345,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-18T13:41:59.885Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aparo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-11-13T20:43:21.000Z","updated_at":"2025-05-18T13:04:25.000Z","dependencies_parsed_at":null,"dependency_job_id":"75741726-20a6-465d-a68c-320faf3e79fb","html_url":"https://github.com/aparo/opensearch-ingest-langdetect","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/aparo/opensearch-ingest-langdetect","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aparo%2Fopensearch-ingest-langdetect","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aparo%2Fopensearch-ingest-langdetect/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aparo%2Fopensearch-ingest-langdetect/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aparo%2Fopensearch-ingest-langdetect/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aparo","download_url":"https://codeload.github.com/aparo/opensearch-ingest-langdetect/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aparo%2Fopensearch-ingest-langdetect/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271746551,"owners_count":24813570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-23T02:00:09.327Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T16:00:53.124Z","updated_at":"2026-03-08T09:37:57.707Z","avatar_url":"https://github.com/aparo.png","language":"Java","funding_links":[],"categories":["Plugins"],"sub_categories":[],"readme":"# OpenSearch Langdetect Ingest Processor\n\nThis is a port of [spinscale's ElasticSearch Langdetect ingest plugin](https://github.com/spinscale/elasticsearch-ingest-langdetect).\nThe code was migrate using my migration script [ElasticSearch to OpenSearch Migration Scripts](https://github.com/aparo/elasticsearch-opensearch-migration-scripts)\n\nUses the [langdetect](https://github.com/YouCruit/language-detection/) plugin to try to find out the language used in a field.\n\nNote that OpenSearch has native support for langdetection nowadays using the\n`inference` ingest processor. See more in\n[the documentation](https://www.elastic.co/guide/en/machine-learning/current/ml-lang-ident.html)\n\n## Installation\n\n| OS    | Command |\n| ----- | ------- |\n| 1.1.0 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-ingest-langdetect/releases/download/1.1.0/ingest-langdetect-1.1.0.zip` |\n| 1.2.0 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-ingest-langdetect/releases/download/1.2.0/ingest-langdetect-1.2.0.zip` |\n| 1.2.2 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-ingest-langdetect/releases/download/1.2.2/ingest-langdetect-1.2.2.zip` |\n| 1.2.3 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-ingest-langdetect/releases/download/1.2.3/ingest-langdetect-1.2.3.zip` |\n\n## Usage\n\n\n```\nPUT _ingest/pipeline/langdetect-pipeline\n{\n  \"description\": \"A pipeline to do whatever\",\n  \"processors\": [\n    {\n      \"langdetect\" : {\n        \"field\" : \"my_field\",\n        \"target_field\" : \"language\"\n      }\n    }\n  ]\n}\n\nPUT /my-index/my-type/1?pipeline=langdetect-pipeline\n{\n  \"my_field\" : \"This is hopefully an english text, that will be detected.\"\n}\n\nGET /my-index/my-type/1\n\n# Expected response\n{\n  \"my_field\" : \"This is hopefully an english text, that will be detected.\",\n  \"language\": \"en\"\n}\n```\n\nYou could also set certain fields that use different analyzers for different languages\n\n```\nPUT _ingest/pipeline/langdetect-analyzer-pipeline\n{\n  \"description\": \"A pipeline to index data into language specific analyzers\",\n  \"processors\": [\n    {\n      \"langdetect\": {\n        \"field\": \"my_field\",\n        \"target_field\": \"lang\"\n      }\n    },\n    {\n      \"script\": {\n        \"source\": \"ctx.language = [:];ctx.language[ctx.lang] = ctx.remove('my_field')\"\n      }\n    }\n  ]\n}\n\nPUT documents\n{\n  \"mappings\": {\n    \"doc\" : {\n      \"properties\" : {\n        \"language\": {\n          \"properties\": {\n            \"de\" : {\n              \"type\": \"text\",\n              \"analyzer\": \"german\"\n            },\n            \"en\" : {\n              \"type\": \"text\",\n              \"analyzer\": \"english\"\n            }\n          }\n        }\n      }\n    }\n  }\n}\n\nPUT /my-index/doc/1?pipeline=langdetect-analyzer-pipeline\n{\n  \"my_field\" : \"This is an english text\"\n}\n\nPUT /my-index/doc/2?pipeline=langdetect-analyzer-pipeline\n{\n  \"my_field\" : \"Das hier ist ein deutscher Text.\"\n}\n\nGET my-index/doc/1\n\nGET my-index/doc/2\n```\n\n## Configuration\n\n| Parameter | Use |\n| --- | --- |\n| field          | Field name of where to read the content from |\n| target_field   | Field name to write the language to |\n| max_length     | Max length of of characters to read, defaults to 10kb, requires a byte size value, like 1mb |\n| ignore_missing | Ignore missing source field. Not throwing exception in that case. Expects for boolean value, defaults to false. |\n| 2.1.0 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-analysis-pinyin/releases/download/2.1.0/opensearch-analysis-pinyin.zip` |\n| 2.2.0 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-analysis-pinyin/releases/download/2.2.0/opensearch-analysis-pinyin.zip` |\n| 2.3.0 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-analysis-pinyin/releases/download/2.3.0/opensearch-analysis-pinyin.zip` |\n| 2.4.0 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-analysis-pinyin/releases/download/2.4.0/opensearch-analysis-pinyin.zip` |\n| 2.6.0 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-analysis-pinyin/releases/download/2.6.0/opensearch-analysis-pinyin.zip` |\n| 2.7.0 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-analysis-pinyin/releases/download/2.7.0/opensearch-analysis-pinyin.zip` |\n| 2.8.0 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-analysis-pinyin/releases/download/2.8.0/opensearch-analysis-pinyin.zip` |\n| 2.9.0 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-analysis-pinyin/releases/download/2.9.0/opensearch-analysis-pinyin.zip` |\n| 2.10.0 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-analysis-pinyin/releases/download/2.10.0/opensearch-analysis-pinyin.zip` |\n| 2.11.0 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-analysis-pinyin/releases/download/2.11.0/opensearch-analysis-pinyin.zip` |\n| 2.11.1 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-analysis-pinyin/releases/download/2.11.1/opensearch-analysis-pinyin.zip` |\n| 2.11.1 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-ingest-langdetect/releases/download/2.11.1/opensearch-ingest-langdetect.zip` |\n| 2.12.0 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-ingest-langdetect/releases/download/2.12.0/opensearch-ingest-langdetect.zip` |\n| 2.13.0 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-ingest-langdetect/releases/download/2.13.0/opensearch-ingest-langdetect.zip` |\n| 2.14.0 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-ingest-langdetect/releases/download/2.14.0/opensearch-ingest-langdetect.zip` |\n| 2.15.0 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-ingest-langdetect/releases/download/2.15.0/opensearch-ingest-langdetect.zip` |\n| 2.16.0 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-ingest-langdetect/releases/download/2.16.0/opensearch-ingest-langdetect.zip` |\n| 2.17.0 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-ingest-langdetect/releases/download/2.17.0/opensearch-ingest-langdetect.zip` |\n| 2.18.0 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-ingest-langdetect/releases/download/2.18.0/opensearch-ingest-langdetect.zip` |\n| 2.19.0 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-ingest-langdetect/releases/download/2.19.0/opensearch-ingest-langdetect.zip` |\n| 2.19.1 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-ingest-langdetect/releases/download/2.19.1/opensearch-ingest-langdetect.zip` |\n| 2.19.2 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-ingest-langdetect/releases/download/2.19.2/opensearch-ingest-langdetect.zip` |\n| 3.0.0 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-ingest-langdetect/releases/download/3.0.0/opensearch-ingest-langdetect.zip` |\n\n## Setup\n\nIn order to install this plugin, you need to create a zip distribution first by running\n\n```bash\ngradle clean check\n```\n\nThis will produce a zip file in `build/distributions`.\n\nAfter building the zip file, you can install it like this\n\n```bash\nbin/plugin install file:///path/to/ingest-langdetect/build/distribution/ingest-langdetect-0.0.1-SNAPSHOT.zip\n```\n\n## Side notes\n\nIn order to cope with the security manager, a special factory is used to load the languages from the classpath.\nYou can check out the `SecureDetectorFactory` class. This implementation also does not use jsonic to prevent the use of reflection when loading the languages.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faparo%2Fopensearch-ingest-langdetect","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faparo%2Fopensearch-ingest-langdetect","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faparo%2Fopensearch-ingest-langdetect/lists"}