{"id":22863764,"url":"https://github.com/yakaz/elasticsearch-analysis-combo","last_synced_at":"2025-07-31T13:16:21.495Z","repository":{"id":3656261,"uuid":"4724412","full_name":"yakaz/elasticsearch-analysis-combo","owner":"yakaz","description":"Elasticsearch Combo Analyzer","archived":false,"fork":false,"pushed_at":"2017-03-01T19:26:50.000Z","size":421,"stargazers_count":85,"open_issues_count":8,"forks_count":21,"subscribers_count":17,"default_branch":"master","last_synced_at":"2025-07-09T02:52:36.247Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"MrSwitch/hello.js","license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yakaz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-06-20T09:22:03.000Z","updated_at":"2023-11-29T04:26:49.000Z","dependencies_parsed_at":"2022-08-20T16:30:33.268Z","dependency_job_id":null,"html_url":"https://github.com/yakaz/elasticsearch-analysis-combo","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/yakaz/elasticsearch-analysis-combo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yakaz%2Felasticsearch-analysis-combo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yakaz%2Felasticsearch-analysis-combo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yakaz%2Felasticsearch-analysis-combo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yakaz%2Felasticsearch-analysis-combo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yakaz","download_url":"https://codeload.github.com/yakaz/elasticsearch-analysis-combo/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yakaz%2Felasticsearch-analysis-combo/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268045808,"owners_count":24186854,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-31T02:00:08.723Z","response_time":66,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-13T11:17:22.088Z","updated_at":"2025-07-31T13:16:21.353Z","avatar_url":"https://github.com/yakaz.png","language":"Java","funding_links":[],"categories":["人工智能"],"sub_categories":["自然语言处理"],"readme":"Elasticsearch Combo Analyzer\n============================\n\nThe Combo Analyzer plugin provides with a new analyzer type that combines the output of multiple analyzers into one.\n\nInstallation\n-----------\n\nSimply run at the root of your ElasticSearch v0.90+ installation:\n\n\tbin/plugin -install com.yakaz.elasticsearch.plugins/elasticsearch-analysis-combo/1.5.1\n\nThis will download the plugin from the Central Maven Repository.\n\nFor older versions of ElasticSearch, you can still use v1.2.0 using the longer command:\n\n\tbin/plugin -url http://oss.sonatype.org/content/repositories/releases/com/yakaz/elasticsearch/plugins/elasticsearch-analysis-combo/1.2.0/elasticsearch-analysis-combo-1.2.0.zip install elasticsearch-analysis-combo\n\nIn order to declare this plugin as a dependency, add the following to your `pom.xml`:\n\n\t\u003cdependency\u003e\n\t    \u003cgroupId\u003ecom.yakaz.elasticsearch.plugins\u003c/groupId\u003e\n\t    \u003cartifactId\u003eelasticsearch-analysis-combo\u003c/artifactId\u003e\n\t    \u003cversion\u003e1.5.1\u003c/version\u003e\n\t\u003c/dependency\u003e\n\nVersion matrix:\n\n    ┌───────────────────────┬──────────────────────────┐\n    │ Combo Analyzer Plugin │ ElasticSearch            │\n    ├───────────────────────┼──────────────────────────┤\n    │ 1.5.x                 │ 1.0.0.RC1 ─► (1.4.3)     │\n    │                       │           ─► (1.3.8)     │\n    ├───────────────────────┼──────────────────────────┤\n    │ 1.4.x                 │ 0.90.3 ─► 0.90.5         │\n    ├───────────────────────┼──────────────────────────┤\n    │ 1.3.x                 │ 0.90.0 ─► 0.90.2         │\n    ├───────────────────────┼──────────────────────────┤\n    │ 1.2.x                 │ 0.19 ─► 0.20             │\n    ├───────────────────────┼──────────────────────────┤\n    │ 1.1.x                 │ 0.19 ─► 0.20             │\n    ├───────────────────────┼──────────────────────────┤\n    │ 1.0.x                 │ 0.19 ─► 0.20             │\n    └───────────────────────┴──────────────────────────┘\n\nDescription\n-----------\n\nIf you have a multilingual index, where each document has its source language, you can analyze the text fields using an analyzer based on the detected language. \nBut what if you can't detect the language, the language detected was wrong, or aggressive stemming deforms the indexed terms? Or if you want to use other language-specific analyzers at search time, but still like to search original word forms indexed in another language?\n\nIn such cases, it helps to have the original words together with the stemmed words in the index.\n\nThe proposed solution merges the terms received from multiple analyzers. It is called the `ComboAnalyzer`.\n\nConfiguration\n-------------\n\nThe plugin provides you with the `combo` analyzer type.  \nIt expects a list of other analyzers to be used, under the `sub_analyzers` property.  \nYou can remove duplicated tokens sharing the same position by setting `true` for the `deduplication` property.\n\nIt is good practice to use the combo analyzer for both index and search.\n\n### Example\n\nFor example, you want to index German text together with other multilingual text in one field. A `combo` setup could look like\n\n\t{\n\t    \"index\" : {\n\t        \"analysis\" : {\n\t            \"analyzer\" : {\n\t                \"default\" : {\n\t                    \"type\" : \"custom\",\n\t                    \"tokenizer\" : \"icu_tokenizer\",\n\t                    \"filter\" : [ \"snowball\", \"icu_folding\" ]\n\t                },\n\t                \"combo\" : {\n\t                    \"type\" : \"combo\",\n\t                    \"sub_analyzers\" : [ \"standard\", \"default\" ]\n\t                }\n\t            },\n\t            \"filter\" : {\n\t                \"snowball\" : {\n\t                    \"type\" : \"snowball\",\n\t                    \"language\" : \"German2\"\n\t                }\n\t            }\n\t        }\n\t    }\n\t}\n\nFor handling German words with umlauts, snowball `German2` filter will expand umlauts (ä to ae, ö to oe, ü to ue). The ICU folding also recognizes german ß as ss, and performs de-accentuation (also known as normalization) on the other languages (also arabic and asian characters). The standard analyzer keeps the original word forms, and is combined with the new default analyzer.\n\nThe result can be checked with the Analyze API of Elasticsearch.\n\n\tcurl -XGET 'localhost:9200/test/_analyze?analyzer=combo\u0026pretty=true' -d 'Ein schöner Tag in Köln'\n\t{\n\t  \"tokens\" : [ {\n\t    \"token\" : \"ein\",\n\t    \"start_offset\" : 0,\n\t    \"end_offset\" : 3,\n\t    \"type\" : \"\u003cALPHANUM\u003e\",\n\t    \"position\" : 1\n\t  }, {\n\t    \"token\" : \"ein\",\n\t    \"start_offset\" : 0,\n\t    \"end_offset\" : 3,\n\t    \"type\" : \"\u003cALPHANUM\u003e\",\n\t    \"position\" : 1\n\t  }, {\n\t    \"token\" : \"schöner\",\n\t    \"start_offset\" : 4,\n\t    \"end_offset\" : 11,\n\t    \"type\" : \"\u003cALPHANUM\u003e\",\n\t    \"position\" : 2\n\t  }, {\n\t    \"token\" : \"schon\",\n\t    \"start_offset\" : 4,\n\t    \"end_offset\" : 11,\n\t    \"type\" : \"\u003cALPHANUM\u003e\",\n\t    \"position\" : 2\n\t  }, {\n\t    \"token\" : \"tag\",\n\t    \"start_offset\" : 12,\n\t    \"end_offset\" : 15,\n\t    \"type\" : \"\u003cALPHANUM\u003e\",\n\t    \"position\" : 3\n\t  }, {\n\t    \"token\" : \"tag\",\n\t    \"start_offset\" : 12,\n\t    \"end_offset\" : 15,\n\t    \"type\" : \"\u003cALPHANUM\u003e\",\n\t    \"position\" : 3\n\t  }, {\n\t   \"token\" : \"in\",\n\t    \"start_offset\" : 16,\n\t    \"end_offset\" : 18,\n\t    \"type\" : \"\u003cALPHANUM\u003e\",\n\t    \"position\" : 4\n\t  }, {\n\t    \"token\" : \"köln\",\n\t    \"start_offset\" : 19,\n\t    \"end_offset\" : 23,\n\t    \"type\" : \"\u003cALPHANUM\u003e\",\n\t    \"position\" : 5\n\t  }, {\n\t    \"token\" : \"koln\",\n\t    \"start_offset\" : 19,\n\t    \"end_offset\" : 23,\n\t    \"type\" : \"\u003cALPHANUM\u003e\",\n\t    \"position\" : 5\n\t  } ]\n\t}\n\nSome tokens may repeat, the offsets and positions may have same values.\nThis should have minimal impact on indexing or relevance scoring. Note that it may even improve scoring, because of the reweighting implied by more matches on some words.\n\nBehind the scene\n----------------\n\nLucene TokenStream API is rather awkward right now, making it hard to run many analyzers over the input again and again.\nBehind the scenes, the `Reader` given to the `Analyzer` must be cloned, in order to feed all the sub-analyzers.\nHopefully it may get easier to get multiple `Reader` in upcoming Lucene versions.\n\nPitfalls\n--------\n\nThere have been reported problems with using multiple time the same `Analyzer` instance.\nYou can recognize such problem by noticing dropped tokens, and a final empty token.\nTest the analysis with a simple word that should neither be tokenized, nor stemmed nor removed,\nyou should see it in the output as many times as you have sub-analyzers,\nand you should not see any other tokens.\n\nWhile this can be detected if it happens inside the same `ComboAnalyzer`, some problem can still arise in an awkward environment\n(like cascading combo analyzers, sharing some `Analyzer` instances, or re-using an `Analyzer` while consuming the `ComboTokenStream`).\n\nThere are two possible fixes to remedy this problem:\n* Disable completely the `TokenStream` reuse. Calls `Analyzer.tokenStream()` instead of `Analyzer.reusableTokenStream()`.\n  Set the `tokenstream_reuse` property to `false` to use this solution.\n* Activate use of token caching. All gotten `TokenStream`s will be completely consumed and cached right after its creation.\n  This loads the information in memory, and may be unsuitable for very long documents.\n  Set the `tokenstream_caching` property to `true` to use this solution.\n\nBoth solution affect memory usage and garbage collection a bit, thus affecting performance a bit too.\nIf you experience no problem with your setup, keep the defaults!\nThere should be no particular reason to use both these options simultaneously.\n\nSee also\n--------\n\nhttps://github.com/elasticsearch/elasticsearch/issues/1128\n\nhttps://issues.apache.org/jira/browse/LUCENE-3392\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyakaz%2Felasticsearch-analysis-combo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyakaz%2Felasticsearch-analysis-combo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyakaz%2Felasticsearch-analysis-combo/lists"}