{"id":13759916,"url":"https://github.com/duydo/elasticsearch-analysis-vietnamese","last_synced_at":"2026-03-12T12:04:56.300Z","repository":{"id":15474136,"uuid":"18207593","full_name":"duydo/elasticsearch-analysis-vietnamese","owner":"duydo","description":" Vietnamese Analysis Plugin for Elasticsearch","archived":false,"fork":false,"pushed_at":"2023-12-07T02:54:54.000Z","size":4680,"stargazers_count":498,"open_issues_count":6,"forks_count":209,"subscribers_count":34,"default_branch":"master","last_synced_at":"2024-08-03T13:03:58.676Z","etag":null,"topics":["analysis-plugin","elasticsearch","java","vietnamese","vietnamese-analysis-plugin"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/duydo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2014-03-28T09:45:31.000Z","updated_at":"2024-08-01T03:17:20.000Z","dependencies_parsed_at":"2023-12-07T03:30:29.535Z","dependency_job_id":"accf8412-c062-49bb-b201-8ed05c5112da","html_url":"https://github.com/duydo/elasticsearch-analysis-vietnamese","commit_stats":null,"previous_names":[],"tags_count":25,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duydo%2Felasticsearch-analysis-vietnamese","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duydo%2Felasticsearch-analysis-vietnamese/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duydo%2Felasticsearch-analysis-vietnamese/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duydo%2Felasticsearch-analysis-vietnamese/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/duydo","download_url":"https://codeload.github.com/duydo/elasticsearch-analysis-vietnamese/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224949644,"owners_count":17397204,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analysis-plugin","elasticsearch","java","vietnamese","vietnamese-analysis-plugin"],"created_at":"2024-08-03T13:01:00.906Z","updated_at":"2025-12-15T22:47:41.675Z","avatar_url":"https://github.com/duydo.png","language":"Java","funding_links":[],"categories":["人工智能","Uncategorized"],"sub_categories":["Uncategorized"],"readme":"# Vietnamese Analysis Plugin for Elasticsearch\n\n[![Test](https://github.com/duydo/elasticsearch-analysis-vietnamese/actions/workflows/test.yml/badge.svg)](https://github.com/duydo/elasticsearch-analysis-vietnamese/actions/workflows/test.yml)\n\nVietnamese Analysis plugin integrates Vietnamese language analysis into Elasticsearch. It uses [C++ tokenizer for Vietnamese](https://github.com/coccoc/coccoc-tokenizer) library developed by\nCocCoc team for their Search Engine and Ads systems.\n\nThe plugin provides `vi_analyzer` analyzer, `vi_tokenizer` tokenizer and `vi_stop` stop filter. The `vi_analyzer` is composed of the `vi_tokenizer` tokenizer, `stop` and `lowercase` filter.\n\n## Example output\n\n```\nGET _analyze\n{\n  \"analyzer\": \"vi_analyzer\",\n  \"text\": \"Cộng hòa Xã hội chủ nghĩa Việt Nam\"\n}\n```\n\nThe above sentence would produce the following terms:\n```\n[\"cộng hòa\", \"xã hội\", \"chủ nghĩa\" ,\"việt nam\"]\n\n```\n\n## Configuration\n\nThe `vi_analyzer` analyzer accepts the following parameters:\n\n- `dict_path` The path to tokenizer dictionary on system. Defaults to `/usr/local/share/tokenizer/dicts`.\n- `keep_punctuation` Keep punctuation marks as tokens. Defaults to `false`.\n- `split_url` If it's enabled (`true`), a domain `duydo.me` is split into  `[\"duy\", \"do\", \"me\"]`.\n  If it's disabled (`false`) `duydo.me` is split into `[\"duydo\", \"me\"]`. Defaults to `false`.\n  \n- `stopwords` A pre-defined stop words list like `_vi_` or an array containing a list of stop words. Defaults to [stopwords.txt](src/main/resources/org/apache/lucene/analysis/vi/stopwords.txt) file. \n- `stopwords_path` The path to a file containing stop words.\n\n\n### Example configuration\nIn this example, we configure the `vi_analyzer` analyzer to keep punctuation marks and to use the custom list of stop words:\n\n```\nPUT my-vi-index-00001\n{\n  \"settings\": {\n    \"analysis\": {\n      \"analyzer\": {\n        \"my_vi_analyzer\": {\n          \"type\": \"vi_analyzer\",\n          \"keep_punctuation\": true,\n          \"stopwords\": [\"rất\", \"những\"]\n        }\n      }\n    }\n  }\n}\n\nGET my-vi-index-00001/_analyze\n{\n  \"analyzer\": \"my_vi_analyzer\",\n  \"text\": \"Công nghệ thông tin Việt Nam rất phát triển trong những năm gần đây.\"\n}\n```\n\nThe above example produces the following terms:\n```\n[\"công nghệ\", \"thông tin\", \"việt nam\", \"phát triển\", \"trong\", \"năm\", \"gần đây\", \".\"]\n\n```\n\nWe can also create a custom analyzer with the `vi_tokenizer`. In following example, we create `my_vi_analyzer` to produce\nboth diacritic and no diacritic tokens in lowercase:\n\n```\nPUT my-vi-index-00002\n{\n  \"settings\": {\n    \"analysis\": {\n      \"analyzer\": {\n        \"my_vi_analyzer\": {\n          \"tokenizer\": \"vi_tokenizer\",\n          \"filter\": [\n            \"lowercase\",\n            \"ascii_folding\"\n          ]\n        }\n      },\n      \"filter\": {\n        \"ascii_folding\": {\n          \"type\": \"asciifolding\",\n          \"preserve_original\": true\n        }\n      }\n    }\n  }\n}\n\nGET my-vi-index-00002/_analyze\n{\n  \"analyzer\": \"my_vi_analyzer\",\n  \"text\": \"Cộng hòa Xã hội chủ nghĩa Việt Nam\"\n}\n```\n\nThe above example produces the following terms:\n```\n[\"cong hoa\", \"cộng hòa\", \"xa hoi\", \"xã hội\", \"chu nghia\", \"chủ nghĩa\", \"viet nam\", \"việt nam\"]\n\n```\n\n## Use Docker\n\nMake sure you have installed both Docker \u0026 docker-compose\n\n### Build the image with Docker Compose\n\n```sh\n# Copy, edit ES version and password for user elastic in file .env. Default password: changeme\ncp .env.sample .env\ndocker compose build\ndocker compose up\n```\n### Verify\n```sh\ncurl -k http://elastic:changeme@localhost:9200/_analyze -H 'Content-Type: application/json' -d '\n{\n  \"analyzer\": \"vi_analyzer\",\n  \"text\": \"Cộng hòa Xã hội chủ nghĩa Việt Nam\"\n}'\n\n# Output\n{\"tokens\":[{\"token\":\"cộng hòa\",\"start_offset\":0,\"end_offset\":8,\"type\":\"\u003cWORD\u003e\",\"position\":0},{\"token\":\"xã hội\",\"start_offset\":9,\"end_offset\":15,\"type\":\"\u003cWORD\u003e\",\"position\":1},{\"token\":\"chủ nghĩa\",\"start_offset\":16,\"end_offset\":25,\"type\":\"\u003cWORD\u003e\",\"position\":2},{\"token\":\"việt nam\",\"start_offset\":26,\"end_offset\":34,\"type\":\"\u003cWORD\u003e\",\"position\":3}]}                                                                                     \n```\n\n## Build from Source\n### Step 1: Build C++ tokenizer for Vietnamese library\n```sh\ngit clone https://github.com/duydo/coccoc-tokenizer.git\ncd coccoc-tokenizer \u0026\u0026 mkdir build \u0026\u0026 cd build\ncmake -DBUILD_JAVA=1 ..\nmake install\n# Link the coccoc shared lib to /usr/lib\nsudo ln -sf /usr/local/lib/libcoccoc_tokenizer_jni.* /usr/lib/\n```\nBy default, the `make install` installs:\n- The lib commands `tokenizer`, `dict_compiler` and `vn_lang_tool` under `/usr/local/bin`\n- The dynamic lib `libcoccoc_tokenizer_jni.so` under `/usr/local/lib/`. The plugin uses this lib directly.\n- The dictionary files under `/usr/local/share/tokenizer/dicts`. The plugin uses this path for `dict_path` by default.\n\nVerify\n```sh\n/usr/local/bin/tokenizer \"Cộng hòa Xã hội chủ nghĩa Việt Nam\"\n# cộng hòa\txã hội\tchủ nghĩa\tviệt nam\n```\n\nRefer [the repo](https://github.com/duydo/coccoc-tokenizer) for more information to build the library.\n\n\n### Step 2: Build the plugin\n\nClone the plugin’s source code:\n\n```sh\ngit clone https://github.com/duydo/elasticsearch-analysis-vietnamese.git\n```\n\nOptionally, edit the `elasticsearch-analysis-vietnamese/pom.xml` to change the version of Elasticsearch (same as plugin version) you want to build the plugin with:\n\n```xml\n...\n\u003cversion\u003e8.7.0\u003c/version\u003e\n...\n ```\n\nBuild the plugin:\n```sh\ncd elasticsearch-analysis-vietnamese\nmvn package\n```\n\n### Step 3: Installation the plugin on Elasticsearch\n\n```sh\nbin/elasticsearch-plugin install file://target/releases/elasticsearch-analysis-vietnamese-8.7.0.zip\n```\n\n## Compatible Versions\nFrom v7.12.11, the plugin uses CocCoc C++ tokenizer instead of the VnTokenizer by Lê Hồng Phương,\nI don't maintain the plugin with the VnTokenizer anymore, if you want to continue developing with it, refer [the branch vntokenizer](https://github.com/duydo/elasticsearch-analysis-vietnamese/tree/vntokenizer).  \n\n| Vietnamese Analysis Plugin | Elasticsearch   |\n|----------------------------|-----------------|\n| master                     | 8.7.0           |\n| develop                    | 8.7.0           |\n| 8.7.0                      | 8.7.0           |\n| 8.4.0                      | 8.4.0 ~ 8.7.1   |\n| 8.0.0                      | 8.0.0 ~ 8.0.x   |\n| 7.16.1                     | 7.16 ~ 7.17.1   |\n| 7.12.1                     | 7.12.1 ~ 7.15.x |     \n| 7.3.1                      | 7.3.1           |   \n| 5.6.5                      | 5.6.5           |\n| 5.4.1                      | 5.4.1           |\n| 5.3.1                      | 5.3.1           |\n| 5.2.1                      | 5.2.1           |\n| 2.4.1                      | 2.4.1           |\n| 2.4.0                      | 2.4.0           |\n| 2.3.5                      | 2.3.5           |\n| 2.3.4                      | 2.3.4           |\n| 2.3.3                      | 2.3.3           |\n| 2.3.2                      | 2.3.2           |\n| 2.3.1                      | 2.3.1           |\n| 2.3.0                      | 2.3.0           |\n| 0.2.2                      | 2.2.0           |\n| 0.2.1.1                    | 2.1.1           |\n| 0.2.1                      | 2.1.0           |\n| 0.2                        | 2.0.0           |\n| 0.1.7                      | 1.7+            |\n| 0.1.6                      | 1.6+            |\n| 0.1.5                      | 1.5+            |\n| 0.1.1                      | 1.4+            |\n| 0.1                        | 1.3             |\n\n\n## Issues\n\nYou might get errors during starting Elasticsearch with the plugin\n\n**1. Error: java.lang.UnsatisfiedLinkError: no libcoccoc_tokenizer_jni in java.library.path ...** (reported in [102](https://github.com/duydo/elasticsearch-analysis-vietnamese/issues/102))\n\nIt happens because of your JVM cannot find the dynamic lib `libcoccoc_tokenizer_jni` in `java.library.path`, try to resolve by doing one of following options:\n- Appending `/usr/local/lib` into environment variable  `LD_LIBRARY_PATH`:\n```sh\nexport LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH\n```\n- Making a symbolic link or copying the file `/usr/local/lib/libcoccoc_tokenizer_jni.so` to `/usr/lib` :\n```sh\n# Make link\nln -sf /usr/local/lib/libcoccoc_tokenizer_jni.so /usr/lib/libcoccoc_tokenizer_jni.so\n\n# Copy \ncp /usr/local/lib/libcoccoc_tokenizer_jni.so /usr/lib\n```\n\n**2. Error: Cannot initialize Tokenizer: /usr/local/share/tokenizer/dicts** (reported in [106](https://github.com/duydo/elasticsearch-analysis-vietnamese/issues/106))\n\nIt happens because of the tokenizer cannot find the dictionary files under `/usr/local/share/tokenizer/dicts`. \nEnsure the path `/usr/local/share/tokenizer/dicts` existed and includes those files: alphabetic, i_and_y.txt, nontone_pair_freq_map.dump, syllable_trie.dump\nd_and_gi.txt, multiterm_trie.dump, numeric. If not, try to build the C++ tokenizer (Step 1) again.\n\n\n## Thanks to\n- [JetBrains](https://www.jetbrains.com) has provided a free license for [IntelliJ IDEA](https://www.jetbrains.com/idea).\n- [CocCoc team](https://coccoc.com) has provided their C++ Vietnamese tokenizer library as open source.\n\n## License\n    \n    This software is licensed under the Apache 2 license, quoted below.\n\n    Copyright by Duy Do\n\n    Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n    use this file except in compliance with the License. You may obtain a copy of\n    the License at\n\n        http://www.apache.org/licenses/LICENSE-2.0\n\n    Unless required by applicable law or agreed to in writing, software\n    distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT\n    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the\n    License for the specific language governing permissions and limitations under\n    the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fduydo%2Felasticsearch-analysis-vietnamese","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fduydo%2Felasticsearch-analysis-vietnamese","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fduydo%2Felasticsearch-analysis-vietnamese/lists"}