Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

https://github.com/duydo/elasticsearch-analysis-vietnamese

Vietnamese Analysis Plugin for Elasticsearch
https://github.com/duydo/elasticsearch-analysis-vietnamese

analysis-plugin elasticsearch java vietnamese vietnamese-analysis-plugin

Last synced: 12 days ago
JSON representation

Vietnamese Analysis Plugin for Elasticsearch

Host: GitHub
URL: https://github.com/duydo/elasticsearch-analysis-vietnamese
Owner: duydo
License: apache-2.0
Created: 2014-03-28T09:45:31.000Z (over 10 years ago)
Default Branch: master
Last Pushed: 2023-12-07T02:54:54.000Z (7 months ago)
Last Synced: 2024-06-20T16:47:32.366Z (16 days ago)
Topics: analysis-plugin, elasticsearch, java, vietnamese, vietnamese-analysis-plugin
Language: Java
Homepage:
Size: 4.46 MB
Stars: 496
Watchers: 34
Forks: 205
Open Issues: 5
Metadata Files:
- Readme: README.md
- License: LICENSE.txt

Lists

awesome-vietnamese-language - ElasticSearch Analysis Vietnamese

README

# Vietnamese Analysis Plugin for Elasticsearch

[![Test](https://github.com/duydo/elasticsearch-analysis-vietnamese/actions/workflows/test.yml/badge.svg)](https://github.com/duydo/elasticsearch-analysis-vietnamese/actions/workflows/test.yml)

Vietnamese Analysis plugin integrates Vietnamese language analysis into Elasticsearch. It uses [C++ tokenizer for Vietnamese](https://github.com/coccoc/coccoc-tokenizer) library developed by
CocCoc team for their Search Engine and Ads systems.

The plugin provides `vi_analyzer` analyzer, `vi_tokenizer` tokenizer and `vi_stop` stop filter. The `vi_analyzer` is composed of the `vi_tokenizer` tokenizer, `stop` and `lowercase` filter.

## Example output

```
GET _analyze
{
"analyzer": "vi_analyzer",
"text": "Cộng hòa Xã hội chủ nghĩa Việt Nam"
}
```

The above sentence would produce the following terms:
```
["cộng hòa", "xã hội", "chủ nghĩa" ,"việt nam"]

```

## Configuration

The `vi_analyzer` analyzer accepts the following parameters:

- `dict_path` The path to tokenizer dictionary on system. Defaults to `/usr/local/share/tokenizer/dicts`.
- `keep_punctuation` Keep punctuation marks as tokens. Defaults to `false`.
- `split_url` If it's enabled (`true`), a domain `duydo.me` is split into `["duy", "do", "me"]`.
If it's disabled (`false`) `duydo.me` is split into `["duydo", "me"]`. Defaults to `false`.

- `stopwords` A pre-defined stop words list like `_vi_` or an array containing a list of stop words. Defaults to [stopwords.txt](src/main/resources/org/apache/lucene/analysis/vi/stopwords.txt) file.
- `stopwords_path` The path to a file containing stop words.

### Example configuration
In this example, we configure the `vi_analyzer` analyzer to keep punctuation marks and to use the custom list of stop words:

```
PUT my-vi-index-00001
{
"settings": {
"analysis": {
"analyzer": {
"my_vi_analyzer": {
"type": "vi_analyzer",
"keep_punctuation": true,
"stopwords": ["rất", "những"]
}
}
}
}
}

GET my-vi-index-00001/_analyze
{
"analyzer": "my_vi_analyzer",
"text": "Công nghệ thông tin Việt Nam rất phát triển trong những năm gần đây."
}
```

The above example produces the following terms:
```
["công nghệ", "thông tin", "việt nam", "phát triển", "trong", "năm", "gần đây", "."]

```

We can also create a custom analyzer with the `vi_tokenizer`. In following example, we create `my_vi_analyzer` to produce
both diacritic and no diacritic tokens in lowercase:

```
PUT my-vi-index-00002
{
"settings": {
"analysis": {
"analyzer": {
"my_vi_analyzer": {
"tokenizer": "vi_tokenizer",
"filter": [
"lowercase",
"ascii_folding"
]
}
},
"filter": {
"ascii_folding": {
"type": "asciifolding",
"preserve_original": true
}
}
}
}
}

GET my-vi-index-00002/_analyze
{
"analyzer": "my_vi_analyzer",
"text": "Cộng hòa Xã hội chủ nghĩa Việt Nam"
}
```

The above example produces the following terms:
```
["cong hoa", "cộng hòa", "xa hoi", "xã hội", "chu nghia", "chủ nghĩa", "viet nam", "việt nam"]

```

## Use Docker

Make sure you have installed both Docker & docker-compose

### Build the image with Docker Compose

```sh
# Copy, edit ES version and password for user elastic in file .env. Default password: changeme
cp .env.sample .env
docker compose build
docker compose up
```
### Verify
```sh
curl -k http://elastic:changeme@localhost:9200/_analyze -H 'Content-Type: application/json' -d '
{
"analyzer": "vi_analyzer",
"text": "Cộng hòa Xã hội chủ nghĩa Việt Nam"
}'

# Output
{"tokens":[{"token":"cộng hòa","start_offset":0,"end_offset":8,"type":"","position":0},{"token":"xã hội","start_offset":9,"end_offset":15,"type":"","position":1},{"token":"chủ nghĩa","start_offset":16,"end_offset":25,"type":"","position":2},{"token":"việt nam","start_offset":26,"end_offset":34,"type":"","position":3}]}
```

## Build from Source
### Step 1: Build C++ tokenizer for Vietnamese library
```sh
git clone https://github.com/duydo/coccoc-tokenizer.git
cd coccoc-tokenizer && mkdir build && cd build
cmake -DBUILD_JAVA=1 ..
make install
# Link the coccoc shared lib to /usr/lib
sudo ln -sf /usr/local/lib/libcoccoc_tokenizer_jni.* /usr/lib/
```
By default, the `make install` installs:
- The lib commands `tokenizer`, `dict_compiler` and `vn_lang_tool` under `/usr/local/bin`
- The dynamic lib `libcoccoc_tokenizer_jni.so` under `/usr/local/lib/`. The plugin uses this lib directly.
- The dictionary files under `/usr/local/share/tokenizer/dicts`. The plugin uses this path for `dict_path` by default.

Verify
```sh
/usr/local/bin/tokenizer "Cộng hòa Xã hội chủ nghĩa Việt Nam"
# cộng hòa xã hội chủ nghĩa việt nam
```

Refer [the repo](https://github.com/duydo/coccoc-tokenizer) for more information to build the library.

### Step 2: Build the plugin

Clone the plugin’s source code:

```sh
git clone https://github.com/duydo/elasticsearch-analysis-vietnamese.git
```

Optionally, edit the `elasticsearch-analysis-vietnamese/pom.xml` to change the version of Elasticsearch (same as plugin version) you want to build the plugin with:

```xml
...
8.7.0
...
```

Build the plugin:
```sh
cd elasticsearch-analysis-vietnamese
mvn package
```

### Step 3: Installation the plugin on Elasticsearch

```sh
bin/elasticsearch-plugin install file://target/releases/elasticsearch-analysis-vietnamese-8.7.0.zip
```

## Compatible Versions
From v7.12.11, the plugin uses CocCoc C++ tokenizer instead of the VnTokenizer by Lê Hồng Phương,
I don't maintain the plugin with the VnTokenizer anymore, if you want to continue developing with it, refer [the branch vntokenizer](https://github.com/duydo/elasticsearch-analysis-vietnamese/tree/vntokenizer).

| Vietnamese Analysis Plugin | Elasticsearch |
|----------------------------|-----------------|
| master | 8.7.0 |
| develop | 8.7.0 |
| 8.7.0 | 8.7.0 |
| 8.4.0 | 8.4.0 ~ 8.7.1 |
| 8.0.0 | 8.0.0 ~ 8.0.x |
| 7.16.1 | 7.16 ~ 7.17.1 |
| 7.12.1 | 7.12.1 ~ 7.15.x |
| 7.3.1 | 7.3.1 |
| 5.6.5 | 5.6.5 |
| 5.4.1 | 5.4.1 |
| 5.3.1 | 5.3.1 |
| 5.2.1 | 5.2.1 |
| 2.4.1 | 2.4.1 |
| 2.4.0 | 2.4.0 |
| 2.3.5 | 2.3.5 |
| 2.3.4 | 2.3.4 |
| 2.3.3 | 2.3.3 |
| 2.3.2 | 2.3.2 |
| 2.3.1 | 2.3.1 |
| 2.3.0 | 2.3.0 |
| 0.2.2 | 2.2.0 |
| 0.2.1.1 | 2.1.1 |
| 0.2.1 | 2.1.0 |
| 0.2 | 2.0.0 |
| 0.1.7 | 1.7+ |
| 0.1.6 | 1.6+ |
| 0.1.5 | 1.5+ |
| 0.1.1 | 1.4+ |
| 0.1 | 1.3 |

## Issues

You might get errors during starting Elasticsearch with the plugin

**1. Error: java.lang.UnsatisfiedLinkError: no libcoccoc_tokenizer_jni in java.library.path ...** (reported in [102](https://github.com/duydo/elasticsearch-analysis-vietnamese/issues/102))

It happens because of your JVM cannot find the dynamic lib `libcoccoc_tokenizer_jni` in `java.library.path`, try to resolve by doing one of following options:
- Appending `/usr/local/lib` into environment variable `LD_LIBRARY_PATH`:
```sh
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
```
- Making a symbolic link or copying the file `/usr/local/lib/libcoccoc_tokenizer_jni.so` to `/usr/lib` :
```sh
# Make link
ln -sf /usr/local/lib/libcoccoc_tokenizer_jni.so /usr/lib/libcoccoc_tokenizer_jni.so

# Copy
cp /usr/local/lib/libcoccoc_tokenizer_jni.so /usr/lib
```

**2. Error: Cannot initialize Tokenizer: /usr/local/share/tokenizer/dicts** (reported in [106](https://github.com/duydo/elasticsearch-analysis-vietnamese/issues/106))

It happens because of the tokenizer cannot find the dictionary files under `/usr/local/share/tokenizer/dicts`.
Ensure the path `/usr/local/share/tokenizer/dicts` existed and includes those files: alphabetic, i_and_y.txt, nontone_pair_freq_map.dump, syllable_trie.dump
d_and_gi.txt, multiterm_trie.dump, numeric. If not, try to build the C++ tokenizer (Step 1) again.

## Thanks to
- [JetBrains](https://www.jetbrains.com) has provided a free license for [IntelliJ IDEA](https://www.jetbrains.com/idea).
- [CocCoc team](https://coccoc.com) has provided their C++ Vietnamese tokenizer library as open source.

## License

This software is licensed under the Apache 2 license, quoted below.

Licensed under the Apache License, Version 2.0 (the "License"); you may not
use this file except in compliance with the License. You may obtain a copy of
the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations under
the License.