{"id":21994413,"url":"https://github.com/codelibs/opensearch-analysis-extension","last_synced_at":"2026-05-06T05:39:42.841Z","repository":{"id":44418370,"uuid":"422364155","full_name":"codelibs/opensearch-analysis-extension","owner":"codelibs","description":null,"archived":false,"fork":false,"pushed_at":"2024-08-09T00:26:16.000Z","size":85,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-08-10T01:39:07.971Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/codelibs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-28T21:45:58.000Z","updated_at":"2024-08-09T00:26:17.000Z","dependencies_parsed_at":"2023-01-29T01:31:34.025Z","dependency_job_id":"1e2d7ff4-a4f2-4ef7-9e5e-27adad1a00e5","html_url":"https://github.com/codelibs/opensearch-analysis-extension","commit_stats":null,"previous_names":[],"tags_count":24,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codelibs%2Fopensearch-analysis-extension","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codelibs%2Fopensearch-analysis-extension/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codelibs%2Fopensearch-analysis-extension/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codelibs%2Fopensearch-analysis-extension/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/codelibs","download_url":"https://codeload.github.com/codelibs/opensearch-analysis-extension/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245052673,"owners_count":20553172,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-29T21:08:50.999Z","updated_at":"2026-05-06T05:39:42.831Z","avatar_url":"https://github.com/codelibs.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OpenSearch Analysis Extension\n\n[![Java CI with Maven](https://github.com/codelibs/opensearch-analysis-extension/actions/workflows/maven.yml/badge.svg)](https://github.com/codelibs/opensearch-analysis-extension/actions/workflows/maven.yml)\n[![Maven Central](https://img.shields.io/maven-central/v/org.codelibs.opensearch/opensearch-analysis-extension)](https://repo1.maven.org/maven2/org/codelibs/opensearch/opensearch-analysis-extension/)\n[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)\n\nA comprehensive text analysis plugin for OpenSearch that provides advanced tokenizers, character filters, and token filters with specialized support for Japanese text processing using Kuromoji analyzer.\n\n## Features\n\n- **Japanese Text Analysis**: Comprehensive Kuromoji-based tokenization and morphological analysis\n- **Character Normalization**: Iteration mark and prolonged sound mark character filters\n- **Advanced Token Filtering**: Number processing, character type filtering, concatenation filters\n- **Reloadable Components**: Dynamic dictionary reloading capabilities\n- **Extensible Architecture**: Easy integration of custom analysis components\n\n## Tech Stack\n\n- **Java**: 21\n- **OpenSearch**: 3.2.0\n- **Lucene**: 10.2.2\n- **Build System**: Maven 3.x\n- **Testing**: JUnit 4.13.2\n\n## Quick Start\n\n### Prerequisites\n\n- OpenSearch 3.2.0 or later\n- Java 21 or later (for building from source)\n\n### Installation\n\n#### From Maven Repository\n```bash\n$OPENSEARCH_HOME/bin/opensearch-plugin install org.codelibs:opensearch-analysis-extension:3.2.0\n```\n\n#### From Local Build\n```bash\n# Clone and build\ngit clone https://github.com/codelibs/opensearch-analysis-extension.git\ncd opensearch-analysis-extension\nmvn package\n\n# Install plugin\n$OPENSEARCH_HOME/bin/opensearch-plugin install file:target/releases/opensearch-analysis-extension-3.2.0.zip\n```\n\n### Basic Usage Example\n\nCreate an index with Japanese text analysis:\n\n```bash\ncurl -XPUT 'http://localhost:9200/sample/' -H 'Content-Type: application/json' -d'\n{\n    \"settings\": {\n        \"index\": {\n            \"analysis\": {\n                \"analyzer\": {\n                    \"japanese_analyzer\": {\n                        \"type\": \"custom\",\n                        \"tokenizer\": \"japanese_tokenizer\",\n                        \"char_filter\": [\"iteration_mark\", \"prolonged_sound_mark\"],\n                        \"filter\": [\"japanese_baseform\", \"kanji_number\", \"char_type\"]\n                    }\n                }\n            }\n        }\n    }\n}'\n```\n\nTest the analyzer:\n\n```bash\ncurl -XGET 'http://localhost:9200/sample/_analyze' -H 'Content-Type: application/json' -d'\n{\n    \"analyzer\": \"japanese_analyzer\",\n    \"text\": \"東京都港区にある会社\"\n}'\n```\n\n## Development\n\n### Building from Source\n\n```bash\n# Clone the repository\ngit clone https://github.com/codelibs/opensearch-analysis-extension.git\ncd opensearch-analysis-extension\n\n# Build the plugin\nmvn clean package\n\n# Run tests\nmvn test\n\n# Skip tests during build\nmvn package -DskipTests=true\n```\n\n### Project Structure\n\n```\nsrc/\n├── main/\n│   ├── java/org/codelibs/opensearch/extension/\n│   │   ├── ExtensionPlugin.java              # Main plugin class\n│   │   ├── analysis/                         # General analysis components\n│   │   │   ├── CharTypeFilterFactory.java\n│   │   │   ├── KanjiNumberFilterFactory.java\n│   │   │   └── ...\n│   │   └── kuromoji/                        # Japanese-specific components\n│   │       └── index/analysis/\n│   │           ├── KuromojiTokenizerFactory.java\n│   │           └── ...\n│   ├── assemblies/plugin.xml                 # Plugin assembly configuration\n│   └── plugin-metadata/\n│       ├── plugin-descriptor.properties      # Plugin metadata\n│       └── plugin-security.policy           # Security policy\n└── test/                                     # Unit and integration tests\n```\n\n### Contributing\n\n1. Fork the repository\n2. Create a feature branch (`git checkout -b feature/amazing-feature`)\n3. Make your changes following the existing code style\n4. Add tests for new functionality\n5. Run the test suite (`mvn test`)\n6. Commit your changes (`git commit -m 'Add amazing feature'`)\n7. Push to the branch (`git push origin feature/amazing-feature`)\n8. Open a Pull Request\n\n## Analysis Components\n\n### Character Filters\n\n#### IterationMarkCharFilter (`iteration_mark`)\nNormalizes iteration mark characters. For example, converts \"学問のすゝめ\" to \"学問のすすめ\".\n\n```json\n{\n  \"char_filter\": [\"iteration_mark\"]\n}\n```\n\n#### ProlongedSoundMarkCharFilter (`prolonged_sound_mark`)\nReplaces various prolonged sound mark characters with `\\u30fc` (KATAKANA-HIRAGANA SOUND MARK).\n\n**Supported Unicode Characters:**\n| Unicode | Name |\n|:-----:|:-----|\n| U002D | HYPHEN-MINUS |\n| UFF0D | FULLWIDTH HYPHEN-MINUS |\n| U2010 | HYPHEN |\n| U2011 | NON-BREAKING HYPHEN |\n| U2012 | FIGURE DASH |\n| U2013 | EN DASH |\n| U2014 | EM DASH |\n| U2015 | HORIZONTAL BAR |\n| U207B | SUPERSCRIPT MINUS |\n| U208B | SUBSCRIPT MINUS |\n| U30FC | KATAKANA-HIRAGANA SOUND MARK |\n\n```json\n{\n  \"char_filter\": [\"prolonged_sound_mark\"]\n}\n```\n\n#### JapaneseIterationMarkCharFilter (`japanese_iteration_mark`)\nKuromoji-specific iteration mark character filter for Japanese text processing.\n\n### Token Filters\n\n#### KanjiNumberFilter (`kanji_number`)\nConverts Kanji number characters (e.g., \"一\") to Arabic numerals (e.g., \"1\").\n\n```json\n{\n  \"filter\": [\"kanji_number\"]\n}\n```\n\n#### CharTypeFilter (`char_type`)\nKeeps tokens based on character type: alphabetic, digit, or letter.\n\n```json\n{\n  \"filter\": {\n    \"my_char_type\": {\n      \"type\": \"char_type\",\n      \"digit\": false,\n      \"alphabetic\": true,\n      \"letter\": true\n    }\n  }\n}\n```\n\n**Character Type Behavior:**\n| Token | Default | digit:false | letter:false |\n|:------|:-------:|:-----------:|:------------:|\n| abc | keep | keep | keep |\n| ab1 | keep | keep | keep |\n| abあ | keep | keep | keep |\n| 123 | keep | remove | keep |\n| 12あ | keep | keep | keep |\n| あいう | keep | keep | remove |\n| #-= | remove | remove | remove |\n\n#### NumberConcatenationFilter (`number_concat`)\nConcatenates tokens with following numbers. Example: \"10\" + \"years\" → \"10years\".\n\n```json\n{\n  \"filter\": {\n    \"numconcat_filter\": {\n      \"type\": \"number_concat\",\n      \"suffix_words_path\": \"suffix.txt\"\n    }\n  }\n}\n```\n\n#### PatternConcatenationFilter (`pattern_concat`)\nConcatenates tokens matching specified patterns.\n\n```json\n{\n  \"filter\": {\n    \"pattern_filter\": {\n      \"type\": \"pattern_concat\",\n      \"pattern1\": \"[0-9]+\",\n      \"pattern2\": \"year(s)?\"\n    }\n  }\n}\n```\n\n#### Additional Token Filters\n\n- **japanese_baseform**: Converts to base forms\n- **japanese_part_of_speech**: Part-of-speech filtering\n- **japanese_readingform**: Reading form conversion\n- **japanese_stemmer**: Japanese stemming\n- **japanese_stop**: Japanese stop word removal\n- **japanese_number**: Japanese number processing\n- **japanese_completion**: Completion suggestions\n- **stop_prefix/stop_suffix**: Prefix/suffix stop word filters\n- **reloadable_keyword_marker**: Dynamic keyword marking\n- **reloadable_stop**: Dynamic stop word filtering\n- **flexible_porter_stem**: Flexible Porter stemming\n- **alphanum_word**: Alphanumeric word processing\n\n### Tokenizers\n\n#### JapaneseTokenizer (`japanese_tokenizer`)\nAdvanced Japanese tokenization using Kuromoji morphological analyzer.\n\n```json\n{\n  \"tokenizer\": {\n    \"my_tokenizer\": {\n      \"type\": \"japanese_tokenizer\",\n      \"mode\": \"extended\",\n      \"discard_punctuation\": false,\n      \"user_dictionary\": \"userdict_ja.txt\"\n    }\n  }\n}\n```\n\n#### ReloadableKuromojiTokenizer (`reloadable_kuromoji`)\nDynamically reloads user dictionary files when updated.\n\n```json\n{\n  \"tokenizer\": {\n    \"reloadable_tokenizer\": {\n      \"type\": \"reloadable_kuromoji\",\n      \"mode\": \"extended\",\n      \"discard_punctuation\": false,\n      \"user_dictionary\": \"userdict_ja.txt\"\n    }\n  }\n}\n```\n\n**Note**: Dictionary updates may affect search results due to term changes.\n\n#### NGramSynonymTokenizer (`ngram_synonym`)\nN-gram tokenization with synonym support.\n\n## Configuration Examples\n\n### Complete Japanese Analysis Setup\n\n```json\n{\n  \"settings\": {\n    \"index\": {\n      \"analysis\": {\n        \"char_filter\": {\n          \"japanese_normalize\": {\n            \"type\": \"mapping\",\n            \"mappings\": [\"iteration_mark\", \"prolonged_sound_mark\"]\n          }\n        },\n        \"tokenizer\": {\n          \"japanese_custom\": {\n            \"type\": \"japanese_tokenizer\",\n            \"mode\": \"extended\",\n            \"user_dictionary\": \"custom_dict.txt\"\n          }\n        },\n        \"filter\": {\n          \"japanese_filters\": {\n            \"type\": \"japanese_baseform\"\n          },\n          \"number_normalize\": {\n            \"type\": \"kanji_number\"\n          },\n          \"char_cleanup\": {\n            \"type\": \"char_type\",\n            \"digit\": true,\n            \"alphabetic\": true,\n            \"letter\": true\n          }\n        },\n        \"analyzer\": {\n          \"japanese_full\": {\n            \"type\": \"custom\",\n            \"char_filter\": [\"iteration_mark\", \"prolonged_sound_mark\"],\n            \"tokenizer\": \"japanese_custom\",\n            \"filter\": [\n              \"japanese_baseform\",\n              \"japanese_part_of_speech\",\n              \"kanji_number\",\n              \"char_type\",\n              \"lowercase\"\n            ]\n          }\n        }\n      }\n    }\n  }\n}\n```\n\n### Multi-Language Analysis\n\n```json\n{\n  \"settings\": {\n    \"analysis\": {\n      \"analyzer\": {\n        \"multilang\": {\n          \"type\": \"custom\",\n          \"tokenizer\": \"standard\",\n          \"char_filter\": [\"prolonged_sound_mark\"],\n          \"filter\": [\n            \"lowercase\",\n            \"char_type\",\n            \"flexible_porter_stem\"\n          ]\n        }\n      }\n    }\n  }\n}\n```\n\n## Testing\n\n```bash\n# Run all tests\nmvn test\n\n# Run specific test class\nmvn test -Dtest=ExtensionPluginTest\n\n# Run tests with verbose output\nmvn test -X\n\n# Generate test coverage report\nmvn jacoco:report\n```\n\n## Version Compatibility\n\n| Plugin Version | OpenSearch Version | Lucene Version | Java Version |\n|:---------------|:-------------------|:---------------|:-------------|\n| 3.2.x | 3.2.0+ | 10.2.2+ | 21+ |\n| 3.1.x | 3.1.0+ | 10.1.x+ | 21+ |\n\n## Performance Considerations\n\n- **Dictionary Size**: Larger user dictionaries impact tokenization performance\n- **Filter Chain**: Minimize filter chain length for better performance  \n- **Reloadable Components**: Use sparingly in high-throughput environments\n- **Memory Usage**: Monitor heap usage with large dictionary files\n\n## Troubleshooting\n\n### Common Issues\n\n**Plugin Installation Fails**\n```bash\n# Check OpenSearch version compatibility\n$OPENSEARCH_HOME/bin/opensearch --version\n\n# Verify plugin version matches OpenSearch version\n$OPENSEARCH_HOME/bin/opensearch-plugin list\n```\n\n**Dictionary Not Loading**\n```bash\n# Check dictionary file permissions and encoding (UTF-8)\n# Verify path in OpenSearch configuration directory\n# Check OpenSearch logs for error messages\n```\n\n**Analysis Not Working**\n```bash\n# Test analyzer configuration\ncurl -XGET 'localhost:9200/_analyze' -d '{\"analyzer\":\"your_analyzer\",\"text\":\"test text\"}'\n\n# Check plugin registration\ncurl -XGET 'localhost:9200/_nodes/plugins'\n```\n\n### Debug Mode\n\nEnable debug logging in `opensearch.yml`:\n\n```yaml\nlogger.org.codelibs.opensearch.extension: DEBUG\n```\n\n## License\n\nLicensed under the Apache License, Version 2.0. See [LICENSE](LICENSE) for details.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodelibs%2Fopensearch-analysis-extension","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcodelibs%2Fopensearch-analysis-extension","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodelibs%2Fopensearch-analysis-extension/lists"}