{"id":18398734,"url":"https://github.com/mongodb-developer/lucene-search-analysis","last_synced_at":"2025-04-07T05:33:37.262Z","repository":{"id":53255763,"uuid":"299773542","full_name":"mongodb-developer/lucene-search-analysis","owner":"mongodb-developer","description":null,"archived":false,"fork":false,"pushed_at":"2022-08-10T19:36:43.000Z","size":563,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":135,"default_branch":"master","last_synced_at":"2024-04-14T11:12:35.116Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"unlicense","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mongodb-developer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-09-30T00:53:00.000Z","updated_at":"2024-01-19T05:26:31.000Z","dependencies_parsed_at":"2022-08-27T07:02:38.017Z","dependency_job_id":null,"html_url":"https://github.com/mongodb-developer/lucene-search-analysis","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongodb-developer%2Flucene-search-analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongodb-developer%2Flucene-search-analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongodb-developer%2Flucene-search-analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongodb-developer%2Flucene-search-analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mongodb-developer","download_url":"https://codeload.github.com/mongodb-developer/lucene-search-analysis/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247601378,"owners_count":20964861,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T02:24:06.826Z","updated_at":"2025-04-07T05:33:35.694Z","avatar_url":"https://github.com/mongodb-developer.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Notice: Repository Deprecation\nThis repository is deprecated and no longer actively maintained. It contains outdated code examples or practices that do not align with current MongoDB best practices. While the repository remains accessible for reference purposes, we strongly discourage its use in production environments.\nUsers should be aware that this repository will not receive any further updates, bug fixes, or security patches. This code may expose you to security vulnerabilities, compatibility issues with current MongoDB versions, and potential performance problems. Any implementation based on this repository is at the user's own risk.\nFor up-to-date resources, please refer to the [MongoDB Developer Center](https://mongodb.com/developer).\n\n\n# Atlas/Lucene Search Analysis\n\n## Introduction\n\nAtlas Search uses [Lucene Analyzers](https://docs.atlas.mongodb.com/reference/atlas-search/analyzers/) to control how the index sets search terms, e.g., where to break up word groupings and whether to consider punctuation. However, without an intimate knowledge of the various Lucene Analyzers, it can be difficult to select the appropriate analyzer for a given field when creating a search index. Inspired by the [Analysis Screen](https://lucene.apache.org/solr/guide/8_6/analysis-screen.html) in Apache Solr, this utility provides two simple ways -- a CLI and a UI -- to test how various analyzers will process a given text.\n\n**Note:** see this [Atlas feature request](https://feedback.mongodb.com/forums/924868-atlas-search/suggestions/41501065-analzye-endpoint-or-analysis-screen)\n\n## Build\n\nThe UI for this tool is implemented as a [Vaadin](https://vaadin.com/) web application. The CLI is implemented as a simple POJO. To build these tools, execute the command `mvn clean package` from the directory containing `pom.xml`.\n\n## Run\n\nUse the command `mvnw -Dspring-boot.run.jvmArguments=\"-Dspring.devtools.restart.enabled=false\"` from the directory containing `pom.xml` to launch the Web UI on port 8080.\n\n![Web UI](src/main/resources/vaadin_app.png)\n\nUse the command `mvn -f pom-cli.xml exec:java -Dexec.args=\"\u003coptions\u003e\"` with the appropriate options to run the CLI. Use the `-h` or `--help` options to see usage instructions.\n\n```bash\nusage: mvn exec:java -Dexec.args=\"\u003coptions\u003e\"\n -h, --help               Prints this message\n -a, --analyzer \u003carg\u003e     Lucene analyzer to use (defaults to 'Standard').\n                         Use 'list' for supported analyzer names.\n -d, --definition \u003carg\u003e   Index definition file containing custom analyzer\n -t, --text \u003carg\u003e         Input text to analyze\n -f, --file \u003carg\u003e         Input text file to analyze\n -l, --language \u003carg\u003e     Language code (used with '--analyzer=Language'\n                         only.  Use 'list' for supported language codes.\n -n, --name \u003carg\u003e         Custom analyzer name\n -o, --operator \u003carg\u003e     Query operator to use (defaults to 'text'). Use\n                         'list' for supported operator names.\n -k, --tokenizer \u003carg\u003e    Tokeniser to use with autocomplete operator\n                         (defaults to 'edgeGram'). Use 'list' for\n                         supported tokenizer names.\n -m, --minGrams \u003carg\u003e     Minimum number of characters per indexed sequence\n                         to use with autocomplete operator (defaults to\n                         '2').\n -x, --maxGrams \u003carg\u003e     Maximum number of characters per indexed sequence\n                         to use with autocomplete operator (defaults to\n                         '3').\n```\n\nYou can also use `java -cp lib/ -jar \u003cpath-to\u003e/atlas-search-analysis-0.0.1.jar \u003coptions\u003e` (Java 11 or later) to run the CLI.\n\n## Examples\n\n### Analyze text using the `lucene.simple` analyzer\n\n```bash\nmvn -f pom-cli.xml -q exec:java -Dexec.args=\"-a simple -t 'hello my-name.is Roy/Kiesler'\"\nUsing org.apache.lucene.analysis.core.SimpleAnalyzer\n[hello] [my] [name] [is] [roy] [kiesler]\n```\n\n### Analyze text using the `lucene.standard` analyzer\n\n```bash\nmvn -f pom-cli.xml -q exec:java -Dexec.args=\"--analyzer standard --text 'hello my-name.is Roy/Kiesler'\"\nUsing org.apache.lucene.analysis.standard.StandardAnalyzer\n[hello] [my] [name.is] [roy] [kiesler]\n```\n\n### Analyze text using the `lucene.whitespace` analyzer\n\n```bash\nmvn -f pom-cli.xml -q exec:java -Dexec.args=\"--analyzer whitespace -t 'hello my-name.is Roy/Kiesler'\"\nUsing org.apache.lucene.analysis.core.WhitespaceAnalyzer\n[hello] [my-name.is] [Roy/Kiesler]\n```\n\n### Analyze text using the `lucene.language` English analyzer\n\n```bash\nmvn -f pom-cli.xml -q exec:java -Dexec.args=\"--analyzer language --language en --text 'running a race'\"\nUsing org.apache.lucene.analysis.en.EnglishAnalyzer\n[run] [race]\n```\n\n### Analyze a text file using the `lucene.language` French analyzer\n\n```bash\ncat \u003c\u003cEOF \u003e\u003e french.txt\nbonjour je m'appelle Roy Kiesler\nEOF\n\nmvn -q exec:java -Dexec.args=\"-a language -l fr -f french.txt\"\nUsing org.apache.lucene.analysis.fr.FrenchAnalyzer\n[bonjou] [apel] [roy] [kiesl]\n```\n\n### Analyze text using a custom analyzer\n\n**Sample 1**\n\n```bash\nmvn -f pom-cli.xml -q exec:java -Dexec.args=\"-a custom -t 'ROCKY II is better than Rocky V' -d index_roman.json -n romanAnalyzer\"\nUsing org.apache.lucene.analysis.custom.CustomAnalyzer\n[rocki] [2] [better] [rocki] [5]\n```\n\n**Sample 2**\n\n```bash\nmvn -f pom-cli.xml -q exec:java -Dexec.args=\"-a custom -t '\u003cdiv\u003e\u003cp\u003eThis is an \u003ca href=\"foo.com\"\u003eHTML\u003c/a\u003e test\u003c/p\u003e\u003c/div\u003e' -d index_html.json -n htmlStrippingAnalyzer\"\nUsing org.apache.lucene.analysis.custom.CustomAnalyzer\n[p] [This] [is] [an] [a] [href] [foo.com] [HTML] [a] [test] [p]\n```\n\n### Analyze text using autocomplete\n\n**Sample 1**\n\n```bash\nmvn -f pom-cli.xml exec:java -Dexec.args=\"-t 'Ribeira Charming Duplex' -o autocomplete\"\nUsing org.apache.lucene.analysis.custom.CustomAnalyzer\nAutocomplete - nGram, minGram(2), maxGram(3)\n[Ri] [Rib] [ib] [ibe] [be] [bei] [ei] [eir] [ir] [ira] [ra] [ra ] [a ] [a C] [ C] [ Ch] [Ch] [Cha] [ha] [har] [ar] [arm] [rm] [rmi] [mi] [min] [in] [ing] [ng] [ng ] [g ] [g D] [ D] [ Du] [Du] [Dup] [up] [upl] [pl] [ple] [le] [lex] [ex]\n```\n\n**Sample 2**\n\n```bash\nmvn -f pom-cli.xml exec:java -Dexec.args=\"-t 'Ribeira Charming Duplex' -o autocomplete -k edgeGram -m 2 -x 15\"\nUsing org.apache.lucene.analysis.custom.CustomAnalyzer\nAutocomplete - edgeNGram, minGram(2), maxGram(15)\n[Ri] [Rib] [Ribe] [Ribei] [Ribeir] [Ribeira] [Ribeira ] [Ribeira C] [Ribeira Ch] [Ribeira Cha] [Ribeira Char] [Ribeira Charm] [Ribeira Charmi] [Ribeira Charmin]\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmongodb-developer%2Flucene-search-analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmongodb-developer%2Flucene-search-analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmongodb-developer%2Flucene-search-analysis/lists"}