{"id":13760295,"url":"https://github.com/dainiusjocas/lucene-grep","last_synced_at":"2025-05-10T10:32:13.377Z","repository":{"id":39789558,"uuid":"328450382","full_name":"dainiusjocas/lucene-grep","owner":"dainiusjocas","description":"Grep-like utility based on Lucene Monitor compiled with GraalVM native-image","archived":false,"fork":false,"pushed_at":"2024-09-03T22:08:38.000Z","size":7243,"stargazers_count":190,"open_issues_count":20,"forks_count":5,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-11-16T17:41:06.321Z","etag":null,"topics":["clojure","graalvm","graalvm-native-image","grep","lucene"],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dainiusjocas.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null},"funding":{"github":["dainiusjocas"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2021-01-10T18:23:15.000Z","updated_at":"2024-10-08T17:56:19.000Z","dependencies_parsed_at":"2023-09-29T11:58:08.757Z","dependency_job_id":null,"html_url":"https://github.com/dainiusjocas/lucene-grep","commit_stats":null,"previous_names":[],"tags_count":99,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dainiusjocas%2Flucene-grep","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dainiusjocas%2Flucene-grep/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dainiusjocas%2Flucene-grep/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dainiusjocas%2Flucene-grep/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dainiusjocas","download_url":"https://codeload.github.com/dainiusjocas/lucene-grep/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253401721,"owners_count":21902716,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clojure","graalvm","graalvm-native-image","grep","lucene"],"created_at":"2024-08-03T13:01:07.272Z","updated_at":"2025-05-10T10:32:13.097Z","avatar_url":"https://github.com/dainiusjocas.png","language":"Clojure","funding_links":["https://github.com/sponsors/dainiusjocas"],"categories":["Clojure","Text Processing"],"sub_categories":[],"readme":"# lucene-grep\nGrep-like utility based on [Lucene Monitor](https://lucene.apache.org/core/8_2_0/monitor/index.html) compiled with GraalVM native-image.\n\n## Features\n\n- Supports Lucene query syntax as described [here](https://lucene.apache.org/core/2_9_4/queryparsersyntax.html)\n- Multiple queries can be provided\n- Queries can be loaded from a file\n- Supports Lucene text analysis configuration for:\n  - char filters\n  - tokenizers\n  - token filters\n  - stemmers for multiple languages\n  - predefined analyzers\n- Support multiple query parsers (classic, complex phrase, standard, simple, and surround)\n- Text output is colored or separated with customizable tags\n- Supports printing file names as hyperlinks for click to open (check support for your terminal [here](https://gist.github.com/egmontkob/eb114294efbcd5adb1944c9f3cb5feda))\n- Text output supports templates\n- Scoring mode (disables highlighting for now)\n- Output can be formatted as JSON of EDN\n- Supports text input from [STDIN](https://en.wikipedia.org/wiki/Standard_streams#Standard_input_(stdin))\n- Supports filtering files with [GLOB](https://en.wikipedia.org/wiki/Glob_(programming)) [file pattern](https://docs.oracle.com/javase/8/docs/api/java/nio/file/FileSystem.html#getPathMatcher-java.lang.String-)\n- Support excluding files from processing with GLOB\n- Compiled with [GraalVM native-image](https://www.graalvm.org/reference-manual/native-image/) tool\n- Supports Linux, MacOS, and Windows\n- Fast startup which makes it usable as CLI utility\n\nStartup and memory as measured with `time` utility on my Linux laptop:\n\u003cimg src=\"docs/time-memory-usage.png\"\nalt=\"Startup time and memory usage\" title=\"Startup time and memory usage\" /\u003e\n\nThe default output has a format: `[FILE_PATH]:[LINE_NUMBER]:[LINE_WITH_A_COLORED_HIGHLIGHT]`\n\nNOTE: Not compatible with `grep`. When compared with `grep` the functionality is limited in most aspects.\n\n## Quickstart\n\n### Brew\n\nMacOS and Linux binaries are provided via `brew`.\n\nInstall:\n\n```\nbrew install dainiusjocas/brew/lmgrep\n```\n\nUpgrade:\n\n```\nbrew upgrade lmgrep\n```\n\n### Docker\n\n`lmgrep` is deployed in the [Docker Hub](https://hub.docker.com/repository/docker/dainiusjocas/lmgrep):\n\n```shell\necho \"Lucene is awesome\" | docker run -i dainiusjocas/lmgrep /lmgrep lucene\n```\n\n### Windows\n\nOn Windows you can install using [scoop](https://scoop.sh/) and the [scoop-clojure](https://github.com/littleli/scoop-clojure) bucket.\n\nOr just follow these concrete steps:\n```shell\n# Note: if you get an error you might need to change the execution policy (i.e. enable Powershell) with\n# Set-ExecutionPolicy RemoteSigned -scope CurrentUser\nInvoke-Expression (New-Object System.Net.WebClient).DownloadString('https://get.scoop.sh')\n\nscoop bucket add scoop-clojure https://github.com/littleli/scoop-clojure\nscoop bucket add extras\nscoop install lmgrep\n```\n\n### Other platforms\n\nJust grab a binary from [Github releases](https://github.com/dainiusjocas/lucene-grep/releases), extract, and place it anywhere on the `$PATH`.\n\nIn case you're running MacOS then give run permissions for the executable binary:\n```shell\nsudo xattr -r -d com.apple.quarantine lmgrep\n```\n\nThen run it:\n```shell\necho \"Lucene is awesome\" | ./lmgrep \"Lucene\"\n```\n\n## Examples \n\nExample of the `lmgrep`:\n```shell\n./lmgrep \"main\" \"*.{clj,edn}\"\n=\u003e\n./src/core.clj:44:(defn -main [\u0026 args]\n./deps.edn:22:   :main-opts   [\"-m\" \"cognitect.test-runner\"]}\n./deps.edn:24:  {:main-opts  [\"-m\" \"clj-kondo.main --lint src test\"]\n./deps.edn:28:  {:main-opts  [\"-m clj.native-image core\"\n```\n\nThe default output is somewhat similar to `grep`, example:\n```shell\ngrep -n -R --include=\\*.{edn,clj} \"main\" ./\n=\u003e\n./deps.edn:22:   :main-opts   [\"-m\" \"cognitect.test-runner\"]}\n./deps.edn:24:  {:main-opts  [\"-m\" \"clj-kondo.main --lint src test\"]\n./deps.edn:26:   :jvm-opts   [\"-Dclojure.main.report=stderr\"]}\n./deps.edn:28:  {:main-opts  [\"-m clj.native-image core\"\n```\n\nSupports input from STDIN:\n```shell\ncat README.md | ./lmgrep \"monitor lucene\"\n```\nTIP: write your Lucene query within double quotes.\n\nVarious options with GLOB file pattern example:\n```shell\n./lmgrep --case-sensitive\\?=false --ascii-fold\\?=true --stem\\?=true --tokenizer=whitespace \"lucene\" \"**/*.md\"\n```\nTIP: write GLOB file patterns within double quotes.\n\nWe can exclude files also with a GLOB pattern.\n```shell\n./lmgrep \"lucene\" \"**/*.md\" --excludes=\"README.md\"\n```\nTIP: a GLOB pattern is treated as recursive if it contains \"**\", otherwise the GLOB is matched only against the file name.\n\nProvide multiple queries:\n```shell\necho \"Lucene is\\n awesome\" |  lmgrep --query=lucene --query=awesome\n=\u003e\n*STDIN*:1:Lucene is\n*STDIN*:2: awesome\n```\n\nProvide Lucene queries in a file:\n```bash\necho \"The quick brown fox jumps over the lazy dog\" | ./lmgrep --queries-file=test/resources/queries.json --format=json\n=\u003e\n{\"line-number\":1,\"line\":\"The quick brown fox jumps over the lazy dog\"}\n```\n\nThe contents of the Lucene queries file is in JSON format, e.g.:\n```json\n[\n  {\n    \"query\": \"fox\"\n  },\n  {\n    \"query\": \"dog\"\n  }\n]\n```\n\nNOTE: when the Lucene queries are specified as a positional argument or with `-q` or `--query` params or with the `--queries-file`, all the queries are concatenated into one list.\n\n## Deviations from Lucene query syntax\n\n- The field names are not supported because there are no field names in a line of text.\n\n## Supported options\n```shell\nUsage: lmgrep [OPTIONS] LUCENE_QUERY [FILES]\nSupported options:\n  -q, --query QUERY                                        Lucene query string(s). If specified then all the positional arguments are interpreted as files.\n      --query-parser QUERY_PARSER                          Which query parser to use, one of: [classic complex-phrase simple standard surround]\n      --queries-file QUERIES_FILE                          A file path to the Lucene query strings with their config. If specified then all the positional arguments are interpreted as files.\n      --queries-index-dir QUERIES_INDEX_DIR                A directory where Lucene Monitor queries are stored.\n      --tokenizer TOKENIZER                                Tokenizer to use, one of: [keyword letter standard unicode-whitespace whitespace]\n      --case-sensitive? CASE_SENSITIVE                     If text should be case sensitive\n      --ascii-fold? ASCII_FOLDED                           If text should be ascii folded\n      --stem? STEMMED                                      If text should be stemmed\n      --stemmer STEMMER                                    Which stemmer to use for token stemming, one of: [arabic armenian basque catalan danish dutch english estonian finnish french german german2 hungarian irish italian kp lithuanian lovins norwegian porter portuguese romanian russian spanish swedish turkish]\n      --presearcher PRESEARCHER              no-filtering  Which Lucene Monitor Presearcher to use, one of: [multipass-term-filtered no-filtering term-filtered]\n      --with-score                                         If the matching score should be computed\n      --format FORMAT                                      How the output should be formatted, one of: [edn json string]\n      --template TEMPLATE                                  The template for the output string, e.g.: file={{file}} line-number={{line-number}} line={{line}}\n      --pre-tags PRE_TAGS                                  A string that the highlighted text is wrapped in, use in conjunction with --post-tags\n      --post-tags POST_TAGS                                A string that the highlighted text is wrapped in, use in conjunction with --pre-tags\n      --excludes EXCLUDES                                  A GLOB that filters out files that were matched with a GLOB\n      --skip-binary-files                                  If a file that is detected to be binary should be skipped. Available for Linux and MacOS only.\n      --[no-]hidden                                        Search in hidden files. Default: true.\n      --max-depth N                                        In case of a recursive GLOB, how deep to search for input files.\n      --with-empty-lines                                   When provided on the input that does not match write an empty line to STDOUT.\n      --with-scored-highlights                             ALPHA: Instructs to highlight with scoring.\n      --[no-]split                                         If a file (or STDIN) should be split by newline.\n      --hyperlink                                          If a file should be printed as hyperlinks.\n      --with-details                                       For JSON and EDN output adds raw highlights list.\n      --word-delimiter-graph-filter WDGF                   WordDelimiterGraphFilter configurationFlags as per https://lucene.apache.org/core/7_4_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/WordDelimiterGraphFilter.html\n      --show-analysis-components                           Just print-out the available analysis components in JSON.\n      --only-analyze                                       When provided output will be analyzed text.\n      --explain                                            Modifies --only-analyze. Output is detailed token info, similar to Elasticsearch Analyze API.\n      --graph                                              Modifies --only-analyze. Output is a string that can be fed to the `dot` program.\n      --analysis ANALYSIS                    {}            The analysis chain configuration\n      --query-parser-conf CONF                             The configuration for the query parser.\n      --concurrency CONCURRENCY              8             How many concurrent threads to use for processing.\n      --queue-size SIZE                      1024          Number of lines read before being processed\n      --reader-buffer-size BUFFER_SIZE                     Buffer size of the BufferedReader in bytes.\n      --writer-buffer-size BUFFER_SIZE                     Buffer size of the BufferedWriter in bytes.\n      --[no-]preserve-order                                If the input order should be preserved.\n      --config-dir DIR                                     A base directory from which to load text analysis resources, e.g. synonym files. Default: current dir.\n      --analyzers-file FILE                                A file that contains definitions of text analyzers. Works in combinations with --config-dir flag.\n      --query-update-buffer-size NUMBER                    Number of queries to be buffered in memory before being committed to the queryindex. Default 100000.\n      --streamed                                           Listens on STDIN for json with both query and a piece of text to be analyzed\n  -h, --help\n```\n\nNOTE: question marks in `zsh` shell must be escaped, e.g. `--case-sensitive\\?=true` or within double quotes e.g. `\"--case-sensitive?=true\"` \n\n## Text Analysis\n\nThe [text analysis](https://www.jocas.lt/blog/post/lucene-text-analysis/) pipeline can be declaratively specified with the `--analysis` flag, e.g.:\n```shell\necho \"\u003cp\u003efoo bars baz\u003c/p\u003e\" | \\\n  ./lmgrep \\\n  --only-analyze \\\n  --analysis='\n  {\n    \"char-filters\": [\n      {\"name\": \"htmlStrip\"},\n      {\n        \"name\": \"patternReplace\",\n         \"args\": {\n           \"pattern\": \"foo\",\n           \"replacement\": \"bar\"\n        }\n      }\n    ],\n    \"tokenizer\": {\"name\": \"standard\"},\n    \"token-filters\": [\n      {\"name\": \"englishMinimalStem\"},\n      {\"name\": \"uppercase\"}\n    ]\n  }\n  '\n=\u003e\n[\"BAR\",\"BAR\",\"BAZ\"]\n```\n\nThe action inside `lmgrep` is as follows:\n- char filters are applied in order:\n  - `htmlStrip` is applied, which removes `\u003cp\u003e` and `\u003c/p\u003e` from the string (i.e. `foo bars baz`)\n  - `patternReplace` is applied, which replaces `foo` with `bar` (i.e. `bar bars baz`)\n- tokenization is performed (i.e. `[bar bars baz]`)\n- token filters are applied in order:\n  - `englishMinimalStem` which stems the tokens (i.e. `[bar bar baz]`)\n  - `uppercase` is applied (i.e. `[BAR BAR BAZ]`)\n- The resulting list of tokens is printed to STDOUT.\n\nYou can peel the analysis config layer by layer and see what are the intermediate results.\n\nFor the full list of supported analysis components see the [documentation](docs/analysis-components.md).\n\n### Default text analysis\n\nIf analysis is not specified then the default analysis pipeline is used which looks like:\n```shell\n--analysis='\n{\n  \"tokenizer\": {\n    \"name\": \"standard\"\n  },\n  \"token-filters\": [\n    {\n      \"name\": \"lowercase\"\n    },\n    {\n      \"name\": \"asciifolding\"\n    },\n    {\n      \"name\": \"englishMinimalStem\"\n    }\n  ]\n}\n'\n```\n\n### Predefined analyzers\n\n```shell\necho \"dogs and cats\" | ./lmgrep --only-analyze --analysis='{\"analyzer\": {\"name\": \"English\"}}'\n=\u003e\n[\"dog\",\"cat\"]\n```\nNote that stopwords were removed and stemming was applied.\n\nThe full list of predefined analyzers can be found [here](docs/predefined-analyzers.md).\n\n### Tips on Analysis Configuration\n\nAnalysis configuration must be a valid JSON and for your case it might make sense to store it in a file.\n\nStore analysis in a file:\n```shell\necho '{\"analyzer\": {\"name\": \"English\"}}' \u003e analysis-conf.json\n```\n\nRun the text analysis:\n```shell\necho \"dogs and cats\" | ./lmgrep --only-analyze --analysis=\"$(cat analysis-conf.json)\"\n```\n\nIf your JSON spans multiple lines ask a little help from `jq`:\n```shell\necho \"dogs and cats\" | ./lmgrep --only-analyze --analysis=$(jq -c . analysis-conf.json)\n```\n\n### What about resources for analyzers?\n\nSome token filters require a file as an argument, e.g. `StopFilterFactory` requires `words` which is a file.\nBy default, the Lucene would load the file under `words` from the classpath.\nHowever, `lmgrep` is a single binary and there the notion of the classpath makes little sense.\nTo support such analysis components that expects files the Lucene class that loads files was patched to support arbitrary files.\n\nE.g. create a stopwords file:\n```shell\necho \"foo\\nbar\" \u003e my-stopwords.txt\n```\nRun the analysis\n```shell\necho \"foo bar baz\" | \\\n  ./lmgrep \\\n  --only-analyze \\\n  --analysis='\n  {\n    \"token-filters\": [\n      {\n        \"name\": \"stop\",\n        \"args\": {\n          \"words\": \"my-stopwords.txt\"\n        }\n      }\n    ]\n  }\n  '\n=\u003e\n[\"baz\"]\n```\nSee the custom stopwords was removed.\n\nCreating files in arbitrary places might be OK for one-off scripts.\nHowever, it creates some mess.\nTherefore, consider creating a folder for your analysis component resources such as: `$HOME/.lmgrep`.\n\n```shell\nexport LMGREP_HOME=$HOME/.lmgrep\necho \"foo\\nbar\" \u003e $LMGREP_HOME/my-stopwords.txt\necho \"foo bar baz\" | \\\n  ./lmgrep \\\n  --only-analyze \\\n  --analysis='\n  {\n    \"token-filters\": [\n      {\n        \"name\": \"stop\",\n        \"args\": {\n          \"words\": \"'$LMGREP_HOME'/my-stopwords.txt\"\n        }\n      }\n    ]\n  }\n  '\n=\u003e\n[\"baz\"]\n```\n\n### Analysis in the queries file\n\nEvery query in the queries file can provide its own configuration, e.g.:\n```json\n[\n  {\n    \"id\": \"0\",\n    \"query\": \"dogs\",\n    \"analysis\": {\n      \"analyzer\": {\n        \"name\": \"English\"\n      }\n    }\n  },\n  {\n    \"id\": \"1\",\n    \"query\": \"dogs\",\n    \"analysis\": {\n      \"tokenizer\": {\n        \"name\": \"whitespace\"\n      }\n    }\n  }\n]\n```\n\nFor each unique analysis configuration a pair of Lucene Analyzer and an internal field name is created.\nThen Lucene Monitor is run over all queries, and their respective fields with their own analyzers for every text input.\n\n## WordDelimiterGraphFilter\n\nUsing WordDelimiterGraphFilter filter might help to tokenize text is various ways, e.g.:\n\n```shell\necho \"test class\" | ./lmgrep \"TestClass\" --word-delimiter-graph-filter=99\n=\u003e\n*STDIN*:1:test class\n```\n\n```shell\necho \"TestClass\" | ./lmgrep \"test class\" --word-delimiter-graph-filter=99\n=\u003e\n*STDIN*:1:TestClass\n```\n\nThe number 99 is a sum of options as described [here](https://lucene.apache.org/core/7_4_0/analyzers-common/constant-values.html#org.apache.lucene.analysis.miscellaneous.WordDelimiterGraphFilter.CATENATE_ALL).\n\n## Phrase Matching with Slop\n\nTo match a phrase you need to put it in double quotes:\n```shell\necho \"GraalVM is awesome\" | ./lmgrep \"\\\"graalvm is\\\"\"\n=\u003e\n*STDIN*:1:GraalVM is awesome\n```\n\nBy default, when phrase terms are not exactly one after another there is no match, e.g.:\n```shell\necho \"GraalVM is awesome\" | ./lmgrep \"\\\"graalvm awesome\\\"\"\n=\u003e\n```\n\nWe can provide a slop parameter i.e. `~2` to allow some number of \"substitutions\" of terms in the document text, e.g.:\n```shell\necho \"GraalVM is awesome\" | ./lmgrep \"\\\"graalvm awesome\\\"~2\"\n=\u003e\n*STDIN*:1:GraalVM is awesome\n```\n\nAs a side effect, when the slop is big enough terms can match out of order, e.g.:\n```shell\necho \"GraalVM is awesome\" | ./lmgrep \"\\\"awesome graalvm\\\"~3\"\n=\u003e\n*STDIN*:1:GraalVM is awesome\n```\nHowever, if order is important there is no way to enforce it Lucene query syntax.\n\n## Lucene query parsers\n\nCurrently, 5 [Lucene query parsers](https://javadoc.io/doc/org.apache.lucene/lucene-queryparser/latest/index.html) are supported:\n\n- classic: [docs](https://javadoc.io/doc/org.apache.lucene/lucene-queryparser/latest/index.html)\n- complex-phrase: [docs](https://javadoc.io/doc/org.apache.lucene/lucene-queryparser/latest/index.html)\n- simple: [docs](https://javadoc.io/doc/org.apache.lucene/lucene-queryparser/latest/index.html)\n- standard: [docs](https://javadoc.io/doc/org.apache.lucene/lucene-queryparser/latest/index.html)\n- surround: [docs](https://javadoc.io/doc/org.apache.lucene/lucene-queryparser/latest/index.html)\n\n### Query parser configuration\n\nAdditional configuration to query parsers can be passed with the `--query-parser-conf` flag, e.g.:\n\n```shell\n./lmgrep \"query\" --query-parser-conf='{\"allow-leading-wildcard\": false}'\n```\n\nThe value must be a JSON string.\nFor the supported configuration values consult the [documentation](docs/query-parsers.md).\n\n## Development\n\nRequirements: \n- Clojure CLI\n- [Babashka](https://github.com/babashka/babashka)\n- Maven\n- GraalVM with the `native-image` tool installed and on `$PATH`\n- GNU Make\n- Docker (just for rebuilding the linux native image).\n\nBuild executable for your platform:\n```shell\nmake build\n```\nIt will create an executable binary file named `lmgrep` stored at the root directory of the repository.\n\nRun the tests:\n```shell\nmake test\n```\n\nLint the core with clj-kondo:\n```shell\nbb lint\n```\n\n## Print results with a custom format\n\n```shell\n./lmgrep --template=\"FILE={{file}} LINE_NR={{line-number}} LINE={{highlighted-line}}\" \"test\" \"**.md\"\n```\n\n| Template Variable     | Notes                                                     |\n|-----------------------|-----------------------------------------------------------|\n| `{{file}}`            | File name                                                 |\n| `{{line-number}}`     | Line number where the text matched the query              |\n| `{{highlighted-line}}`| Line that matched the query with highlighters applied     |\n| `{{line}}`            | Line that matched the query                               |\n| `{{score}}`           | Score of the match (summed)                               |\n\nWhen `{{highlighted-line}}` is used then `--pre-tags` and `--post-tags` options are available, e.g.:\n```shell\necho \"some text to to match\" | lmgrep \"text\" --pre-tags=\"\u003cem\u003e\" --post-tags=\"\u003c/em\u003e\" --template=\"{{highlighted-line}}\"\n=\u003e\nsome \u003cem\u003etext\u003c/em\u003e to to match\n```\n\n## Scoring\n\nThe main thing to understand is that scoring is for every line separately in the context of that one line as a whole corpus.\n\nAnother consideration is that scoring is summed up for every line of all the matches. E.g. query \"one two\" is rewritten by Lucene into two term queries.\n\nEach individual score is BM25 which is default in Lucene.\n\n## `--only-analyze`\n\nGreat for debugging.\n\nThe output is a list tokens after analyzing the text, e.g.:\n```shell\necho \"Dogs and CAt\" | ./lmgrep --only-analyze     \n# =\u003e [\"dog\",\"and\",\"cat\"]\n```\n\nIn combination with `--explain` flag outputs the detailed analyzed text similar to Elasticsearch Analyze API, e.g.:\n```shell\necho \"Dogs and CAt\" | ./lmgrep --only-analyze --explain | jq\n# =\u003e [\n  {\n    \"token\": \"dog\",\n    \"position\": 0,\n    \"positionLength\": 1,\n    \"type\": \"\u003cALPHANUM\u003e\",\n    \"end_offset\": 4,\n    \"start_offset\": 0\n  },\n  {\n    \"end_offset\": 8,\n    \"positionLength\": 1,\n    \"position\": 1,\n    \"start_offset\": 5,\n    \"type\": \"\u003cALPHANUM\u003e\",\n    \"token\": \"and\"\n  },\n  {\n    \"position\": 2,\n    \"token\": \"cat\",\n    \"positionLength\": 1,\n    \"end_offset\": 12,\n    \"type\": \"\u003cALPHANUM\u003e\",\n    \"start_offset\": 9\n  }\n]\n```\n\nTo draw a token graph you can use the `--graph` flag, e.g.:\n```shell\necho \"FooBar-Baz\" | ./lmgrep --word-delimiter-graph-filter=99 --only-analyze --graph\n# =\u003e\ndigraph tokens {\n  graph [ fontsize=30 labelloc=\"t\" label=\"\" splines=true overlap=false rankdir = \"LR\" ];\n  // A2 paper size\n  size = \"34.4,16.5\";\n  edge [ fontname=\"Helvetica\" fontcolor=\"red\" color=\"#606060\" ]\n  node [ style=\"filled\" fillcolor=\"#e8e8f0\" shape=\"Mrecord\" fontname=\"Helvetica\" ]\n\n  0 [label=\"0\"]\n  -1 [shape=point color=white]\n  -1 -\u003e 0 []\n  0 -\u003e 2 [ label=\"foobar / FooBar\"]\n  0 -\u003e 1 [ label=\"foo / Foo\"]\n  1 [label=\"1\"]\n  1 -\u003e 2 [ label=\"bar / Bar\"]\n  2 [label=\"2\"]\n  2 -\u003e 3 [ label=\"baz / Baz\"]\n  -2 [shape=point color=white]\n  3 -\u003e -2 []\n}\n```\n\nThe `--graph` flag makes the text analysis output into a valid GraphViz program that can be fed to `dot` which draws a picture out of the text, magic.\n\nIf you have [GraphViz](https://graphviz.org/download/) installed on your machine, a one-liner to save the image of the text graph:\n```shell\necho \"FooBar-Baz\" | ./lmgrep --word-delimiter-graph-filter=99 --only-analyze --graph | dot -Tpng -o token-graph.png\n```\nThe output image looks should look like:\n\u003cimg src=\"docs/token-graph.png\"\nalt=\"Token Graph\" title=\"Token Graph\" /\u003e\n\nIf you also have ImageMagic installed you can preview the token graph with this one-liner on Ubuntu:\n```shell\necho \"FooBar-Baz\" | ./lmgrep --word-delimiter-graph-filter=99 --only-analyze --graph | dot -Tpng | display\n```\n\nOr MacOS:\n```shell\necho \"FooBar-Baz\" | ./lmgrep --word-delimiter-graph-filter=99 --only-analyze --graph | dot -Tpng | open -a Preview.app -f\n```\n\n## Streamed matching\n\nStart `lmgrep` process once and wait for the input from STDIN that includes both: text and the query.\nUsing such a technique avoids the \"cold start\" issues when with the stream of text the query is known only when the text is known.\n\nExample:\n\n```shell\necho '{\"query\": \"nike~\", \"text\": \"I am selling nikee\"}' | ./lmgrep --streamed --with-score --format=json --query-parser=simple\n#=\u003e {\"line-number\":1,\"line\":\"I am selling nikee\",\"score\":0.09807344}\n```\n\nIs equivalent to:\n\n```shell\necho  \"I am selling nikee\" | ./lmgrep --query=\"nike~\" --with-score --format=json --query-parser=simple\n#=\u003e {\"line-number\":1,\"line\":\"I am selling nikee\",\"score\":0.09807344}\n```\n\nAll other options are also applicable.\n\n## Custom Builds\n\n### Raudikko or Voikko stemming for Finnish Language\n\nNOTE: The project is re-architected in a way that Raudikko token filter definition is in the subdirectory.\nAlso, it was put under the deps.edn alias.\nHowever clever this change is, the uberjar builder has a hard time.\nThe solution now is to modify deps.edn file so that the raudikko dependency is put under top level :deps.\nTools build also has a hard time building an uberjar.\n\n```shell\n(export LMGREP_FEATURE_RAUDIKKO=true \u0026\u0026 bb generate-reflection-config \u0026\u0026 make build)\n```\n\n## Environment variables\n\nCheck the [docs](docs/env-vars.md).\n\n## Future work\n\n- [ ] Optimize matching by [processing lines in batches](https://github.com/dainiusjocas/lucene-grep/issues/3)\n\n## License\n\nCopyright \u0026copy; 2022 [Dainius Jocas](https://www.jocas.lt).\n\nDistributed under The Apache License, Version 2.0.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdainiusjocas%2Flucene-grep","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdainiusjocas%2Flucene-grep","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdainiusjocas%2Flucene-grep/lists"}