{"id":15520714,"url":"https://github.com/proycon/lexmatch","last_synced_at":"2025-04-23T04:29:50.810Z","repository":{"id":62441995,"uuid":"408466579","full_name":"proycon/lexmatch","owner":"proycon","description":"Simple lexicon matcher against a text","archived":false,"fork":false,"pushed_at":"2024-07-03T19:07:18.000Z","size":49,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-22T10:21:12.957Z","etag":null,"topics":["lexical-search","nlp"],"latest_commit_sha":null,"homepage":"https://git.sr.ht/~proycon/lexmatch","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/proycon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-09-20T14:02:53.000Z","updated_at":"2025-02-01T18:58:19.000Z","dependencies_parsed_at":"2022-11-01T21:51:56.458Z","dependency_job_id":null,"html_url":"https://github.com/proycon/lexmatch","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proycon%2Flexmatch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proycon%2Flexmatch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proycon%2Flexmatch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proycon%2Flexmatch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/proycon","download_url":"https://codeload.github.com/proycon/lexmatch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250371209,"owners_count":21419532,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["lexical-search","nlp"],"created_at":"2024-10-02T10:29:00.819Z","updated_at":"2025-04-23T04:29:50.554Z","avatar_url":"https://github.com/proycon.png","language":"Rust","funding_links":[],"categories":["\u003ca name=\"text-processing\"\u003e\u003c/a\u003eText processing"],"sub_categories":[],"readme":"# Lexmatch\n\nThis is a simple lexicon matching tool that, given a lexicon of words or\nphrases, identifies all matches in a given target text, returning their exact\npositions. It can be used compute a frequency list for a lexicon, on a target\ncorpus.\n\nThe implementation uses suffix arrays or hash tables. The text must be\nplain-text UTF-8. For the former implementation (default), it is limited to\n2^32 bytes (about 4GB). For the latter implementation (`--tokens`/`--cjk`),\nthere is no such limit. The offsets outputted will be UTF-8 *byte* positions.\n\nThis tool only does exact (or case insensitive) matching, if you need fuzzy\nmatching against lexicons, check out [analiticcl](https://github.com/proycon/analiticcl)\ninstead.\n\n## Installation\n\nYou can build and install the latest stable release using Rust's package manager:\n\n```\ncargo install lexmatch\n```\n\nor if you want the development version after cloning this repository:\n\n```\ncargo install --path .\n```\n\nNo cargo/rust on your system yet? Do ``sudo apt install cargo`` on Debian/ubuntu based systems, ``brew install rust`` on mac, or use [rustup](https://rustup.rs/).\n\n## Usage\n\nSee ``lexmatch --help``.\n\nSimple example:\n\n```\n$ lexmatch --lexicon lexicon.lst corpus.txt\n```\n\nThe lexicon must be plain-text UTF-8 containing one entry per line, an entry\nneed not be a single word and is not constrained in length. If the lexicon\nconsists of Tab Separated Values (TSV), then only the first column is\nconsidered, the rest is ignored.\n\nInstead of a lexicon you can also provide the patterns to query on the command line using ``--query``.\n\nBy default, you will get a TSV file with a column for the text, the occurrence count, and\none with the begin position (UTF-8 byte position) for each match (dynamic columns):\n\n```\n$ lexmatch --query good --query bad /nettmp/republic.short.txt \nReading text from /tmp/republic.short.txt...\nBuilding suffix array (this may take a while)...\nSearching...\ngood    4       193     3307    3480    278\nbad     3       201     3315    3488\n```\n\nMatching is case sensitive by default, add `--no-case` for case insensitive\nbehaviour (all input and output will be lowercase, this may in rare cases cause\nthe UTF-8 offsets to no longer be valid on the original text).\n\nFor verbose output, add ``--verbose``. This produces cleaner TSV (tab seperated\nvalues) output that you can easily import in for example the [STAM\ntools](https://github.com/annotation/stam-tools):\n\n```\n$ lexmatch --verbose --query good --query bad /nettmp/republic.short.txt\nText    BeginUtf8Offset EndUtf8Offset\nReading text from /tmp/republic.short.txt...\nBuilding suffix array (this may take a while)...\nSearching...\ngood    193     197\ngood    3307    3311\ngood    3480    3484\ngood    278     282\nbad     201     204\nbad     3315    3318\nbad     3488    3491\n```\n\nYou may provide multiple lexicons as well as multiple test files, the output\nwill output the lexicon and/or test file in such cases. If multiple lexicons match, they are all returned (delimited by a semicolon). The order of the\nresults is arbitrary.\n\nIf you don't care for the exact positions but rather want to compute a\nfrequency list with the number of occurrences for each item in the lexicon or\npassed through ``--query``, then pass ``--count-only``:\n\n```\n$ lexmatch --count-only --query good --query bad /tmp/republic.short.txt\nReading text from /tmp/republic.short.txt...\nBuilding suffix array (this may take a while)...\nSearching...\ngood    4\nbad\t3\n```\n\nYou can configure a minimum frequency threshold using ``--freq``.\n\nRather than match all of the lexicon against the text, you can also iterate\nover tokens in the text and check if they occur in the lexicon. This uses a\nhash map instead of a suffix array and is typically faster. It is more limited,\nhowever, and can not be used with frequency thresholds or counting. It will\nalways produce verbose output (similar to ``--verbose``):\n\n```\n$ lexmatch --tokens --query good --query bad /nettmp/republic.short.txt\nText    BeginUtf8Offset EndUtf8Offset\nReading text from /tmp/republic.short.txt...\ngood    193     197\nbad     201     204\ngood    278     282\ngood    3307    3311\nbad     3315    3318\ngood    3480    3484\nbad     3488    3491\n```\n\nUnlike before, you will find the matches are now returned in reading order.\n\nIf you add `--coverage` then you will get an extra last line with some coverage\nstatistics. This is useful to see how much of the text is covered by your\nlexicon.\n\n```\n#coverage (tokens) = 7/627 = 0.011164274322169059\n```\n\nCoverage can also be computed line-by-line and matching against multiple lexicons, we can also read directly from stdin rather than from file by passing `-` as filename:\n\n```\n$ echo \"Is this good or bad?\\nIt is quite good.\" | lexmatch --coverage-matrix --query good --query bad  -\nReading text from -...\nLine            query\nIs this good or bad?    0.4\nIt is quite good.       0.25\n```\n\nThis can be used as a simple lexicon-based method for language detection:\n\n```\n$ echo \"Do you know what language this is?\\nUnd was ist das hier genau?\\nÇa va assez bien je crois\" | lexmatch -i  --coverage-matrix --lexicon ~X/en.lst --lexicon ~X/de.lst --lexicon ~X/fr.lst  -                     \nReading lexicon...\nReading lexicon...\nReading lexicon...\nReading text from -...\nLine            /home/proycon/exp/en.lst        /home/proycon/exp/de.lst        /home/proycon/exp/fr.lst        Total\ndo you know what language this is?      1       0       0.14285714285714285     1.1428571428571428\nund was ist das hier genau?     0.16666666666666666     0.8333333333333334      0.16666666666666666     1.1666666666666667\nça va assez bien je crois       0.2     0.2     0.6     1\n```\n\nWhen using ``--tokens`` (or `--coverage-matrix`) we rely on whitespace and punctuation to delimit\ntokens. This does not work for languages such as Chinese, Japanese and Korean\nthat are not delimited in such a way. For such languages, similar linear search\nbehaviour can be attained by passing ``--cjk`` instead, with an integer value\nrepresenting the maximum character length to explore. A greedy search will then\nbe performed that favours longer patterns over shorter ones.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproycon%2Flexmatch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fproycon%2Flexmatch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproycon%2Flexmatch/lists"}