{"id":21393136,"url":"https://github.com/xiamx/lemma","last_synced_at":"2025-06-27T04:33:33.639Z","repository":{"id":37664202,"uuid":"96734626","full_name":"xiamx/lemma","owner":"xiamx","description":"A Morphological Parser (Analyser) / Lemmatizer written in Elixir.","archived":false,"fork":false,"pushed_at":"2025-06-22T15:05:18.000Z","size":370,"stargazers_count":50,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-22T15:41:05.528Z","etag":null,"topics":["elixir","erlang","lemmatization","lemmatizer","morphological-analyser","morphology","nlp"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xiamx.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-07-10T04:06:40.000Z","updated_at":"2025-06-22T15:05:20.000Z","dependencies_parsed_at":"2022-08-30T04:21:03.605Z","dependency_job_id":null,"html_url":"https://github.com/xiamx/lemma","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/xiamx/lemma","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xiamx%2Flemma","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xiamx%2Flemma/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xiamx%2Flemma/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xiamx%2Flemma/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xiamx","download_url":"https://codeload.github.com/xiamx/lemma/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xiamx%2Flemma/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261319539,"owners_count":23140906,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elixir","erlang","lemmatization","lemmatizer","morphological-analyser","morphology","nlp"],"created_at":"2024-11-22T14:09:01.161Z","updated_at":"2025-06-27T04:33:33.615Z","avatar_url":"https://github.com/xiamx.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"# \u003cimg src=\"http://cs.mcgill.ca/~mxia3/images/lemma.png\" alt=\"Lemma\"\u003e\n\n[![Build Status](https://travis-ci.org/xiamx/lemma.svg?branch=master)](https://travis-ci.org/xiamx/lemma)\n[![Hex.pm](https://img.shields.io/hexpm/v/lemma.svg)](https://hex.pm/packages/lemma)\n[![license](https://img.shields.io/github/license/xiamx/lemma.svg)](https://github.com/xiamx/lemma/blob/master/LICENSE)\n\n\nA Morphological Parser (Analyser) / Lemmatizer written in Elixir. It is implemented using a textbook classic method relying in an abstraction called [Finite State Transducer](https://github.com/xiamx/gen_fst). \n\n**What are morphological parsing and lemmatization?**\n\n\u003e For grammatical reasons, documents are going to use different forms of a word, such as organize, organizes, and organizing. Additionally, there are families of derivationally related words with similar meanings, such as democracy, democratic, and democratization. In many situations, it seems as if it would be useful for a search for one of these words to return documents that contain another word in the set.\n\u003e \u003cbr/\u003e The goal of both stemming and lemmatization is to reduce inflectional forms and sometimes derivationally related forms of a word to a common base form. For instance:\n\u003e \u003cbr/\u003e\u003cbr/\u003eam, are, is ⇒ be \n\u003e \u003cbr/\u003ecar, cars, car's, cars' ⇒ car\n\u003e \u003cbr/\u003e\u003cbr/\u003eThe result of this mapping of text will be something like:\n\u003e \u003cbr/\u003e_the boy's cars are different colors_ ⇒\n\u003e the boy car be differ color. \n\u003e \u003cbr/\u003e -- [Stanford NLP Group](https://nlp.stanford.edu/IR-book/html/htmledition/stemming-and-lemmatization-1.html)\n\n**Not for production use, this library is neither cpu nor memory efficient**\n\n## Example Usage\n\n```elixir\n# Initialize a new morphological parser for English\nparser = Lemma.new :en\nassert {:ok, \"play\"} == parser |\u003e Lemma.parse(\"plays\")\n```\n\nDocumentation can be found at [https://hexdocs.pm/lemma](https://hexdocs.pm/lemma).\n## Installation\n\nIf [available in Hex](https://hex.pm/docs/publish), the package can be installed\nby adding `lemma` to your list of dependencies in `mix.exs`:\n\n```elixir\ndef deps do\n  [{:lemma, \"~\u003e 0.1.0\"}]\nend\n```\n\n## Benchmarking\n\n### Lemmatize a paragraph of words\n\nA simple benchmarking script is provided to test the performance of performing lemmatization on a paragraph of words. The script can be invoked with `mix run benchmarks/lemmatizing.exs`\n\n#### Sample output\n\n```\nBenchmark using 430 fixture words\nOperating System: Windows\nCPU Information: Intel(R) Core(TM) i5-5300U CPU @ 2.30GHz\nNumber of Available Cores: 4\nAvailable memory: 12.753915904 GB\nElixir 1.5.0\nErlang 19.2\nBenchmark suite executing with the following configuration:\nwarmup: 2.00 s\ntime: 5.00 s\nparallel: 1\ninputs: none specified\nEstimated total run time: 7.00 s\n\n\nBenchmarking Lemmatize input sequential...\n\nName                                 ips        average  deviation         median\nLemmatize input sequential          4.31      232.14 ms    68.17%      110.00 ms\n```\n\n### Compile-time vs Runtime parser\n\nTo evaluate the difference between building the parser at compile time vs. building the parser at runtime, we have another benchmarking script.\nThe script can be invoked with `mix run benchmarks/runtime_vs_compiletime.exs`\n\n#### Sample output\n\n```\nOperating System: Windows\nCPU Information: Intel(R) Core(TM) i5-5300U CPU @ 2.30GHz\nNumber of Available Cores: 4\nAvailable memory: 12.753915904 GB\nElixir 1.5.0\nErlang 19.2\nBenchmark suite executing with the following configuration:\nwarmup: 2.00 s\ntime: 5.00 s\nparallel: 1\ninputs: none specified\nEstimated total run time: 14.00 s\n\n\nBenchmarking Compiled parser...\nBenchmarking Dynamic parser...\n\nName                      ips        average  deviation         median\nCompiled parser       15.91 K       62.86 us   123.32%         0.0 us\nDynamic parser         3.80 K      262.91 us   118.03%      160.00 us\n\nComparison:\nCompiled parser       15.91 K\nDynamic parser         3.80 K - 4.18x slower\n```\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxiamx%2Flemma","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxiamx%2Flemma","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxiamx%2Flemma/lists"}