{"id":21426093,"url":"https://github.com/bcdh/lemator","last_synced_at":"2025-07-08T17:36:23.063Z","repository":{"id":135733816,"uuid":"222094873","full_name":"BCDH/lemator","owner":"BCDH","description":"A brute-force lemmatizer and disambiguator.","archived":false,"fork":false,"pushed_at":"2019-11-16T12:18:35.000Z","size":18,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-23T07:29:02.459Z","etag":null,"topics":["dictionary","disambiguation","lemmatization","lookup"],"latest_commit_sha":null,"homepage":"","language":"PowerShell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BCDH.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-11-16T12:18:30.000Z","updated_at":"2019-11-16T12:21:22.000Z","dependencies_parsed_at":null,"dependency_job_id":"934c32fa-f443-400c-ad1c-375c0a572a81","html_url":"https://github.com/BCDH/lemator","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BCDH%2Flemator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BCDH%2Flemator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BCDH%2Flemator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BCDH%2Flemator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BCDH","download_url":"https://codeload.github.com/BCDH/lemator/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243933437,"owners_count":20370986,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dictionary","disambiguation","lemmatization","lookup"],"created_at":"2024-11-22T21:39:55.619Z","updated_at":"2025-03-16T21:14:28.016Z","avatar_url":"https://github.com/BCDH.png","language":"PowerShell","funding_links":[],"categories":[],"sub_categories":[],"readme":"**LEMATOR** (\u003c Serb. лематор \u003c лемати, to thrash, beat up; pun intended) is a brute-force lemmatizer and disambiguator written for PowerShell.\n\nIt's a simple tool based on dictionary look-ups. It does no fancy morphological analysis. The function `Use-Lemmatizer` looks up tokens from a TEI XML file in a morphological lexicon and records all the possible lemmas for a given token (separated by '|' if the look-up was non-ambiguous) in the TEI XML output file:\n\n![Скриншот 2019-11-16 11.30.10](https://i.imgur.com/lA7zApA.png)\n\nThe non-recognized tokens (i.e. those not present in the morphological lexicon) are logged by frequency so that they can be used to improve the morphological lexicon:\n\n![Скриншот 2019-11-16 12.10.05](https://i.imgur.com/weddtP3.png)\n\nIn a second step, Lemator's `Use-Disambiguator` function can be used to manually disambiguate ambiguous lemmas (separated by '|') from a TEI XML file. To speed the process up, users can disambiguate multiple contexts at the same time:  \n\n![Скриншот 2019-11-16 12.15.01](https://i.imgur.com/upIt7qy.jpg)\n\nFor details on how to use and configure Lemator, see below.\n\n\n## Use-Lemmatizer\n\nThis function:\n- collects all the words from the source xml file using a given XPATH (see variable `$XPATH_W` in `lemator.psm1`)\n- looks up the lemma values in the dictionary file for each collected token\n- records the lemma value(s) in an attribute for each token (see variable `$ATTR_LEMMA` in `lemator.psm1`)\n- outputs the fully reconstructed, lemmatized XML file.\n\nPARAMETER Source\u003cbr/\u003e\nInput XML, mandatory.\n\nPARAMETER Target\u003cbr/\u003e\nOutput file; optional; by default adds a suffix .lemmatized before the .xml suffix, so that\nthe input file `VSK.P13.sample.xml` will be saved as `VSK.P13-sample.lemmatized.xml`.\n\nPARAMETER Type\u003cbr/\u003e\n- With type \"AllLemmas\", for a given word form, the lemmatizer will look up all possible lemmas in the dictionary file, reduce them to a unique list and record them like this \"lemma1|lemma2|lemma3\".\n- With type \"UniqueLemmas\", the lemmatizer will record lemmas only for non-ambiguous tokens, i.e. only for those tokens that have exactly one possible lemma listed in the dictionary file.\n\nPARAMETER Dictionary\u003cbr/\u003e\nDictionary file, optional. Default value: \"slaws.dic\"\n\nPARAMETER ProcessTagged\u003cbr/\u003e\nIf enabled, the lemmatizer will process all the tokens from the xml source, regardless of their lemma attribute, otherwise it will skip word elements that already have a non-empty lemma attribute.\n\nPARAMETER Unknown\u003cbr/\u003e\nOutput text file (utf8) containing a sorted list of tokens without a corresponding dictionary-based lemma by their frequency.\n\nEXAMPLE\u003cbr/\u003e\n```powershell\nUse-Lemmatizer VSK.P13-sample.xml\n```\n\nEXAMPLE\u003cbr/\u003e\n```powershell\nUse-Lemmatizer `\n    -Source VSK.P13-sample.xml `\n    -Target \"$PSScriptRoot\\.temp\\result.xml\" `\n    -Dictionary \"$PSScriptRoot\\.temp\\slaws.dic\" `\n    -Type UniqueLemmas\n```\n\n## Use-Disambiguator\n\nThe function:\n- collects all tokens annotated with multiple lemmas (see `$XPATH_W` variable in `lemator.psm1` (for instance: `//tei:w[contains(@lemma, '|')]`) as well as their context (sentences, or words tokens and after)\n- presents to the user a configurable number of sentences (contexts) in which the given multi-lemma combo appears and the option to chose which of the possible lemmas is appropriate in the given context\n- outputs the fully reconstructed XML file containing only the disambiguated lemma value in the lemma attribute for each of the contexts\n\nPARAMETER Source\u003cbr/\u003e\nInput XML, mandatory.\n\nPARAMETER Target\u003cbr/\u003e\nOutput file; optional; by default adds a suffix .disambiguated before the .xml suffix, so that the input file `VSK.P13.sample.xml` will be saved as `VSK.P13-sample.disambiguated.xml`.\n\nPARAMETER Context\u003cbr/\u003e\n- With context \"Sentence\", the disambiguator will declare the context for each multilemma word to be a sentence, i.e. the `tei:s` node which is the parent of the  given `tei:w[contains(@lemma, '|')]`. Optional parameter. If the parameter is missing, the default is sentence.\n\n- With context \"Siblings\", the disambiguator will declare the context for each multilemma combo to be a configurable number of siblings to the given `tei:w[contains(@lemma, '|')]` both on the left and right. This will be used for files that don't have explicit tags for sentences.\n\nPARAMETER SiblingsNumber\u003cbr/\u003e\nThis parameter is allowed only if we have `-Context Siblings`. The number indicates how many sibling-nodes (both on the left and the right) should be picked up by the script as the context for the given token.\n\nPARAMETER ContextsPerWindow\u003cbr/\u003e\nThis parameter indicates the number of contexts that the user will be considering at a time.\nDefault vlaue: 5.\n\nEXAMPLE\n```powershell\n  Use-Disambiguator `\n      -Source VSK.P13-sample.xml `\n      -Context \"Sentence\" `\n      -ContextsPerWindow 3\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbcdh%2Flemator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbcdh%2Flemator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbcdh%2Flemator/lists"}