{"id":33278640,"url":"https://github.com/kuhumcst/cstlemma","last_synced_at":"2026-01-11T09:40:39.076Z","repository":{"id":4963494,"uuid":"6121049","full_name":"kuhumcst/cstlemma","owner":"kuhumcst","description":"Lemmatiser for Danish, Dutch, English, German, Polish, Romanian, Russian and tens of other languages, that uses affix rules (affix: prefix, infix, suffix, circumfix). Rules are obtained by supervised learning from a full form - lemma list.","archived":false,"fork":false,"pushed_at":"2025-06-26T06:55:40.000Z","size":1527,"stargazers_count":36,"open_issues_count":1,"forks_count":6,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-06-26T07:38:51.088Z","etag":null,"topics":["affix","dutch","german","infix","lemmatiser","lemmatizer","prefix","suffix"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"PulsedLight3D/LIDARLite_Basics","license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kuhumcst.png","metadata":{"files":{"readme":"README.md","changelog":"Changelog","contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2012-10-08T07:43:33.000Z","updated_at":"2025-06-26T06:55:44.000Z","dependencies_parsed_at":"2025-04-01T08:22:38.188Z","dependency_job_id":"02430bc7-4d88-4606-88d2-cc13966e8d13","html_url":"https://github.com/kuhumcst/cstlemma","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/kuhumcst/cstlemma","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kuhumcst%2Fcstlemma","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kuhumcst%2Fcstlemma/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kuhumcst%2Fcstlemma/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kuhumcst%2Fcstlemma/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kuhumcst","download_url":"https://codeload.github.com/kuhumcst/cstlemma/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kuhumcst%2Fcstlemma/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":285731803,"owners_count":27222214,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-22T02:00:05.934Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["affix","dutch","german","infix","lemmatiser","lemmatizer","prefix","suffix"],"created_at":"2025-11-17T10:00:42.714Z","updated_at":"2025-11-22T04:01:46.809Z","avatar_url":"https://github.com/kuhumcst.png","language":"C++","funding_links":[],"categories":["Tools"],"sub_categories":["Lemmatization"],"readme":"# CSTLEMMA - the CST Lemmatiser\n\n\n**This distribution contains the following directories and files:**\n\n* doc\n    * This directory contains documentation of the program.\n* src\n    * This directory contains source code and a Makefile.\n    * You also need to clone source code from the following repositories to `lib/parsesgml`, `lib/letterfunc` and `lib/hashmap` respectively:\n        * https://github.com/kuhumcst/parsesgml\n        * https://github.com/kuhumcst/letterfunc\n        * https://github.com/kuhumcst/hashmap\n    * NOTE: Fetching libraries and compiling the code can be done automatically by running `makecstlemma.bash`.\n    * ALTERNATIVELY: Place the three included source directories in the same root directory that the `cstlemma` directory is in and then run `make` inside `cstlemma/src`.\n* Changelog\n    * A document describing changes between versions.\n* COPYING\n    * The full text of the GNU public licence.\n* README.md\n    * This file.\n\n\n\n**CSTLEMMA has been compiled and run on the following platforms:**\n\u003ctable\u003e\n\u003cthead\u003e\u003ctd\u003e\nPlatform\u003c/td\u003e\u003ctd\u003eCompiler(s)\u003c/td\u003e\u003c/thead\u003e\n\u003ctr\u003e\u003ctd\u003e\nWindows\u003c/td\u003e\u003ctd\u003eBorland C++ 5 and Microsoft Visual C++ 6.0 and later\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003e\nLinux\u003c/td\u003e\u003ctd\u003eGNU C++ 3.3.1 and later\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\nBoth 32 and 64 bit versions can be made.\n\n**Installation**\n\n* Linux/Mac:\n    1. Download (e.g. git clone) cstlemma, parsesgml, letterfunc and hashmap. If you are going to use the Makefile that comes with cstlemma, locate each of these packages in separate subdirectories under the same directory, and call these subdirectories cstlemma, parsesgml, letterfunc and hashmap, respectively. You can run `makecstlemma.bash` to do all of this automatically.\n    2. Change directory to the 'cstlemma/src' directory.\n    3. Run 'make' (both Linux and Mac) or 'make all' (only Linux). To get rid of object files, run\n    4. 'make clean'.\n\n**Running**\n\nFor running the CST lemmatiser you need as a minimum a file containing flex\nrules. The absolute minimal set of flex rules is the empty set, in which case\nthe lemmatiser assumes that all words in your input text are perfectly\nlemmatised already.\n\nThus, for checking that the lemmatiser runs OK, you could do the following:\n\n        touch my_empty_rule_file\n        ./cstlemma -L -f my_empty_rule_file -i my_text_file.txt\n\nThis would create a file my_text_file.txt.lemma that has two tab-separated\ncolumns: the left column contains a word from your text and the right column\ncontains the same word, converted to lower-case. The -L option tells the\nprogram lemmatise (as opposed to generating flex rules or creating a binary\ndictionary). The -f and -i options tell the program which rules and which input\ntext to read.\n\nThe lemmatiser \"cstlemma\" can generate flex rules from a full-form dictionary,\nbut a better way to obtain flex rules for use by cstlemma is to use the program\ncalled \"affixtrain\" (https://github.com/kuhumcst/affixtrain). The flex rules\nthat cstlemma can produce only look at words from the end (\"suffix\"-oriented).\nAffixtrain, on the other hand, can look at the beginning, the end and in several\nplaces inside a word, all at once. For languages like German and Dutch, were\nmorphological changes deep inside words often follow strict rules, affixtrain\nhas a clear advantage over the old suffix based algorithm implemented in\ncstlemma, but also languages that only have suffix morphology can have words\nthat carry important information in other places than near the end about word\nclass and therefore morphology. A disadavantage of training flex rules with\naffixtrain is that it can take a long time, perhaps days if the training data\nconsists of millions of unique full form - lemma pairs.\n\nThe full-form dictionary used to train flex rules can also be used to generate\na binary dictionary, which the program can use to even better lemmatise your\ninput text. For this task you can use cstlemma (with the -D option).\n\n**Online availability**\n\nCSTLEMMA is demonstrated at CST's website\n(http://ada.sc.ku.dk/tools/index.php?lang=en)\nand an integrated webservice in the CLARIN-DK infrastructure\n(https://clarin.dk/).\n\n**References**\n\nBart Jongejan and Hercules Dalianis. 2009. Automatic\ntraining of lemmatization rules that handle morphological\nchanges in pre-, in- and suffixes alike. In\n*Proceedings of the Joint Conference of the 47th\nAnnual Meeting of the ACL and the 4th International\nJoint Conference on Natural Language Processing\nof the AFNLP*, pages 145–153. Association\nfor Computational Linguistics.\nhttps://www.aclweb.org/anthology/P09-1017.pdf\n\n**Appraisal of lemmatization (author: ChatGPT)**\n\n  There once was a word, quite a beast\n  \n  Its endings were many, at least\n  \n  But with lemmatization\n  \n  It found liberation\n  \n  And made perfect sense in the least\n\n**Contact info**\n\nFor questions and remarks about the program, please feel free to contact us.\n\nOur postal address is:\n\n    Center for Sprogteknologi\n    University of Copenhagen\n    Njalsgade 140\n    2300 Copenhagen S.\n    Denmark\n\nOn the internet, you can visit us at [www.cst.ku.dk](http://www.cst.ku.dk)\nHere you can also try the CST lemmatiser for Danish and many other languages.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkuhumcst%2Fcstlemma","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkuhumcst%2Fcstlemma","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkuhumcst%2Fcstlemma/lists"}