{"id":24517222,"url":"https://github.com/apertium/apertium-weighting-tools","last_synced_at":"2025-09-11T17:23:26.423Z","repository":{"id":80141645,"uuid":"203861161","full_name":"apertium/apertium-weighting-tools","owner":"apertium","description":"Scripts for weighting morphological analyzers","archived":false,"fork":false,"pushed_at":"2020-03-06T14:11:29.000Z","size":69,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-05-27T12:08:55.590Z","etag":null,"topics":["apertium-tools","morphological-analysis"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apertium.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-08-22T19:39:18.000Z","updated_at":"2020-02-19T23:57:18.000Z","dependencies_parsed_at":"2023-04-05T13:47:09.554Z","dependency_job_id":null,"html_url":"https://github.com/apertium/apertium-weighting-tools","commit_stats":{"total_commits":30,"total_committers":1,"mean_commits":30.0,"dds":0.0,"last_synced_commit":"78ed6ac1121786489533472e49a562cf20995ae6"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apertium%2Fapertium-weighting-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apertium%2Fapertium-weighting-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apertium%2Fapertium-weighting-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apertium%2Fapertium-weighting-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apertium","download_url":"https://codeload.github.com/apertium/apertium-weighting-tools/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243506849,"owners_count":20301766,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apertium-tools","morphological-analysis"],"created_at":"2025-01-22T01:31:45.651Z","updated_at":"2025-03-15T10:44:30.655Z","avatar_url":"https://github.com/apertium.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# apertium-weighting-tools\nThe project aims at implementing a set of algorithms for weighting\ntransducers.\n\n- [apertium-weighting-tools](#apertium-weighting-tools)\n  * [Dependencies](#dependencies)\n  * [Models Description](#models-description)\n      - [lt-weight](#lt-weight)\n        * [Methodology](#methodology)\n        * [Usage](#usage)\n        * [Example](#example)\n    + [Weightlist generation algorithms](#weightlist-generation-algorithms)\n      - [annotated-corpus-to-weightlist](#annotated-corpus-to-weightlist)\n        * [Methodology](#methodology-1)\n        * [Usage](#usage-1)\n        * [Example](#example-1)\n        * [Limitations](#limitations)\n        * [Future work](#future-work)\n      - [unannotated-corpus-to-weightlist](#unannotated-corpus-to-weightlist)\n        * [Methodology](#methodology-2)\n        * [Usage](#usage-2)\n        * [Example](#example-2)\n        * [Future work](#future-work-1)\n      - [equal-weightlist](#equal-weightlist)\n        * [Methodology](#methodology-3)\n        * [Usage](#usage-3)\n        * [Example](#example-3)\n      - [analysis-length-reweight](#analysis-length-reweight)\n        * [Methodology](#methodology-4)\n        * [Usage](#usage-4)\n        * [Example](#example-4)\n        * [Future work](#future-work-2)\n      - [w2v-weightlist](#w2v-weightlist)\n        * [Methodology](#methodology-5)\n        * [Usage](#usage-5)\n        * [Example](#example-5)\n        * [Limitations](#limitations-1)\n        * [Future work](#future-work-3)\n  * [Evaluation](#evaluation)\n    + [eval/corpus_split.py](#eval-corpus-splitpy)\n      - [Usage](#usage-6)\n    + [eval/\\*\\_fit.py](#eval-----fitpy)\n      - [Usage](#usage-7)\n    + [Example](#example-6)\n    + [Results](#results)\n  * [Appendix](#appendix)\n    + [Xerox regexp](#xerox-regexp)\n\n## Dependencies\n**lt-weight**\n- `lttoolbox`\n- `hfst`\n\n**Most of the weighting scripts**\n- `python3`\n\n**Word2vec scripts**\n- `tqdm` (A python package for showing the progress)\n- `gensim` (A python package for training word2vec models)\n\n`pip install tqdm gensim`\n\n**Evaluation scripts**\n- `tabulate`\n(A python package for generating evaluation results in Markdown table format)\n\n`pip install tabulate`\n\n## Models Description\n#### lt-weight\n\n##### Methodology\nAdd weights to a compiled dictionary using a set of weightlists.\n- The weightlists are written as Xerox regexp which is more powerful\nthan using weighted string pairs as it permits matching all analyses\nof certain prefixes, suffixes.\nBUT, take care when using complex regexp since composing a transducer\nwith a complex regexp transducer is time and memory consuming\n(TODO: WHAT IS THE EXACT COMPLEXITY?)\n- The script can make use of a sequence of a weightlists and it\nconsiders each weightlist to be a fallback one for paths that weren't\nweighted by the preceding weightlist.\n- It's also advised to use a default weightlist - in the form\n`[?*]::LARGE_WEIGHT` - as the last weightlist  such that paths that\nweren't part of any of the weightlists aren't dropped from the final\ncomposed transducer and are given large default weight.\n- Note: The script makes use of composition of weighted FSTs using\n`hfst-compose` and assumes that the weights belong to a tropical\nsemi-ring.\nSo, if the input fst had a path mapping `cat` to `dog` with a weight\nof `1` and the weightlist was in form `[d o g]::2` then the weighted\nfst will map `cat` to `dog` with a weight of `3 (1+2)`\nTODO: SO?? ADVANTAGE IN ANY WAY?\n\n##### Usage\nTODO: PRINT USAGE\n\n##### Example\nFor the ambiguous English word `saw`, `apertium-eng` already gives it\nfour distinct morphological analyses:\n\n\n`^saw/saw\u003cn\u003e\u003csg\u003e/saw\u003cvblex\u003e\u003cinf\u003e/saw\u003cvblex\u003e\u003cpres\u003e/saw\u003cvblex\u003e\u003cimp\u003e/see\u003cvblex\u003e\u003cpast\u003e$`\nLet's assume we want to priortize the analyses as follows:\n1) `^saw/see\u003cvblex\u003e\u003cpast\u003e$` as the verb see is somehow common in\nlanguage.\n2) `^saw/saw\u003cn\u003e\u003csg\u003e$` as it's both a popular movie and a useful tool.\n3) `^saw/saw\u003cvblex\u003e\u003cimp\u003e$` as imperative verbs seem to occur a lot in\nthe linguist's opinion.\n4) Any other possible should be given high default weight.\n\n```\n$ cat saw.att \n0       1       s       s\n1       2       a       a\n2       3       w       w\n3       9       ε       \u003cn\u003e\n9       8       ε       \u003csg\u003e\n3       6       ε       \u003cvblex\u003e\n6       8       ε       \u003cinf\u003e\n6       8       ε       \u003cpres\u003e\n6       8       ε       \u003cimp\u003e\n1       4       a       e\n4       5       w       e\n5       7       ε       \u003cvblex\u003e\n7       8       ε       \u003cpast\u003e\n8\n\n$ lt-comp lr saw.att saw.bin\nmain@standard 10 13\n\n$ echo '[s e e %\u003cvblex%\u003e %\u003cpast%\u003e]::1' \u003e wl1\n$ echo '[s a w %\u003cn%\u003e %\u003csg%\u003e]::2' \u003e wl2\n$ echo '[?* %\u003cvblex%\u003e %\u003cimp%\u003e]::3' \u003e wl3\n$ echo '[?*]::4' \u003e wl4\n\n$ ./lt-weight saw.bin weighted_saw.bin wl1 wl2 wl3 wl4\nReading from /tmp/tmp.mOItp9wzUO/transducer.hfst and\n/tmp/tmp.mOItp9wzUO/weighted-regexp.hfst, writing to\n/tmp/tmp.mOItp9wzUO/weighted-transducer.hfst\nComposing text(/tmp/tmp.mOItp9wzUO/transducer.att) and xre(?)...\nReading from /tmp/tmp.mOItp9wzUO/transducer.hfst and\n/tmp/tmp.mOItp9wzUO/weighted-regexp.hfst, writing to\n/tmp/tmp.mOItp9wzUO/weighted-transducer.hfst\nComposing text(/tmp/tmp.mOItp9wzUO/transducer.att) and xre(?)...\nReading from /tmp/tmp.mOItp9wzUO/transducer.hfst and\n/tmp/tmp.mOItp9wzUO/weighted-regexp.hfst, writing to\n/tmp/tmp.mOItp9wzUO/weighted-transducer.hfst\nComposing text(/tmp/tmp.mOItp9wzUO/transducer.att) and xre(?)...\nReading from /tmp/tmp.mOItp9wzUO/transducer.hfst and\n/tmp/tmp.mOItp9wzUO/weighted-regexp.hfst, writing to\n/tmp/tmp.mOItp9wzUO/weighted-transducer.hfst\nComposing text(/tmp/tmp.mOItp9wzUO/transducer.att) and xre(?)...\nmain@standard 10 13\n\n$ echo 'saw' | lt-proc weighted_saw.bin -W\n\n\n^saw/see\u003cvblex\u003e\u003cpast\u003e\u003cW:1.000000\u003e/saw\u003cn\u003e\u003csg\u003e\u003cW:2.000000\u003e/saw\u003cvblex\u003e\u003cimp\u003e\u003cW:3.000000\u003e/saw\u003cvblex\u003e\u003cpres\u003e\u003cW:4.000000\u003e/saw\u003cvblex\u003e\u003cinf\u003e\u003cW:4.000000\u003e$\n```\n\n**Note**:\n- wl1 and wl2 can be merged into a single weightlist since they\naren't actually acting as fallback for each other.\n- On the other hand, if wl3 and wl4 were merged into a single\nweightlist then the analysis `saw\u003cvblex\u003e\u003cimp\u003e` would actually be\nadded twice to the weighted transducer with two different weights 3\nand 4!\n\n### Weightlist generation algorithms\n#### annotated-corpus-to-weightlist\n##### Methodology\nGenerate a weightlist given an annotated corpus.\nThe annotated corpus is in form `^surface_form/analyzed_form$`.\nThe script will estimate the weight for the analyzed form by\ncalculating the probability of the analysis in the corpus.\n`P(analysis) = Count(analysis) / size of corpus`\nThis model acts as a benchmark model for the other unsupervised\ntechniques.\n\nTo account for the OOV analyses, laplace smoothing is used such that:\n- the weight for analyses that aren't part of the corpus is\n`1 / (size of corpus + number of unique analysis in corpus + 1)`\n- the weight for an analysis that is part of the corpus is\n`(1 + Count(analysis)) / (size of corpus + number of unique analysis\nin corpus + 1)`\n\nA tweak to the weightlist generation (`--tag_weightlist`) was also\nadded to give priority to analyses with tags that are common in the\ncorpus.\ni.e: If the noun `\u003cn\u003e` tag is highly probable in the corpus then we\nmight want to weight OOV analyses with a \u003cn\u003e tag with a lower weight\n(higher probablility).\nHowever, using the tag weightlist won't make the weights\nprobabilistic.\n\n##### Usage\nTODO: PRINT USAGE\n\n##### Example\nTODO\n##### Limitations\n- The `--tag_weightlist` seems to be slow since it envolves the\ncomposition of the original fst with a regexp-fst in the form [?*\u003ctag\u003e].\n\n##### Future work\n- Make use of advanced techniques other than the unigram counts\n(example: n-grams).\n\n#### unannotated-corpus-to-weightlist\n##### Methodology\nTo generate a tagged corpus similar to that used in the\n`annotated-corpus-to-weightlist` method, the script will:\n- analyze a raw corpus using a compiled unweighted dictionary.\n- make use of a constraint grammar to discard immpossible analyses or\nselect certain ones.\n- write all the possible analyses to a file using the following\nformat for each line `^surface_form/analyzed_form$`.\n- use the `annotated-corpus-to-weightlist` to estimate a weightlist.\n\n##### Usage\nTODO: PRINT USAGE\n\n##### Example\nTODO\n\n##### Future work\n- Investigate the effect of using a constraint grammar and perform a\nthorough error analysis\n\n#### equal-weightlist\n##### Methodology\nGenerate a simple weightlist with the same weight for all analyses.\nThis model acts as the baseline for all the other techniques.\n\n##### Usage\nTODO: PRINT USAGE\n\n##### Example\nTODO\n\n#### analysis-length-reweight\n##### Methodology\nUse `hfst-reweight` to directly give a weight of one to all the edges\nof the finite state transducer.\n\n##### Usage\nTODO: PRINT USAGE\n\n##### Example\nTODO\n\n##### Future work\n- Compare the results of using `equal-weightlist` and\n`analysis-length-reweight`\nWeighting epsilon input might be the reason for the drift in the\nresults of both methods.\n\n#### w2v-weightlist\n##### Methodology\nGenerate a weightlist for words based on a word2vec CBOW (continuous\nbag of words) model.\n- First, use the whole raw corpus to train a word2vec model.\n- Then, use a sliding window and for each center word, predict the\nmost probable words given the current context.\n- Finally, If the center word isn't ambiguous, just increment the\ncount of its analysis.\nElse, For each ambiguous analysis of the center word, Count the\nnumber of similar words that aren't ambiguous AND have tags matching\nthe tag of the center word's analysis.\n\n##### Usage\nTODO: PRINT USAGE\n\n##### Example\nTODO\n\n##### Limitations\n- Training a word2vec model using a raw corpus is both time and\nmemory intense (that's why the script currently avoids loading the\nwhole data into memory at the same time)\n\n##### Future work\n- Refactor the word2vec training function\n\n## Evaluation\nTo evaluate and compare the performance of the weighting methods,\ncross validation is used.\n\n### eval/corpus_split.py\nA script to divide a tagged corpus into n folds (where n is a\nparameter)\n\n#### Usage\nTODO: PRINT USAGE\n\n### eval/\\*\\_fit.py\nEach weighting method has a script for training n models and report\nthe metrics for each one in a tabular form\n#### Usage\nTODO: PRINT USAGE\n\n### Example\n```\nREPO=\"../../apertium-kaz\"\nBIN=\"${REPO}/kaz.automorf.bin\"\nTAGGED_CORPUS=\"${REPO}/corpus/kaz.tagged\"\nUNTAGGED_CORPUS=\"${REPO}/kaz-clean\"\nCONSTRAINT_GRAMMAR=\"${REPO}/kaz.rlx.bin\"\nBASE_DIR=$(mktemp -d)\nFOLDS_DIR=\"folds\"\nCLEANED_CORPUS=\"${BASE_DIR}/kaz.cleaned\"\n\napertium-cleanstream -n \u003c \"$TAGGED_CORPUS\" \u003e \"$CLEANED_CORPUS\"\npython eval/corpus_split.py -o \"$FOLDS_DIR\" \"$CLEANED_CORPUS\"\n\npython eval/unigram_fit.py -i \"$FOLDS_DIR\" -b \"$BIN\" -o temp_uni_bin\npython eval/equalweight_fit.py -i \"$FOLDS_DIR\" -b \"$BIN\" -o temp_eq_bin\npython eval/constraintgrammar_fit.py -i \"$FOLDS_DIR\" -cg \"$CONSTRAINT_GRAMMAR\" -corpus \"$UNTAGGED_CORPUS\" -b \"$BIN\" -o temp_cg_bin\npython eval/analysis_length_fit.py -i \"$FOLDS_DIR\" -b \"$BIN\" -o temp_ana_bin\npython eval/w2v_fit.py -i \"$FOLDS_DIR\" -b \"$BIN\" -o temp_w2v_bin -corpus \"$UNTAGGED_CORPUS\"\n\necho 'Uni'\npython eval/metrics_report.py -i \"$FOLDS_DIR\" -b temp_uni_bin\necho 'Eq'\npython eval/metrics_report.py -i \"$FOLDS_DIR\" -b temp_eq_bin\necho 'Cg'\npython eval/metrics_report.py -i \"$FOLDS_DIR\" -b temp_cg_bin\necho 'Length'\npython eval/metrics_report.py -i \"$FOLDS_DIR\" -b temp_ana_bin\necho 'W2V'\npython eval/metrics_report.py -i \"$FOLDS_DIR\" -b temp_w2v_bin\n```\n### Results\nTODO: Add the results on apertium repos\n\n## Appendix\n### Xerox regexp\nAfter a series of shot-gun debugging, I found this file\n(ftp://ftp.cis.upenn.edu/pub/cis639/public_html/docs/xfst.html)\nand it was a great help in understanding how to use XEROX regexp them\n-especially the `Commands \u003c-\u003e RE operators` section.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapertium%2Fapertium-weighting-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapertium%2Fapertium-weighting-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapertium%2Fapertium-weighting-tools/lists"}