{"id":18887162,"url":"https://github.com/thunlp-mt/mask-align","last_synced_at":"2025-07-13T02:05:50.862Z","repository":{"id":49953203,"uuid":"365977676","full_name":"THUNLP-MT/Mask-Align","owner":"THUNLP-MT","description":"Code for our paper \"Mask-Align: Self-Supervised Neural Word Alignment\" in ACL 2021","archived":false,"fork":false,"pushed_at":"2021-05-10T09:00:37.000Z","size":2438,"stargazers_count":60,"open_issues_count":7,"forks_count":20,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-28T10:54:19.137Z","etag":null,"topics":["machine-translation","self-supervised-learning","word-alignment"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/THUNLP-MT.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-05-10T08:52:30.000Z","updated_at":"2024-05-30T09:52:44.000Z","dependencies_parsed_at":"2022-08-23T16:40:15.755Z","dependency_job_id":null,"html_url":"https://github.com/THUNLP-MT/Mask-Align","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FMask-Align","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FMask-Align/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FMask-Align/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FMask-Align/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/THUNLP-MT","download_url":"https://codeload.github.com/THUNLP-MT/Mask-Align/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248973100,"owners_count":21191913,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["machine-translation","self-supervised-learning","word-alignment"],"created_at":"2024-11-08T07:34:54.081Z","updated_at":"2025-04-14T22:33:46.612Z","avatar_url":"https://github.com/THUNLP-MT.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Mask-Align: Self-Supervised Neural Word Alignment\n\nThis is the implementation of our work \u003ca href=\"https://arxiv.org/abs/2012.07162\"\u003eMask-Align: Self-Supervised Neural Word Alignment\u003c/a\u003e. \n\n```bibtex\n@inproceedings{chen2021maskalign,\n   title={Mask-Align: Self-Supervised Neural Word Alignment},\n   author={Chi Chen and Maosong Sun and Yang Liu},\n   booktitle={Association for Computational Linguistics (ACL)},\n   year={2021}\n}\n```\n\nThe implementation is built on top of [THUMT](https://github.com/THUNLP-MT/THUMT).\n\n## Contents \n* [Introduction](#introduction)\n* [Prerequisites](#prerequisites)\n* [Usage](#usage)\n* [Configs](#configs)\n* [Visualization](#visualization)\n* [Contact](#contact)\n\n## Introduction\n\n**Mask-Align** is a self-supervised neural word aligner.\nIt parallelly masks out each target token and predicts it conditioned on both source and the remaining target tokens. The source token that contributes most to recovering a masked target token will be aligned to that target token.\n\n\n![](figs/mask-align.gif)\n\n## Prerequisites\n\n* PyTorch\n* NLTK\n* remi *\n* pyecharts *\n* pandas *\n* matplotlib *\n* seaborn *\n\n\\*: *optional*, only used for [Visualization](#visualization).\n\n## Usage\n\n### Data Preparation\n\nTo get the data used in our paper, you can follow the instructions in [https://github.com/lilt/alignment-scripts](https://github.com/lilt/alignment-scripts).\n\nTo train an aligner with your own data, you should pre-process it yourself. Usually this includes tokenization, BPE, etc. You can find a simple guide [here](https://github.com/THUNLP-MT/THUMT/blob/master/docs/walkthrough.md#data-preparation). \n\nNow we have the pre-processed parallel training data (`train.src`, `train.tgt`), validation data (optional) (`valid.src`, `valid.tgt`) and test data (`test.src`, `test.tgt`). An example 3-sentence German–English parallel training corpus is:\n\n```\n# train.src\nwiederaufnahme der sitzungsperiode\nfrau präsidentin , zur geschäfts @@ordnung .\nich bitte sie , sich zu einer schweigeminute zu erheben .\n\n# train.tgt\nresumption of the session\nmadam president , on a point of order .\nplease rise , then , for this minute ' s silence .\n```\n\nThe next step is to shuffle the training set, which proves to be helpful for improving the results.\n\n```python\npython thualign/scripts/shuffle_corpus.py --corpus train.src train.tgt\n```\n\nThe resulting files `train.src.shuf` and `train.tgt.shuf` rearrange the sentence pairs randomly.\n\nThen we need to generate vocabulary from the training set.\n\n```python\npython thualign/scripts/build_vocab.py train.src.shuf vocab.train.src\npython thualign/scripts/build_vocab.py train.tgt.shuf vocab.train.tgt\n```\n\nThe resulting files `vocab.train.src.txt` and `vocab.train.tgt.txt` are final source and target vocabularies used for model training.\n\n### Training\n\nAll experiments are configured via config files in `thualign/configs`, see [Configs](#configs) for more details.. We provide an example config file `thualign/configs/user/example.config`. You can easily use it by **making three changes**: \n\n1. change `device_list`, `update_cycle` and `batch_size` to match your machine configuration; \n\n2. change ` exp_dir` and `output` to your own experiment directory\n\n3. change `train/valid/test_input` and `vocab` to your data paths;\n\nWhen properly configured, you can use the following command to train an alignment model described in the config file\n\n```sh\nbash thualign/bin/train.sh -s thualign/configs/user/example.config\n```\n\nor more simply\n\n```sh\nbash thualign/bin/train.sh -s example\n```\n\nThe configuration file is an **INI** file and is parsed through [configparser](https://docs.python.org/zh-cn/3.7/library/configparser.html). By adding a new section, you can easily customize some configs while keep other configs unchanged. \n\n```ini\n[DEFAULT]\n...\n\n[small_budget]\nbatch_size = 4500\nupdate_cycle = 8\ndevice_list = [0]\nhalf = False\n```\n\nUse `-e` option to run this `small_budget` section\n\n```sh\nbash thualign/bin/train.sh -s example -e small_budget\n```\n\nYou can also monitor the training process through [tensorboard](https://www.tensorflow.org/tensorboard/)\n\n```sh\ntensorboard --logdir=[output]\n```\n\n### Test\n\nAfter training, the following command can be used to generate attention weights (`-g`), generate data for attention visualization (`-v`), and test its AER (`-t`) if `test_ref` is provided.\n\n```sh\nbash thualign/bin/test.sh -s [CONFIG] -e [EXP] -gvt\n```\n\nFor example, to test the model trained with the configs in `example.config`\n\n```sh\nbash thualign/bin/test.sh -s example -gvt\n```\n\nYou might get the following output\n```\nalignment-soft.txt: 14.4% (87.7%/83.5%/9467)\n```\n\nThe alignment results (`alignment.txt`) along with other test results are stored in `[output]/test` by default.\n\n## Configs\n\nMost of the configuration of Mask-Align is done through configuration files in `thualign/configs`. The model reads the basic configs first, followed by the user-defined configs.\n\n### Basic Config\n\nPredefined configs for experiments to use.\n\n- `base.config`: basic configs for training, validation and test\n\n- `model.config`: define different models with their hyperparameters\n\n### User Config\n\nCustomized configs that must describe the following configuration and maybe other experiment-specific parameters:\n\n- `train/valid/test_input`: paths of input parallel corpuses\n- `vocab`: paths of vocabulary files generated from `thualign/scripts/build_vocab.py`\n- `output`: path to save the model outputs\n- `model`: which model to use\n- `batch_size`: the batch size (number of tokens) used in the training stage.\n- `update_cycle`: the number of iterations for updating model parameters. The default value is 1. If you have only 1 GPU and want to obtain the same translation performance with using 4 GPUs, simply set this parameter to 4. Note that the training time will also be prolonged.\n- `device_list`: the list of GPUs to be used in training. Use the `nvidia-smi` command to find unused GPUs. If the unused GPUs are *gpu0* and *gpu1*, set this parameter as `device_list=[0,1]`.\n- `half`: set this to `True` if you wish to use half-precision training. This will speeds up the training procedure. Make sure that you have the GPUs with half-precision support.\n\nHere is a minimal experiment config:\n\n```ini\n### thualign/configs/user/example.config\n[DEFAULT]\n\ntrain_input = ['train.src', 'train.tgt']\nvalid_input = ['valid.src', 'valid.tgt']\nvocab = ['vocab.src.txt', 'vocab.tgt.txt']\ntest_input = ['test.src', 'test.tgt']\ntest_ref = test.talp\n\nexp_dir = exp\nlabel = agree_deen\noutput = ${exp_dir}/${label}\n\nmodel = mask_align\n\nbatch_size = 9000\nupdate_cycle = 1\ndevice_list = [0,1,2,3]\nhalf = True\n```\n\n## Visualization\n\nTo better understand and analyze the model, Mask-Align supports the following two types of visulizations.\n\n### Training Visualization\n\nAdd `eval_plot = True` in your config file to turn on visualization during training. This will plot 5 attention maps from evaluation in the tensorboard. \n\n\u003cimg src=\"figs/viz_training.png\" width=500/\u003e\n\nThese packages are required for training visualization:\n* pandas\n* matplotlib\n* seaborn\n\n### Attention Visualization\n\nUse `-v` in the test command to generate `alignment_vizdata.pt` first. It is stored in `[output]/test` by default. To visualize it, using this script\n\n```sh\npython thualign/scripts/visualize.py [output]/test/alignment_vizdata.pt [--port PORT]\n```\n\nThis will start a local service that plots the attention weights for all the test sentence pairs. You can access it through a web browser.\n\n\u003cimg src=\"figs/viz_attn.png\" width=500/\u003e\n\nThese packages are required for training visualization:\n* remi\n* pyecharts\n\n## Contact\n\nIf you have questions, suggestions and bug reports, please email [chenchi19@mails.tsinghua.edu.cn](mailto:chenchi19@mails.tsinghua.edu.cn).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthunlp-mt%2Fmask-align","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthunlp-mt%2Fmask-align","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthunlp-mt%2Fmask-align/lists"}