{"id":50200866,"url":"https://github.com/maveryn/punctuation-restoration","last_synced_at":"2026-06-11T14:00:38.512Z","repository":{"id":45251421,"uuid":"268829960","full_name":"maveryn/punctuation-restoration","owner":"maveryn","description":"Punctuation Restoration using Transformer Models for High-and Low-Resource Languages","archived":false,"fork":false,"pushed_at":"2024-07-29T06:43:01.000Z","size":11225,"stargazers_count":228,"open_issues_count":8,"forks_count":65,"subscribers_count":8,"default_branch":"master","last_synced_at":"2026-03-26T03:41:13.787Z","etag":null,"topics":["bangla","bert","deep-learning","punctuation-marks","punctuation-restoration","pytorch","transformer-models"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/maveryn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2020-06-02T14:54:10.000Z","updated_at":"2026-03-03T10:38:34.000Z","dependencies_parsed_at":"2022-08-12T11:51:00.940Z","dependency_job_id":null,"html_url":"https://github.com/maveryn/punctuation-restoration","commit_stats":null,"previous_names":["maveryn/punctuation-restoration"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/maveryn/punctuation-restoration","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maveryn%2Fpunctuation-restoration","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maveryn%2Fpunctuation-restoration/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maveryn%2Fpunctuation-restoration/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maveryn%2Fpunctuation-restoration/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/maveryn","download_url":"https://codeload.github.com/maveryn/punctuation-restoration/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maveryn%2Fpunctuation-restoration/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34201842,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-11T02:00:06.485Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bangla","bert","deep-learning","punctuation-marks","punctuation-restoration","pytorch","transformer-models"],"created_at":"2026-05-25T22:00:42.189Z","updated_at":"2026-06-11T14:00:38.506Z","avatar_url":"https://github.com/maveryn.png","language":"Python","funding_links":[],"categories":["Developer Tools \u0026 Libraries"],"sub_categories":["🚀 How to contribute"],"readme":"# Punctuation Restoration using Transformer Models\n\nThis repository contins official implementation of the paper [*Punctuation Restoration using Transformer Models for High-and Low-Resource Languages*](https://aclanthology.org/2020.wnut-1.18/) accepted at the EMNLP workshop [W-NUT 2020](http://noisy-text.github.io/2020/).\n\n\n## Data\n\n#### English\nEnglish datasets are provided in `data/en` directory. These are collected from [here](https://drive.google.com/file/d/0B13Cc1a7ebTuMElFWGlYcUlVZ0k/view).\n\n#### Bangla\nBangla datasets are provided in `data/bn` directory.\n\n\n## Model Architecture\nWe fine-tune a Transformer architecture based language model (e.g., BERT) for the punctuation restoration task.\nTransformer encoder is followed by a bidirectional LSTM and linear layer that predicts target punctuation token at\neach sequence position.\n![](./assets/model_architectue.png)\n\n\n## Dependencies\nInstall PyTorch following instructions from [PyTorch website](https://pytorch.org/get-started/locally/). Remaining\ndependencies can be installed with the following command\n```bash\npip install -r requirements.txt\n```\n\n\n## Training\nTo train punctuation restoration model with optimal parameter settings for English run the following command\n```\npython src/train.py --cuda=True --pretrained-model=roberta-large --freeze-bert=False --lstm-dim=-1 \n--language=english --seed=1 --lr=5e-6 --epoch=10 --use-crf=False --augment-type=all  --augment-rate=0.15 \n--alpha-sub=0.4 --alpha-del=0.4 --data-path=data --save-path=out\n```\nTo train for Bangla the corresponding command is\n```\npython src/train.py --cuda=True --pretrained-model=xlm-roberta-large --freeze-bert=False --lstm-dim=-1 \n--language=bangla --seed=1 --lr=5e-6 --epoch=10 --use-crf=False --augment-type=all  --augment-rate=0.15 \n--alpha-sub=0.4 --alpha-del=0.4 --data-path=data --save-path=out\n```\n\n#### Supported models for English\n```\nbert-base-uncased\nbert-large-uncased\nbert-base-multilingual-cased\nbert-base-multilingual-uncased\nxlm-mlm-en-2048\nxlm-mlm-100-1280\nroberta-base\nroberta-large\ndistilbert-base-uncased\ndistilbert-base-multilingual-cased\nxlm-roberta-base\nxlm-roberta-large\nalbert-base-v1\nalbert-base-v2\nalbert-large-v2\n```\n\n#### Supported models for Bangla\n```\nbert-base-multilingual-cased\nbert-base-multilingual-uncased\nxlm-mlm-100-1280\ndistilbert-base-multilingual-cased\nxlm-roberta-base\nxlm-roberta-large\n```\n\n\n## Pretrained Models\nYou can find pretrained mdoels for RoBERTa-large model with augmentation for English [here](https://drive.google.com/file/d/17BPcnHVhpQlsOTC8LEayIFFJ7WkL00cr/view?usp=sharing)  \nXLM-RoBERTa-large model with augmentation for Bangla can be found [here](https://drive.google.com/file/d/1X2udyT1XYrmCNvWtFpT_6jrWsQejGCBW/view?usp=sharing)\n\n\n\n## Inference\nYou can run inference on unprocessed text file to produce punctuated text using `inference` module. Note that if the \ntext already contains punctuation they are removed before inference. \n\nExample script for English:\n```bash\npython inference.py --pretrained-model=roberta-large --weight-path=roberta-large-en.pt --language=en \n--in-file=data/test_en.txt --out-file=data/test_en_out.txt\n```\nThis should create the text file with following output:\n```text\nTolkien drew on a wide array of influences including language, Christianity, mythology, including the Norse Völsunga saga, archaeology, especially at the Temple of Nodens, ancient and modern literature and personal experience. He was inspired primarily by his profession, philology. his work centred on the study of Old English literature, especially Beowulf, and he acknowledged its importance to his writings. \n```\n\nSimilarly, For Bangla\n```bash\npython inference.py --pretrained-model=xlm-roberta-large --weight-path=xlm-roberta-large-bn.pt --language=bn  \n--in-file=data/test_bn.txt --out-file=data/test_bn_out.txt\n```\nThe expected output is\n```text\nবিংশ শতাব্দীর বাংলা মননে কাজী নজরুল ইসলামের মর্যাদা ও গুরুত্ব অপরিসীম। একাধারে কবি, সাহিত্যিক, সংগীতজ্ঞ, সাংবাদিক, সম্পাদক, রাজনীতিবিদ এবং সৈনিক হিসেবে অন্যায় ও অবিচারের বিরুদ্ধে নজরুল সর্বদাই ছিলেন সোচ্চার। তার কবিতা ও গানে এই মনোভাবই প্রতিফলিত হয়েছে। অগ্নিবীণা হাতে তার প্রবেশ, ধূমকেতুর মতো তার প্রকাশ। যেমন লেখাতে বিদ্রোহী, তেমনই জীবনে কাজেই \"বিদ্রোহী কবি\"। তার জন্ম ও মৃত্যুবার্ষিকী বিশেষ মর্যাদার সঙ্গে উভয় বাংলাতে প্রতি বৎসর উদযাপিত হয়ে থাকে। \n```\n\nPlease note that *Comma* includes commas, colons and dashes, *Period* includes full stops, exclamation marks \nand semicolons and *Question* is just question marks. \n\n\n## Test\nTrained models can be tested on processed data using `test` module to prepare result.\n\nFor example, to test the best preforming English model run following command\n```bash\npython src/test.py --pretrained-model=roberta-large --lstm-dim=-1 --use-crf=False --data-path=data/test\n--weight-path=weights/roberta-large-en.pt --sequence-length=256 --save-path=out\n```\nPlease provide corresponding arguments for `pretrained-model`, `lstm-dim`, `use-crf` that were used during training the\nmodel. This will run test for all data available in `data-path` directory.\n\n\n## Cite this work\n\n```\n@inproceedings{alam-etal-2020-punctuation,\n    title = \"Punctuation Restoration using Transformer Models for High-and Low-Resource Languages\",\n    author = \"Alam, Tanvirul  and\n      Khan, Akib  and\n      Alam, Firoj\",\n    booktitle = \"Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020)\",\n    month = nov,\n    year = \"2020\",\n    address = \"Online\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://www.aclweb.org/anthology/2020.wnut-1.18\",\n    pages = \"132--142\",\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaveryn%2Fpunctuation-restoration","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaveryn%2Fpunctuation-restoration","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaveryn%2Fpunctuation-restoration/lists"}