{"id":13715257,"url":"https://github.com/ottokart/punctuator","last_synced_at":"2025-05-07T04:30:40.312Z","repository":{"id":28953552,"uuid":"32479621","full_name":"ottokart/punctuator","owner":"ottokart","description":"An LSTM RNN for restoring missing punctuation in unsegmented text.","archived":false,"fork":false,"pushed_at":"2016-09-24T14:46:02.000Z","size":615,"stargazers_count":79,"open_issues_count":4,"forks_count":26,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-11-14T03:34:25.735Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ottokart.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-03-18T19:27:35.000Z","updated_at":"2024-06-17T14:34:14.000Z","dependencies_parsed_at":"2022-08-17T18:45:25.652Z","dependency_job_id":null,"html_url":"https://github.com/ottokart/punctuator","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ottokart%2Fpunctuator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ottokart%2Fpunctuator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ottokart%2Fpunctuator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ottokart%2Fpunctuator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ottokart","download_url":"https://codeload.github.com/ottokart/punctuator/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252813619,"owners_count":21808360,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T00:00:56.676Z","updated_at":"2025-05-07T04:30:39.492Z","avatar_url":"https://github.com/ottokart.png","language":"Python","funding_links":[],"categories":["List"],"sub_categories":[],"readme":"# Punctuator\nAn LSTM RNN for restoring missing punctuation in text.\nThere's now a bidirectional model also available: https://github.com/ottokart/punctuator2\n\nModel is trained in two stages (second stage is optional though):\n\n1. First stage is trained on punctuation annotated text. Here the model learns to restore puncutation based on textual features only.\n2. Optional second stage can be trained on punctuation *and* pause annotated text. In this stage the model learns to combine pause durations with textual features and adapts to the target domain. If pauses are omitted then only adaptation is performed. Second stage with pause durations can be used for example for restoring punctuation in automatic speech recognition system output.\n\n# Requirements\n* Python (tested with v.2.7.5)\n* Numpy (tested with v.1.8.0)\n\n# Requirements for data:\n\n* Vocabulary file to specify the input vocabulary of the model. One word per line.\n* Cleaned text files for training and validation of the first phase model. Punctuation marks that are not going to be restored by the model should be removed or replaced with appropriate substitute (e.g. questionmarks can be replaced with periods etc). Each punctuation symbol must be surrounded by spaces.\n\n  Example:\n  ```to be ,COMMA or not to be ,COMMA that is the question .PERIOD```\n* *(Optional)* Pause annotated text files for training and validation of the second phase model. These should be cleaned in the same way as the first phase data. Pause durations in seconds should be marked after each word with a special tag `\u003csil=0.200\u003e`. Punctuation mark, if any, must come after the pause tag.\n\n  Example:\n  ```to \u003csil=0.000\u003e be \u003csil=0.100\u003e ,COMMA or \u003csil=0.000\u003e not \u003csil=0.000\u003e to \u003csil=0.000\u003e be \u003csil=0.150\u003e ,COMMA that \u003csil=0.000\u003e is \u003csil=0.000\u003e the \u003csil=0.000\u003e question \u003csil=1.000\u003e .PERIOD```\n\n  Second phase data can also be without pause annotations to do just target domain adaptation (PHASE2['USE_PAUSES'] in **conf.py** should be changed accordingly then).\n  \nMake sure that first words of sentences don't have capitalized first letters. This would give the model unfair hints about period locations.\n\n# Configuration\n\nConfiguration is located in **conf.py**.\n\nPunctuation annotation symbols are specified in the PUNCTUATIONS dictionary. *.PERIOD* and *,COMMA* are the default and the dictionary should also include *space* (no punctuation).\n\nLocation of the vocabulary file is specified in VOCABULARY_FILE.\n\nThe locations of the data files can be configured in PHASE1['TRAIN_DATA'], PHASE1['DEV_DATA'], PHASE2['TRAIN_DATA'] and PHASE2['DEV_DATA'].\n\nChanging some configuration options (batch size, data files, pause usage, punctuations or vocabulary) may require deleting the *data* directory so the data files will be reconverted during the next run.\n\n# Usage\n\nRun `python main.py \u003cmodel_name\u003e`.\n\nModel name is optional. Default is 'model'.\n\nA small example dataset is also included. \n\n**Tools might not work properly yet!**\n\n\n# Citing\n\nThe software is described in [this](https://phon.ioc.ee/dokuwiki/lib/exe/fetch.php?media=people:tanel:interspeech2015-paper-punct.pdf) paper:\n\n    @inproceedings{tilk2015,\n      author    = {Ottokar Tilk and Tanel Alum{\\\"a}e},\n      title     = {{LSTM} for Punctuation Restoration in Speech Transcripts},\n      booktitle = {Interspeech 2015},\n      year      = {2015},\n      address   = {Dresden, Germany}\n    }\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fottokart%2Fpunctuator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fottokart%2Fpunctuator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fottokart%2Fpunctuator/lists"}