{"id":34113957,"url":"https://github.com/time-in-translation/preprocess-corpora","last_synced_at":"2026-04-02T01:04:26.020Z","repository":{"id":38272085,"uuid":"144151807","full_name":"time-in-translation/preprocess-corpora","owner":"time-in-translation","description":"Creating (parallel) corpora from scratch using Uplug tooling","archived":false,"fork":false,"pushed_at":"2023-06-29T14:30:11.000Z","size":788,"stargazers_count":2,"open_issues_count":1,"forks_count":1,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-12-17T03:25:05.371Z","etag":null,"topics":["parallel-corpus","preprocessing"],"latest_commit_sha":null,"homepage":"https://time-in-translation.hum.uu.nl","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/time-in-translation.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-08-09T12:48:16.000Z","updated_at":"2023-07-18T06:52:16.000Z","dependencies_parsed_at":"2023-02-01T05:01:08.445Z","dependency_job_id":"4fd056af-b49e-4fda-92ef-4ea2d134c694","html_url":"https://github.com/time-in-translation/preprocess-corpora","commit_stats":{"total_commits":59,"total_committers":3,"mean_commits":"19.666666666666668","dds":"0.13559322033898302","last_synced_commit":"40e3e34955e45fa46b5c75bfe2f05282c151f36d"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/time-in-translation/preprocess-corpora","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/time-in-translation%2Fpreprocess-corpora","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/time-in-translation%2Fpreprocess-corpora/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/time-in-translation%2Fpreprocess-corpora/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/time-in-translation%2Fpreprocess-corpora/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/time-in-translation","download_url":"https://codeload.github.com/time-in-translation/preprocess-corpora/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/time-in-translation%2Fpreprocess-corpora/sbom","scorecard":{"id":885757,"data":{"date":"2025-08-11","repo":{"name":"github.com/time-in-translation/preprocess-corpora","commit":"40e3e34955e45fa46b5c75bfe2f05282c151f36d"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":1.7,"checks":[{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Code-Review","score":0,"reason":"Found 0/28 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 2 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":0,"reason":"26 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: PYSEC-2024-230 / GHSA-248v-346w-9cwc","Warn: Project is vulnerable to: PYSEC-2023-135 / GHSA-xqr8-7jwr-rhp7","Warn: Project is vulnerable to: PYSEC-2024-60 / GHSA-jjg7-2v4v-x38h","Warn: Project is vulnerable to: GHSA-cpwx-vrp4-4pq7","Warn: Project is vulnerable to: GHSA-gmj6-6f8f-6699","Warn: Project is vulnerable to: GHSA-h5c8-rqwp-cp95","Warn: Project is vulnerable to: GHSA-h75v-3vvj-5mfj","Warn: Project is vulnerable to: GHSA-q2x7-8rv6-6q7h","Warn: Project is vulnerable to: PYSEC-2024-167 / GHSA-cgvx-9447-vcch","Warn: Project is vulnerable to: GHSA-8qvm-5x2c-j2w7","Warn: Project is vulnerable to: GHSA-mr82-8j83-vxmv","Warn: Project is vulnerable to: GHSA-9hjg-9r4m-mvj7","Warn: Project is vulnerable to: GHSA-9wx4-h78v-vm56","Warn: Project is vulnerable to: PYSEC-2023-74 / GHSA-j8r2-6x86-q33q","Warn: Project is vulnerable to: GHSA-3749-ghw9-m3mg","Warn: Project is vulnerable to: PYSEC-2025-41 / GHSA-53q9-r3pm-6pq6","Warn: Project is vulnerable to: PYSEC-2024-252 / GHSA-5pcm-hx3q-hm94","Warn: Project is vulnerable to: GHSA-887c-mr87-cxwp","Warn: Project is vulnerable to: PYSEC-2024-251 / GHSA-pg7h-5qx3-wjr3","Warn: Project is vulnerable to: PYSEC-2024-250","Warn: Project is vulnerable to: PYSEC-2024-259","Warn: Project is vulnerable to: GHSA-g7vv-2v7x-gj9p","Warn: Project is vulnerable to: GHSA-34jh-p97f-mpxf","Warn: Project is vulnerable to: PYSEC-2023-212 / GHSA-g4mx-q9vg-27p4","Warn: Project is vulnerable to: GHSA-pq67-6m6q-mj2v","Warn: Project is vulnerable to: PYSEC-2023-192 / GHSA-v845-jxx5-vc9f"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-24T09:58:15.606Z","repository_id":38272085,"created_at":"2025-08-24T09:58:15.606Z","updated_at":"2025-08-24T09:58:15.606Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31293631,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-01T21:15:39.731Z","status":"ssl_error","status_checked_at":"2026-04-01T21:15:34.046Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["parallel-corpus","preprocessing"],"created_at":"2025-12-14T19:22:10.425Z","updated_at":"2026-04-02T01:04:26.006Z","avatar_url":"https://github.com/time-in-translation.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# preprocess-corpora\n\nThis repository contains Python scripts to preprocess and sentence-align parallel (or monolingual) corpora. \nThe repository originally heavily relied upon the software applications [Uplug](https://bitbucket.org/tiedemann/uplug/src/master/) and (in lesser respect) [TreeTagger](http://www.cis.uni-muenchen.de/~schmid/tools/TreeTagger/) to work.\nNowadays, we also have support for [NLTK](https://www.nltk.org/), [Spacy](https://spacy.io/), and [Stanza](https://stanfordnlp.github.io/stanza/).\n\n## Installation\n\nFirst, make sure to have installed [Uplug](https://bitbucket.org/tiedemann/uplug/src/master/) and [TreeTagger](http://www.cis.uni-muenchen.de/~schmid/tools/TreeTagger/).\nUsers intending to use [NLTK](https://www.nltk.org/) and/or [Spacy](https://spacy.io/) and/or [Stanza](https://stanfordnlp.github.io/stanza/) can skip this step.\n\nThen, create a [virtual environment](https://docs.python.org/3/library/venv.html) and activate it:\n\n    $ python -m venv venv\n    $ source venv/bin/activate\n\nThen, install the requirements in this virtual environment via:\n\n    $ pip install -r requirements.txt\n\nFinally, create the executables `preprocess` and `align` via:    \n \n    $ pip install --editable .\n\nIf you intend to use [NLTK](https://www.nltk.org/) for tokenization, be sure to download the [Punkt models](https://www.nltk.org/api/nltk.tokenize.html?highlight=punkt#module-nltk.tokenize.punkt):\n\n    $ python\n    \u003e\u003e\u003e import nltk\n    \u003e\u003e\u003e nltk.download('punkt')\n\nIf you intend to use [Spacy](https://spacy.io/) for tokenization and tagging, be sure to download the language models (small models suffice, be sure to replace `en_core_web_sm` with the model for your language):\n\n    $ python -m spacy download en_core_web_sm\n\nWith [Stanza](https://stanfordnlp.github.io/stanza/), models will be downloaded on-the-fly.\n\n## Usage\n\n### Preprocessing\n\nThe `preprocess` script allows preprocessing raw text and then to tokenize and tag the text in the [XML format used in OPUS](http://opus.nlpl.eu/).\n\nRun `preprocess` to process all unformatted .txt-files in a folder. \n\nUsage:\n\n    $ preprocess [OPTIONS] FOLDER_IN FOLDER_OUT {de|en|nl|sv|ca|es|fr|it|pt|ro|bg|pl|ru|br|hi|ar|mx}\n\nOptions:\n\n- `--from_word` to use .docx-files as input, rather than .txt-files.\n- `--tokenizer` to tokenize the files; choose either:\n    - `uplug` (requires installation of Uplug (and language support in Uplug)).\n    - `nltk` (requires installation of the Punkt models (and language support in Punkt))\n    - `spacy` (requires installation of the Spacy models (and language support in Spacy))\n    - `stanza` (requires installation of the Stanza models (and language support in Stanza))\n    - `treetagger` (use the very naive tokenization in the *treetagger-xml* package (not recommended!))\n- `--tag` to tag the files (requires installation of Spacy or TreeTagger (and language support))\n- `--dialog` to detect dialogs in the generated .xml-files.\n\n\n### Alignment\n\nRun `align` to sentence-align .xml-files in a working directory. Requires installation of Uplug.\n\nUsage:\n\n    $ align [OPTIONS] WORKING_DIR [[de|en|nl|sv|ca|es|fr|it|pt|ro|bg|pl|ru|br|hi|ar|mx]]...\n\n### Supported languages\n\n| Genus      | Language   | ISO | Preprocessing | Tokenization | Tagging |\n|------------|------------|-----|:-------------:|:------------:|:-------:|\n| Germanic   | German     | de  |       ✔       |      ✔       |    ✔    |\n| Germanic   | English    | en  |       ✔       |      ✔       |    ✔    |\n| Germanic   | Dutch      | nl  |       ✔       |      ✔       |    ✔    |\n| Germanic   | Swedish    | sv  |       ✔       |   ✔ (NLTK)   |    ✗    |\n| Romance    | Catalan    | ca  |       ✔       |  ✔ (Stanza)  |    ✔    |\n| Romance    | Spanish    | es  |       ✔       |      ✔       |    ✔    |\n| Romance    | French     | fr  |       ✔       |      ✔       |    ✔    |\n| Romance    | Italian    | it  |       ✔       |      ✔       |    ✔    |\n| Romance    | Portuguese | pt  |       ✔       |  ✔ (Uplug)   |    ✔    |\n| Romance    | Romanian   | ro  |       ✔       |  ✔ (Stanza)  |    ✔    |\n| Hellenic   | Greek      | el  |       ✔       |  ✔ (Stanza)  |    ✔    |\n| Slavic     | Belarusian | be  |       ✔       |  ✔ (Stanza)  |    ✔    |\n| Slavic     | Bulgarian  | bg  |       ✔       |  ✔ (Stanza)  |    ✔    |\n| Slavic     | Czech      | cs  |       ✔       |  ✔ (Stanza)  |    ✔    |\n| Slavic     | Croatian   | hr  |       ✔       |  ✔ (Spacy)   |    ✔    |\n| Slavic     | Lithuanian | lt  |       ✔       |  ✔ (Spacy)   |    ✔    |\n| Slavic     | Latvian    | lv  |       ✔       |  ✔ (Stanza)  |    ✔    |\n| Slavic     | Macedonian | mk  |       ✔       |  ✔ (Spacy)   |    ✔    |\n| Slavic     | Polish     | pl  |       ✔       |  ✔ (Spacy)   |    ✔    |\n| Slavic     | Russian    | ru  |       ✔       |  ✔ (Uplug)   |    ✔    |\n| Slavic     | Slovak     | sk  |       ✔       |  ✔ (Stanza)  |    ✔    |\n| Slavic     | Slovenian  | sl  |       ✔       |  ✔ (Stanza)  |    ✔    |\n| Slavic     | Serbian    | sr  |       ✔       |  ✔ (Stanza)  |    ✗    |\n| Slavic     | Ukrainian  | uk  |       ✔       |  ✔ (Spacy)   |    ✔    |\n| Celtic     | Breton     | br  |       ✔       |      ✗       |    ✗    |\n| Indo-Aryan | Hindi      | hi  |       ✔       |  ✔ (Stanza)  |    ✔    |\n\nSome comments:\n- For Dutch, for tokenization, Uplug can potentially use [Alpino](https://www.let.rug.nl/vannoord/alp/Alpino/) (recommended).\n- For Swedish, consider using [Stagger](https://www.ling.su.se/english/nlp/tools/stagger) for part-of-speech tagging.\n- Spanish varieties (Mexican Spanish (mx) and Rioplatense Spanish (ar)) are supported by referring to the Spanish parameters.\n- Note that the Portuguese NLTK Punkt parameters are based upon Brazilian Portuguese.\n- For Hindi, we use [RNNTagger](https://www.cis.uni-muenchen.de/~schmid/tools/RNNTagger/) instead of TreeTagger.\n\n## Tests\n\nRun the tests via\n\n    $ python -m unittest discover\n\nIn `preprocess_corpora/tests/data/alice`, you can find the example corpus used in the tests.\nThis corpus was compiled from Lewis Carroll's Alice in Wonderland and its translations into German, French, and Italian.\nThe source files are available through [Project Gutenberg](http://www.gutenberg.org/).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftime-in-translation%2Fpreprocess-corpora","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftime-in-translation%2Fpreprocess-corpora","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftime-in-translation%2Fpreprocess-corpora/lists"}