{"id":15620768,"url":"https://github.com/akb89/witokit","last_synced_at":"2025-04-28T17:14:52.588Z","repository":{"id":62589284,"uuid":"156668425","full_name":"akb89/witokit","owner":"akb89","description":"A Python toolkit to generate a tokenized dump of Wikipedia for NLP","archived":false,"fork":false,"pushed_at":"2024-05-03T19:48:10.000Z","size":49,"stargazers_count":11,"open_issues_count":1,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-28T17:14:44.671Z","etag":null,"topics":["dump","multilingual","nlp","tokenize","wikipedia","wikipedia-dump"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/akb89.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-11-08T07:39:49.000Z","updated_at":"2024-12-30T22:23:54.000Z","dependencies_parsed_at":"2024-10-22T20:47:03.656Z","dependency_job_id":null,"html_url":"https://github.com/akb89/witokit","commit_stats":{"total_commits":51,"total_committers":2,"mean_commits":25.5,"dds":"0.019607843137254943","last_synced_commit":"ccdb6054a8b3591f3d7a840cf02e2c924df20ef5"},"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akb89%2Fwitokit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akb89%2Fwitokit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akb89%2Fwitokit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akb89%2Fwitokit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/akb89","download_url":"https://codeload.github.com/akb89/witokit/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251352624,"owners_count":21575864,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dump","multilingual","nlp","tokenize","wikipedia","wikipedia-dump"],"created_at":"2024-10-03T09:02:28.051Z","updated_at":"2025-04-28T17:14:52.568Z","avatar_url":"https://github.com/akb89.png","language":"Python","funding_links":[],"categories":["Urdu Datasets"],"sub_categories":["Urdu Monolingual Corpora"],"readme":"# WiToKit\n[![GitHub release][release-image]][release-url]\n[![Build][travis-image]][travis-url]\n[![MIT License][license-image]][license-url]\n\nWelcome to `WiToKit`, a Python toolkit to download and generate preprocessed Wikipedia dumps for all languages.\n\nWiToKit can be used to converts a Wikipedia archive into a single .txt file, one (tokenized) sentence per line.\n\n*Note: WiToKit currently only supports `xx-pages-articles.xml.xx.bz2` Wikipedia archives corresponding to articles, templates, media/file descriptions, and primary meta-pages.*\n\n## Install\n\nAfter a git clone, run:\n\n```bash\npython3 setup.py install\n```\n\n## Use\n\n### Download\nTo download a .bz2-compressed Wikipedia XML dump, do:\n```bash\nwitokit download ⁠\\\n  --lang lang_wp_code \\\n  --date wiki_date \\\n  --output /abs/path/to/output/dir/where/to/store/bz2/archives \\\n  --num-threads num_cpu_threads\n```\n\nFor example, to download the latest English Wikipedia, do:\n```bash\nwitokit download ⁠--lang en --date latest --output /abs/path/to/output/dir --num-threads 2\n```\n\nThe `--lang` parameter expects the WP (language) code corresponding\nto the desired Wikipedia archive.\nCheck out the full list of Wikipedias with their corresponding WP codes [here](https://en.wikipedia.org/wiki/List_of_Wikipedias).\n\nThe `--date` parameter expects a string corresponding to one of the dates\nfound under the Wikimedia dump site corresponding to a given Wikipedia dump\n(e.g. https://dumps.wikimedia.org/enwiki/ for the English Wikipedia).\n\n**Important** Keep num-threads \u003c= 3 to avoid rejection from Wikimedia servers\n\n### Extract\nTo extract the content of the downloaded .bz2 archives, do:\n\n```bash\nwitokit extract \\\n  --input /abs/path/to/downloaded/wikipedia/bz2/archives \\\n  --num-threads num_cpu_threads\n```\n\n### Process\nTo preprocess the content of the extracted XML archives and output a single .txt file, tokenize, one sentence per line:\n```bash\nwitokit process \\\n  --input /abs/path/to/wikipedia/extracted/xml/archives \\\n  --output /abs/path/to/single/output/txt/file \\\n  --lower \\  # if set, will lowercase text\n  --num-threads num_cpu_threads\n```\n\nPreprocessing for all languages is performed with [Polyglot](https://github.com/aboSamoor/polyglot).\n\n### Sample\nYou can also use WiToKit to sample the content of a preprocess .txt file, using:\n```bash\nwitokit sample \\\n  --input /abs/path/to/witokit/preprocessed/txt/file \\\n  --percent \\  # percentage of total lines to keep\n  --balance  # if set, will balance sampling, otherwise, will take top n sentences only\n```\n\n[release-image]:https://img.shields.io/github/release/akb89/witokit.svg?style=flat-square\n[release-url]:https://github.com/akb89/witokit/releases/latest\n[pypi-image]:https://img.shields.io/pypi/v/witokit.svg?style=flat-square\n[pypi-url]:https://pypi.org/project/witokit/\n[travis-image]:https://img.shields.io/travis/akb89/witokit.svg?style=flat-square\n[travis-url]:https://travis-ci.org/akb89/witokit\n[license-image]:http://img.shields.io/badge/license-MIT-000000.svg?style=flat-square\n[license-url]:LICENSE.txt\n[req-url]:https://requires.io/github/akb89/witokit/requirements/?branch=master\n[req-image]:https://img.shields.io/requires/github/akb89/witokit.svg?style=flat-square\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakb89%2Fwitokit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fakb89%2Fwitokit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakb89%2Fwitokit/lists"}