{"id":18862135,"url":"https://github.com/yohasebe/wp2txt","last_synced_at":"2025-04-04T20:11:38.764Z","repository":{"id":3510618,"uuid":"4568155","full_name":"yohasebe/wp2txt","owner":"yohasebe","description":"A command-line toolkit to extract text content and category data from Wikipedia dump files","archived":false,"fork":false,"pushed_at":"2023-05-13T12:01:00.000Z","size":19996,"stargazers_count":172,"open_issues_count":3,"forks_count":37,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-03-27T09:05:18.057Z","etag":null,"topics":["corpus","machine-learning","nlp","ruby","wikipedia","wikipedia-dump"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yohasebe.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2012-06-06T02:56:02.000Z","updated_at":"2025-03-03T04:05:25.000Z","dependencies_parsed_at":"2023-02-14T15:16:02.529Z","dependency_job_id":null,"html_url":"https://github.com/yohasebe/wp2txt","commit_stats":{"total_commits":130,"total_committers":6,"mean_commits":"21.666666666666668","dds":0.05384615384615388,"last_synced_commit":"3c4bc0269fd2f2c14f594b993b5f6df0ed08cd72"},"previous_names":[],"tags_count":27,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yohasebe%2Fwp2txt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yohasebe%2Fwp2txt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yohasebe%2Fwp2txt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yohasebe%2Fwp2txt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yohasebe","download_url":"https://codeload.github.com/yohasebe/wp2txt/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247242678,"owners_count":20907134,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["corpus","machine-learning","nlp","ruby","wikipedia","wikipedia-dump"],"created_at":"2024-11-08T04:33:24.081Z","updated_at":"2025-04-04T20:11:38.740Z","avatar_url":"https://github.com/yohasebe.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src='https://raw.githubusercontent.com/yohasebe/wp2txt/master/image/wp2txt-logo.svg' width=\"400\" /\u003e\n\nA command-line toolkit to extract text content and category data from Wikipedia dump files\n\n## About\n\nWP2TXT extracts text and category data from Wikipedia dump files (encoded in XML / compressed with Bzip2), removing MediaWiki markup and other metadata.\n\n## Changelog\n\n**May 2023**\n\n- Problems caused by too many parallel processors are addressed by setting the upper limit on the number of processors to 8. \n\n**April 2023**\n\n- File split/delete issues fixed\n\n**January 2023**\n\n- Bug related to command line arguments fixed\n- Code cleanup introducing Rubocop\n\n**December 2022**\n\n- Docker images available via Docker Hub\n\n**November 2022**\n\n- Code added to suppress \"Invalid byte sequence error\" when an ilegal UTF-8 character is input.\n\n**August 2022**\n\n- A new option `--category-only` has been added. When this option is enabled, only the title and category information of the article is extracted.\n- A new option `--summary-only` has been added. If this option is enabled, only the title, category information, and opening paragraphs of the article will be extracted.\n- Text conversion with the current version of WP2TXT is *more than 2x times faster* than the previous version due to parallel processing of multiple files (the rate of speedup depends on the CPU cores used for processing).\n\n## Screenshot\n\n\u003cimg src='https://raw.githubusercontent.com/yohasebe/wp2txt/master/image/screenshot.png' width=\"800\" /\u003e\n\n**Environment**\n\n- WP2TXT 1.0.1\n- MacBook Pro (2021 Apple M1 Pro)\n- enwiki-20220720-pages-articles.xml.bz2 (19.98 GB)\n\nIn the above environment, the process (decompression, splitting, extraction, and conversion) to obtain the plain text data of the English Wikipedia takes less than 1.5 hours.\n\n## Features\n\n- Converts Wikipedia dump files in various languages\n- Creates output files of specified size\n- Allows specifying ext elements (page titles, section headers, paragraphs, list items) to be extracted\n- Allows extracting category information of the article\n- Allows extracting opening paragraphs of the article\n\n## Setting Up\n\n### WP2TXT on Docker\n\n1. Install [Docker Desktop](https://www.docker.com/products/docker-desktop/) (Mac/Windows/Linux)\n2. Execute `docker` command in a terminal:\n\n```shell\ndocker run -it -v /Users/me/localdata:/data yohasebe/wp2txt\n```\n\n- Make sure to Replace `/Users/me/localdata` with the full path to the data directory in your local computer\n\n3. The Docker image will begin downloading and a bash prompt will appear when finished.\n4. The `wp2txt` command will be avalable anywhare in the Docker container. Use the `/data` directory as the location of the input dump files and the output text files.\n\n**IMPORTANT:**\n\n- Configure Docker Desktop resource settings (number of cores, amount of memory, etc.) to get the best performance possible.\n- When running the `wp2txt` command inside a Docker container, be sure to set the output directory to somewhere in the mounted local directory specified by the `docker run` command.\n\n### WP2TXT on MacOS and Linux\n\nWP2TXT requires that one of the following commands be installed on the system in order to decompress `bz2` files:\n\n- `lbzip2` (recommended)\n- `pbzip2`\n- `bzip2`\n\nIn most cases, the `bzip2` command is pre-installed on the system. However, since `lbzip2` can use multiple CPU cores and is faster than `bzip2`, it is recommended that you install it additionally. WP2TXT will attempt to find the decompression command available on your system in the order listed above.\n\nIf you are using MacOS with Homebrew installed, you can install `lbzip2` with the following command:\n\n    $ brew install lbzip2\n\n### WP2TXT on Windows\n\nInstall [Bzip2 for Windows](http://gnuwin32.sourceforge.net/packages/bzip2.htm) and set the path so that WP2TXT can use the bunzip2.exe command. Alternatively, you can extract the Wikipedia dump file in your own way and process the resulting XML file with WP2TXT.\n\n## Installation\n\n### WP2TXT command\n\n    $ gem install wp2txt\n\n## Wikipedia Dump File\n\nDownload the latest Wikipedia dump file for the desired language at a URL such as\n\n    https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2\n\nHere, `enwiki` refers to the English Wikipedia. To get the Japanese Wikipedia dump file, for instance, change this to `jawiki` (Japanese). In doing so, note that there are two instances of `enwiki` in the URL above.\n\nAlternatively, you can also select Wikipedia dump files created on a specific date from [here](http://dumps.wikimedia.org/backup-index.html). Make sure to download a file named in the following format:\n\n    xxwiki-yyyymmdd-pages-articles.xml.bz2\n\nwhere `xx` is language code such as `en` (English)\" or `ja` (japanese), and  `yyyymmdd` is the date of creation (e.g. `20220801`).\n\n## Basic Usage\n\nSuppose you have a folder with a wikipedia dump file and empty subfolders organized as follows:\n\n```\n.\n├── enwiki-20220801-pages-articles.xml.bz2\n├── /xml\n├── /text\n├── /category\n└── /summary\n```\n\n### Decompress and Split\n\nThe following command will decompress the entire wikipedia data and split it into many small (approximately 10 MB) XML files.\n\n    $ wp2txt --no-convert -i ./enwiki-20220801-pages-articles.xml.bz2 -o ./xml\n\n**Note**: The resulting files are not well-formed XML. They contain part of the orignal XML extracted from the Wikipedia dump file, taking care to ensure that the content within the \u003cpage\u003e tag is not split into multiple files.\n\n### Extract plain text from MediaWiki XML\n\n    $ wp2txt -i ./xml -o ./text\n\n\n### Extract only category info from MediaWiki XML\n\n    $ wp2txt -g -i ./xml -o ./category\n\n### Extract opening paragraphs from MediaWiki XML\n\n    $ wp2txt -s -i ./xml -o ./summary\n\n### Extract directly from bz2 compressed file\n\nIt is possible (though not recommended) to 1) decompress the dump files, 2) split the data into files, and 3) extract the text just one line of command. You can automatically remove all the intermediate XML files with `-x` option.\n\n    $ wp2txt -i ./enwiki-20220801-pages-articles.xml.bz2 -o ./text -x\n\n## Sample Output\n\nOutput contains title, category info, paragraphs\n\n    $ wp2txt -i ./input -o /output\n\n- [English Wikipedia](https://raw.githubusercontent.com/yohasebe/wp2txt/master/data/output_samples/testdata_en.txt)\n- [Japanese Wikipedia](https://raw.githubusercontent.com/yohasebe/wp2txt/master/data/output_samples/testdata_ja.txt)\n\nOutput containing title and category only\n\n    $ wp2txt -g -i ./input -o /output\n\n- [English Wikipedia](https://raw.githubusercontent.com/yohasebe/wp2txt/master/data/output_samples/testdata_en_category.txt)\n- [Japanese Wikipedia](https://raw.githubusercontent.com/yohasebe/wp2txt/master/data/output_samples/testdata_ja_category.txt)\n\nOutput containing title, category, and summary\n\n    $ wp2txt -s -i ./input -o /output\n\n- [English Wikipedia](https://raw.githubusercontent.com/yohasebe/wp2txt/master/data/output_samples/testdata_en_summary.txt)\n- [Japanese Wikipedia](https://raw.githubusercontent.com/yohasebe/wp2txt/master/data/output_samples/testdata_ja_summary.txt)\n\n## Command Line Options\n\nCommand line options are as follows:\n\n    Usage: wp2txt [options]\n    where [options] are:\n      -i, --input                      Path to compressed file (bz2) or decompressed file (xml), or path to directory containing files of the latter format\n      -o, --output-dir=\u003cs\u003e             Path to output directory\n      -c, --convert, --no-convert      Output in plain text (converting from XML) (default: true)\n      -a, --category, --no-category    Show article category information (default: true)\n      -g, --category-only              Extract only article title and categories\n      -s, --summary-only               Extract only article title, categories, and summary text before first heading\n      -f, --file-size=\u003ci\u003e              Approximate size (in MB) of each output file (default: 10)\n      -n, --num-procs                  Number of proccesses (up to 8) to be run concurrently (default: max num of available CPU cores minus two)\n      -x, --del-interfile              Delete intermediate XML files from output dir\n      -t, --title, --no-title          Keep page titles in output (default: true)\n      -d, --heading, --no-heading      Keep section titles in output (default: true)\n      -l, --list                       Keep unprocessed list items in output\n      -r, --ref                        Keep reference notations in the format [ref]...[/ref]\n      -e, --redirect                   Show redirect destination\n      -m, --marker, --no-marker        Show symbols prefixed to list items, definitions, etc. (Default: true)\n      -b, --bz2-gem                    Use Ruby's bzip2-ruby gem instead of a system command\n      -v, --version                    Print version and exit\n      -h, --help                       Show this message\n\n## Caveats\n\n* Some data, such as mathematical formulas and computer source code, will not be converted correctly.\n* Some text data may not be extracted correctly for various reasons (incorrect matching of begin/end tags, language-specific formatting rules, etc.).\n* The conversion process can take longer than expected. When dealing with a huge data set such as the English Wikipedia on a low-spec environment, it can take several hours or more.\n\n## Useful Links\n\n* [Wikipedia Database backup dumps](http://dumps.wikimedia.org/backup-index.html)\n\n## Author\n\n* Yoichiro Hasebe (\u003cyohasebe@gmail.com\u003e)\n\n## References\n\nThe author will appreciate your mentioning one of these in your research.\n\n* Yoichiro HASEBE. 2006. [Method for using Wikipedia as Japanese corpus.](http://ci.nii.ac.jp/naid/110006226727) _Doshisha Studies in Language and Culture_ 9(2), 373-403.\n* 長谷部陽一郎. 2006. [Wikipedia日本語版をコーパスとして用いた言語研究の手法](http://ci.nii.ac.jp/naid/110006226727). 『言語文化』9(2), 373-403.\n\nOr use this BibTeX entry:\n\n```\n@misc{wp2txt_2023,\n  author = {Yoichiro Hasebe},\n  title = {WP2TXT: A command-line toolkit to extract text content and category data from Wikipedia dump files},\n  url = {https://github.com/yohasebe/wp2txt},\n  year = {2023}\n}\n```\n\n## License\n\nThis software is distributed under the MIT License. Please see the LICENSE file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyohasebe%2Fwp2txt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyohasebe%2Fwp2txt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyohasebe%2Fwp2txt/lists"}