{"id":13341984,"url":"https://github.com/studerw/wiki-dump-parser","last_synced_at":"2025-03-11T23:30:46.210Z","repository":{"id":49938575,"uuid":"233170180","full_name":"studerw/wiki-dump-parser","owner":"studerw","description":"Java tool to Wikimedia dumps into Java Article pojos for test or fake data.","archived":false,"fork":false,"pushed_at":"2023-12-05T22:24:00.000Z","size":1197,"stargazers_count":5,"open_issues_count":3,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-10-24T10:08:44.868Z","etag":null,"topics":["fake-data","java","wiki","wikiextractor","wikipedia","wikipedia-dump"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/studerw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-01-11T03:27:46.000Z","updated_at":"2023-04-03T17:09:07.000Z","dependencies_parsed_at":"2024-10-24T02:55:04.940Z","dependency_job_id":"8854f848-0099-42ca-b6be-c3a30cb023d0","html_url":"https://github.com/studerw/wiki-dump-parser","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/studerw%2Fwiki-dump-parser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/studerw%2Fwiki-dump-parser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/studerw%2Fwiki-dump-parser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/studerw%2Fwiki-dump-parser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/studerw","download_url":"https://codeload.github.com/studerw/wiki-dump-parser/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243129602,"owners_count":20241041,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fake-data","java","wiki","wikiextractor","wikipedia","wikipedia-dump"],"created_at":"2024-07-29T19:27:12.586Z","updated_at":"2025-03-11T23:30:45.721Z","avatar_url":"https://github.com/studerw.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Wiki Dump Parser\n![WIKIPEDIA_LOGO](https://github.com/studerw/wiki-dump-parser/blob/master/wikipedia_logo.png)\n\n![travisci-passing](https://api.travis-ci.org/studerw/wiki-dump-parser.svg?branch=master)\n[![APL v2](https://img.shields.io/badge/license-Apache%202-blue.svg)](http://www.apache.org/licenses/LICENSE-2.0.html)\n\n\n----\nThis is a small utility to create random text files to use for loading test data into Elastic Search or a CMS system.\nIt uses Wikipedia to get a massive amount of documents from the dumps you can download and extract, parses them,\nand returns them to use as Java pojoso of type _Article_ which contains the article's text, id, url, trimmed text,\nand title. \n\nTo get one or more dump files, I have successfully used this mirror: [Latest English Wiki Dump](https://dumps.wikimedia.org/enwiki/latest/).\n\nThe dump is divided up into dozens of bzipped archives. This tool expects that you've downloaded the current archive files (i.e.)\nwithout all the revisions and metadata. \n\nFor example, to download a single archive of a few hundred MB, eventually parsing into several hundred Wikipedia articles,\ndo the following:\n\n* Go to [Latest English Wiki Dump](https://dumps.wikimedia.org/enwiki/latest/).\n* Download files in the format of `enwiki-latest-pages-articles{n}.xml-{random_hash}.bz2`, where `n` while be some index from 1 to about 30 or so. \n  - Note that you don't need the files in the format `enwiki-latest-pages-articles-multistream{n}.xml-{random_hash}.bz2`.\n* Download one of these files, for example `enwiki-latest-pages-articles1.xml-{random_hash}.bz2`.\n* [Bunzip2](https://linux.die.net/man/1/bunzip2) the downloaded file: `bunzip2 \u003cfile\u003e`. The `bz2` extension will be removed, and the resulting file will now\nbe much larger. \n* Checkout the [WikiExtractor](https://github.com/attardi/wikiextractor) Python tool, e.g. `git clone https://github.com/attardi/wikiextractor.git`.\n* I used Python 3.7, but supposed 2.x works too. From the tool's directory, run `python3 WikiExtractor.py  --json  -o output enwiki-latest-pages-articles1.xml-{random_hash}` \n* The `output` folder will contain several directories of the articles first letter, e.g. _AA_ or _AB_ and inside each of these folders\nwill be dozens more files named `wiki_00`, `wiki_01`, etc.\n* Inside each one of these files is dozens are JSON objects describing an article. This tool takes a directory, finds all the files, and returns\nyou a list of _Articles_ which you could then push to ElasticSearch or anywhere else.\n    \n## Quick WikiExtractorfor Dump File\nThis was more fully described above, but for quick reference:\n```\nbunzip2 enwiki-latest-pages-articles{n}.xml-{random_hash}.bz2\npython3 WikiExtractor.py  --json  -o output_json_1 enwiki-latest-pages-articles1.xml-p10p30302\n```\n\n## Build\n\nTo build the jar, checkout the source and run:\n\n```bash\nmvn clean install\n```\n\n## Usage\n\nOnce, you've built it and installed or deployed it to your own repository, \nadd the following to your Maven build file:\n\n```\n  \u003cdependency\u003e\n    \u003cgroupId\u003ecom.studerw.wiki\u003c/groupId\u003e\n    \u003cartifactId\u003ewiki-dump-parser\u003c/artifactId\u003e\n    \u003cversion\u003e1.0.0-SNAPSHOT\u003c/version\u003e\n  \u003c/dependency\u003e\n```\n\nUse the _WikiParser_ class as follows:\n\n```java\n    Stream\u003cArticle\u003e articles = wikiParser.parseByDirectory(\"./output\");\n    articles.forEach(article -\u003e System.out.println(\"Title: \"+ article.getTitle());\n```\n\n## Examples\nThere are examples of json files in `src/test/resources/json` and usage of the tool in `src/test/com/studerw/wiki/WikiParserTest`.\n\n## Logging\nThe API uses [SLF4J](http://www.slf4j.org/).\nYou can use any implementation like Logback or Log4j.\n\nSpecific Loggers that you can tune:\n\n* `com.studerw.wiki` - set to debug for parsing output.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstuderw%2Fwiki-dump-parser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstuderw%2Fwiki-dump-parser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstuderw%2Fwiki-dump-parser/lists"}