{"id":27947945,"url":"https://github.com/zulko/wiki_dump_extractor","last_synced_at":"2025-07-26T13:39:20.925Z","repository":{"id":280299492,"uuid":"941546242","full_name":"Zulko/wiki_dump_extractor","owner":"Zulko","description":"Code to extract and process wikipedia pages from a XML dump","archived":false,"fork":false,"pushed_at":"2025-05-31T17:18:34.000Z","size":297,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-01T05:05:19.829Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Zulko.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-03-02T14:53:34.000Z","updated_at":"2025-05-31T17:18:37.000Z","dependencies_parsed_at":"2025-03-02T16:19:08.643Z","dependency_job_id":"da8263c6-ecd5-48b4-8ec4-b17f8595cba3","html_url":"https://github.com/Zulko/wiki_dump_extractor","commit_stats":null,"previous_names":["zulko/wiki_dump_extractor"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Zulko/wiki_dump_extractor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zulko%2Fwiki_dump_extractor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zulko%2Fwiki_dump_extractor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zulko%2Fwiki_dump_extractor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zulko%2Fwiki_dump_extractor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Zulko","download_url":"https://codeload.github.com/Zulko/wiki_dump_extractor/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zulko%2Fwiki_dump_extractor/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267175742,"owners_count":24047921,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-26T02:00:08.937Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-07T14:39:21.627Z","updated_at":"2025-07-26T13:39:20.901Z","avatar_url":"https://github.com/Zulko.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Wiki dump extractor\n\nA python library to extract and analyze pages from a wiki dump.\n\nThis library is used in particular in the [Landnotes](https://github.com/Zulko/landnotes) project to extract and analyze pages from the Wikipedia dump.\n\nThe project is hosted on [GitHub](https://github.com/zulko/wiki_dump_extractor) an the HTML documentation is available [here](https://zulko.github.io/wiki_dump_extractor/).\n\n## Scope\n\nMake the wikipedia dumps easier to work with:\n\n- Extract pages from a wiki dump\n- Be easy to install and run\n- Be fast (can iterate over 50,000 pages / secong using Avro)\n- Be memory efficient\n- Allow for batch processing and parallel processing\n\nProvide utilities for page analysis:\n\n- Date parsing\n- Section extraction\n- Text cleaning\n- and more.\n\n## Usage\n\nTo simply iterate over the pages in the dump:\n\n```python\nfrom wiki_dump_extractor import WikiDumpExtractor\n\ndump_file = \"enwiki-20220301-pages-articles-multistream.xml.bz2\"\nextractor = WikiDumpExtractor(file_path=dump_file)\nfor page in extractor.iter_pages(limit=1000):\n    print(page.title)\n```\n\nTo extract the pages by batches (here we save the pages separate CSV files):\n\n```python\nfrom wiki_dump_extractor import WikiDumpExtractor\n\ndump_file = \"enwiki-20220301-pages-articles-multistream.xml.bz2\"\nextractor = WikiDumpExtractor(file_path=dump_file)\nbatches = extractor.iter_page_batches(batch_size=1000, limit=10)\nfor i, batch in enumerate(batches):\n    df = pandas.DataFrame([page.to_dict() for page in batch])\n    df.to_csv(f\"batch_{i}.csv\")\n```\n\n### Converting the dump to Avro\n\nThere are many reasons why you might want to convert the dump to Avro. The original `xml.bz2` dump is 22Gb but very slow to read from (250/s), the uncompressed dump is 107Gb, relatively fast to read (this library uses lxml which reads thousands of pages per second), however 50% of the pages in there are empty redirect pages.\n\nThe following code converts the batch to a 28G avro dump that only contains the 12 million real pages, stores redirects in a fast LMDB database, and creates an index for quick page lookups. The operation takes ~40 minutes depending on your machine.\n\n```python\nfrom wiki_dump_extractor import WikiXmlDumpExtractor\n\nfile_path = \"enwiki-20250201-pages-articles-multistream.xml\"\nextractor = WikiXmlDumpExtractor(file_path=file_path)\nignored_fields = [\"timestamp\", \"page_id\", \"revision_id\", \"redirect_title\"]\nextractor.extract_pages_to_avro(\n    output_file=\"wiki_dump.avro\",\n    redirects_db_path=\"redirects.lmdb\",  # LMDB database for fast redirect lookups\n    ignored_fields=ignored_fields,\n)\n```\n\nThen index the pages for fast lookups:\n\n```python\nfrom wiki_dump_extractor import WikiAvroDumpExtractor\n\nextractor = WikiAvroDumpExtractor(file_path=\"wiki_dump.avro\")\nextractor.index_pages(page_index_db=\"page_index.lmdb\")\n```\n\nLater on, read the Avro file and use redirects and index as follows (reads the 12 million pages in ~3-4 minutes depending on your machine):\n\n```python\nfrom wiki_dump_extractor import WikiAvroDumpExtractor\n\n# Create extractor\nextractor = WikiAvroDumpExtractor(\n    file_path=\"wiki_dump.avro\",\n    index_dir=\"page_index.lmdb\"  # Use the index for faster lookups\n)\n\n# Get pages with automatic redirect resolution\npages = extractor.get_page_batch_by_title(\n    [\"Page Title 1\", \"Page Title 2\"]\n)\n```\n\n## Installation\n\n```bash\npip install wiki-dump-extractor\n```\n\nOr from the source in development mode:\n\n```bash\npip install -e .\n```\n\nTo use the LLM-specific module (that would be mostly if you are on a project like Landnotes), use\n\n```bash\npip install wiki-dump-extractor[llm]\n```\n\nOr locally:\n```bash\npip install -e \".[llm]\"\n```\n\nTo install with tests, use `pip install -e \".[dev]\"` then run the tests with `pytest` in the root directory.\n\n### Requirements for running the LLM utils\n\n```bash\n# Add the Cloud SDK distribution URI as a package source\necho \"deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main\" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list\n\n# Import the Google Cloud public key\ncurl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -\n\n# Update the package list and install the Cloud SDK\nsudo apt-get update \u0026\u0026 sudo apt-get install google-cloud-sdk\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzulko%2Fwiki_dump_extractor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzulko%2Fwiki_dump_extractor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzulko%2Fwiki_dump_extractor/lists"}