{"id":20114680,"url":"https://github.com/gatenlp/wpextract","last_synced_at":"2025-06-25T15:07:43.929Z","repository":{"id":202629659,"uuid":"573084559","full_name":"GateNLP/wpextract","owner":"GateNLP","description":"Create datasets from WordPress sites for research or archiving","archived":false,"fork":false,"pushed_at":"2025-03-31T14:19:59.000Z","size":2033,"stargazers_count":4,"open_issues_count":4,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-06-20T18:59:47.858Z","etag":null,"topics":["corpus","crawler","nlp","text-extraction","text-mining","web-scraping","wordpress"],"latest_commit_sha":null,"homepage":"https://wpextract.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GateNLP.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-12-01T17:01:26.000Z","updated_at":"2025-05-11T04:29:22.000Z","dependencies_parsed_at":"2025-04-12T23:21:57.550Z","dependency_job_id":"110601e1-bf63-46e8-b7c2-51fcc62c5731","html_url":"https://github.com/GateNLP/wpextract","commit_stats":null,"previous_names":["gatenlp/wordpress-site-extractor","gatenlp/wpextract"],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/GateNLP/wpextract","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GateNLP%2Fwpextract","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GateNLP%2Fwpextract/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GateNLP%2Fwpextract/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GateNLP%2Fwpextract/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GateNLP","download_url":"https://codeload.github.com/GateNLP/wpextract/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GateNLP%2Fwpextract/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261897003,"owners_count":23226648,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["corpus","crawler","nlp","text-extraction","text-mining","web-scraping","wordpress"],"created_at":"2024-11-13T18:31:16.982Z","updated_at":"2025-06-25T15:07:43.905Z","avatar_url":"https://github.com/GateNLP.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# WPextract - WordPress Site Extractor\n\n\u003ca href=\"https://pypi.org/project/wpextract/\"\u003e\u003cimg alt=\"PyPI - Version\" src=\"https://img.shields.io/pypi/v/wpextract\"\u003e\u003c/a\u003e\n\u003ca href=\"https://anaconda.org/conda-forge/wpextract\"\u003e\u003cimg alt=\"Conda Version\" src=\"https://img.shields.io/conda/vn/conda-forge/wpextract\"\u003e\u003c/a\u003e\n\u003ca href=\"https://zenodo.org/doi/10.5281/zenodo.12725781\"\u003e\u003cimg src=\"https://zenodo.org/badge/573084559.svg\" alt=\"DOI\"\u003e\u003c/a\u003e\n\n**WPextract is a tool to create datasets from WordPress sites.**\n\n- Archives posts, pages, tags, categories, media (including files), comments, and users\n- Uses the WordPress API to guarantee 100% accurate and complete content\n- Resolves internal links and media to IDs\n- Automatically parses multilingual sites to create parallel datasets\n\n\n## Quickstart\n\nSee the [complete documentation](https://wpextract.readthedocs.io/) for more detailed usage.\n\n1. Install with `pipx`\n    ```shell-session\n    $ pipx install wpextract\n    ```\n2. Download site data\n    ```shell-session\n    $ wpextract download \"https://example.org\" out_dl\n    ```\n3. Process into a dataset\n    ```shell-session\n    $ wpextract extract out_dl out_data\n    ```\n\n## About WPextract\n\nWPextract was built by [Freddy Heppell](https://freddyheppell.com) of the [GATE Project](https://gate.ac.uk) at the [School of Computer Science, University of Sheffield](https://sheffield.ac.uk/cs), originally created to scrape mis/disinformation websites for research.\n\n## License\n\nAvailable under the Apache 2.0 license. See [LICENSE](LICENSE) for more information.\n\n## Citing\n\n\u003e [!NOTE]\n\u003e This software was developed for our EMNLP 2023 paper [_Analysing State-Backed Propaganda Websites: a New Dataset and Linguistic Study_](https://aclanthology.org/2023.emnlp-main.349/). The code has been updated since the paper was written; for archival purposes, the precise version used for the study is [available on Zenodo](https://zenodo.org/records/10008086).\n\nWe'd love to hear about your use of our tool, you can [email us](mailto:frheppell1@sheffield.ac.uk) to let us know! Feel free to create issues and/or pull requests for new features or bugs.\n\nIf you use this tool in published work, please cite [our EMNLP paper](https://aclanthology.org/2023.emnlp-main.349/):\n\n\u003e Freddy Heppell, Kalina Bontcheva, and Carolina Scarton. 2023. Analysing State-Backed Propaganda Websites: a New Dataset and Linguistic Study. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5729–5741, Singapore. Association for Computational Linguistics.\n\nPermanent references to each release of this software are available from [Zenodo](https://zenodo.org/doi/10.5281/zenodo.12725781).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgatenlp%2Fwpextract","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgatenlp%2Fwpextract","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgatenlp%2Fwpextract/lists"}