Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/gatenlp/wpextract
Create datasets from WordPress sites for research or archiving
https://github.com/gatenlp/wpextract
corpus crawler nlp text-extraction text-mining web-scraping wordpress
Last synced: about 2 months ago
JSON representation
Create datasets from WordPress sites for research or archiving
- Host: GitHub
- URL: https://github.com/gatenlp/wpextract
- Owner: GateNLP
- License: apache-2.0
- Created: 2022-12-01T17:01:26.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2024-08-13T15:49:01.000Z (5 months ago)
- Last Synced: 2024-10-14T06:27:59.814Z (3 months ago)
- Topics: corpus, crawler, nlp, text-extraction, text-mining, web-scraping, wordpress
- Language: Python
- Homepage: https://wpextract.readthedocs.io
- Size: 1.88 MB
- Stars: 3
- Watchers: 3
- Forks: 0
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- Contributing: .github/CONTRIBUTING.md
- License: LICENSE
- Citation: CITATION.cff
Awesome Lists containing this project
README
# WPextract - WordPress Site Extractor
**WPextract is a tool to create datasets from WordPress sites.**
- Archives posts, pages, tags, categories, media (including files), comments, and users
- Uses the WordPress API to guarantee 100% accurate and complete content
- Resolves internal links and media to IDs
- Automatically parses multilingual sites to create parallel datasets
## Quickstart
See the [complete documentation](https://wpextract.readthedocs.io/) for more detailed usage.
1. Install with `pipx`
```shell-session
$ pipx install wpextract
```
2. Download site data
```shell-session
$ wpextract download "https://example.org" out_dl
```
3. Process into a dataset
```shell-session
$ wpextract extract out_dl out_data
```
## About WPextract
WPextract was built by [Freddy Heppell](https://freddyheppell.com) of the [GATE Project](https://gate.ac.uk) at the [School of Computer Science, University of Sheffield](https://sheffield.ac.uk/cs), originally created to scrape mis/disinformation websites for research.
## License
Available under the Apache 2.0 license. See [LICENSE](LICENSE) for more information.
## Citing
> [!NOTE]
> This software was developed for our EMNLP 2023 paper [_Analysing State-Backed Propaganda Websites: a New Dataset and Linguistic Study_](https://aclanthology.org/2023.emnlp-main.349/). The code has been updated since the paper was written; for archival purposes, the precise version used for the study is [available on Zenodo](https://zenodo.org/records/10008086).
We'd love to hear about your use of our tool, you can [email us](mailto:[email protected]) to let us know! Feel free to create issues and/or pull requests for new features or bugs.
If you use this tool in published work, please cite [our EMNLP paper](https://aclanthology.org/2023.emnlp-main.349/):
> Freddy Heppell, Kalina Bontcheva, and Carolina Scarton. 2023. Analysing State-Backed Propaganda Websites: a New Dataset and Linguistic Study. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5729–5741, Singapore. Association for Computational Linguistics.
Permanent references to each release of this software are available from [Zenodo](https://zenodo.org/doi/10.5281/zenodo.12725781).