https://github.com/lilingxi01/wikidl

WikiDL is an efficient wikipedia data dump downloader for researchers. It uses multiprocessing for maxing out the bandwidth and CPUs, and is friendly with task schedulers like Slurm.
https://github.com/lilingxi01/wikidl

cli downloader dump python research wikimedia

Last synced: over 1 year ago
JSON representation

WikiDL is an efficient wikipedia data dump downloader for researchers. It uses multiprocessing for maxing out the bandwidth and CPUs, and is friendly with task schedulers like Slurm.

• Host: GitHub
• URL: https://github.com/lilingxi01/wikidl
• Owner: lilingxi01
• License: gpl-2.0
• Created: 2023-04-30T17:30:46.000Z (about 3 years ago)
• Default Branch: main
• Last Pushed: 2024-11-23T08:25:25.000Z (over 1 year ago)
• Last Synced: 2025-04-04T12:55:27.677Z (over 1 year ago)
• Topics: cli, downloader, dump, python, research, wikimedia
• Language: Python
• Homepage: https://wikidl.lingxi.li
• Size: 38.1 KB
• Stars: 2
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README