https://github.com/mardi4nfdi/wikifilter
Simple script to filter wikidumps for wiki-tags
https://github.com/mardi4nfdi/wikifilter
dump filter filter-wikidumps wiki-tags wikipedia
Last synced: 4 months ago
JSON representation
Simple script to filter wikidumps for wiki-tags
- Host: GitHub
- URL: https://github.com/mardi4nfdi/wikifilter
- Owner: MaRDI4NFDI
- Created: 2014-02-20T10:44:28.000Z (over 12 years ago)
- Default Branch: master
- Last Pushed: 2025-03-30T09:06:55.000Z (over 1 year ago)
- Last Synced: 2025-03-30T10:19:43.703Z (over 1 year ago)
- Topics: dump, filter, filter-wikidumps, wiki-tags, wikipedia
- Language: Python
- Size: 102 KB
- Stars: 1
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
wikiFilter
==========
Simple script to filter wikidumps for wiki-tags
To filter all pages that contain math from enwiki you can do the following
```
git clone https://github.com/physikerwelt/wikiFilter
cd wikiFilter
mkdir wout
wget https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2
./wikiFilter.py
```
All options of wikiFilter can be seen via
```
./wikiFilter.py --help
usage: wikiFilter.py [-h] [-f FILE] [-s SIZE] [-d DIR] [-t TAG] [-v] [-T]
extract wikipages that contain the math tag
optional arguments:
-h, --help show this help message and exit
-f FILE, --filename FILE
the bz2-file to be split and filtered (default:
enwiki-latest-pages-articles.xml.bz2)
-s SIZE, --splitsize SIZE
the number of pages contained in each split (default:
1000000)
-d DIR, --outputdir DIR
the directory name where the files go (default: wout)
-t TAG, --tagname TAG
the tag to search for (default: math)
-v, --verbosity
-T, --template include all templates (default: False)
```