Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/siznax/wptools

Wikipedia tools (for Humans): easily extract data from Wikipedia, Wikidata, and other MediaWikis
https://github.com/siznax/wptools

api-client commons data-science glam linked-open-data mediawiki mediawiki-api open-data python restbase wikidata wikimedia-commons wikipedia wikipedia-api

Last synced: 3 days ago
JSON representation

Wikipedia tools (for Humans): easily extract data from Wikipedia, Wikidata, and other MediaWikis

Host: GitHub
URL: https://github.com/siznax/wptools
Owner: siznax
License: mit
Created: 2012-02-06T04:10:27.000Z (almost 13 years ago)
Default Branch: master
Last Pushed: 2023-08-05T19:06:41.000Z (over 1 year ago)
Last Synced: 2024-10-29T17:35:48.078Z (2 months ago)
Topics: api-client, commons, data-science, glam, linked-open-data, mediawiki, mediawiki-api, open-data, python, restbase, wikidata, wikimedia-commons, wikipedia, wikipedia-api
Language: Python
Homepage:
Size: 1.14 MB
Stars: 574
Watchers: 23
Forks: 78
Open Issues: 38
Metadata Files:
- Readme: README.rst
- Changelog: HISTORY.rst
- Contributing: CONTRIBUTING.md
- License: LICENSE

Awesome Lists containing this project

README

        Wikipedia tools (for Humans)

============================

.. image:: https://img.shields.io/pypi/v/wptools.svg

        :target: https://pypi.python.org/pypi/wptools/

.. image:: https://travis-ci.org/siznax/wptools.svg?branch=master

        :target: https://travis-ci.org/siznax/wptools

.. image:: https://coveralls.io/repos/github/siznax/wptools/badge.svg?branch=master

        :target: https://coveralls.io/github/siznax/wptools

Python and command-line MediaWiki access for Humans

- get page extracts, image, Infobox data, Wikidata, and more

- get a random page, category, or site

- get page statistics

- get category members

- get site info and stats

- get data in any language

This package is intended to make it as easy as possible to get data

from MediaWiki instances, expose more Wikidata, and extend Wikimedia

APIs just for kicks. We say "(for Humans)" because that is a goal_.

Questions, feedback, and especially contributions_ are welcome!

Install

-------

.. code-block:: bash

    $ pip install wptools

    ✨🦄✨

Example

-------

.. code-block:: python

    >>> import wptools

Get a page object:

.. code-block:: python

    >>> page = wptools.page('Gandhi')

Get `API:Query`_ data:

.. _`API:Query`: https://www.mediawiki.org/wiki/API:Query

.. code-block:: python

    >>> page.get_query()

    en.wikipedia.org (query) Gandhi

    en.wikipedia.org (imageinfo) File:Portrait Gandhi.jpg

    Mahatma Gandhi (en) data

    {

      aliases:  M K Gandhi, Mohandas Gandhi, Bapu, Gandhi, M...

      assessments:  Pakistan, Alternative Views, South Afric...

      description:  pre-eminent leader of Indian nationalism ...

      extext:  Mahātmā **Mohandas Karamchand Gandhi** ( ; H...

      extract:  
Mahātmā Mohandas Karamchand Gandhi {u'size': 2951123, 'kind': 'query-pageimage', u...

      label: Mahatma Gandhi

      length: 262,790

      links:  10 Janpath, 14th Dalai Lama, 1915 Singapore M...

      modified:  page

      pageid: 19379

      random: Salt

      redirected:  {u'to': u'Mahatma Gandhi', u'from': u'Gandhi'}

      redirects:  {u'ns': 0, u'pageid': 55342, u'title': u'M...

      requests:  query, imageinfo

      title: Mahatma Gandhi

      url: https://en.wikipedia.org/wiki/Mahatma_Gandhi

      url_raw: https://en.wikipedia.org/wiki/Mahatma_Gandhi?action=raw

      watchers: 1,811

      wikibase: Q1001

      wikidata_url: https://www.wikidata.org/wiki/Q1001

    }


Get `API:Parse`_ data:

.. _`API:Parse`: https://www.mediawiki.org/wiki/API:Parse

.. code-block:: python

    >>> page.get_parse()

    en.wikipedia.org (parse) Gandhi

    en.wikipedia.org (imageinfo) File:MKGandhi.jpg

    Mahatma Gandhi (en) data

    {

      image:  {u'size': 2951123, 'kind': 'parse-image', u'des...

      infobox:  known_for, other_names, image, signature, bi...

      iwlinks:  https://biblio.wiki/wiki/Mohandas_K._Gandhi,...

      pageid: 19379

      parsetree:  Redirect...

      requests:  parse, imageinfo

      title: Mahatma Gandhi

      wikibase: Q1001

      wikidata_url: https://www.wikidata.org/wiki/Q1001

      wikitext:  {{Redirect|Gandhi}}{{pp-move-indef}}{{pp...

    }

Get Wikidata_:

.. _Wikidata: https://www.wikidata.org/w/api.php

.. code-block:: python

    >>> page = wptools.page(wikibase='Q1001')

    >>> page.get_wikidata()

    www.wikidata.org (wikidata) Q1001

    www.wikidata.org (labels) Q1280678|P535|P434|Q1860|P3762|Q668|P12...

    www.wikidata.org (labels) P119|Q1930187|P691|P18|P19|P1066|P509|P...

    www.wikidata.org (labels) Q6512732|Q1568|P972|Q84|P1430|P31|Q2140...

    www.wikidata.org (labels) P1576|Q4964182|P1368|P140|Q22336956|P12...

    en.wikipedia.org (imageinfo) File:Portrait Gandhi.jpg

    Mahatma Gandhi (en) data

    {

      aliases:  M K Gandhi, Mohandas Gandhi, Bapu, Gandhi, M...

      claims:  P646, P535, P906, P434, P648, P3762, P1711, ...

      description:  pre-eminent leader of Indian nationalism ...

      image:  {u'size': 2951123, 'kind': 'wikidata-image', u'...

      label: Mahatma Gandhi

      labels:  Q1280678, P535, Q131149, P434, Q1860, P3762,...

      modified:  wikidata

      requests:  wikidata, labels, labels, labels, labels, im...

      title: Mahatma_Gandhi

      what: human

      wikibase: Q1001

      wikidata:  Geni.com profile ID (P2600), National Libr...

      wikidata_pageid: 1330

      wikidata_url: https://www.wikidata.org/wiki/Q1001

    }

Get RESTBase_ data:

.. _RESTBase: https://www.mediawiki.org/wiki/RESTBase

.. code-block:: python

    >>> page.get_restbase('/page/summary/')

    en.wikipedia.org (restbase) /page/summary/Gandhi

    Mahatma Gandhi (en) data

    {

      description:  pre-eminent leader of Indian nationalism ...

      exhtml:  
Mahātmā Mohandas Karamchand Gandhi...

      exrest:  Mahātmā Mohandas Karamchand Gandhi (; Hindust...

      image:  {'kind': 'restbase-original', u'width': 2024, '...

      pageid: 19379

      requests:  restbase

      title: Mahatma_Gandhi

      url: https://en.wikipedia.org/wiki/Gandhi

      url_raw: https://en.wikipedia.org/wiki/Gandhi?action=raw

    }

Get all the things (at once):

.. code-block:: python

    >>> page.get()

    en.wikipedia.org (query) Gandhi

    en.wikipedia.org (parse) 19379

    www.wikidata.org (wikidata) Q1001

    www.wikidata.org (labels) Q1280678|P535|P434|Q1860|P3762|Q668|P12...

    www.wikidata.org (labels) P119|Q1930187|P691|P18|P19|P1066|P509|P...

    www.wikidata.org (labels) Q6512732|Q1568|P972|Q84|P1430|P31|Q2140...

    www.wikidata.org (labels) P1576|Q4964182|P1368|P140|Q22336956|P12...

    en.wikipedia.org (restbase) /page/summary/Mahatma_Gandhi

    en.wikipedia.org (imageinfo) File:MKGandhi.jpg|File:Portrait Gandhi.jpg

    Mahatma Gandhi (en) data

    {

      aliases:  M K Gandhi, Mohandas Gandhi, Bapu, Gandhi, M...

      assessments:  Pakistan, Alternative Views, South Afric...

      claims:  P646, P535, P906, P434, P648, P3762, P1711, ...

      description:  pre-eminent leader of Indian nationalism ...

      exhtml:  
Mahātmā Mohandas Karamchand Gandhi...

      exrest:  Mahātmā Mohandas Karamchand Gandhi (; Hindust...

      extext:  Mahātmā **Mohandas Karamchand Gandhi** ( ; H...

      extract:  
Mahātmā Mohandas Karamchand Gandhi {u'size': 2951123, 'kind': 'query-pageimage', u...

      infobox:  known_for, other_names, image, signature, bi...

      iwlinks:  https://biblio.wiki/wiki/Mohandas_K._Gandhi,...

      label: Mahatma Gandhi

      labels:  Q1280678, P535, Q131149, P434, Q1860, P3762,...

      length: 262,790

      links:  10 Janpath, 14th Dalai Lama, 1915 Singapore M...

      modified:  wikidata, page

      pageid: 19379

      parsetree:  Redirect...

      random: Salt

      redirected:  {u'to': u'Mahatma Gandhi', u'from': u'Gandhi'}

      redirects:  {u'ns': 0, u'pageid': 55342, u'title': u'M...

      requests:  query, parse, wikidata, labels, labels, labe...

      title: Mahatma_Gandhi

      url: https://en.wikipedia.org/wiki/Mahatma_Gandhi

      url_raw: https://en.wikipedia.org/wiki/Mahatma_Gandhi?action=raw

      watchers: 1,811

      what: human

      wikibase: Q1001

      wikidata:  Geni.com profile ID (P2600), National Libr...

      wikidata_pageid: 1330

      wikidata_url: https://www.wikidata.org/wiki/Q1001

      wikitext:  {{Redirect|Gandhi}}{{pp-move-indef}}{{pp...

    }


Get more (expensive) data:

.. code-block:: python

    >>> page.get_more()

    en.wikipedia.org (querymore) Gandhi

    Mahatma Gandhi (en) data

    {

      categories:  Category:1869 births, Category:1948 death...

      contributors: 2,606

      files:  File:Aum Om red.svg, File:Commons-logo.svg, Fi...

      languages:  {u'lang': u'af', u'title': u'Mahatma Gand...

      pageid: 19379

      redirected:  {u'to': u'Mahatma Gandhi', u'from': u'Gandhi'}

      requests:  querymore

      title: Mahatma Gandhi

      views: 19,242

    }

Get data in `another language`_:

.. _`another language`: https://github.com/siznax/wptools/wiki/Language-Codes

.. code-block:: python

    >>> page = wptools.page(lang='zh')

    zh.wikipedia.org (random) 🍰

    哈莉特·塔布曼 (zh) data

    {

      pageid: 211070

      title: 哈莉特·塔布曼

    }

Get data from `another wiki`_:

.. _`another wiki`: https://meta.wikimedia.org/wiki/List_of_Wikipedias

.. code-block:: python

    >>> page = wptools.page(wiki='en.wikiquote.org')

    en.wikiquote.org (random) 🍪

    Malala_Yousafzai (en)

    {

      pageid: 146817

      title: Malala_Yousafzai

    }

Documentation

-------------

See our wiki_.

Please enjoy!

@siznax 👹

.. _contributions: https://github.com/siznax/wptools/blob/master/CONTRIBUTING.md

.. _goal: http://docs.python-requests.org/en/master/user/intro/

.. _wiki: https://github.com/siznax/wptools/wiki