{"id":13608973,"url":"https://github.com/zverok/wikipedia_ql","last_synced_at":"2025-04-05T14:09:42.942Z","repository":{"id":43841198,"uuid":"376362887","full_name":"zverok/wikipedia_ql","owner":"zverok","description":"Query language for efficient data extraction from Wikipedia","archived":false,"fork":false,"pushed_at":"2022-02-16T12:46:29.000Z","size":5417,"stargazers_count":350,"open_issues_count":0,"forks_count":5,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-03-29T13:11:12.261Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zverok.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.rst","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-06-12T18:49:54.000Z","updated_at":"2025-02-26T13:21:51.000Z","dependencies_parsed_at":"2022-08-30T13:21:24.307Z","dependency_job_id":null,"html_url":"https://github.com/zverok/wikipedia_ql","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zverok%2Fwikipedia_ql","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zverok%2Fwikipedia_ql/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zverok%2Fwikipedia_ql/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zverok%2Fwikipedia_ql/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zverok","download_url":"https://codeload.github.com/zverok/wikipedia_ql/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247345856,"owners_count":20924102,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T19:01:31.430Z","updated_at":"2025-04-05T14:09:42.889Z","avatar_url":"https://github.com/zverok.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# WikipediaQL: querying structured data from Wikipedia\n\n**WikipediaQL** is an _experimental query language_ and Python library for querying structured data from Wikipedia. It looks like this:\n\n```\n$ wikipedia_ql --page \"Guardians of the Galaxy (film)\" \\\n    '{\n      page@title as \"title\";\n      section[heading=\"Cast\"] as \"cast\" \u003e\u003e {\n          li \u003e\u003e text:matches(\"^(.+?) as (.+?):\") \u003e\u003e {\n              text-group[group=1] as \"actor\";\n              text-group[group=2] as \"character\"\n          }\n      };\n      section[heading=\"Critical response\"] \u003e\u003e {\n          sentence:contains(\"Rotten Tomatoes\") as \"RT ratings\" \u003e\u003e {\n              text:matches(\"\\d+%\") as \"percent\";\n              text:matches(\"(\\d+) (critic|review)\") \u003e\u003e text-group[group=1] as \"reviews\";\n              text:matches(\"[\\d.]+/10\") as \"overall\"\n          }\n      }\n    }'\n\nRT ratings:\n  overall: 7.8/10\n  percent: 92%\n  reviews: '334'\ncast:\n- actor: Chris Pratt\n  character: Peter Quill / Star-Lord\n- actor: Zoe Saldaña\n  character: Gamora\n- actor: Dave Bautista\n  character: Drax the Destroyer\n- actor: Vin Diesel\n  character: Groot\n- actor: Bradley Cooper\n  character: Rocket CREDITED ONLY AS \"Rocket\" PER OFFICIAL SYNOPSIS!\n- actor: Lee Pace\n  character: Ronan the Accuser\n- actor: Michael Rooker\n  character: Yondu Udonta\n- actor: Karen Gillan\n  character: Nebula\n- actor: Djimon Hounsou\n  character: Korath\n- actor: John C. Reilly\n  character: Rhomann Dey\n- actor: Glenn Close\n  character: Irani Rael\n- actor: Benicio del Toro\n  character: Taneleer Tivan / The Collector\ntitle: Guardians of the Galaxy (film)\n```\n\n### How?\n\nWikipediaQL-the-tool does roughly this:\n\n* Parses query in WikipediaQL-the-language;\n* Uses [MediaWiki API](https://en.wikipedia.org/w/api.php) to fetch pages' metadata;\n* Uses [Parsoid API](https://www.mediawiki.org/wiki/Parsoid/API) to fetch each page's content semantic HTML;\n* Applies selectors from the query to page to extract structured data.\n\n**The WikipediaQL development is covered in ongoing series of articles:** (newest first)\n\n* Jan'22: [Wikipedia and irregular data: how much can you fetch in one expression?](https://zverok.substack.com/p/wikipedia-and-irregular-data-how)\n* Dec'21: [Wikipedia as the data source: taming the irregular](https://zverok.substack.com/p/wikipediaql-1)\n* Oct'21: [Why Wikipedia matters, and how to make sense of it (programmatically)](https://zverok.substack.com/p/wikipedia)\n\n[Subscribe to follow](https://zverok.substack.com/)\n\n### Why?\n\n\u003e “Sometimes magic is just someone spending more time on something than anyone else might reasonably expect.” — Raymond Joseph Teller\n\nWikipedia is the most important open knowledge project: basically, the \"table of contents\" of all the human data. While it might be incomplete or misleading in details, the amount of data is incredible, and its organization makes all the data accessible to humans.\n\nOTOH, the data is semi-structured and quite hard to extract automatically. This project is an experiment in making this data accessible to machines—or, rather, to humans with programming languages. The main goal is to develop an easy to use and memorize, unambiguous and powerful query language and support it by the reference implementation.\n\n_See [FAQ](#faq) below for justifications of parsing Wikipedia instead of just using already formalized Wikidata (and of parsing HTML instead of Wikipedia markup)._\n\n## Usage\n\n`$ pip install wikipedia_ql`\n\n```\n$ wikipedia_ql --page \"Page name\" query_text\n# or\n$ wikipedia_ql query_text_with_page\n```\n\nUsage as Python library:\n\n```python\nfrom wikipedia_ql import media_wiki\n\nwikipedia = media_wiki.Wikipedia()\n\ndata = wikipedia.query(query_text)\n```\n\nFull WikipediaQL query looks like this:\n```\nfrom \u003csource\u003e {\n  \u003cselectors\u003e\n}\n```\n\nWhen using `--page` parameter to the executable, you need only to pass selectors in the query text.\n\n_Source_ is Wikipedia article name, or category name, or (in the future) other ways of specifying multiple pages. _Selectors_ are similar to CSS; they are nested in one another with `selector { other; selectors }`, or (shortcut) `selector \u003e\u003e other_selector`. All terminal selectors (e.g., doesn't having others nested) produce values in the output; the value can be associated with a name with `as \"valuename\"`.\n\nSee below for a list of selectors and sources currently supported and the future ones.\n\n### Examples\n\nSimple query for some info from the page:\n\n```\n$ wikipedia_ql --page \"Pink Floyd\" \\\n    'section[heading=\"Discography\"] \u003e\u003e li \u003e\u003e {\n        a as \"title\";\n        text:matches(\"\\((.+)\\)\") \u003e\u003e text-group[group=1] as \"year\";\n    }'\n\n- title: The Piper at the Gates of Dawn\n  year: '1967'\n- title: A Saucerful of Secrets\n  year: '1968'\n- title: More\n  year: '1969'\n- title: Ummagumma\n  year: '1969'\n#     ...and so on...\n```\n\nMulti-page query from pages of some category (only from inside Python):\n```python\nquery = r'''\nfrom category:\"2020s American time travel television series\" {\n    page@title as \"title\";\n    section[heading=\"External links\"] \u003e\u003e {\n      li \u003e\u003e text:matches(\"^(.+?) at IMDb\") \u003e\u003e text-group[group=1] \u003e\u003e a@href as \"imdb\"\n    }\n}\n'''\n\n# iquery returns generator, fetching pages as you go\nfor row in wikipedia.iquery(query):\n  print(row)\n# {'title': 'Agents of S.H.I.E.L.D.', 'imdb': 'https://www.imdb.com/title/tt2364582/'}\n# {'title': 'The Flash (2014 TV series)', 'imdb': 'https://www.imdb.com/title/tt3107288/'}\n# {'title': 'Legends of Tomorrow', 'imdb': 'https://www.imdb.com/title/tt4532368/'}\n# ....\n```\n\nNavigating through pages in one query (note the `-\u003e` which means \"perform subquery in the page by link\"):\n```\n$ wikipedia_ql --page Björk \\\n    'section[heading=\"Discography\"] \u003e\u003e li \u003e\u003e a -\u003e {\n        page@title as \"title\";\n        .infobox-image \u003e\u003e img \u003e\u003e @src as \"cover\"\n    }'\n\n- cover: https://upload.wikimedia.org/wikipedia/en/thumb/7/77/Bj%C3%B6rk-Debut-1993.png/220px-Bj%C3%B6rk-Debut-1993.png\n  title: Debut (Björk album)\n- cover: https://upload.wikimedia.org/wikipedia/en/thumb/3/3f/Bjork_Post.png/220px-Bjork_Post.png\n  title: Post (Björk album)\n- cover: https://upload.wikimedia.org/wikipedia/en/thumb/a/af/Bj%C3%B6rk_-_Homogenic.png/220px-Bj%C3%B6rk_-_Homogenic.png\n  title: Homogenic\n...\n```\n\nAs the page source should be fetched from Wikipedia every time, and it can be a major slowdown when experimenting, `wikipedia_ql` implements super-naive caching:\n\n```py\nwikipedia = media_wiki.Wikipedia(cache_folder='some/folder')\nwikipedia.query('from \"Pink Floyd\" { page@title }') # fetches page from Wikipedia, then runs query\nwikipedia.query('from \"Pink Floyd\" { page@title }') # gets the cached by prev.request contents from some/folder\n```\n\n_(**Caution!** as it was said, for now, the cache is super-naive: it just stores page contents in the specified folder forever. You might delete it from the cache manually, though: there are just `PageName.meta.json` and `PageName.json` files.)_\n\n## Current state \u0026 Planned features\n\n### Query language\n\n```\nfrom \u003csource\u003e {\n  \u003cselectors\u003e\n}\n```\n\n**Source** can be:\n\n* [x] `\"Page title\"`\n* [x] `category:\"Category title\"`\n* [ ] `\"Page title\",\"Other page title\"` (several pages at once)\n* [ ] `geo:\"\u003clat\u003e, \u003clng\u003e\"`\n* [ ] `prefix:\"Page title prefix\"`\n* [ ] `search:\"Search string\"`\n  * _Maybe several search types, as Wikipedia has several search APIs_\n\n**Selectors** are CSS-alike selectors, `type.class[attr=\"value\"][otherattr=\"othervalue\"]`. Note, that **unlike CSS**, nesting (any `child` inside `parent`) is performed not with spaces (`parent child`), but with `parent \u003e\u003e child`.\n\n* Singular selectors:\n  * [x] regular CSS selectors, like `a` or `table.wikitable`\n    * _Note again, that `selector nested_selector` is **not** supported, so you need to `li \u003e\u003e a` to say \"all links inside the list items\"_\n  * [ ] `section`\n    * [x] `section[heading=\"Section heading\"]`: fetch everything inside section with the specified heading (full heading text must match);\n    * [x] `section:first` (useful to fetch article intro)\n    * [ ] `section`: all sections;\n    * [ ] `section[level=3]`: all sections of particular level\n    * [ ] more powerful `heading` value patterns would be supported (probably in CSS-alike manner: `heading^=\"Starts from\"` and so on)\n  * [ ] `text`\n    * [x] `text:matches(\"pattern\")`: part of the document matching pattern (Python's regexp); document's structure would be preserved, so you can nest CSS and other WikipediaQL selectors inside: `li \u003e\u003e text:matches(\"^(.+?) as\") \u003e\u003e a@href as \"link\"`\n    * [x] `text`: without pattern specification, just selects the entire text of the parent element;\n    * [ ] pattern flags, like `text:imatches(\"pattern\")` (case-insensitive)\n    * [ ] handle inline images `alt` attribute as text\n  * [x] `text-group`: should be directly nested in `text` pattern, refers to capture a group of the regexp; see the first example in the README;\n    * [x] `text-group[group=1]`: group by number\n    * [x] `text-group[group=\"name\"]`: named groups\n  * [ ] `sentence`\n    * [x] `sentence:contains(\"pattern\")`: find sentence where pattern matches (whole sentence is selected)\n    * [x] `sentence`: all sentences in the scope\n    * [x] `sentence:first`\n    * [ ] other CSS pseudo-selectors for `sentence`\n    * [ ] pattern flags (same as for text)\n    * [ ] _more suitable sentence tokenizer will be used, probably: currently we are relying on nltk, which is too powerful (and large dependency) for our simplistic needs_\n  * [x] `page`: refers (from any scope) to the entire current page; useful for re-nesting fetched data in a logical way and to include metadata attributes in output (see below)\n  * [ ] `\u003cselector\u003e@\u003cattribute\u003e`:\n      * [x] `\u003ccss_selector\u003e@\u003ctag_attribute\u003e`\n        * [x] expand URLs\n      * [ ] `page@\u003cpage_attribute\u003e`\n        * [x] `title`\n        * [ ] other\n        * [ ] smart fetching metadata\n      * [x] as a free-standing selector (`@title` on the top level to fetch \"current page's title\" instead of `page@title`, and so on)\n  * [x] `table-data` for data tables: `from \"Kharkiv\" { section[heading=\"Climate\"] \u003e\u003e table \u003e\u003e table-data \u003e\u003e tr[title^=\"Average high\"] \u003e\u003e td[column=\"Jan\"] }`; see [docs](https://github.com/zverok/wikipedia_ql/blob/main/docs/Tables.md)\n  * [x] _infoboxes_ are covered with the same `table-data` quasi-selector, see [showcase](https://github.com/zverok/wikipedia_ql/blob/main/docs/showcase/Infoboxes.md)\n  * [ ] `hatnote` to fetch and process [Hatnotes](https://en.wikipedia.org/wiki/Wikipedia:Hatnote) (special links at the top of the page/section, saying \"for more information, see [here]\", \"this page is about X, if you want Y, go [there]\" and so on)\n  * [ ] `:has`: like [CSS `:has` pseudo-class](https://developer.mozilla.org/en-US/docs/Web/CSS/:has) but support all the WikipediaQL selectors, so one might say `from category:\"Marvel Cinematic Universe films\" { :has(@category*=\"films\") { ...work with pages... } }` to drop from the result-set pages of the [category](https://en.wikipedia.org/wiki/Category:Marvel_Cinematic_Universe_films) which aren't movies.\n  * [ ] (?) `:primary` or something like that (maybe `:largest`), to select the most important thing in the scope (for example, `section[heading=\"Discography\"] \u003e\u003e ul:primary` will probably fetch the list of albums, while the section might have other, smaller lists, like the enumeration of studios where recordings were done)\n  * _TBC on \"as-good-examples-found\" basis_\n* Selector operations:\n  * [x] sequence: `parent \u003e\u003e child`\n  * [x] group: `{ selector1; selector2; selector3 }` fetch all the selectors in the result set\n  * [ ] immediate child: `parent \u003e child`; maybe (come good reasons) other CSS relations like `sibling1 + sibling2`\n  * [x] **follow link**: `section[\"Discography\"] \u003e\u003e li \u003e\u003e a -\u003e { selectors working inside the fetched page }`, to allow expressing page navigation in a singular query\n* Marking information to extract:\n  * [x] extract unnamed data: every terminal selector puts extracted value in resultset (the resultset then will look like several nested arrays)\n  * [x] `as \"variablename\"`: every terminal selector with associated name puts extracted value as `{\"name\": value}`; there is still some uncertainty on how it all should be structured, but mostly the right thing is done\n  * [ ] `as :type` and `as \"name\":type` for typecasting values\n    * [ ] (?) simpe types (`as \"year\":int`) maybe wouldn't be that necessary, as the conversion can be easily done in the client\n    * [ ] `as :html` (as opposed to current \"content text only\" extraction) might be useful in many cases\n    * [ ] converting a large section of the document into composite type should shine. Things like `infobox as :hash` or `wikitable as :dataframe` will change the usability of data extraction significantly\n* Other query language features:\n  * [ ] comments (probably `// text`, as `#` can be a start of valid CSS selector)\n  * [ ] (maybe?) named/reusable subquires\n\n### Other features/problems\n\n* **Speed** is not stellar now (roughly, a few seconds per page). This is due to a) Parsoid API doesn't support batch-fetching pages, so each page is fetched with a separate HTTP request, and b) this is an unoptimized prototype (so parsing takes a lot more than it could). To fix this, we plan to implement:\n  * [ ] Less naive page caching\n  * [ ] Profiling and optimizations (like probably using naked `lxml` instead of `BeautifulSoup`, and simpler sentence tokenizer)\n  * [ ] Support for async/parallel processing (as far as I can understand, _in Python_ async I/O would be the most useful optimization; but multi-threaded selectors processing will bring no gain due to GIL?)\n* [ ] More robust edge case handling is planned, like links to absent pages, redirects (including redirecting to the middle of another page), disambiguation pages, etc.\n* [ ] Exposing HTTP client dependency to the client code, allowing request logging, custom caching strategies, etc.\n* Some potential far-future features:\n  * [ ] using other languages Wikipedias: nothing in WikipediaQL makes it bound to English Wikipedia only\n  * [ ] (maybe?) other MediaWiki-based sites (like Wikvoyage, Wiktionary, etc.): more high-level selectors (like `infobox` and `wikitable`) would be irrelevant though, and may need a replacement with site-specific ones\n  * [ ] (?) Request analyzer (e.g., prediction of efficiency and number of requests)\n\n## Roadmap\n\n* **0.0.7** more Wikipedia API support (page metadata, page lists etc.)\n* **0.1.0** efficiency \u0026 robustness of existing features\n* **0.2.0** documentation and principal portability to other languages\n* (maybe) online client-side demo, using [Pyodide](https://pyodide.org/en/stable/)?..\n\n## FAQ\n\n### Why not use [Wikidata](https://www.wikidata.org/) (or other structured Wikipedia-based data source, like [DBPedia](https://www.dbpedia.org/))?\n\nWikidata is a massive effort to represent Wikipedia in a computable form. But currently, it contains much less data than Wikipedia itself; and much less accessible for _investigatory_ data extraction (TODO: good examples!) While it gets improved constantly, I wanted to tackle the problem from a different angle and see how accessible we can make Wikipedia itself, with all of its semi-structuredness.\n\n### Why not fetch and parse page sources in Wikitext?\n\nSome similar projects (say, [wtf_wikipedia](https://github.com/spencermountain/wtf_wikipedia)) work by fetching page source in Wikitext form, and parsing it for data extraction. This road looks pretty tempting (and for several years, I went it myself with the previous iteration: [infoboxer Ruby project](https://github.com/molybdenum-99/infoboxer)). The problem here is that at first sight, Wikitext is better structured: large chunks of data are represented by [templates](https://en.wikipedia.org/wiki/Wikipedia:Templates) like `{{Infobox; field=value, ...}}` so it really _seems_ like a better source for data extraction. There are two huge problems with this approach, though:\n\n1. The list of templates is infinite and unpredictable. While ten city-related pages would have `{{Infobox city` in a pretty similar form, the eleventh will have `{{Geobox capital` with all the different fields and conventions—but in HTML they would render to the similarly-looking `\u003ctable class=\"infobox\"`. Or, some TV series will represent a list of episodes with just a plain table markup, while the other will use a sophisticated `{{Episode list` template. And it all might change with time (some spring cleanup replacing all the template names or converting some regular text to a template). The HTML version is _much_ more stable and predictable.\n2. To library/QL users, having Wikitext\u0026templates as the base would mean that writing queries requires intimate knowledge of Wikitext format and conventions and frequently looking at the page's source (via \"Edit\") to find out how something is represented in Wikitext. OTOH, HTML/CSS-based approach relies on widely known elements like headers, lists, links, tables, and simple grouping objects, like sections and sentences. Most of them are unambiguously deduced by looking at the page in the browser or, in the worst case, by \"inspect element\" (to find out its particular class/id).\n\n## Prior work\n\nThis project is the N-th iteration of the ideas about providing \"common knowledge\" in a computable form. Most of the previous work was done in Ruby and centered around **[Reality](https://github.com/molybdenum-99/reality)** project; and included, amongst other things, **[Infoboxer](https://github.com/molybdenum-99/infoboxer)** the Wikipedia parser/high-level client, and [MediaWiktory](https://github.com/molybdenum-99/mediawiktory) the idiomatic low-level MediaWiki client. Some of that work still to be incorporated into WikipediaQL and sister projects.\n\nThat project was once inspired by [\"integrated knowledge\"](https://www.wolfram.com/knowledgebase/) feature of Wolfram Language, I've talked about it (and other topics leading to this project) in a [Twitter thread](https://twitter.com/zverok/status/1397651367909629953) (yes).\n\nThe WikipediaQL syntax seems to be subconsciously inspired by [qsx](https://github.com/danburzo/qsx) selectors language. (By _subconsciously_ I mean I don't remember thinking \"Oh, I should do something similar\", but the day I've published WikipediQL, [past.codes](https://past.codes/) service have reminded me I starred `qsx` in December 2020. I started to think about WikipediaQL syntax in June 2021, but there are striking similarities, so it should be related to some indirect inspiration by that project.)\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzverok%2Fwikipedia_ql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzverok%2Fwikipedia_ql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzverok%2Fwikipedia_ql/lists"}