{"id":16010207,"url":"https://github.com/nichtich/jq-wikidata","last_synced_at":"2025-03-18T02:32:26.800Z","repository":{"id":138803600,"uuid":"183804748","full_name":"nichtich/jq-wikidata","owner":"nichtich","description":"jq module to process Wikidata JSON format","archived":false,"fork":false,"pushed_at":"2019-06-04T19:55:51.000Z","size":73,"stargazers_count":11,"open_issues_count":0,"forks_count":1,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-02-28T06:21:22.095Z","etag":null,"topics":["jq","json","wikibase","wikidata"],"latest_commit_sha":null,"homepage":null,"language":"JSONiq","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nichtich.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-04-27T17:51:50.000Z","updated_at":"2024-12-13T11:35:26.000Z","dependencies_parsed_at":null,"dependency_job_id":"fb68067e-190b-49e7-ac6c-efe7352bf049","html_url":"https://github.com/nichtich/jq-wikidata","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nichtich%2Fjq-wikidata","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nichtich%2Fjq-wikidata/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nichtich%2Fjq-wikidata/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nichtich%2Fjq-wikidata/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nichtich","download_url":"https://codeload.github.com/nichtich/jq-wikidata/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243894541,"owners_count":20365048,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["jq","json","wikibase","wikidata"],"created_at":"2024-10-08T13:05:01.121Z","updated_at":"2025-03-18T02:32:26.789Z","avatar_url":"https://github.com/nichtich.png","language":"JSONiq","funding_links":[],"categories":[],"sub_categories":[],"readme":"# jq-wikidata\n\n[![Build Status](https://travis-ci.org/nichtich/jq-wikidata.svg?branch=master)](https://travis-ci.org/nichtich/jq-wikidata)\n\n\u003e jq module to process Wikidata JSON format\n\nThis git repository contains a module for the [jq data transformation language](https://stedolan.github.io/jq/) to process entity data from [Wikidata](https://www.wikidata.org) or other [Wikibase](http://wikiba.se/) instances serialized in its JSON format.\n\nSeveral methods exist [to get entity data from Wikidata](https://www.wikidata.org/wiki/Wikidata:Data_access).\nThis module is designed to process entities [in their JSON serialization](https://www.mediawiki.org/wiki/Wikibase/DataModel/JSON)\nespecially for large numbers of entities.  Please also consider using a dedicated client such as\n[wikidata-cli] instead.\n\n[wikidata-cli]: https://www.npmjs.com/package/wikidata-cli\n\n## Table of Contents\n\n* [Install](#install)\n* [Usage](#usage)\n  * [Process JSON dumps](#process-json-dumps)\n  * [Per-item access](#per-item-access)\n  * [Reduce entity data](#reduce-entity-data)\n* [API](#api)\n  * [Reduce item](#reduce-item)\n  * [Reduce property](#reduce-property)\n  * [Reduce labels](#reduce-labels)\n  * [Reduce descriptions](#reduce-descriptions)\n  * [Reduce aliases](#reduce-aliases)\n  * [Reduce sitelinks ](#reduce-sitelinks)\n  * [Reduce claims](#reduce-claims)\n  * [Reduce claim](#reduce-claim)\n  * [Reduce references](#reduce-references)\n  * [Reduce lexeme](#reduce-lexeme)\n  * [Reduce forms](#reduce-forms)\n  * [Reduce info](#reduce-info)\n  * [Stream an array of entities](#stream-an-array-of-entities)\n* [Contributing](#contributing)\n* [License](#license)\n\n## Install\n\nInstallation requires [jq](https://stedolan.github.io/jq/) version 1.5 or newer.\n\nPut `wikidata.jq` to a place where jq can [find it as module](https://stedolan.github.io/jq/manual/#Modules).\nOne way to do so is to check out this repository to directory `~/.jq/wikidata/`:\n\n~~~sh\nmkdir -p ~/.jq \u0026\u0026 git clone https://github.com/nichtich/jq-wikidata.git ~/.jq/wikidata\n~~~\n\n## Usage\n\nThe shortest method to use functions of this jq module is to directly `include` the module. Try to process a single Wikidata entity (see below for details about [per-item acces](#per-item-access)):\n\n~~~sh\nwget http://www.wikidata.org/wiki/Special:EntityData/Q42.json\njq 'include \"wikidata\"; .entities[].labels|reduceLabels' Q42.json\n~~~\n\nIt is recommended to put Wikidata entities in a newline delimited JSON file:\n\n~~~sh\njq -c .entities[] Q42.json \u003e entities.ndjson\njq -c 'include \"wikidata\"; .labels|reduceLabels' entities.ndjson\n~~~\n\nMore complex scripts should better be put into a `.jq` file:\n\n~~~jq\ninclude \"wikidata\";\n\n.labels|reduceLabels\n~~~\n\nThe file can then be processed this way:\n\n~~~sh\njq -f script.jq entities.ndjson\n~~~\n\n### Process JSON dumps\n\nWikidata JSON dumps are made available at \u003chttps://dumps.wikimedia.org/wikidatawiki/entities/\u003e.\nThe current dumps exceed 35GB even in its most compressed form. The file contains one large JSON\narray so it should better be converted into a stream of JSON objects for further processing.\n\nWith a fast and stable internet connection it's possible to process the dump on-the fly like this:\n\n~~~sh\ncurl -s https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.json.bz2 \\\n  | bzcat | jq -nc --stream 'include \"wikidata\"; ndjson' | jq .id\n~~~\n\n### Per-item access\n\nJSON data for single entities can be ontained via the\n[Entity Data URL](https://www.wikidata.org/wiki/Special:EntityData). Examples:\n\n* \u003chttps://www.wikidata.org/wiki/Special:EntityData/Q42.json\u003e\n* \u003chttps://www.wikidata.org/wiki/Special:EntityData/L3006.json\u003e\n* \u003chttps://www.wikidata.org/wiki/Special:EntityData/L3006-F1.json\u003e\n\nThe module function `entity_data_url` creates these URLs from Wikidata\nitentifier strings. The resulting data is wrapped in JSON object; unwrap with\n`.entities|.[]`:\n\n~~~bash\ncurl $(echo Q42 | jq -rR 'include \"wikidata\"; entity_data_url') | jq '.entities|.[]'\n~~~\n\nAs mentioned above you better use [wikidata-cli] for accessing small sets of items:\n\n~~~bash\nwd d Q42\n~~~\n\nTo get sets of items that match a given criteria either use SPARL or MediaWiki API module\n[wbsearchentities] and/or MediaWiki API module [wbgetentities].\n\n[wbsearchentities]: https://www.wikidata.org/w/api.php?action=help\u0026modules=wbsearchentities\n[wbgetentities]: https://www.wikidata.org/w/api.php?action=help\u0026modules=wbgetentities\n\n### Reduce entity data\n\nUse function [reduceEntity](#reduce-entity) or more specific functions\n([reduceInfo](#reduce-info), [reduceItem](#reduce-item),\n[reduceProperty](#reduceProperty), [reduceLexeme](#reduceLexeme)) to\nreduce the JSON data structure without loss of essential information.\n\nFurher select only some specific fields if needed:\n\n~~~jq\njq '{id,labels}' entities.ndjson\n~~~\n\n## API\n\n### Reduce Entity\n\nApplies [reduceInfo](#reduce-info) and one of [reduceItem](#reduce-item),\n[reduceProperty](#reduce-property), [reduceLexeme](#reduce-lexeme).\n\n~~~jq\nreduceEntity\n~~~\n\n### Reduce item\n\nSimplifies labels, descriptions, aliases, claims, and sitelinks of an item.\n\n~~~jq\nreduceItem\n~~~\n\n### Reduce property\n\nSimplifies labels, descriptions, aliases, and claims of a property.\n\n~~~jq\nreduceProperty\n~~~\n\n### Reduce labels\n\n~~~jq\n.labels|reduceLabels\n~~~\n\n### Reduce descriptions\n\n~~~jq\n.descriptions|reduceDescriptions\n~~~\n\n### Reduce aliases\n\n~~~jq\n.aliases|reduceAliases\n~~~\n\n### Reduce sitelinks\n\n~~~jq\n.sitelinks|reduceSitelinks\n~~~\n\n### Reduce lexeme\n\nSimplifies lemmas, forms, and senses of a lexeme entity.\n\n~~~jq\nreduceLexeme\n~~~\n\n### Reduce forms\n\n~~~jq\n.forms|reduceForms\n~~~\n\n### Reduce senses\n\n~~~jq\n.senses|reduceSenses\n~~~\n\n### Reduce claims\n\nRemoves unnecessary fields `.id`, `.hash`, `.type`, `.property` and simplifies\nvalues for each claim.\n\n~~~jq\n.claims|reduceClaims\n~~~\n\n### Reduce claim\n\nReduces a single claim value.\n\n~~~jq\n.claims.P26[]|reduceClaim\n~~~\n\n### Reduce references\n\n...\n\n### Reduce forms\n\nOnly lexemes have forms.\n\n~~~\n.forms|reduceForms\n~~~\n\n### Reduce info\n\n~~~jq\nreduceInfo\n~~~\n\nRemoves additional information fields `pageid`, `ns`, `title`, `lastrevid`, and `modified`.\n\nTo remove selected field see jq function [`del`](https://stedolan.github.io/jq/manual/#del(path_expression)).\n\n### Stream an array of entities\n\nModule function `ndjson` can be used to process a stream with an array of\nentities into a list of entities:\n\n~~~sh\nbzcat latest-all.json.bz2 | jq -n --stream 'import \"wikidata\"; ndjson'\n~~~\n\nAlternative, possibly more performant methods to process array of entities [are described here](https://lucaswerkmeister.de/posts/2017/09/03/wikidata+dgsh/):\n\n~~~sh\nbzcat latest-all.json.bz2 | head -n-1 | tail -n+2 | sed 's/,$//'\n~~~\n\n## Contributing\n\nThe source code is hosted at \u003chttps://github.com/nichtich/jq-wikidata\u003e.\n\nBug reports and feature requests [are welcome](https://github.com/nichtich/jq-wikidata/issues/new)!\n\n## License\n\nMade available under the MIT License by Jakob Voß.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnichtich%2Fjq-wikidata","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnichtich%2Fjq-wikidata","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnichtich%2Fjq-wikidata/lists"}