{"id":15592158,"url":"https://github.com/derhuerst/build-wikipedia-feed","last_synced_at":"2025-04-15T05:14:58.850Z","repository":{"id":22910796,"uuid":"97652558","full_name":"derhuerst/build-wikipedia-feed","owner":"derhuerst","description":"Build a hyperdb of Wikipedia articles.","archived":false,"fork":false,"pushed_at":"2022-04-07T21:49:10.000Z","size":62,"stargazers_count":24,"open_issues_count":3,"forks_count":4,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-28T16:11:51.499Z","etag":null,"topics":["database","hyperdb","p2p","wikipedia"],"latest_commit_sha":null,"homepage":"https://github.com/derhuerst/build-wikipedia-feed","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"isc","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/derhuerst.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"license.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-07-18T23:33:12.000Z","updated_at":"2024-05-26T11:55:28.000Z","dependencies_parsed_at":"2022-07-27T03:47:08.155Z","dependency_job_id":null,"html_url":"https://github.com/derhuerst/build-wikipedia-feed","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derhuerst%2Fbuild-wikipedia-feed","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derhuerst%2Fbuild-wikipedia-feed/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derhuerst%2Fbuild-wikipedia-feed/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derhuerst%2Fbuild-wikipedia-feed/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/derhuerst","download_url":"https://codeload.github.com/derhuerst/build-wikipedia-feed/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248325473,"owners_count":21084932,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","hyperdb","p2p","wikipedia"],"created_at":"2024-10-02T23:56:11.583Z","updated_at":"2025-04-15T05:14:58.833Z","avatar_url":"https://github.com/derhuerst.png","language":"JavaScript","funding_links":["https://patreon.com/derhuerst"],"categories":[],"sub_categories":[],"readme":"# build-wikipedia-feed\n\n**Build a [hyperdrive](https://npmjs.com/package/hyperdrive) feed of Wikipedia articles**, including historical revisions.\n\n[![npm version](https://img.shields.io/npm/v/build-wikipedia-feed.svg)](https://www.npmjs.com/package/build-wikipedia-feed)\n[![build status](https://img.shields.io/travis/derhuerst/build-wikipedia-feed.svg)](https://travis-ci.org/derhuerst/build-wikipedia-feed)\n![ISC-licensed](https://img.shields.io/github/license/derhuerst/build-wikipedia-feed.svg)\n[![chat on gitter](https://badges.gitter.im/derhuerst.svg)](https://gitter.im/derhuerst)\n[![support me on Patreon](https://img.shields.io/badge/support%20me-on%20patreon-fa7664.svg)](https://patreon.com/derhuerst)\n\n\n## Rationale\n\n### Problem\n\n[Wikipedia](https://en.wikipedia.org/wiki/Wikipedia) is an incredibly important collection of knowledge on the internet. It is free to read and edit for everyone. Though, it is important to know that **it is stored on only a handful of servers in a handful of countries controlled by a single organisation**. This mainly causes two problems:\n\n- Currently, it is **too easy to censor** the Wikipedia. We need a system that supports redundancy without any additional effort.\n- It **does not work offline**. Making an offline copy is complicated. Also, you usually have to download all articles for a language.\n\n### Solution\n\nLet's **store Wikipedia's content in a [peer-to-peer (P2P) system](https://en.wikipedia.org/wiki/Peer-to-peer)**. By leveraging software from [the *Dat* project](https://docs.datproject.org), we won't have to reinvent the wheel. [The *Dat* protocol](https://github.com/datproject/docs/blob/master/papers/dat-paper.pdf) efficiently only syncs changes between to versions of data, allows for sparse \u0026 live replication and is completely [distributed](https://en.wikipedia.org/wiki/Peer-to-peer#Unstructured_networks).\n\nThis tool can extract articles from a [Wikipedia dump](https://dumps.wikimedia.org/enwiki) or download it directly, and store it in a *Dat* archive. See below for more details.\n\n\n## Installing\n\n```shell\nnpm install -g build-wikipedia-feed\n```\n\n\n## Usage\n\nThis module exposes several command line building blocks.\n\n### read *all* revisions of every article\n\nPipe [a `stub-meta-history*` XML file](https://dumps.wikimedia.org/enwiki/) into `wiki-revisions-list`. You will get an [ndjson](http://ndjson.org) list of page revisions.\n\n```shell\ncurl -s 'https://dumps.wikimedia.org/enwiki/20181001/enwiki-20181001-stub-meta-history1.xml.gz' | gunzip | wiki-revisions-list \u003erevisions.ndjson\n```\n\n### read the *most recent* revision of every article\n\nPipe [a `stub-meta-current*` XML file](https://dumps.wikimedia.org/enwiki/) file into `wiki-revisions-list`. You will get an [ndjson](http://ndjson.org) list of page revisions.\n\n```shell\ncurl -s 'https://dumps.wikimedia.org/enwiki/20181001/enwiki-20181001-stub-meta-current1.xml.gz' | gunzip | wiki-revisions-list \u003erevisions.ndjson\n```\n\n### read articles being *edited right now*\n\nUse `wiki-live-revisions`. You will get an [ndjson](http://ndjson.org) list of page revisions.\n\n```shell\nwiki-live-revisions \u003erevisions.ndjson\n```\n\n### fetch \u0026 store revisions in a [hyperdrive](https://npmjs.com/package/hyperdrive)\n\nUse `wiki-store-revisions` to write the HTML content of all revisions in `revisions.ndjson` into a [hyperdrive](https://npmjs.com/package/hyperdrive). The archive will be created under `p2p-wiki` in [your system's data directory](https://github.com/sindresorhus/env-paths#usage).\n\n```shell\ncat revisions.ndjson | wiki-store-revisions\n```\n\n\n## Related\n\n- [distributed-wikipedia-mirror](https://github.com/ipfs/distributed-wikipedia-mirror) – Putting wikipedia on IPFS\n- [`fetch-wikipedia-page-revision`](https://github.com/derhuerst/fetch-wikipedia-page-revision#fetch-wikipedia-page-revision) – Fetch a revision of a Wikipedia page as mobile HTML.\n- [`wikipedia-edits-stream`](https://github.com/derhuerst/wikipedia-edits-stream#wikipedia-edits-stream) – A live stream of page edits on Wikipedia.\n- [`commons-photo-url`](https://github.com/derhuerst/commons-photo-url#commons-photo-url) – Download Wikimedia Commons photos.\n- [`wiki-article-name-encoding`](https://github.com/derhuerst/wiki-article-name-encoding#wiki-article-name-encoding) – Encode \u0026 decode Wiki(pedia) article names/slugs.\n\n\n## Contributing\n\nIf you have a question or have difficulties using `build-wikipedia-feed`, please double-check your code and setup first. If you think you have found a bug or want to propose a feature, refer to [the issues page](https://github.com/derhuerst/build-wikipedia-feed/issues).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fderhuerst%2Fbuild-wikipedia-feed","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fderhuerst%2Fbuild-wikipedia-feed","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fderhuerst%2Fbuild-wikipedia-feed/lists"}