{"id":15023142,"url":"https://github.com/chlodalejandro/wikimedia-streams","last_synced_at":"2026-03-03T08:43:04.887Z","repository":{"id":37851581,"uuid":"384770034","full_name":"ChlodAlejandro/wikimedia-streams","owner":"ChlodAlejandro","description":"Receive events from Wikimedia wikis using the Wikimedia Event Platforms' EventStreams.","archived":false,"fork":false,"pushed_at":"2024-10-07T10:41:43.000Z","size":3159,"stargazers_count":6,"open_issues_count":7,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-10-11T02:03:25.334Z","etag":null,"topics":["events","eventsource","eventstreams","mediawiki","sse","wikimedia","wikipedia"],"latest_commit_sha":null,"homepage":"https://chlodalejandro.github.io/wikimedia-streams/","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ChlodAlejandro.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-07-10T18:58:02.000Z","updated_at":"2024-08-31T21:06:54.000Z","dependencies_parsed_at":"2023-10-03T19:11:47.783Z","dependency_job_id":"b5524bae-af0e-4487-945d-37a89d73fcae","html_url":"https://github.com/ChlodAlejandro/wikimedia-streams","commit_stats":{"total_commits":64,"total_committers":4,"mean_commits":16.0,"dds":0.09375,"last_synced_commit":"72d4a013a348212b31d399d555621f99eab5f57e"},"previous_names":[],"tags_count":23,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChlodAlejandro%2Fwikimedia-streams","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChlodAlejandro%2Fwikimedia-streams/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChlodAlejandro%2Fwikimedia-streams/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChlodAlejandro%2Fwikimedia-streams/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ChlodAlejandro","download_url":"https://codeload.github.com/ChlodAlejandro/wikimedia-streams/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219864035,"owners_count":16555943,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["events","eventsource","eventstreams","mediawiki","sse","wikimedia","wikipedia"],"created_at":"2024-09-24T19:58:46.120Z","updated_at":"2026-03-03T08:43:04.809Z","avatar_url":"https://github.com/ChlodAlejandro.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# wikimedia-streams\n\u003cimg align=\"right\" height=\"70\" alt=\"wikimedia-streams logo\" src=\"https://raw.githubusercontent.com/ChlodAlejandro/wikimedia-streams/master/assets/wikimedia-streams.png\"\u003e\n\n[![npm version](https://img.shields.io/npm/v/wikimedia-streams.svg?style=flat-square)](https://www.npmjs.org/package/wikimedia-streams)\n[![npm downloads](https://img.shields.io/npm/dm/wikimedia-streams.svg?style=flat-square)](http://npm-stat.com/charts.html?package=wikimedia-streams)\n\nwikimedia-streams connects to Wikimedia's [Event Platform EventStreams](https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams) in order to serve real-time changes to Wikimedia wikis. This entire library is typed, which makes parameter handling well-documented and defined.\n\nThis package works best with TypeScript, but also works with plain JavaScript.\n\nBy default, this package requires an [`EventEmitter`](https://nodejs.org/docs/latest/api/events.html#class-eventemitter) polyfill when used on a browser. Special output files with a bundled EventEmitter polyfill for userscripts and gadgets are available; see below for more information. Really old browser may also need an [`EventSource`](https://developer.mozilla.org/en-US/docs/Web/API/EventSource) polyfill; this must be loaded separately, as this package doesn't provide a version bundled with such a polyfill. On Node.js, native `EventEmitter` is used, and [eventsource](https://www.npmjs.com/package/eventsource) is used as an EventSource polyfill. This dependency structure allows the package to have the same signature in the browser and in Node.\n\nThis package always supports the latest version of the spec. Minor version bumps on the spec count as breaking changes on this package. Refer to the following table for spec versions supported by each version.\n\n| Package version       | [Spec version](https://stream.wikimedia.org/?spec) |\n|-----------------------|----------------------------------------------------|\n| `2.0.0`\u0026ndash;`3.0.0` | `0.8.0`, `0.9.0`                                    |\n| `0.1.0`\u0026ndash;`2.0.0` | `0.7.3`                                            |\n\n## Setup\n\nCreate a new WikimediaStream with the following:\n\n```ts\nimport WikimediaStream from \"wikimedia-streams\";\n\n// \"recentchange\" can be replaced with any valid stream. \nconst stream = new WikimediaStream(\"recentchange\");\n```\n\nIf you're using CommonJS imports, you'll need to add `.default` after `require()`.\n```ts\nconst WikimediaStream = require(\"wikimedia-streams\").default;\nconst stream = new WikimediaStream(\"recentchange\");\n```\n\nAdditional files are available under `dist/browser` for browser use:\n* `index.js` – for use in `\u003cscript\u003e` tags and non-wiki pages (requires an EventEmitter polyfill)\n  * `WikimediaStreams` global exists, `WikimediaStreams` namespace is **NOT** exported\n* `bundle.js` – for use in userscripts\n    * `WikimediaStreams` global exists, `WikimediaStreams` namespace is **NOT** exported\n* `lib.js` – for use in MediaWiki-namespace JS files and gadgets\n    * `WikimediaStreams` global does **NOT** exist, `WikimediaStreams` namespace is exported\n\nIf you're using wikimedia-streams in a browser, you have multiple options:\n* If you're using a bundler (Webpack, Browserify, etc.), you can use the same code as above.\n* If you're using a script tag (through JSDelivr, etc.), you'll need to load\n  both wikimedia-streams and an `EventEmitter` polyfill.\n  ```html\n  \u003c!-- Load `eventemitter3` for an EventEmitter polyfill. --\u003e\n  \u003cscript src=\"https://tools-static.wmflabs.org/cdnjs/ajax/libs/eventemitter3/5.0.1/index.min.js\" /\u003e\n  \u003c!-- Try to self-host wikimedia-streams if you can! --\u003e\n  \u003cscript src=\"https://cdn.jsdelivr.net/npm/wikimedia-streams@latest\" /\u003e\n  \u003cscript\u003e\n  \tconst stream = new WikimediaStream.default(\"recentchange\");\n  \u003c/script\u003e\n  ```\n* If you're using `mw.loader.load` (userscripts), you have two options:\n  * You can load both wikimedia-streams and an `EventEmitter` polyfill.\n    ```js\n    await mw.loader.load(\"https://tools-static.wmflabs.org/cdnjs/ajax/libs/eventemitter3/5.0.1/index.min.js\");\n    await mw.loader.load(\"\u003cURL to a reupload of dist/browser/index.min.js\u003e\");\n    const stream = new WikimediaStream.default(\"recentchange\");\n    ```\n  * You can also load a version of wikimedia-streams that includes an `EventEmitter` polyfill.\n    Use this in case you would like to upload the library on-wiki or would like to cut down on\n    request count.\n    ```js\n    await mw.loader.load(\"\u003cURL to a reupload of dist/browser/bundle.min.js\u003e\");\n    const stream = new WikimediaStream.default(\"recentchange\");\n    ```\n* If you're developing a gadget, you should probably use a MediaWiki-namespace JS file\n  for security reasons. If `dist/browser/lib.js` is uploaded as `MediaWiki:Gadget-wikimedia-streams.js`,\n  you can import it using a gadget dependency.\n  ```wikitext\n  \u003c!-- MediaWiki:Gadgets-definition --\u003e\n  * mygadget[ResourceLoader |dependencies=ext.gadget.wikimedia-streams]|mygadget.js\n  * wikimedia-streams[ResourceLoader |package |hidden]|wikimedia-streams.js\n  ```\n  ```js\n  // MediaWiki:Gadget-mygadget.js\n  mw.loader.using(\"ext.gadget.wikimedia-streams\").then(function (require) {\n      var WikimediaStream = require(\"wikimedia-streams\").default;\n  \t  var stream = new WikimediaStream(\"recentchange\");\n  });\n  \n  // or if `|package` is set in mygadget's definition\n  var WikimediaStream = require(\"ext.gadget.wikimedia-streams\").default;\n  var stream = new WikimediaStream(\"recentchange\");\n  ```\n\n## Usage\nAfter setup, you can listen to sent events using `.on`.\n\n```ts\nstream.on(\"recentchange\", (data, event) =\u003e {\n\tif (data.wiki === \"enwiki\") {\n\t\t// Edits from the English Wikipedia\n\t\tconsole.log(data.title); // Output the page title.\n\t}\n});\n```\n\nDon't forget to close the stream when you're done (or else Node will remain open).\n\n```ts\nstream.close();\n```\n\nYou can also use `.on(\"mediawiki.recentchange\")` to listen to recent changes. A full list of streams and their available aliases are provided below.\n\n### Available streams\n\n| **Stream**                                                                                                          | **Aliases** | **Description**                                                                                      |\n|---------------------------------------------------------------------------------------------------------------------|---|------------------------------------------------------------------------------------------------------|\n| [eventgate-main.test.event](https://stream.wikimedia.org/v2/stream/eventgate-main.test.event)                       | `test` | Testing event.                                                                                       |\n| [mediawiki.page-create](https://stream.wikimedia.org/v2/stream/mediawiki.page-create)                               | `page-create` | Newly-created pages.                                                                                 |\n| [mediawiki.page-delete](https://stream.wikimedia.org/v2/stream/mediawiki.page-delete)                               | `page-delete` | Deleted pages.                                                                                       |\n| [mediawiki.page-links-change](https://stream.wikimedia.org/v2/stream/mediawiki.page-links-change)                   | `page-links-change` | Changes to page links.                                                                               |\n| [mediawiki.page-move](https://stream.wikimedia.org/v2/stream/mediawiki.page-move)                                   | `page-move` | Page moves.                                                                                          | \n| [mediawiki.page-properties-change](https://stream.wikimedia.org/v2/stream/mediawiki.page-properties-change)         | `page-properties-change` | Changes to page properties.                                                                          |\n| [mediawiki.page-undelete](https://stream.wikimedia.org/v2/stream/mediawiki.page-undelete)                           | `page-undelete` | Undeleted pages.                                                                                     | \n| [mediawiki.recentchange](https://stream.wikimedia.org/v2/stream/mediawiki.recentchange)                             | `recentchange` | Recent changes. The recent changes schema is drastically different from the schema of other streams. |\n| [mediawiki.revision-create](https://stream.wikimedia.org/v2/stream/mediawiki.revision-create)                       | `revision-create` | Edits to pages.                                                                                      |\n| [mediawiki.revision-tags-change](https://stream.wikimedia.org/v2/stream/mediawiki.revision-tags-change)             |  | Changes to revision tags. Added in v0.4.0.                                                            |\n| [mediawiki.revision-visibility-change](https://stream.wikimedia.org/v2/stream/mediawiki.revision-visibility-change) | | Changes to revision visibility (caused by suppression or revision deletion).                         |\n\n### Removed streams\n| **Stream**                                                                                                          | **Aliases**      | **Description**                                                                                                         |\n|---------------------------------------------------------------------------------------------------------------------|------------------|-------------------------------------------------------------------------------------------------------------------------|\n| [mediawiki.revision-score](https://stream.wikimedia.org/v2/stream/mediawiki.revision-score)                         | `revision-score` | ORES scores for edits to pages. Removed as of v2.0.0 (09-14-2023; [T342116](https://phabricator.wikimedia.org/T342116)) |\n\n### Multiple streams\nYou can listen to multiple streams at once by passing an array as the parameter when creating a WikimediaStream.\n\n```ts\nimport WikimediaStream from \"wikimedia-streams\";\n\nconst stream = new WikimediaStream([\"page-create\", \"revision-create\"]);\n\nstream.on(\"page-create\", (data, event) =\u003e {\n\tif (data.database === \"enwiki\") {\n\t\t// Page created on the English Wikipedia.\n\t}\n});\nstream.on(\"revision-create\", (data, event) =\u003e {\n\tif (data.database === \"enwiki\") {\n\t\t// Page edited on the English Wikipedia.\n\t}\n});\n```\n\n## Filtering\n\nYou can filter a stream using masks. An event must match the provided mask to be accepted.\nFilters are built using the `filter` function, and can only filter one stream type at a time\nto ensure proper typing.\n\n```ts\nconst filter = stream.filter(\"mediawiki.recentchange\");\n```\n\nThree filter modes are provided; these mirror the types used by [Pywikibot](https://doc.wikimedia.org/pywikibot/stable/api_ref/pywikibot.comms.html#comms.eventstreams.EventStreams.register_filter) for parity:\n* `none` skips the event if it matches the mask. If it skips no event, it proceeds to `all` filters.\n* `all` skips the event if it does not match all `all` filters. If it skips no event, it proceeds to `any` filters.\n* `any` skips the event if it does not match any `any` filters.\n\n```ts\nconst filter1 = stream.filter(\"mediawiki.recentchange\");\nfilter1.none({ type: \"categorize\" })\n\t.on((event) =\u003e {\n\t\t// Only edits that aren't \"categorize\" types will be accessible here.\n\t});\n\nconst filter2 = stream.filter(\"mediawiki.recentchange\");\nfilter2\n\t.all({ type: \"edit\" })\n\t.all({ wiki: \"enwiki\" })\n\t.on((event) =\u003e {\n\t\t// Only edits on the English Wikipedia will be accessible here.\n\t});\n\nconst filter3 = stream.filter(\"mediawiki.recentchange\");\nfilter3\n\t.any({ type: \"commonswiki\" })\n\t.any({ wiki: \"enwiki\" })\n\t.on((event) =\u003e {\n\t\t// Only changes on the English Wikipedia and Wikimedia Commons will be accessible here.\n\t});\n```\n\nNote that you are supposed to chain the filter functions together and in order. Type assistance\nwill not be available otherwise. Due to how the types are constructed, compile-time errors are\nemitted to ensure proper use of the code. This is not available in JavaScript, and can lead to\nunexpected behavior if filters are used improperly.\n\n```ts\n// This is an example of IMPROPER usage!!!\n\nconst filter = stream.filter(\"mediawiki.recentchange\");\n\nfilter.all({ type: \"categorize\" })\n\t.on((event) =\u003e {\n\t\t// This will never be called.\n\t});\n\nfilter.all({ type: \"edit\" })\n\t.on((event) =\u003e {\n\t\t// This will never be called.\n\t});\n\n// By using the above two, the functions in `on` will never be called, since the event will\n// only pass through the filter if the edit has a type of both \"categorize\" and \"edit\", which\n// is impossible.\n\n// This is the correct way to clone filters:\nconst filter2 = stream.filter(\"mediawiki.recentchange\");\nfilter2.clone().all({ type: \"categorize\" })\n\t.on((event) =\u003e {\n\t\t// This will be called.\n\t});\nfilter2.clone().all({ type: \"categorize\" })\n\t.on((event) =\u003e {\n\t\t// This will be called.\n\t});\n```\n\n```ts\n// This is an example of IMPROPER usage!!!\n\nstream.filter(\"mediawiki.recentchange\")\n\t.all({ wiki: \"enwiki\" })\n\t.none({ type: \"categorize\" }) // This will fail on compile time.\n\t.on((event) =\u003e {\n\t\t// Though this will correctly provide English Wikipedia new/edit/log events,\n\t\t// types *may* be incorrect.\n\t});\n```\n\nDue to [limitations in TypeScript](https://github.com/microsoft/TypeScript/issues/4196),\nthe received type may be too broad compared to the actual values of the types.\n\n### Examples\n1. Get all edits from the English Wikipedia.\n\t```ts\n\tstream.filter(\"mediawiki.recentchange\")\n\t.all({ wiki: \"enwiki\" })\n\t.all({ type: \"edit\" })\n\t.on((event) =\u003e {\n\t\tconsole.log(`New edit from ${event.user} on \"${event.title}\"`)\n\t});\n\t```\n2. Get all log events from the English Wikipedia.\n\t```ts\n\tstream.filter(\"mediawiki.recentchange\")\n\t.all({ wiki: \"enwiki\" })\n\t.all({ type: \"log\" })\n\t.on((event) =\u003e {\n\t\tconsole.log(`${event.user} performed ${event.log_type}/${event.log_action} on \"${event.title}\"`)\n\t});\n\t```\n3. Get edits from all wikis with a byte difference of greater than 500.\n\t```ts\n\tstream.filter(\"mediawiki.recentchange\")\n\t.all({ wiki: \"enwiki\" })\n\t.all({ type: \"edit\" })\n\t.on((event) =\u003e {\n\t\t// Byte difference is a computed value. This must take place in manual filter.\n\t\tconst byteDiff = event.length.new - event.length.old;\n\t\tif (Math.abs(byteDiff) \u003e 500) {\n\t\t\tconsole.log(`${byteDiff \u003e 0 ? `+${byteDiff}` : byteDiff} bytes ${event.user} on \"${event.title}\"`)\n\t\t}\n\t});\n\t```\n## Resuming streams\n\u003e [!NOTE]\n\u003e This feature is not available in browsers.\n\nAfter every received event, the stream stores the ID of the last event that was sent.\nThis ID can be used to continue streams, so that you don't miss any events. You can\nalso save this ID to a file when gracefully stopping for a restart, and use it again\nat a later time. Note that streams cannot be replayed indefinitely; EventStreams may\nonly hold an event [for a certain duration](https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams#Historical_Consumption).\n\n```ts\nconst stream = new WikimediaStream( 'recentchange' );\n\n// Stopping!\nfs.writeFileSync( 'last-event.json', JSON.stringify( stream.lastEventId ) );\nstream.close();\n\n// Restarting!\nconst stream2 = new WikimediaStream( 'recentchange', {\n\tlastEventId: JSON.parse( fs.readFileSync( 'last-event.json' ).toString( 'utf8' ) )\n} );\n```\n\nWhen re-opening a previously closed stream, the library will automatically resume\nfrom the last event that it processed. To avoid this, instantiate a new `WikimediaStream`.\n\n## User agent\n\u003e [!NOTE]\n\u003e This feature is not available in browsers.\n\nWikimedia sites require developers to follow the [User-Agent policy](https://meta.wikimedia.org/wiki/User-Agent_policy), which requires a descriptive user agent to be sent with requests. By default, wikimedia-streams will send a generic `wikimedia-streams/${VERSION}` User-Agent header. You can set a custom user agent by providing the `headers.User-Agent` option when creating the stream object.\n```ts\nconst stream = new WikimediaStream(\"recentchange\", {\n    headers: {\n        \"User-Agent\": \"MyCoolTool/1.0 (https://example.com/MyCoolTool)\"\n    }\n});\n```\n\n## Canary events\n[Canary events](https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams#Canary_Events)\nare events that are sent to ensure that the stream is still active. These events are filtered\nout by wikimedia-streams by default. To enable them, set the `enableCanary` option to `true`.\nNote that you will be required to filter out these events yourself, or process them accordingly.\n\n## License\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0.\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n\nType documentation is partially derived from https://stream.wikimedia.org/?doc, also licensed under the Apache License, Version 2.0. `spec.json` is downloaded from https://stream.wikimedia.org/?spec, also licensed under the Apache License, Version 2.0.\n\n## Disclaimer\nYou are expected to follow the Wikimedia Foundation [Terms of Use](https://foundation.wikimedia.org/wiki/Terms_of_Use) when accessing EventStreams. The package developer(s) are not liable for any damage caused by you using this package.\n\nIf you're developing a bot that runs on Wikimedia wikis which edits based on changes found on EventStreams, be sure to follow the [bot best practices](https://www.mediawiki.org/wiki/Manual:Creating_a_bot#General_guidelines_for_running_a_bot) when making edits or other changes. This includes setting a proper user agent (required by [policy](https://meta.wikimedia.org/wiki/User-Agent_policy)), which is supported by this package.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchlodalejandro%2Fwikimedia-streams","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchlodalejandro%2Fwikimedia-streams","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchlodalejandro%2Fwikimedia-streams/lists"}