{"id":13451868,"url":"https://github.com/wikimedia/eventgate","last_synced_at":"2025-04-30T04:35:02.312Z","repository":{"id":39584102,"uuid":"159851652","full_name":"wikimedia/eventgate","owner":"wikimedia","description":"Library and HTTP Service for validating JSONSchema events and producing them (to Kafka or elsewhere)","archived":false,"fork":false,"pushed_at":"2024-07-18T03:30:25.000Z","size":765,"stargazers_count":22,"open_issues_count":1,"forks_count":10,"subscribers_count":22,"default_branch":"master","last_synced_at":"2025-04-27T05:04:57.629Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wikimedia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-11-30T16:42:25.000Z","updated_at":"2024-04-19T05:31:45.000Z","dependencies_parsed_at":"2023-01-17T16:15:41.491Z","dependency_job_id":"784d074c-39ea-4ffe-a6ee-7a87994a0866","html_url":"https://github.com/wikimedia/eventgate","commit_stats":{"total_commits":491,"total_committers":21,"mean_commits":23.38095238095238,"dds":0.4582484725050916,"last_synced_commit":"bad4f1c09b103841b65e682d3fae10b91a19a821"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wikimedia%2Feventgate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wikimedia%2Feventgate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wikimedia%2Feventgate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wikimedia%2Feventgate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wikimedia","download_url":"https://codeload.github.com/wikimedia/eventgate/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251644262,"owners_count":21620617,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T07:01:05.227Z","updated_at":"2025-04-30T04:35:02.283Z","avatar_url":"https://github.com/wikimedia.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/wikimedia/eventgate.svg?branch=master)](https://travis-ci.org/wikimedia/eventgate)\n[![Coverage Status](https://coveralls.io/repos/github/wikimedia/eventgate/badge.svg?branch=master)](https://coveralls.io/github/wikimedia/eventgate?branch=master)\n\n# EventGate\n\nEventGate takes in JSON events via HTTP POST, validates and then produces them\nto a pluggable destination (usually to Kafka). Valid events pass through the\ngate, invalid ones do not.\n\nThroughout this codebase, an 'event' is a parsed JSON object with\na strict JSONSchema, and a 'schema' refers to an instance of a JSONSchema for\nan event.\n\n# Usage\n\nFor configuration of the default EventGate implementation (see below), edit\nconfig.yaml and run `npm start`.  The service will listen for HTTP POST\nrequests with Content-Type set to `application/json` at `/v1/events`.  The POST\nbody should be an array of JSON event objects.  (The default EventGate uses\nKafka, so you must have a running Kafka instance to produce to.)\n\n\n# Architecture\n\nThe EventGate service is a generic HTTP POST JSON event intake, event schema validation\nand event producing service.  The schema validation and event produce implementation\nis left up to the user.  This service ships with a schema URL based\nvalidator and Kafka produce implementation (using node-rdkafka), but you can\nplug in your own by implementing a factory module that returns an instantiated `EventGate`\nand use it via `eventgate_factory_module` application config.  See documentation below for more on\nthe default Kafka EventGate.\n\n## EventGate implementation configuration\n\nThe `EventGate` class is generic enough to be used with any type of validation and event\nproduction via function injection.  The HTTP route that handles POSTing of events needs to have an\ninstantiated EventGate instance.  To make this configurable (without editing the route code), the\nroute in routes/events.js will look for `app.conf.eventgate_factory_module` and require it.\nThis module is expected to export a function named `factory` that takes a conf `options` object,\na bunyan `logger`, and an optional `metrics` object that conforms to the node-statsd interface\n(This will actually be provided from\n[service-runner app metrics](https://github.com/wikimedia/service-runner#metric-reporting), and\nthe Express router used to handle the events (in case your EventGate implementation wants to\nadd any extra routes at this time).  E.g.\n\n```javascript\nfunction myEventGateFactory(options, logger, metrics, router) { ... }\n````\nThe factory function should return a Promise of an instantiated EventGate.\n\nOnce the EventGate Promise resolves, the `/v1/events` HTTP route will be added and will use the\ninstantiated EventGate to validate and produce incoming events.\n\n\n## EventGate class\n\nThe `EventGate` class in lib/eventgate.js handles event validation and produce logic.\nIt is instantiated with `validate` and a `produce` functions that each take a single\n`event` and extra `context` object.  `validate` should either return a Promise of the validated\nevent (possibly augmented, e.g. field defaults populated) or throw a `ValidationError`.\n`produce` is expected to return a Promise of the `produce` result, or throw an\n`Error`.\n\nOnce instantiated with these injected functions, an `EventGate` instance can be used\nby calling the `process` function with an array of events and any extra `context`.\n`process` will catch both validation and produce errors, and will return an object\nwith errors for each event grouped by the type of error.  E.g. if you passed in 3 events,\nand 1 was valid, 1 was invalid, and another encountered an unknown error, `process`\nwill return an object like:\n\n```$javascript\n{\n    'success': [{ ... }],\n    'invalid': [{ ... }],\n    'error':   [{ ... }]\n}\n```\nwith each array containing an `EventStatus` instance representing the event's process result.\n\nThroughout the code, functions that are injected into to constructors are expected\nto take a single `event` object and a `context` object.\nE.g. `validate(event, context)`, `produce(event, context)`, etc.  These\nfunctions should be able to do their job with only these arguments.\nAny additional logic or context should be bound into the function, either by partially applying\nor creating a closure.  Example:\n\nIf you wanted to write all incoming events to a file based on some field in the event, you\ncould write an EventGate `produce` function like this:\n\n```javascript\nconst _ require('lodash');\nconst writeFileAsync = promisify(fs.writeFile);\n/**\n * @return a function that writes event to the file at event[file_path_field]\n */\nfunction makeProduceFunction(options) {\n    const file_path_field = options.file_path_field;\n    return (event, context = {}) =\u003e {\n        const path = _.get(event, file_path_field);\n        return writeFileAsync(path, JSON.stringify(event));\n    }\n}\n\n// Instantiate a new EventGate using this configured produce function closure:\nconst eventGate = new EventGate({\n    // ...,\n    const options = {\n        file_path_field: 'file_path'\n    }\n    produce: makeProduceFunction(options);\n    // ...,\n})\n\n// This eventGate will produce every event by calling the function returned by makeProduceFunction.\n```\n\n### Error events\n\nThe EventGate constructor also takes an optional `mapToErrorEvent` function.\n`mapToErrorEvent` takes an `event` and an `error` and returns a new error event representing\nthe error.  This new error event should be suitable for producing through the same\ninstantiated `EventGate`.  If `mapToErrorEvent` is provided and event processing fails for any\nreason, those errors will be converted to event errors via this function, and then produced.\nThere should be no difference between the kind of events that `mapToErrorEvent` returns and\nthe kind of events that your instantiated `EventGate` can handle.\n`eventGate.process([errorEvents])` should work.  If your `mapToErrorEvent` implementation\nreturns `null` for any given failed `event`, no error event for that error will be\nproduced.  This allows `mapToErrorEvent` implementations to decide what types of\nErrors should be produced.\n\n# Default EventGate - Schema URI validation \u0026 producing with Kafka\n\nIf `eventgate_factory_module` is not specified, this service will use provided configuration\nto instantiate and use an EventGate that validates events with JSONSchemas discovered via\nschema URLs.  Depending on configuration, the default EventGate can write events to\na file, and/or produce them to Kafka.\n\n## EventValidator class \u0026 event schema URLs\n\nWhile the `EventGate` class that handles event validation and production can\nbe configured to do validation in any way, the default EventGate uses the\n`EventValidator` class to validate events with schemas obtained from\nschema URIs in the events themselves. Every event should contain a URI\nto the JSONSchema for the event.  `EventValidator` will extract and use those URIs to look up\n(and cache) those schemas to use for event validation.  The `EventValidator` instance\nused by the default EventGate can request schema URIs from the local filesystem with\n`file://` or remote ones via `http://` based URIs.  The field in the each event where the schema\nURI is located is configured by the `schema_uri_field` config, which defaults to `$schema`.\n(`$schema` is a standard field used by JSONSchemas to locate their metaschemas, so it makes\nsense to store an event's schema in this field as well.)\nWhen an event is received, the schema URI will be extracted from the `schema_uri_field`.\nThe JSONSchema at the URI will be downloaded and used to validate the event.\nThe extracted schema URI can optionally be prefixed with `schema_base_uri` and suffixed with\n`schema_file_extension`.  Setting a `schema_base_uri` will allow set hostname relative URIs,\ne.g. '/path/to/event-schema/1.0.0' in the events, while configuring the base location of your schema\nrepositories, e.g. 'http://my.schema-repo.org/schemas' If you\nuse the defaults, all of your events should have a $schema field set to\na resolvable schema URI.\n\nEventValidator always supports draft-07 JSONSchemas (via AJV).  It also\nsupports additional JSONSchemas, defaulting to also supporting draft-04.\nIf you need to modify or change this, you can override the `metaSchemas` array\noption to the EventValidator constructor.\n\nBy default, EventValidator will validate all (non-meta) schemas with AJV's\njson-schema-secure schema.  This prevents schemas from including potentially\nrisky features that could facilitate DOS attacks.\nSee: https://github.com/epoberezkin/ajv#security-considerations\nTo allow insecure schemas, you can set `allowInsecureSchemas: true` in\nthe EventValidator constructor options.\n\n## Streams\nA 'stream' here refers to the destination name of an event.  It is closely related\nto Kafka's concept of a topic.  Much of the time a stream might correspond 1:1 with\na Kafka topic.  If you don't care about the topic name that is used for a any given event,\nyou don't need to configure a `stream_field`.  The default behavior is to sanitize an event's\nschema URI and use it for the Kafka topic name.  E.g. if an event's schema URI is\n`/ui/element/button-push`, the topic name will end up being `ui_element_button-push`.\nHowever, if `stream_field` is configured and present in an event, its value will be\nused as the destination Kafka topic of that event. If you need finer control over\nevent -\u003e Kafka topic mapping, you should implement your own Kafka produce function\n(see\n[eventgate-wikimedia](https://gerrit.wikimedia.org/r/plugins/gitiles/eventgate-wikimedia/+/refs/heads/master/eventgate-wikimedia.js))\\\nfor an example.)\n\n# Configuration\n\nConfiguration is passed to the service via the `config.yaml` file, which\nhas a `services` object.  This object contains a service named `eventgate`.\nThe `conf` object of this service will be passed to the `eventgate_factory_module`.\nTo use a custom EventGate implementation, set `eventgate_factory_module` to your\njavascript module that exports a `factory` function that instantiate an EventGate with\n`options`.  See the section above entitled 'EventGate implementation configuration'.\n\n## Default EventGate configuration\n\nThe following values in `conf` will be used to instantiate a a default EventGate\nthat extracts JSONSchemas from schema URIs, validates events using those\nschemas, and then produces them to an output file and or Kafka.\nIf `kafka.conf` is set, than Kafka will be used.  Since node-rdkafka-factory\nis an optional dependency, please make sure it (and node-rdkafka) is properly\ninstalled if you use this.\n\nAll `*_field` configs point to a field in an event, and use\ndotted path notation to access sub-objects.\nE.g. \"value\" will be extracted from `{ meta: { stream: \"value\" } }` if\n`stream_field` is set to 'meta.stream'.\n\nProperty                    |         Default | Description\n----------------------------|-----------------|--------------------------\n`port`                      |            6927 | port on which the service will listen\n`interface`                 |       localhost | hostname on which to listen\n`user_agent`                |        eventgate | The UserAgent seen when making remote http requests (e.g. for remote schemas)\n`schema_uri_field`          |         $schema | The value extracted from this field will be used (with `schema_base_uris` and `schema_file_extension`) to download the event's JSONSchema for validation.\n`schema_base_uris`          |       undefined | If given, a relative schema URI will be prepended with each of these base URIs to build schema URLs.  The resulting URLs will each be requested, and the first existent schema found at that URL will be used. This allows you to configure multiple schema repositories/registries where your schema might be located.  E.g. you could use this if you wanted to have some schemas locally for reliability, but remote for resolvability.\n`schema_file_extension`     |       undefined | If given, this will be appended to every extracted schema URI unless the filename in the URI already has an extension.\n`stream_field`              |       undefined | The name of the stream this event belongs to. If not set, `schema_uri_field` will be used (and sanitized) instead.\n`output_path`               |          stdout | Path to file to write valid events to, or 'stdout'. If undefined, events will not be output to a file.\n`kafka.conf`                |       undefined | [node-rdkafka](https://blizzard.github.io/node-rdkafka/current/) / [librdkafka](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md) configuration.  This will be passed directly to the node-rdkafka `kafka.Producer` constructor.  Make sure you set kafka.conf['metadata.broker.list'].  If undefined, events will not be produced to Kafka.\n`kafka.topic_conf`          |       undefined | node-rdkafka (and librdkafka) topic specific configuration.  This will be passed directly to the node-rdkafka `kafka.Producer` constructor.\n\n# eventgate-wikimedia implementation and use as a dependency\nThe Wikimedia Foundation runs this EventGate service as a dependency of [eventgate-wikimedia](https://gerrit.wikimedia.org/g/eventgate-wikimedia/+/refs/heads/master).  WMF implements a custom\nEventGate factory in [eventgate-wikimedia.js](https://gerrit.wikimedia.org/r/plugins/gitiles/eventgate-wikimedia/+/refs/heads/master/eventgate-wikimedia.js).\n\nIf you are using EventGate as an npm dependency in a custom implementation repository, you will\nneed to configure service-runner in your config.yaml file to run the EventGate express app.  Set:\n\n```yaml\nservices:\n  - name: EventGate-mycustom-implementation\n    # Load the eventgate module which is installed as an npm dependency\n    module: eventgate\n    # Make service-runner start running via tha exported app function.\n    entrypoint: app\n    # ...\n```\n\n# /v1/_test/events route\nIf you are using EventGate as a service, and if `test_events` is configured,\na `GET /v1/_test/events` route will be added. When requested, the `test_events` will be produced as if\nthey were POSTed to /v1/events. This is useful for readiness probes that want to make sure the service can\nproduce events end to end.\n\n# service-template-node\n\nThis service is based on Wikimedia's [service-template-node](https://github.com/wikimedia/service-template-node).  It is a fork of that 'template'\nrepository.  See also the [ServiceTemplateNode documentation](https://www.mediawiki.org/wiki/ServiceTemplateNode).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwikimedia%2Feventgate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwikimedia%2Feventgate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwikimedia%2Feventgate/lists"}