{"id":17281399,"url":"https://github.com/mr-karan/nomad-events-sink","last_synced_at":"2025-04-14T09:51:41.898Z","repository":{"id":37093881,"uuid":"426582267","full_name":"mr-karan/nomad-events-sink","owner":"mr-karan","description":"An events collection agent which processes Nomad Events and dumps to external sink providers like HTTP","archived":false,"fork":false,"pushed_at":"2023-06-21T05:45:50.000Z","size":316,"stargazers_count":52,"open_issues_count":5,"forks_count":9,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-06T05:03:47.258Z","etag":null,"topics":["nomad"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mr-karan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-11-10T10:40:57.000Z","updated_at":"2025-03-09T08:50:30.000Z","dependencies_parsed_at":"2024-06-19T00:25:38.854Z","dependency_job_id":"6db8ad37-d741-425b-96bd-ecbb4c6aea81","html_url":"https://github.com/mr-karan/nomad-events-sink","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mr-karan%2Fnomad-events-sink","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mr-karan%2Fnomad-events-sink/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mr-karan%2Fnomad-events-sink/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mr-karan%2Fnomad-events-sink/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mr-karan","download_url":"https://codeload.github.com/mr-karan/nomad-events-sink/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248859657,"owners_count":21173337,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["nomad"],"created_at":"2024-10-15T09:45:46.285Z","updated_at":"2025-04-14T09:51:41.876Z","avatar_url":"https://github.com/mr-karan.png","language":"Go","funding_links":[],"categories":["Utilities","Infrastructure setup"],"sub_categories":["Monitoring and Logging"],"readme":"\u003ca href=\"https://zerodha.tech\"\u003e\u003cimg src=\"https://zerodha.tech/static/images/github-badge.svg\" align=\"right\" /\u003e\u003c/a\u003e\n\n# Nomad Events Sink\n\n![](docs/logo.png)\n\nNomad Events Sink is an events collection agent which uses [Nomad Events SDK](https://www.nomadproject.io/api-docs/events) to fetch events.\n\nEvents can help debug the cluster state and alert operators about new deployments, failing allocations, node updates etc. Nomad emits these events in a channel and provides an SDK to access these events.\n\nThe missing piece was a tool to ingest these events and analyze/visualize them later. This is where `nomad-events-sink` agent comes into picture and helps to ingest these events in multiple configurable **Sinks**.\n\nThe tool is designed to be generic in order to handle many use-cases of processing events. Multiple [Sink Providers](./internal/sinks/provider/provider.go) can be created and events will be handled by them. Common usecases included storing events for long term use (Vector-\u003eLoki), alerting on cluster state changes (Slack/Rocketchat webhooks) etc.\n\n## How does it work?\n\n- A subscription for each topic is created and new events are watched.\n- Whenever a new event comes, it gets added to a `Sink` channel.\n- Multiple background workers (Goroutines) are listening to this channel, processing events in FIFO.\n- These workers batch events and push to upstream providers (for eg HTTP/S3 etc).\n- A background worker commits the event index state to a file on disk. This is done so that the program is able to pickup events from where it left before shutting down.\n\n### Batching Events\n\nThis program uses 2 batch strategies in order to avoid an inefficient process of making a request for every incoming event:\n\n- `idle_timeout`: If a batch is in memory for more than `idle_timeout` duration, it is flushed to providers.\n- `events_count`: If a batch has more events than `events_counts`, it is flushed to providers.\n\n## Deploy\n\nYou can choose one of the various deployment options:\n\n### Nomad\n\nTo deploy on a Nomad cluster, you can refer to [this jobspec](./deploy/job.nomad).\n\n### Binary\n\nGrab the latest release from [Releases](https://github.com/mr-karan/nomad-events-sink/releases).\n\nTo run:\n\n```\n$ ./nomad-events-sink.bin --config config.toml\n```\n\n### Docker\n\n```\ndocker pull ghcr.io/mr-karan/nomad-events-sink:latest\n```\n\n### Quick Start\n\nRefer to [this example](./examples/README.md) for quickly getting started with ingesting Deployment Events to Loki using Vector.\n\n![](./docs/loki.png)\n\n## Configuration\n\nRefer to `config.sample.toml` for a list of configurable values.\n\n### Environment Variables\n\nAll config variables can also be populated as env vairables by prefixing `NOMAD_EVENTS_SINK_` and replacing `.` with `__`.\n\nFor eg: `app.data_dir` becomes `NOMAD_EVENTS_SINK_app__data_dir`.\n\nNomad API client reads the following environment variables:\n\n- `NOMAD_TOKEN`\n- `NOMAD_ADDR`\n- `NOMAD_REGION`\n- `NOMAD_NAMESPACE`\n- `NOMAD_HTTP_AUTH`\n- `NOMAD_CACERT`\n- `NOMAD_CAPATH`\n- `NOMAD_CLIENT_CERT`\n- `NOMAD_CLIENT_KEY`\n- `NOMAD_TLS_SERVER_NAME`\n- `NOMAD_SKIP_VERIFY`\n\nYou can read about them in detail [here](https://www.nomadproject.io/docs/runtime/environment).\n\n## Sink Providers\n\n- Currently only **HTTP** Provider is implemented. There's a full [working example](./examples/README.md) of ingesting events to the HTTP server provided by Vector and dumping them to Loki.\n\n## Contribution\n\nPlease feel free to open a new issue for bugs, new features (like Sink Providers), feedback etc.\n\n## LICENSE\n\n[LICENSE](./LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmr-karan%2Fnomad-events-sink","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmr-karan%2Fnomad-events-sink","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmr-karan%2Fnomad-events-sink/lists"}