{"id":13785570,"url":"https://github.com/seatgeek/nomad-firehose","last_synced_at":"2025-10-26T14:30:42.807Z","repository":{"id":22671605,"uuid":"96985847","full_name":"seatgeek/nomad-firehose","owner":"seatgeek","description":"Firehose all nomad job, allocation, nodes and evaluations changes to rabbitmq, kinesis or stdout","archived":false,"fork":false,"pushed_at":"2020-09-03T16:44:35.000Z","size":216,"stargazers_count":115,"open_issues_count":20,"forks_count":33,"subscribers_count":13,"default_branch":"master","last_synced_at":"2024-11-17T22:34:44.522Z","etag":null,"topics":["consul","firehose","hashicorp","hashicorp-nomad","nomad","rabbitmq"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/seatgeek.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-07-12T08:48:16.000Z","updated_at":"2023-12-11T09:17:34.000Z","dependencies_parsed_at":"2022-08-07T10:16:03.192Z","dependency_job_id":null,"html_url":"https://github.com/seatgeek/nomad-firehose","commit_stats":null,"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seatgeek%2Fnomad-firehose","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seatgeek%2Fnomad-firehose/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seatgeek%2Fnomad-firehose/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seatgeek%2Fnomad-firehose/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/seatgeek","download_url":"https://codeload.github.com/seatgeek/nomad-firehose/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238343232,"owners_count":19456203,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["consul","firehose","hashicorp","hashicorp-nomad","nomad","rabbitmq"],"created_at":"2024-08-03T19:01:01.926Z","updated_at":"2025-10-26T14:30:42.480Z","avatar_url":"https://github.com/seatgeek.png","language":"Go","funding_links":[],"categories":["Infrastructure setup"],"sub_categories":["Monitoring and Logging"],"readme":"# nomad-firehose\n\n`nomad-firehose` is a tool meant to enable teams to quickly build logic around nomad task events without hooking into Nomad API.\n\n## Running\n\nThe project has build artifacts for Linux, Darwin and Windows in the [GitHub releases tab](https://github.com/seatgeek/nomad-firehose/releases).\n\nA Docker container is also provided at [seatgeek/nomad-firehose](https://hub.docker.com/r/seatgeek/nomad-firehose/tags/)\n\n## Requirements\n\n- Go 1.11\n\n## Building\n\nTo build a binary, run the following\n\n```shell\n# get this repo\ngo get github.com/seatgeek/nomad-firehose\n\n# go to the repo directory\ncd $GOPATH/src/github.com/seatgeek/nomad-firehose\n\n# build the `nomad-firehose` binary\nmake build\n```\n\nThis will create a `nomad-firehose` binary in your `$GOPATH/bin` directory.\n\n## Configuration\n\nAny `NOMAD_*` env that the native `nomad` CLI tool supports are supported by this tool.\n\nAny `CONSUL_*` env that the native `consul` CLI tool supports are supported by this tool.\n\nThe most basic requirement is `export NOMAD_ADDR=http://\u003cip\u003e:4646` and `export CONSUL_HTTP_ADDR=\u003cip\u003e:8500`.\n\n### Consul\n\n`nomad-firehose` will use Consul to maintain leader-ship and store last event time processed (saved on quit or every 10s).\n\nThis mean you can run more than 1 process of each firehose, and only one will actually do any work.\n\nSaving the last event time mean that restarting the process won't firehose all old changes to your sink, reducing duplicated events.\n\nBy default, the Consul lock is maintained in KV at `nomad-firehose/${type}.lock` and the last event time is stored in KV at `nomad-firehose/${type}.value`. You can change the prefix from `nomad-firehose` by setting `NOMAD_FIREHOSE_CONSUL_PREFIX` to your desired prefix.\n\n#### Consul ACL Token Permissions\n\nIf the Consul cluster being used is running ACLs, the following ACL policy will allow the required access:\n\n```hcl\nkey \"nomad-firehose\" {\n  policy = \"write\"\n}\nsession \"\" {\n  policy = \"write\"\n}\n```\n\nIf you've set a custom prefix, specify that in the `key` ACL entry instead.\n\n### Kafka\n\nTo connect to Kafka with TLS, set the SINK_KAFKA_CA_CERT_PATH to the path to your CA cert file.\nTo use SASL/PLAIN authentication, set `$SINK_KAFKA_USER` and `$SINK_KAFKA_PASSWORD` environment variables.\n\n\n## Usage\n\nThe `nomad-firehose` binary has several helper subcommands.\n\nThe sink type is configured using `$SINK_TYPE` environment variable. Valid values are:\n- `amqp`\n- `kinesis`\n- `nsq`\n- `redis`\n- `kafka`\n- `mongo`\n- `sqs`\n- `eventbridge`\n- `stdout`\n- `syslog`\n\nThe `amqp` and `rabbitmq` sinks are configured using `$SINK_AMQP_CONNECTION` (`amqp://guest:guest@127.0.0.1:5672/`), `$SINK_AMQP_EXCHANGE`, `$SINK_AMQP_ROUTING_KEY`, and `$SINK_AMQP_WORKERS` (default: `1`) environment variables.\n\nThe `http` sink is configured using `$SINK_HTTP_ADDRESS` (`localhost:8080/allocations`)` environment variable.\n\nThe `kafka` sink is configured using `$SINK_KAFKA_BROKERS` (`kafka1:9092,kafka2:9092,kafka3:9092`), and `$SINK_KAFKA_TOPIC` environment variables.\n\nThe `kinesis` sink is configured using `$SINK_KINESIS_STREAM_NAME` and `$SINK_KINESIS_PARTITION_KEY` environment variables.\n\nThe `mongo` sink is configured using `$SINK_MONGODB_CONNECTION` (`mongodb://localhost:27017/`), `$SINK_MONGODB_DATABASE` and `$SINK_MONGODB_COLLECTION` environment variables.\n\nThe `nsq` sink is configured using `$SINK_NSQ_ADDR` and `$SINK_NSQ_TOPIC_NAME` environment variables.\n\nThe `redis` sink is configured using `$SINK_REDIS_URL` (`redis://[user]:[password]@127.0.0.1[:5672]/0`) and `$SINK_REDIS_KEY` environment variables.\n\nThe `stdout` sink does not have any configuration, it will simply output the JSON to stdout for debugging.\n\nThe `syslog` sink is configured using `$SINK_SYSLOG_PROTO` (e.g. `tcp`, `udp` - leave empty if logging to a local syslog socket), `$SINK_SYSLOG_ADDR` (e.g. `127.0.0.1:514` - leave empty if logging to a local syslog socket), and `$SINK_SYSLOG_TAG` (default: `nomad-firehose`).\n\nThe `sqs` sink is configured using `$SINK_SQS_QUEUE_NAME` which is the name of the queue in SQS. This queue is expected to be a FIFO queue.\nThe URL of the queue is inferred by the presence of the `AWS_REGION` and `AWS_ACCOUNT_ID` env variables.\n\nThe `eventbridge` sink is configured using `$SINK_EVENT_BUS_NAME` which is the name of the bus in Event Bridge. The environment variables `SINK_EVENT_BUS_DETAIL_TYPE` and `SINK_EVENT_BUS_SOURCE` are used to configure the schema when creating Event Bus rules.\n\n### `allocations`\n\n`nomad-firehose allocations` will monitor all allocation changes in the Nomad cluster and emit each task state as a new firehose event to the configured sink.\n\nThe allocation output is different from the [default API response](https://www.nomadproject.io/api/allocations.html), as the tool will emit an event per new [TaskStates](https://www.nomadproject.io/docs/http/allocs.html), rather than all the previous events.\n\n```json\n{\n    \"Name\": \"job.task[0]\",\n    \"AllocationID\": \"1ef2eba2-00e4-3828-96d4-8e58b1447aaf\",\n    \"DesiredStatus\": \"run\",\n    \"DesiredDescription\": \"\",\n    \"ClientStatus\": \"running\",\n    \"ClientDescription\": \"\",\n    \"JobID\": \"logrotate\",\n    \"GroupName\": \"cron\",\n    \"TaskName\": \"logrotate\",\n    \"EvalID\": \"bf926150-ed30-6c13-c597-34d7a3165fdc\",\n    \"TaskState\": \"running\",\n    \"TaskFailed\": false,\n    \"TaskStartedAt\": \"2017-06-30T19:58:28.325895579Z\",\n    \"TaskFinishedAt\": \"0001-01-01T00:00:00Z\",\n    \"TaskEvent\": {\n        \"Type\": \"Task Setup\",\n        \"Time\": 1498852707712617200,\n        \"FailsTask\": false,\n        \"RestartReason\": \"\",\n        \"SetupError\": \"\",\n        \"DriverError\": \"\",\n        \"DriverMessage\": \"\",\n        \"ExitCode\": 0,\n        \"Signal\": 0,\n        \"Message\": \"Building Task Directory\",\n        \"KillReason\": \"\",\n        \"KillTimeout\": 0,\n        \"KillError\": \"\",\n        \"StartDelay\": 0,\n        \"DownloadError\": \"\",\n        \"ValidationError\": \"\",\n        \"DiskLimit\": 0,\n        \"DiskSize\": 0,\n        \"FailedSibling\": \"\",\n        \"VaultError\": \"\",\n        \"TaskSignalReason\": \"\",\n        \"TaskSignal\": \"\"\n    }\n}\n```\n\n### `nodes`\n\n`nomad-firehose nodes` will monitor all node changes in the Nomad cluster and emit a firehose event per change to the configured sink.\n\nThe output will be equal to the [Nomad Node API structure](https://www.nomadproject.io/api/nodes.html)\n\n### `evaluations`\n\n`nomad-firehose evaluations` will monitor all evaluation changes in the Nomad cluster and emit a firehose event per change to the configured sink.\n\nThe output will be equal to the [Nomad Evaluation API structure](https://www.nomadproject.io/api/evaluations.html)\n\n### `jobs`\n\n`nomad-firehose jobs` will monitor all job changes in the Nomad cluster and emit a firehose event per change to the configured sink.\n\nThe output will be equal to the *full* [Nomad Job API structure](https://www.nomadproject.io/api/jobs.html#read-job)\n\n### `jobliststubs`\n\n`nomad-firehose jobliststubs` will monitor all job changes in the Nomad cluster and emit a firehose event per change to the configured sink.\n\nThe output will be equal to the job list [Nomad Job API structure](https://www.nomadproject.io/api/jobs.html#list-jobs)\n\n### `deployments`\n\n`nomad-firehose deployments` will monitor all deployment changes in the Nomad cluster and emit a firehose event per change to the configured sink.\n\nThe output will be equal to the *full* [Nomad Deployment API structure](https://www.nomadproject.io/api/deployments.html)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fseatgeek%2Fnomad-firehose","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fseatgeek%2Fnomad-firehose","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fseatgeek%2Fnomad-firehose/lists"}