{"id":17281385,"url":"https://github.com/mr-karan/nomad-vector-logger","last_synced_at":"2025-05-06T20:09:51.032Z","repository":{"id":47702315,"uuid":"504381071","full_name":"mr-karan/nomad-vector-logger","owner":"mr-karan","description":"A daemon which continuously watches jobs running in a Nomad cluster and templates out a Vector configuration file which can be used to collect application logs enriched with Nomad metadata.","archived":false,"fork":false,"pushed_at":"2024-06-05T18:06:24.000Z","size":73,"stargazers_count":53,"open_issues_count":4,"forks_count":15,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-10-16T09:45:52.330Z","etag":null,"topics":["nomad","vector"],"latest_commit_sha":null,"homepage":"https://mrkaran.dev/posts/nomad-logging/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mr-karan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-06-17T03:27:03.000Z","updated_at":"2024-10-13T16:08:50.000Z","dependencies_parsed_at":"2023-02-14T07:31:10.263Z","dependency_job_id":null,"html_url":"https://github.com/mr-karan/nomad-vector-logger","commit_stats":null,"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mr-karan%2Fnomad-vector-logger","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mr-karan%2Fnomad-vector-logger/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mr-karan%2Fnomad-vector-logger/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mr-karan%2Fnomad-vector-logger/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mr-karan","download_url":"https://codeload.github.com/mr-karan/nomad-vector-logger/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221583860,"owners_count":16847622,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["nomad","vector"],"created_at":"2024-10-15T09:45:44.592Z","updated_at":"2024-10-26T21:18:15.895Z","avatar_url":"https://github.com/mr-karan.png","language":"Go","funding_links":[],"categories":["Infrastructure setup"],"sub_categories":["Monitoring and Logging"],"readme":"\u003ca href=\"https://zerodha.tech\"\u003e\u003cimg src=\"https://zerodha.tech/static/images/github-badge.svg\" align=\"right\" /\u003e\u003c/a\u003e\n\n# nomad-vector-logger\n\nA daemon which continuously watches for deployments in a [Nomad](https://www.nomadproject.io/) cluster and generates a [Vector](https://vector.dev/) configuration file, which can be used to collect logs enriched with Nomad **metadata**.\n\nEach log event is annotated with the following metadata:\n\n- Namespace of the application\n- Node where the deployment is running\n- Job name\n- Group name\n- Task name\n- Allocation ID\n\n## Why\n\n### Problem\n\nCurrently, Nomad stores all application logs inside `$NOMAD_DATA_DIR/$NOMAD_ALLOC_DIR/logs/` directory. The limitation is that these logs don't have any information about the task/job/allocation etc. Suppose there are multiple deployments on the same host. In that case, no central log collecting agent can distinguish and process these logs uniquely.\n\nFor the `docker` driver, this is a non-issue since logging of tasks with the docker driver is configured with [`logging`](https://www.nomadproject.io/docs/drivers/docker#config-1) stanza.\n\nUsers running deployments with `raw_exec` and `exec` as the task driver will find that no such configuration exists as mentioned in this [GitHub Issue](https://github.com/hashicorp/nomad/issues/10219).\n\n### Solution\n\n- `nomad-vector-logger` is a daemon that runs in the background, periodically polling for `Allocations` on the node.\n- It then generates a `vector` configuration to collect logs from the allocation's log directory. It enriches the log event with relevant metadata.\n- `vector` is started with a [`--watch-config`](https://vector.dev/docs/administration/management/#reloading) flag, which automatically live-reloads `vector` whenever config changes. A config change can happen whenever an allocation is _created/stopped/restarted_.\n\nYou can see a sample [config file](./sample/nomad.toml) that is generated by this daemon. This config file can be used in addition to other `vector` config files to provide the config for the rest of the pipeline (additional transformations, sinks etc.).\n\n#### Before\n\nLogs without any metdata on `/opt/nomad/data/alloc/$ALLOC_ID/alloc/logs`:\n\n```\n==\u003e proxy.stdout.0 \u003c==\n192.168.29.76 - - [03/Sep/2022:17:30:36 +0000] \"GET / HTTP/1.1\" 200 27 \"-\" \"Mozilla/5.0 (X11; Linux x86_64; rv:103.0) Gecko/20100101 Firefox/103.0\" \"-\"\n```\n\n#### After\n\nThis is an example JSON log collected from `nginx` task running with `raw_exec` task driver on Nomad, collected using `vector`:\n\n```json\n{\n    \"file\": \"/opt/nomad/data/alloc/64a2f9fd-e003-0bb3-b5cd-838125283a06/alloc/logs/proxy.stdout.0\",\n    \"host\": \"pop-os\",\n    \"message\": \"192.168.29.76 - - [03/Sep/2022:17:30:36 +0000] \\\"GET / HTTP/1.1\\\" 200 27 \\\"-\\\" \\\"Mozilla/5.0 (X11; Linux x86_64; rv:103.0) Gecko/20100101 Firefox/103.0\\\" \\\"-\\\"\",\n    \"nomad\": {\n        \"alloc_id\": \"64a2f9fd-e003-0bb3-b5cd-838125283a06\",\n        \"group_name\": \"nginx\",\n        \"job_name\": \"nginx\",\n        \"namespace\": \"default\",\n        \"node_name\": \"pop-os\",\n        \"task_name\": \"proxy\"\n    },\n    \"source_type\": \"file\",\n    \"timestamp\": \"2022-09-03T17:30:42.569487273Z\"\n}\n```\n\n## Dev Setup\n\n```\nmake dev\n```\n\nYou can refer to a local dev suite which runs this program in a Nomad cluster. The [jobspec](./dev/deployment.nomad) can also be used as a reference for production deployment.\n\n## Deployment Notes\n\n- This program is meant to be run inside a Nomad cluster and should have proper ACL to fetch `Allocation:*` events. You can use this ACL policy to generate a token:\n\n```hcl\nnamespace \"*\" {\n  policy = \"read\"\n}\n\nnode {\n  policy = \"read\"\n}\n\nagent {\n  policy = \"read\"\n}\n```\n\n- It's preferable to run it as a `system` job. Each program allocation will be responsible for configuring `vector` to collect logs from that particular log directory on the host.\n\nYou can choose one of the various deployment options:\n\n### Binary\n\nGrab the latest release from [Releases](https://github.com/mr-karan/nomad-vector-logger/releases).\n\nTo run:\n\n```\n$ ./nomad-vector-logger.bin --config config.toml\n```\n\n### Nomad\n\nView a sample deployment file at [dev/deployment.nomad](./dev/deployment.nomad).\n\n### Docker \n\nDocker images are available on [GitHub](https://github.com/mr-karan/nomad-vector-logger/pkgs/container/nomad-vector-logger).\n\n## Configuration\n\nRefer to [config.sample.toml](./config.sample.toml) for a list of configurable values.\n\n### Environment Variables\n\nAll config variables can also be populated as env variables by prefixing `NOMAD_VECTOR_LOGGER_` and replacing `.` with `__`.\n\nFor eg: `app.data_dir` becomes `NOMAD_VECTOR_LOGGER_app__data_dir`.\n\n## Contribution\n\nPlease feel free to open a new issue for bugs, feedback etc.\n\n## LICENSE\n\n[LICENSE](./LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmr-karan%2Fnomad-vector-logger","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmr-karan%2Fnomad-vector-logger","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmr-karan%2Fnomad-vector-logger/lists"}