https://github.com/mr-karan/nomad-vector-logger
A daemon which continuously watches jobs running in a Nomad cluster and templates out a Vector configuration file which can be used to collect application logs enriched with Nomad metadata.
https://github.com/mr-karan/nomad-vector-logger
nomad vector
Last synced: 5 months ago
JSON representation
A daemon which continuously watches jobs running in a Nomad cluster and templates out a Vector configuration file which can be used to collect application logs enriched with Nomad metadata.
- Host: GitHub
- URL: https://github.com/mr-karan/nomad-vector-logger
- Owner: mr-karan
- License: mit
- Created: 2022-06-17T03:27:03.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2024-06-05T18:06:24.000Z (over 1 year ago)
- Last Synced: 2024-10-16T09:45:52.330Z (12 months ago)
- Topics: nomad, vector
- Language: Go
- Homepage: https://mrkaran.dev/posts/nomad-logging/
- Size: 71.3 KB
- Stars: 53
- Watchers: 2
- Forks: 15
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-nomad - mr-karan/nomad-vector-logger - Nomad-vector-logger is a daemon that monitors Nomad cluster jobs and generates Vector configuration files to collect logs enriched with Nomad metadata for improved centralized logging and observability. (Infrastructure setup / Monitoring and Logging)
README
# nomad-vector-logger
A daemon which continuously watches for deployments in a [Nomad](https://www.nomadproject.io/) cluster and generates a [Vector](https://vector.dev/) configuration file, which can be used to collect logs enriched with Nomad **metadata**.
Each log event is annotated with the following metadata:
- Namespace of the application
- Node where the deployment is running
- Job name
- Group name
- Task name
- Allocation ID
## Why
### Problem
Currently, Nomad stores all application logs inside `$NOMAD_DATA_DIR/$NOMAD_ALLOC_DIR/logs/` directory. The limitation is that these logs don't have any information about the task/job/allocation etc. Suppose there are multiple deployments on the same host. In that case, no central log collecting agent can distinguish and process these logs uniquely.
For the `docker` driver, this is a non-issue since logging of tasks with the docker driver is configured with [`logging`](https://www.nomadproject.io/docs/drivers/docker#config-1) stanza.
Users running deployments with `raw_exec` and `exec` as the task driver will find that no such configuration exists as mentioned in this [GitHub Issue](https://github.com/hashicorp/nomad/issues/10219).
### Solution
- `nomad-vector-logger` is a daemon that runs in the background, periodically polling for `Allocations` on the node.
- It then generates a `vector` configuration to collect logs from the allocation's log directory. It enriches the log event with relevant metadata.
- `vector` is started with a [`--watch-config`](https://vector.dev/docs/administration/management/#reloading) flag, which automatically live-reloads `vector` whenever config changes. A config change can happen whenever an allocation is _created/stopped/restarted_.
You can see a sample [config file](./sample/nomad.toml) that is generated by this daemon. This config file can be used in addition to other `vector` config files to provide the config for the rest of the pipeline (additional transformations, sinks etc.).
#### Before
Logs without any metdata on `/opt/nomad/data/alloc/$ALLOC_ID/alloc/logs`:
```
==> proxy.stdout.0 <==
192.168.29.76 - - [03/Sep/2022:17:30:36 +0000] "GET / HTTP/1.1" 200 27 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:103.0) Gecko/20100101 Firefox/103.0" "-"
```
#### After
This is an example JSON log collected from `nginx` task running with `raw_exec` task driver on Nomad, collected using `vector`:
```json
{
"file": "/opt/nomad/data/alloc/64a2f9fd-e003-0bb3-b5cd-838125283a06/alloc/logs/proxy.stdout.0",
"host": "pop-os",
"message": "192.168.29.76 - - [03/Sep/2022:17:30:36 +0000] \"GET / HTTP/1.1\" 200 27 \"-\" \"Mozilla/5.0 (X11; Linux x86_64; rv:103.0) Gecko/20100101 Firefox/103.0\" \"-\"",
"nomad": {
"alloc_id": "64a2f9fd-e003-0bb3-b5cd-838125283a06",
"group_name": "nginx",
"job_name": "nginx",
"namespace": "default",
"node_name": "pop-os",
"task_name": "proxy"
},
"source_type": "file",
"timestamp": "2022-09-03T17:30:42.569487273Z"
}
```
## Dev Setup
```
make dev
```
You can refer to a local dev suite which runs this program in a Nomad cluster. The [jobspec](./dev/deployment.nomad) can also be used as a reference for production deployment.
## Deployment Notes
- This program is meant to be run inside a Nomad cluster and should have proper ACL to fetch `Allocation:*` events. You can use this ACL policy to generate a token:
```hcl
namespace "*" {
policy = "read"
}
node {
policy = "read"
}
agent {
policy = "read"
}
```
- It's preferable to run it as a `system` job. Each program allocation will be responsible for configuring `vector` to collect logs from that particular log directory on the host.
You can choose one of the various deployment options:
### Binary
Grab the latest release from [Releases](https://github.com/mr-karan/nomad-vector-logger/releases).
To run:
```
$ ./nomad-vector-logger.bin --config config.toml
```
### Nomad
View a sample deployment file at [dev/deployment.nomad](./dev/deployment.nomad).
### Docker
Docker images are available on [GitHub](https://github.com/mr-karan/nomad-vector-logger/pkgs/container/nomad-vector-logger).
## Configuration
Refer to [config.sample.toml](./config.sample.toml) for a list of configurable values.
### Environment Variables
All config variables can also be populated as env variables by prefixing `NOMAD_VECTOR_LOGGER_` and replacing `.` with `__`.
For eg: `app.data_dir` becomes `NOMAD_VECTOR_LOGGER_app__data_dir`.
## Contribution
Please feel free to open a new issue for bugs, feedback etc.
## LICENSE
[LICENSE](./LICENSE)