{"id":15697998,"url":"https://github.com/dacbd/vercel-log-drain","last_synced_at":"2025-05-08T21:16:27.916Z","repository":{"id":247336633,"uuid":"825514569","full_name":"dacbd/vercel-log-drain","owner":"dacbd","description":"A simple log-drain you can deploy to export log messages from Vercel to AWS Cloudwatch.","archived":false,"fork":false,"pushed_at":"2025-04-08T02:08:48.000Z","size":146,"stargazers_count":12,"open_issues_count":4,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-08T21:16:22.341Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dacbd.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-07-08T01:36:19.000Z","updated_at":"2025-05-01T21:28:57.000Z","dependencies_parsed_at":"2024-08-28T02:29:59.300Z","dependency_job_id":"99da9e96-cb7a-413b-abcd-4b167e802b43","html_url":"https://github.com/dacbd/vercel-log-drain","commit_stats":null,"previous_names":["dacbd/vercel-log-drain"],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dacbd%2Fvercel-log-drain","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dacbd%2Fvercel-log-drain/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dacbd%2Fvercel-log-drain/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dacbd%2Fvercel-log-drain/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dacbd","download_url":"https://codeload.github.com/dacbd/vercel-log-drain/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253149620,"owners_count":21861740,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-03T19:22:31.235Z","updated_at":"2025-05-08T21:16:27.887Z","avatar_url":"https://github.com/dacbd.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# vercel-log-drain\n\nA simple log-drain you can deploy to export log messages from Vercel to one or more sources!\n\n## Drivers\n\n### AWS CloudWatch\n\n\u003e *Available with the `cloudwatch` [feature](#cargo-features) (enabled by default).*\n\nTo use the CloudWatch driver, you'll need to either:\n\n- add an environment variable for `VERCEL_LOG_DRAIN_CLOUDWATCH_ENABLED=true`\n- add the `--cloudwatch-enabled` cli flag\n\nThe log drain will create new log groups and log streams if they are not present.\nLog groups follow this scheme: `/vercel/{project_name}/{vercel_source}`, `project_name` is self-explaining, and `vercel_source` is one of the following `build`, `edge`, `external`, `lambda`, and `static`\nThe log stream is the vercel deployment ID.\n\n#### Permissions\n\nAWS permissions used:\n\n```\nlogs:DescribeLogGroups\nlogs:DescribeLogStreams\nlogs:CreateLogGroup\nlogs:CreateLogStream\nlogs:PutLogEvents\nlogs:PutRetentionPolicy\n```\n\n#### Terraform examples\n\nBelow are [`aws_iam_role`][], [`aws_iam_role_policy`][] and [`aws_iam_policy_document`][] definitions which grant a minimal set of permissions required to push logs to CloudWatch:\n\n[`aws_iam_role`]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role\n[`aws_iam_role_policy`]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy\n[`aws_iam_policy_document`]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document\n\n\n\u003cdetails\u003e\u003csummary\u003eSimple 🏠 \u003c/summary\u003e\n\n```hcl\nresource \"aws_iam_role\" \"vercel_log_drain\" {\n  name               = \"vercel-log-drain\"\n  description        = \"Role to be used by the vercel log drain deployment\"\n  assume_role_policy = data.aws_iam_policy_document.vercel_log_drain_assume.json\n}\ndata \"aws_iam_policy_document\" \"vercel_log_drain_assume\" {\n    # depends on how you intend to deploy/run the service\n}\nresource \"aws_iam_role_policy\" \"vercel_log_drain_policy\" {\n  name   = \"vercel-log-drain-policy\"\n  role   = aws_iam_role.vercel_log_drain.id\n  policy = data.aws_iam_policy_document.vercel_log_drain_permissions.json\n}\ndata \"aws_iam_policy_document\" \"vercel_log_drain_permissions\" {\n  statement {\n    actions = [\n      \"logs:DescribeLogGroups\",\n      \"logs:DescribeLogGroups\",\n      \"logs:DescribeLogStreams\",\n      \"logs:CreateLogGroup\",\n      \"logs:CreateLogStream\",\n      \"logs:PutLogEvents\",\n      \"logs:PutRetentionPolicy\",\n    ]\n    resources = [\n      \"*\"\n    ]\n  }\n}\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\u003csummary\u003eAdvanced 🏘️ \u003c/summary\u003e\n\n```hcl\ndata \"aws_caller_identity\" \"current\" {}\nvariable \"aws_region\" {\n  type        = string\n  description = \"AWS region to run in\"\n}\nresource \"aws_iam_role\" \"vercel_log_drain\" {\n  name               = \"vercel-log-drain\"\n  description        = \"Role to be used by the vercel log drain deployment\"\n  assume_role_policy = data.aws_iam_policy_document.vercel_log_drain_assume.json\n}\ndata \"aws_iam_policy_document\" \"vercel_log_drain_assume\" {\n    # An sts:AssumeRole policy for the service. This varies depending on how\n    # you intend to deploy/run the service.\n}\nresource \"aws_iam_role_policy\" \"vercel_log_drain_policy\" {\n  name   = \"vercel-log-drain-policy\"\n  role   = aws_iam_role.vercel_log_drain.id\n  policy = data.aws_iam_policy_document.vercel_log_drain_permissions.json\n}\ndata \"aws_iam_policy_document\" \"vercel_log_drain_permissions\" {\n  statement {\n    actions = [\n      \"logs:CreateLogGroup\",\n      \"logs:PutRetentionPolicy\",\n    ]\n    resources = [\n      provider::aws::arn_build(\n        \"aws\",\n        \"logs\",\n        var.aws_region,\n        data.aws_caller_identity.current.account_id,\n        \"log-group:/vercel/*\"\n      )\n    ]\n  }\n\n  statement {\n    actions = [\n      \"logs:CreateLogStream\",\n      \"logs:DescribeLogStreams\",\n      \"logs:PutLogEvents\",\n    ]\n    resources = [\n      provider::aws::arn_build(\n        \"aws\",\n        \"logs\",\n        var.aws_region,\n        data.aws_caller_identity.current.account_id,\n        \"log-group:/vercel/*:*\"\n      )\n    ]\n  }\n\n  statement {\n    actions = [\n      \"logs:DescribeLogGroups\",\n    ]\n    resources = [\"*\"]\n  }\n}\n```\n\n\u003c/details\u003e\n\n\n### [Grafana Loki](https://grafana.com/docs/loki/latest/)\n\n\u003e *Available with the `loki` [feature](#cargo-features) (enabled by default).*\n\nTo use the loki driver, you'll need to set up:\n\n- `--loki-enabled` (or the env var `VERCEL_LOG_DRAIN_LOKI_ENABLED=true`)\n- `--loki-url` (or the env var `VERCEL_LOG_DRAIN_LOKI_URL`)\n- (optional, if you have basic auth) `--loki-basic-auth-user` and `--loki-basic-auth-pass` (or the corresponding env vars `VERCEL_LOG_DRAIN_LOKI_USER` and `VERCEL_LOG_DRAIN_LOKI_PASS`)\n\n## Configuration\n\n| CLI Flag                 | Environment Variable                 | Default Value | Description                              |\n| ------------------------ | ------------------------------------ | ------------- | ---------------------------------------- |\n| `-l, --log`              | `VERCEL_LOG_DRAIN_LOG_LEVEL`         | `INFO`        | Log level                                |\n| `-i, --ip`               | `VERCEL_LOG_DRAIN_IP`                | `\"0.0.0.0\"`   | IP address to bind to                    |\n| `-p, --port`             | `VERCEL_LOG_DRAIN_PORT`              | `8000`        | Port number                              |\n| `--vercel-verify`        | `VERCEL_VERIFY`                      | -             | Vercel verification token                |\n| `--vercel-secret`        | `VERCEL_SECRET`                      | -             | Vercel secret                            |\n| `--enable-metrics`       | `VERCEL_LOG_DRAIN_ENABLE_METRICS`    | -             | Enable prometheus metrics endpoint       |\n| `--metrics-prefix`       | `VERCEL_LOG_DRAIN_METRICS_PREFIX`    | \"drain\"       | the shared prefix to use for all metrics |\n| `--enable-cloudwatch`    | `VERCEL_LOG_DRAIN_ENABLE_CLOUDWATCH` | -             | Enable CloudWatch integration            |\n| `--enable-loki`          | `VERCEL_LOG_DRAIN_ENABLE_LOKI`       | -             | Enable Loki integration                  |\n| `--loki-url`             | `VERCEL_LOG_DRAIN_LOKI_URL`          | `\"\"`          | Loki URL                                 |\n| `--loki-basic-auth-user` | `VERCEL_LOG_DRAIN_LOKI_USER`         | `\"\"`          | Loki basic auth username                 |\n| `--loki-basic-auth-pass` | `VERCEL_LOG_DRAIN_LOKI_PASS`         | `\"\"`          | Loki basic auth password                 |\n\n## Setting up (in Vercel)\n\nVercel requires that you host the application over HTTP or HTTPS, and have it be accessible from the public internet.\n\n`vercel-log-drain` itself only supports HTTP – so you should put an HTTPS load balancer in front of it.\n\nTo add new log drains in Vercel:\n\n1. Go to the [Vercel account dashboard](https://vercel.com/account).\n2. Find the team to configure, and click `...` → Manage\n3. Select Log Drains\n\nThe parameters you'll need for `vercel-log-drain` are:\n\n* Delivery format: JSON\n* Custom secret: the Vercel secret you set with `--vercel-secret` or `VERCEL_SECRET`\n* Endpoint: `https://${your_hostname}/vercel`\n* (optional) Custom headers: add a random secret header value, and configure your load balancer to require that header (to help filter out bot noise)\n\nPass the value of the `x-vercel-verify` header (provided by Vercel) to `vercel-log-drain` with the `--vercel-verify` argument or `VERCEL_VERIFY` environment variable.\n\n\u003e [!NOTE]\n\u003e Vercel *does not* sign the initial verification request, and expects the endpoint to return HTTP 200 OK and the `x-vercel-verify` to that request.\n\u003e\n\u003e Configuring Vercel to send an extra custom header *and* requiring it in your HTTPS load balancer should allow it to drop the majority of bot traffic *without* it touching `vercel-log-drain` or exposing your account's `x-vercel-verify` token.\n\n## Operation\n\nAs written in my deployment this handled about `~8M` requests per month, with an avg response time (LB -\u003e target) of `1-1.5ms` with an avg memory usage of `~5MB`.\nA 3 node deployment (for redundency) with `100m` CPU and `128MB` memory reservations should be able to go quite far.\nThe response times above are a bit unfair because the system is designed to always responsed to vercel as fast as possible, adding the messages to an internal queue which processes the messages async from the actual POST request which is was receieved from.\n\nIf you click Vercel's test log drain button when you are setting up your deployment you may see some messages fail to parse this is because a few of the test messages dont fully follow their documented structure (some fields are missing)\n\nNo effort has really been made yet to optimize the code, still it is performant enough to handle anything, but feel free to contribute optimizations or idiomatic code corrections, I wrote this in a vacuum.\n\n### JSON logging in vercel\n\nIf you have structured JSON logging ie the contents of `messaage` is a json string, the service attempts to parse it as json so a fully JSON message can be pass downstream, vs a string containing json.\n\nExample: `{ \"message\": { \"method\": \"GET\" } }` vs `{ \"message\": \"{ \\\"method\\\": \\\"GET\\\" }\" }`\n\nThis helps with log queries in cloudwatch or if modified your downsteam system to search or filter on data not just provided by vercel but also your own JSON logging in the deployed application.\n\n## Cargo features\n\n`cargo` will build `vercel-log-drain` with **all**\n[features](https://doc.rust-lang.org/cargo/reference/features.html) by default:\n\nFeature      | Description\n------------ | --------\n`cloudwatch` | [AWS CloudWatch](#aws-cloudwatch) driver\n`loki`       | [Grafana Loki](#grafana-loki) driver\n\nIf you want a smaller binary, you could disable all of them with\n`--no-default-features`, and then only re-enable the features you use.\n\nFor example, to build `vercel-log-drain` with only AWS CloudWatch support:\n\n```sh\ncargo build --release --no-default-features --features cloudwatch\n```\n\nThis can also be used when building the Docker image:\n\n```sh\ndocker build -t vercel-log-drain --build-arg 'BUILD_ARGS=--no-default-features --features cloudwatch' .\n```\n\n## Testing\n\n```bash\ncargo build\n\n# run the server\n./target/debug/vercel-log-drain --enable-metrics --vercel-secret \"deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef\" --vercel-verify verify --log DEBUG\n```\n\nIf you're using the nix environment, there are some helpful scripts for running and sending test payloads to the server!\n\n```bash\n# build\ncargo build\n\n# run the server\nrun  # you'll need to add env vars or your options here! example (i have an http sink server running on :8080 that is logging all requests incoming):\nrun --enable-loki --loki-url http://localhost:8080/ingest\n\n# send test payloads\ntest_drain ./src/fixtures/sample_1.json\n```\n\n## Related vercel documentation\n\n- [Vercel JSON Log Drains](https://vercel.com/docs/observability/log-drains-overview/log-drains-reference#json-log-drains)\n- [Vercel Secure Log Drains](https://vercel.com/docs/observability/log-drains-overview/log-drains-reference#secure-log-drains)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdacbd%2Fvercel-log-drain","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdacbd%2Fvercel-log-drain","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdacbd%2Fvercel-log-drain/lists"}