{"id":16053165,"url":"https://github.com/denschub/nginx-syslog-postgres-bridge","last_synced_at":"2025-06-25T02:34:36.478Z","repository":{"id":244331961,"uuid":"814910105","full_name":"denschub/nginx-syslog-postgres-bridge","owner":"denschub","description":"Receives syslog messages from nginx and stores them in a PostgreSQL database.","archived":false,"fork":false,"pushed_at":"2025-05-12T21:48:28.000Z","size":263,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-12T22:36:07.001Z","etag":null,"topics":["analytics","nginx","observability","ops","syslog"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/denschub.png","metadata":{"files":{"readme":"README.md","changelog":"Changelog.md","contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":".github/SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-06-14T01:09:13.000Z","updated_at":"2025-05-12T21:48:33.000Z","dependencies_parsed_at":"2024-12-10T21:01:07.512Z","dependency_job_id":"daec197a-53d5-44f0-95fb-772ad5d5198c","html_url":"https://github.com/denschub/nginx-syslog-postgres-bridge","commit_stats":{"total_commits":36,"total_committers":1,"mean_commits":36.0,"dds":0.0,"last_synced_commit":"1a059c1100d838c85882693eea9d561118190a84"},"previous_names":["denschub/nginx-syslog-postgres-bridge"],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/denschub/nginx-syslog-postgres-bridge","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denschub%2Fnginx-syslog-postgres-bridge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denschub%2Fnginx-syslog-postgres-bridge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denschub%2Fnginx-syslog-postgres-bridge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denschub%2Fnginx-syslog-postgres-bridge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/denschub","download_url":"https://codeload.github.com/denschub/nginx-syslog-postgres-bridge/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denschub%2Fnginx-syslog-postgres-bridge/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261791515,"owners_count":23210134,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","nginx","observability","ops","syslog"],"created_at":"2024-10-09T01:40:52.641Z","updated_at":"2025-06-25T02:34:36.466Z","avatar_url":"https://github.com/denschub.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# nginx-syslog-postgres-bridge\n\nA bridge to connect nginx' syslog output for `access_log` to a PostgreSQL database. Transferring `error_log` is not supported.\n\nIt is highly recommended that the PostgreSQL used for this tool supports the TimescaleDB extension. It works fine with either the cloud-hosted option, or a [self-hosted TimescaleDB][selfhosted-timescale]. If TimescaleDB support is detected, the database migrations automatically set up the `access_log` table as a Hypertable, with partitioning on the `event_ts`, and a 365 day retention policy.\n\nRunning this on a plain PostgreSQL works - but performance will take a hit for larger datasets, especially query performance. You also have to manually delete old entries if you want to.\n\n## Data consistency and completeness\n\nnginx does not store failed deliveries. If this service is down, log lines will simply be dropped by nginx. Invalid datagrams will be dropped. Log lines that do not fit within a single UDP datagram (~65KiB) will, [as spec'ed][rfc5426], result in an incomplete JSON document and thus be dropped as well.\n\nThe data resulting from this tool should be considered good enough for simple statistical analysis and occasional tracing. It does not replace a full end-to-end tracing setup with a coverage guarantee.\n\n## Security considerations\n\nThis bridge does not run any authentication, authorization, or validation. Any valid JSON datagram received will be stored in the database. While it would be possible to allow-list certain source IPs in this application, doing that in a firewall probably makes more sense.\n\nAll data sent to this application is sent unencrypted over UDP. While there are syslog transport mechanisms via TCP and encryption, [nginx does not support those][nginx-syslog]. If logging data is sent over an untrusted network, encrypted tunneling is recommended since the log format includes PII (namely, the user's IP).\n\n## Performance considerations\n\nBecause nginx is just firing UDP datagrams towards this application with no regard for anything, this application is designed to process incoming UDP traffic as fast as possible. Each incoming UDP datagram is immediately spawned off into a different task to make room for more UDP traffic. This results in an application that can handle pretty much all traffic - but due to background processing queues and the latency of storing things in a database, memory usage can grow. The `queue-size` setting limits how many valid log entries can be stored asynchronously. The default value of 10k entries guarantees that even in the worst case of 10k log entries, each using 64KiB (which, in practice, is impossible), the app is limited to ~650MiB memory usage.\n\nIn a local benchmark, with nginx, PostgreSQL, and this application sharing a Docker environment with 5 CPU cores of an Apple M1 Max, I was able to achieve a peak traffic of ~22k req/s. This bridge was able to handle all the incoming log entries without issues. However, during the 30 seconds of the burn-in test, a total of ~660k requests have been responded to. Most of them did not immediately end up in the database, as PostgreSQL inserts in their current form are rather slow, so the memory usage peaked at ~480 MiB. Backfilling those log entries into the database took roughly 8 minutes. It's therefore not a good idea to expose this application to a constant load of more than 1k req/s.\n\nShort bursts of traffic with longer pauses in between to clear the backlog are fine. Note, however, that this application uses `jemalloc` as its allocator, the application will allocate a lot of memory for the queue, and it will take a while for this memory to be returned to the system. If handling large spikes of traffic is a concern, check [`jemalloc`s tuning documentation][jemalloc-tuning] for information on how to free memory faster.\n\nFor constantly high loads, this application can be optimized by a) batching `INSERT` queries in transactions and b) running transactions in parallel. However, as the currently possible load exceeds any load realistically expected in its environment, these optimizations are ignored for now.\n\n## Required nginx configuration\n\nnginx needs to be configured with a special log format. [Check the dedicated documentation page for details](./docs/nginx_config.md).\n\n## Deployment and configuration\n\nA container image is pushed to [the GitHub Container registry at `ghcr.io/denschub/nginx-syslog-postgres-bridge:latest`][ghcr], and [to Docker Hub as `denschub/nginx-syslog-postgres-bridge:latest`][dockerhub]. The container exposes port 8514.\n\nConfiguration of the server is done with either environment variables or via CLI arguments. Make sure to set `DATABASE_URL`/`--database-url` to a valid PostgreSQL connection URL like `postgres://postgres@127.0.0.1/nginx_logs`. The database needs to exist before starting the server, but the server startup procedure will take care of all database migrations.\n\nReleased binaries are available for all stable releases. Check the [Releases section on GitHub][github-releases] for the latest release, and you'll find a `.zip` with a pre-built binary.\n\nAdditional settings are available, for example a custom limit for the maximum queue length. Run with `--help` to see all details.\n\n## License\n\n[MIT](/LICENSE).\n\n[dockerhub]: https://hub.docker.com/repository/docker/denschub/nginx-syslog-postgres-bridge/general\n[ghcr]: https://github.com/denschub/nginx-syslog-postgres-bridge/pkgs/container/nginx-syslog-postgres-bridge\n[github-releases]: https://github.com/denschub/nginx-syslog-postgres-bridge/releases\n[jemalloc-tuning]: https://github.com/jemalloc/jemalloc/blob/dev/TUNING.md\n[nginx-syslog]: https://nginx.org/en/docs/syslog.html\n[rfc5426]: https://www.rfc-editor.org/rfc/rfc5426\n[selfhosted-timescale]: https://docs.timescale.com/self-hosted/latest\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdenschub%2Fnginx-syslog-postgres-bridge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdenschub%2Fnginx-syslog-postgres-bridge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdenschub%2Fnginx-syslog-postgres-bridge/lists"}