{"id":20333960,"url":"https://github.com/braedon/kong-log-bridge","last_synced_at":"2026-05-11T06:57:12.030Z","repository":{"id":146707711,"uuid":"281618277","full_name":"braedon/kong-log-bridge","owner":"braedon","description":"Transform Kong request logs and forward them to Elasticsearch.","archived":false,"fork":false,"pushed_at":"2023-09-25T19:04:20.000Z","size":39,"stargazers_count":2,"open_issues_count":1,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-14T16:06:26.582Z","etag":null,"topics":["elasticsearch","kong","logging","privacy","security"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/braedon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-22T08:19:49.000Z","updated_at":"2021-12-09T07:33:29.000Z","dependencies_parsed_at":"2024-11-14T20:37:04.518Z","dependency_job_id":"de94e399-865b-4e63-8332-67fbf991f347","html_url":"https://github.com/braedon/kong-log-bridge","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/braedon%2Fkong-log-bridge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/braedon%2Fkong-log-bridge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/braedon%2Fkong-log-bridge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/braedon%2Fkong-log-bridge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/braedon","download_url":"https://codeload.github.com/braedon/kong-log-bridge/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241852158,"owners_count":20030969,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","kong","logging","privacy","security"],"created_at":"2024-11-14T20:35:25.288Z","updated_at":"2026-05-11T06:57:11.993Z","avatar_url":"https://github.com/braedon.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Kong Request Log Bridge\n====\nTransform Kong request logs and forward them to Elasticsearch. Redact request logs for improved privacy and security, and index them directly into Elasticsearch, without the need for complex and heavyweight tools like Logstash.\n\n[Source Code](https://github.com/braedon/kong-log-bridge) | [Docker Image](https://hub.docker.com/r/braedon/kong-log-bridge)\n\n# Usage\nThe service is distributed as a docker image. Released versions can be found on Docker Hub (note that no `latest` version is provided):\n\n```bash\n\u003e sudo docker pull braedon/kong-log-bridge:\u003cversion\u003e\n```\n\nThe docker image exposes a REST API on port `8080`. It is configured by passing options after the image name:\n```bash\n\u003e sudo docker run --rm --name kong-log-bridge \\\n    -p \u003chost port\u003e:8080 \\\n    braedon/kong-log-bridge:\u003cversion\u003e \\\n        -e \u003celasticsearch node\u003e \\\n        --convert-ts \\\n        --convert-qs-bools \\\n        --hash-ip \\\n        --hash-auth \\\n        --hash-cookie\n```\nRun with the `-h` flag to see details on all the available options.\n\nNote that all options can be set via environment variables. The environment variable names are prefixed with `KONG_LOG_BRIDGE_OPT`, e.g. `KONG_LOG_BRIDGE_OPT_CONVERT_TS=true` is equivalent to `--convert-ts`. CLI options take precedence over environment variables.\n\n## Input\nKong JSON request logs can be `POST`ed to the `/logs` endpoint. This is designed for logs to be sent by the [Kong HTTP Log plugin](https://docs.konghq.com/hub/kong-inc/http-log/). See the Kong documentation for details on how to enable and configure the plugin.\n\nThis is currently the only supported input method, but more may be added in the future.\n\n## Transformation\nRequest logs are passed through largely unchanged by default, but you probably want to enable at least one transformation.\n\n### Timestamp Conversion `--convert-ts`\nKong request logs include a number of UNIX timestamps (some in milliseconds rather than seconds). These are not human readable, and require explicit mappings to be used in Elasticsearch. Enabling this option will convert these timestamps to [RFC3339 date-time strings](https://www.ietf.org/rfc/rfc3339.txt) for readability and automatic Elasticsearch mapping.\n\nFields converted:\n```\n - service.created_at\n - service.updated_at\n - route.created_at\n - route.updated_at\n - started_at\n - tries[].balancer_start\n```\n\n### Querystring Bool Conversion `--convert-qs-bools`\nWhen a URL includes a query string parameter without a `=` or value, Kong uses the boolean `true` as the value. This can cause Elasticsearch mapping conflicts if is parameter is sometimes provided with a string value. This option enables converting any boolean `true` value to an empty string.\n\n### Client IP Hashing `--hash-ip`\nThis option enables hashing the `client_ip` field to avoid storing sensitive user IP addresses.\n\nSpecific raw IP addresses can be exposed with the `--expose-ip` option. This option adds a `raw_client_ip` field to logs for requests from the specified IP address hash. This option should only be used where accessing the raw IP is strictly necessary, e.g. to investigate an IP that's sending malicious requests.\n\n### Authorization Hashing `--hash-auth`\nThis option enables hashing the `credentials` part of the [`Authorization` request header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Authorization) (`request.headers.authorization` field) to avoid storing credentials/tokens.\n\n```\nAuthorization: Bearer some_secret_token -\u003e Bearer 7ftgstREEBqhHrQNgj6MVA\n```\n\n### Cookie Hashing `--hash-cookie`\nThis option enables hashing the `value` part of the [`Cookie` request header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cookie) (`request.headers.cookie` field) and [`Set-Cookie` response header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie) (`response.headers.set-cookie` field) to avoid storing sensitive cookies.\n\n```\nCookie: some_cookie=some_session -\u003e some_cookie=q1EXmTUdD0Bvm8_jHrQizw\nSet-Cookie: some_cookie=some_session; Secure; HttpOnly; SameSite=Lax -\u003e some_cookie=q1EXmTUdD0Bvm8_jHrQizw; Secure; HttpOnly; SameSite=Lax\n```\n\n### Field Hashing and Nulling `--hash-path`/`--null-path`\nArbitrary request log fields can be hashed or converted to null by specifying their path with these options. Provide the desired option multiple times to specify multiple paths.\n\nPaths describe how to traverse the JSON structure of the request logs to find a field. They consist of a hierarchy of object fields to traverse from the root JSON object, separated by periods (`.`). The `[]` suffix on a field indicates its value is an array, and should be iterated.\n\ne.g. `--hash-path tries[].ip` will hash the `ip` of every upstream \"try\" in the `tries` array.\n\nPaths don't need to end at specific value - they can specify an entire object or array.\n\ne.g. `--null-path request.headers` will convert the entire `request.headers` object to null, effectively removing it from the log.\n\nIf a path doesn't match any field in a given request log it will be ignored.\n\n### Object Limits `--limit-request-headers`/`--limit-request-querystring`\nRequests can contain arbitrary numbers of headers and query string parameters. This can create large numbers of fields in the destination Elasticsearch index, potentially causing performance and indexing issues.\n\nTo mitigate this issue, the number of keys in the `request.headers` and `request.querystring` fields are limited to 100 by default - subsequent keys are dropped. The limits can be changed by the `--limit-request-headers` and `--limit-request-querystring` options.\n\n## Output\nTransformed logs are indexed in Elasticsearch.\n\nThis is currently the only supported output method, but more may be added in the future.\n\n### Elasticsearch Nodes `-e`/`--es-node` (required)\nThe address of at least one Elasticsearch node must be provided via this option. The port should be included if non-standard (`9200`). Provide the option multiple times to specify multiple nodes in a cluster.\n\n### Elasticsearch Index `--es-index`\nThe Elasticsearch index to send logs to. [Elasticsearch index date math](https://www.elastic.co/guide/en/elasticsearch/reference/current/date-math-index-names.html) can be used. Defaults to `\u003ckong-requests-{now/d}\u003e`.\n\n### Elasticsearch Security\nA number of options exist to support Elasticsearch server and client SSL, and basic authentication. See the `-h` output for details.\n\n# Development\nTo run directly from the git repo, run the following in the root project directory:\n```bash\n\u003e pip3 install -r requirements.txt\n\u003e python3 main.py [OPTIONS]\n```\nTo run tests (as usual, from the root project directory), use:\n```bash\n\u003e python3 -m unittest\n```\nNote that these tests currently only cover the log transformation functionality - there are no automated system tests as of yet.\n\nTo build a docker image directly from the git repo, run the following in the root project directory:\n```bash\n\u003e sudo docker build -t \u003cyour repository name and tag\u003e .\n```\n\nTo develop in a docker container, first build the image, and then run the following in the root project directory:\n```bash\n\u003e sudo docker run --rm -it --name kong-log-bridge --entrypoint bash -v $(pwd):/app \u003cyour repository name and tag\u003e\n```\nThis will mount all the files inside the container, so editing tests or application code will be synced live. You can run the tests with `python -m unittest`.\n\nSend me a PR if you have a change you want to contribute!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbraedon%2Fkong-log-bridge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbraedon%2Fkong-log-bridge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbraedon%2Fkong-log-bridge/lists"}