{"id":26164727,"url":"https://github.com/optum/kong-kafka-log","last_synced_at":"2025-04-14T15:24:09.029Z","repository":{"id":48235668,"uuid":"259454582","full_name":"Optum/kong-kafka-log","owner":"Optum","description":"Log Kong Transactions to Kafka","archived":false,"fork":false,"pushed_at":"2024-06-18T17:32:24.000Z","size":1282,"stargazers_count":16,"open_issues_count":0,"forks_count":14,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-14T15:22:03.811Z","etag":null,"topics":["api-gateway","kafka","kong","logging","optum"],"latest_commit_sha":null,"homepage":"","language":"Lua","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Optum.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-27T21:04:38.000Z","updated_at":"2024-06-18T17:32:27.000Z","dependencies_parsed_at":"2024-06-18T22:00:53.049Z","dependency_job_id":null,"html_url":"https://github.com/Optum/kong-kafka-log","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Optum%2Fkong-kafka-log","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Optum%2Fkong-kafka-log/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Optum%2Fkong-kafka-log/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Optum%2Fkong-kafka-log/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Optum","download_url":"https://codeload.github.com/Optum/kong-kafka-log/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248904962,"owners_count":21180890,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api-gateway","kafka","kong","logging","optum"],"created_at":"2025-03-11T15:40:00.376Z","updated_at":"2025-04-14T15:24:09.003Z","avatar_url":"https://github.com/Optum.png","language":"Lua","funding_links":[],"categories":[],"sub_categories":[],"readme":"# kong-kafka-log\n\nThis plugin publishes request and response logs to a [Kafka](https://kafka.apache.org/) topic or to file/stdout.\n\n## Supported Kong Releases\nKong \u003e= 3.7.x\n\n## Installation\nRecommended:\n```\n$ luarocks install kong-kafka-log\n```\nOther:\n```\n$ git clone https://github.com/Optum/kong-kafka-log.git /path/to/kong/plugins/kong-kafka-log\n$ cd /path/to/kong/plugins/kong-kafka-log\n$ luarocks make *.rockspec\n```\n\n## Configuration\n\n### Enabling globally for Kafka Logging\n\n```bash\n$ curl -X POST http://kong:8001/plugins \\\n    --data \"name=kong-kafka-log\" \\\n    --data \"config.bootstrap_servers=localhost:9092\" \\\n    --data \"config.topic=kong-log\" \\\n    --data \"config.ask_id=MYASKID-00000000\" \\\n    --data \"config.app_name=GatewayStageEnvironment\" \\\n    --data \"config.timeout=10000\" \\\n    --data \"config.keepalive=60000\" \\\n    --data \"config.ssl=false\" \\\n    --data \"config.ssl_verify=false\" \\\n    --data \"config.producer_request_acks=1\" \\\n    --data \"config.producer_request_timeout=2000\" \\\n    --data \"config.producer_request_limits_messages_per_request=200\" \\\n    --data \"config.producer_request_limits_bytes_per_request=1048576\" \\\n    --data \"config.producer_request_retries_max_attempts=10\" \\\n    --data \"config.producer_request_retries_backoff_timeout=100\" \\\n    --data \"config.producer_async=true\" \\\n    --data \"config.producer_async_flush_timeout=1000\" \\\n    --data \"config.producer_async_buffering_limits_messages_in_memory=50000\"\n```\n### Enabling globally for Stdout Logging\n\n```bash\n$ curl -X POST http://kong:8001/plugins \\\n    --data \"name=kong-kafka-log\" \\\n    --data \"config.log_to_file=true\" \\\n    --data \"config.log_to_kafka=false\" \\\n    --data \"config.ask_id=MYASKID-00000000\" \\\n    --data \"config.app_name=GatewayStageEnvironment\" \\\n\n```\n\n### Parameters\n\nHere's a list of all the parameters which can be used in this plugin's configuration:\n\n| Form Parameter | default | description |\n| --- \t\t\t\t\t\t| --- | --- |\n| `name` \t\t\t\t\t                        |       | The name of the plugin to use, in this case `kafka-log` |\n| `config.log_to_file` \t                    |       | Option to log messages to file specified in `log_to_file_path`|\n| `config.log_to_file_path` \t                    |       | Location of file to output message to defaults to `/dev/stdout`|\n| `config.log_to_kafka` \t                    |       | Option to disable kafka logging |\n| `config.bootstrap_servers` \t                    |       | List of bootstrap brokers in `host:port` format |\n| `config.topic` \t\t\t                        |       | Topic to publish to |\n| `config.ask_id` \t\t\t                        |       | ASK Id |\n| `config.app_name` \t\t\t                    |       | Application name using logging utility |\n| `config.timeout`   \u003cbr /\u003e \u003csmall\u003eOptional\u003c/small\u003e | 10000 | Socket timeout in millis |\n| `config.keepalive` \u003cbr /\u003e \u003csmall\u003eOptional\u003c/small\u003e | 60000 | Keepalive timeout in millis |\n| `config.ssl` \u003cbr /\u003e \u003csmall\u003eOptional\u003c/small\u003e | false | Enable SSL Connection |\n| `config.ssl_verify` \u003cbr /\u003e \u003csmall\u003eOptional\u003c/small\u003e | false | Enable SSL Verification |\n| `config.producer_request_acks` \u003cbr /\u003e \u003csmall\u003eOptional\u003c/small\u003e                              | 1       | The number of acknowledgments the producer requires the leader to have received before considering a request complete. Allowed values: 0 for no acknowledgments, 1 for only the leader and -1 for the full ISR |\n| `config.producer_request_timeout` \u003cbr /\u003e \u003csmall\u003eOptional\u003c/small\u003e                           | 2000    | Time to wait for a Produce response in millis |\n| `config.producer_request_limits_messages_per_request` \u003cbr /\u003e \u003csmall\u003eOptional\u003c/small\u003e       | 200     | Maximum number of messages to include into a single Produce request |\n| `config.producer_request_limits_bytes_per_request` \u003cbr /\u003e \u003csmall\u003eOptional\u003c/small\u003e \t     | 1048576 | Maximum size of a Produce request in bytes |\n| `config.producer_request_retries_max_attempts` \u003cbr /\u003e \u003csmall\u003eOptional\u003c/small\u003e \t         | 10      | Maximum number of retry attempts per single Produce request |\n| `config.producer_request_retries_backoff_timeout` \u003cbr /\u003e \u003csmall\u003eOptional\u003c/small\u003e\t     \t | 100     | Backoff interval between retry attempts in millis |\n| `config.producer_async` \u003cbr /\u003e \u003csmall\u003eOptional\u003c/small\u003e                                     | true    | Flag to enable asynchronous mode |\n| `config.producer_async_flush_timeout` \u003cbr /\u003e \u003csmall\u003eOptional\u003c/small\u003e                       | 1000    | Maximum time interval in millis between buffer flushes in in asynchronous mode |\n| `config.producer_async_buffering_limits_messages_in_memory` \u003cbr /\u003e \u003csmall\u003eOptional\u003c/small\u003e | 50000   | Maximum number of messages that can be buffered in memory in asynchronous mode |\n\n## Log Format\n\n```\n{\n  \"application\": {\n    \"askId\": \"ASKID000-000000\",\n    \"name\": \"GatewaySampleInstance\"\n  },\n  \"destHost\": {\n    \"port\": 443,\n    \"ipv4\": 179347885,\n    \"path\": \"/Backend/api/path\",\n    \"hostname\": \"api-service.company.com\"\n  },\n  \"name\": \"my.api.service.name\",\n  \"device\": {\n    \"vendor\": \"Optum\",\n    \"ip4\": 176365362,\n    \"hostname\": \"kong-507-g7rck\",\n    \"product\": \"kong-kafka-log\"\n  },\n  \"receivedTime\": 1588059064647,\n  \"msg\": \"STARGATE_PROXY_TX\",\n  \"sourceHost\": {\n    \"port\": 443,\n    \"ip4\": 179301896\n  },\n  \"logClass\": \"SECURITY_SUCCESS\",\n  \"request\": {\n    \"out\": 675,\n    \"method\": \"GET\",\n    \"request\": \"https://gateway.company.com:443/api/proxy/path/service/v1\",\n    \"Optum_CID_Ext\": \"27097e07-fa15-4bbb-9a9e-7cea46abc422#1\",\n    \"in\": 339\n  },\n  \"eventClass\": \"200\",\n  \"sourceUser\": {\n    \"tokenHash\": \"3502a5bc96d4468c6974e8b415eb8899b501b1ca6dc717ee4e07ff655dbaebb6\", --\u003e sha256(authentication token)\n    \"name\": \"consumer.name\"\n  }\n}\n```\n\nAs the log format is custom for our requirements for Kafka logging, you can fork and replace the ```/src/basic.lua``` with your desired logging format.\n\n## Known issues and limitations\n\nKnown limitations:\n\n1. There is no support for Authentication (Would like to see MTLS Auth or [SASL](https://kafka.apache.org/protocol.html#sasl_handshake) written into underlying dependency [library](https://github.com/doujiang24/lua-resty-kafka))\n2. There is no support for message compression\n\n## Quickstart\n\n1. Install `kong-kafka-log` via `luarocks`:\n\n    ```\n    luarocks install kong-kafka-log\n    ```\n\n2. Load the `kong-kafka-log` in `Kong`:\n\n    ```\n    KONG_PLUGINS=bundled,kong-kafka-log bin/kong start\n    ```\n\n2. Add `kong-kafka-log` plugin globally:\n\n    ```\n    curl -X POST http://localhost:8001/plugins \\\n        --data \"name=kong-kafka-log\" \\\n        --data \"config.bootstrap_servers=localhost:9093\" \\\n        --data \"config.ask_id=testaskid\" \\\n        --data \"config.app_name=gatewayappname\" \\\n        --data \"config.ssl=true\" \\\n        --data \"config.topic=example-topic\"\n    ```\n\n3. As requests are made, check your Kafka instance!\n\n## Implementation details\n\nOriginal source rewritten from [kong-plugin-kafka-log](https://github.com/yskopets/kong-plugin-kafka-log) by [yskopets](https://github.com/yskopets), Big Thanks!  \nThis plugin makes use of [lua-resty-kafka](https://github.com/doujiang24/lua-resty-kafka) as a dependency. Need version \u003e= ```v0.09```\n\n## Maintainers\n[jeremyjpj0916](https://github.com/jeremyjpj0916)  \n[rsbrisci](https://github.com/rsbrisci)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foptum%2Fkong-kafka-log","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foptum%2Fkong-kafka-log","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foptum%2Fkong-kafka-log/lists"}