{"id":13775838,"url":"https://github.com/grepplabs/kafka-proxy","last_synced_at":"2026-02-02T01:11:52.792Z","repository":{"id":40505141,"uuid":"121521103","full_name":"grepplabs/kafka-proxy","owner":"grepplabs","description":"Proxy connections to Kafka cluster. Connect through SOCKS Proxy, HTTP Proxy or to cluster running in Kubernetes.","archived":false,"fork":false,"pushed_at":"2024-11-15T13:00:57.000Z","size":11911,"stargazers_count":500,"open_issues_count":40,"forks_count":86,"subscribers_count":15,"default_branch":"master","last_synced_at":"2024-11-15T13:41:28.896Z","etag":null,"topics":["kafka","kafka-gateway","kafka-proxy","oauthbearer","proxy","sasl","socks5"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/grepplabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"custom":"https://paypal.me/grepplabs?locale.x=en_GB"}},"created_at":"2018-02-14T14:39:49.000Z","updated_at":"2024-11-15T12:53:29.000Z","dependencies_parsed_at":"2023-02-19T10:00:53.572Z","dependency_job_id":"b16cfb2e-0073-4dcb-bd5d-264b155277fb","html_url":"https://github.com/grepplabs/kafka-proxy","commit_stats":null,"previous_names":[],"tags_count":39,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grepplabs%2Fkafka-proxy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grepplabs%2Fkafka-proxy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grepplabs%2Fkafka-proxy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grepplabs%2Fkafka-proxy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/grepplabs","download_url":"https://codeload.github.com/grepplabs/kafka-proxy/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225031353,"owners_count":17410046,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","kafka-gateway","kafka-proxy","oauthbearer","proxy","sasl","socks5"],"created_at":"2024-08-03T17:01:52.130Z","updated_at":"2026-02-02T01:11:52.786Z","avatar_url":"https://github.com/grepplabs.png","language":"Go","funding_links":["https://paypal.me/grepplabs?locale.x=en_GB"],"categories":["\u003ca id=\"d03d494700077f6a65092985c06bf8e8\"\u003e\u003c/a\u003e工具","Operations"],"sub_categories":["\u003ca id=\"57b8e953d394bbed52df2a6976d98dfa\"\u003e\u003c/a\u003eSocks","Tools"],"readme":"## kafka-proxy\n\n[![Build Status](https://github.com/grepplabs/kafka-proxy/actions/workflows/build.yaml/badge.svg)](https://github.com/grepplabs/kafka-proxy/actions/workflows/build.yaml)\n[![Docker Hub](https://img.shields.io/badge/docker-latest-blue.svg)](https://hub.docker.com/r/grepplabs/kafka-proxy)\n[![Docker Pulls](https://img.shields.io/docker/pulls/grepplabs/kafka-proxy)](https://hub.docker.com/r/grepplabs/kafka-proxy)\n\nThe Kafka Proxy is based on idea of [Cloud SQL Proxy](https://github.com/GoogleCloudPlatform/cloudsql-proxy). \nIt allows a service to connect to Kafka brokers without having to deal with SASL/PLAIN authentication and SSL certificates.  \n\nIt works by opening tcp sockets on the local machine and proxying connections to the associated Kafka brokers\nwhen the sockets are used. The host and port in [Metadata](http://kafka.apache.org/protocol.html#The_Messages_Metadata)\nand [FindCoordinator](http://kafka.apache.org/protocol.html#The_Messages_FindCoordinator)\nresponses received from the brokers are replaced by local counterparts.\nFor discovered brokers (not configured as the boostrap servers), local listeners are started on random ports.\nThe dynamic local listeners feature can be disabled and an additional list of external server mappings can be provided.\n\nThe Proxy can terminate TLS traffic and authenticate users using SASL/PLAIN. The credentials verification method\nis configurable and uses golang plugin system over RPC.\n\nThe proxies can also authenticate each other using a pluggable method which is transparent to other Kafka servers and clients.\nCurrently, the Google ID Token for service accounts is implemented i.e. proxy client requests and sends service account JWT and proxy server receives and validates it against Google JWKS.\n\nKafka API calls can be restricted to prevent some operations e.g. topic deletion or produce requests.\n\n\nSee:\n* [Kafka Proxy with Amazon MSK](https://gist.github.com/everesio/262e11c6e5cebf56f1d5111c8cd7da3f)\n* [A Guide To The Kafka Protocol](https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol)\n* [Kafka protocol guide](http://kafka.apache.org/protocol.html)\n\n\n### Supported Kafka versions\nFollowing table provides overview of supported Kafka versions (specified one and all previous Kafka versions).\nAs not every Kafka release adds new messages/versions which are relevant to the Kafka proxy, newer Kafka versions can also work.\n\n\n| Kafka proxy version | Kafka version |\n|---------------------|---------------|\n|                     | from 0.11.0   |\n| 0.2.9               | to 2.8.0      |\n| 0.3.1               | to 3.4.0      |\n| 0.3.11              | to 3.7.0      |\n| 0.3.12              | to 3.9.0      |\n| 0.4.2               | to 4.0.0      |\n\n### Install binary release\n\n1. Download the latest release\n\n   Linux\n\n        curl -Ls https://github.com/grepplabs/kafka-proxy/releases/download/v0.4.3/kafka-proxy-v0.4.3-linux-amd64.tar.gz | tar xz\n\n   macOS\n\n        curl -Ls https://github.com/grepplabs/kafka-proxy/releases/download/v0.4.3/kafka-proxy-v0.4.3-darwin-amd64.tar.gz | tar xz\n\n2. Move the binary in to your PATH.\n\n    ```\n    sudo mv ./kafka-proxy /usr/local/bin/kafka-proxy\n    ```\n\n### Building\n\n    make clean build\n\n### Docker images\n\nDocker images are available on [Docker Hub](https://hub.docker.com/r/grepplabs/kafka-proxy/tags).\n\nYou can launch a kafka-proxy container for trying it out with\n\n    docker run --rm -p 30001-30003:30001-30003 grepplabs/kafka-proxy:0.4.3 \\\n              server \\\n            --bootstrap-server-mapping \"localhost:19092,0.0.0.0:30001\" \\\n            --bootstrap-server-mapping \"localhost:29092,0.0.0.0:30002\" \\\n            --bootstrap-server-mapping \"localhost:39092,0.0.0.0:30003\" \\\n            --dial-address-mapping \"localhost:19092,172.17.0.1:19092\" \\\n            --dial-address-mapping \"localhost:29092,172.17.0.1:29092\" \\\n            --dial-address-mapping \"localhost:39092,172.17.0.1:39092\" \\\n            --debug-enable\n\nKafka-proxy will now be reachable on `localhost:30001`, `localhost:30002` and `localhost:30003`, connecting to kafka brokers\nrunning in docker (network bridge gateway `172.17.0.1`) advertising PLAINTEXT listeners on `localhost:19092`, `localhost:29092` and `localhost:39092`.\n\n### Docker images with precompiled plugins\n\nDocker images with precompiled plugins located in `/opt/kafka-proxy/bin/` are tagged with `\u003crelease\u003e-all`.\n\nYou can launch a kafka-proxy container with auth-ldap plugin for trying it out with\n\n    docker run --rm -p 30001-30003:30001-30003 grepplabs/kafka-proxy:0.4.3-all \\\n                  server \\\n                --bootstrap-server-mapping \"localhost:19092,0.0.0.0:30001\" \\\n                --bootstrap-server-mapping \"localhost:29092,0.0.0.0:30002\" \\\n                --bootstrap-server-mapping \"localhost:39092,0.0.0.0:30003\" \\\n                --dial-address-mapping \"localhost:19092,172.17.0.1:19092\" \\\n                --dial-address-mapping \"localhost:29092,172.17.0.1:29092\" \\\n                --dial-address-mapping \"localhost:39092,172.17.0.1:39092\" \\\n                --debug-enable \\\n                --auth-local-enable  \\\n                --auth-local-command=/opt/kafka-proxy/bin/auth-ldap  \\\n                --auth-local-param=--url=ldap://172.17.0.1:389  \\\n                --auth-local-param=--start-tls=false \\\n                --auth-local-param=--bind-dn=cn=admin,dc=example,dc=org  \\\n                --auth-local-param=--bind-passwd=admin  \\\n                --auth-local-param=--user-search-base=ou=people,dc=example,dc=org  \\\n                --auth-local-param=--user-filter=\"(\u0026(objectClass=person)(uid=%u)(memberOf=cn=kafka-users,ou=realm-roles,dc=example,dc=org))\"\n\n\n### Help output\n\n    Run the kafka-proxy server\n\n    Usage:\n      kafka-proxy server [flags]\n\n      Flags:\n            --auth-gateway-client-command string                   Path to authentication plugin binary\n            --auth-gateway-client-enable                           Enable gateway client authentication\n            --auth-gateway-client-log-level string                 Log level of the auth plugin (default \"trace\")\n            --auth-gateway-client-magic uint                       Magic bytes sent in the handshake\n            --auth-gateway-client-method string                    Authentication method\n            --auth-gateway-client-param stringArray                Authentication plugin parameter\n            --auth-gateway-client-timeout duration                 Authentication timeout (default 10s)\n            --auth-gateway-server-command string                   Path to authentication plugin binary\n            --auth-gateway-server-enable                           Enable proxy server authentication\n            --auth-gateway-server-log-level string                 Log level of the auth plugin (default \"trace\")\n            --auth-gateway-server-magic uint                       Magic bytes sent in the handshake\n            --auth-gateway-server-method string                    Authentication method\n            --auth-gateway-server-param stringArray                Authentication plugin parameter\n            --auth-gateway-server-timeout duration                 Authentication timeout (default 10s)\n            --auth-local-command string                            Path to authentication plugin binary\n            --auth-local-enable                                    Enable local SASL/PLAIN authentication performed by listener - SASL handshake will not be passed to kafka brokers\n            --auth-local-log-level string                          Log level of the auth plugin (default \"trace\")\n            --auth-local-mechanism string                          SASL mechanism used for local authentication: PLAIN or OAUTHBEARER (default \"PLAIN\")\n            --auth-local-param stringArray                         Authentication plugin parameter\n            --auth-local-timeout duration                          Authentication timeout (default 10s)\n            --bootstrap-server-mapping stringArray                 Mapping of Kafka bootstrap server address to local address (host:port,host:port(,advhost:advport))\n            --debug-enable                                         Enable Debug endpoint\n            --debug-listen-address string                          Debug listen address (default \"0.0.0.0:6060\")\n            --default-listener-ip string                           Default listener IP (default \"0.0.0.0\")\n            --deterministic-listeners                              Enable deterministic listeners (listener port = min port + broker id).\n            --dial-address-mapping stringArray                     Mapping of target broker address to new one (host:port,host:port). The mapping is performed during connection establishment\n            --dynamic-advertised-listener string                   Advertised address for dynamic listeners. If left empty, default-listener-ip is used. Supports templating with {{.brokerId}} for dynamic hostnames and a fixed port if provided.\n            --dynamic-listeners-disable                            Disable dynamic listeners.\n            --dynamic-sequential-min-port int                      If set to non-zero, makes the dynamic listener use a sequential port starting with this value rather than a random port every time.\n            --external-server-mapping stringArray                  Mapping of Kafka server address to external address (host:port,host:port). A listener for the external address is not started\n            --forbidden-api-keys ints                              Forbidden Kafka request types. The restriction should prevent some Kafka operations e.g. 20 - DeleteTopics\n            --forward-proxy string                                 URL of the forward proxy. Supported schemas are socks5 and http\n            --gssapi-auth-type string                              GSSAPI auth type: KEYTAB or USER (default \"KEYTAB\")\n            --gssapi-disable-pa-fx-fast                            Used to configure the client to not use PA_FX_FAST.\n            --gssapi-keytab string                                 krb5.keytab file location\n            --gssapi-krb5 string                                   krb5.conf file path, default: /etc/krb5.conf (default \"/etc/krb5.conf\")\n            --gssapi-password string                               Password for auth type USER\n            --gssapi-realm string                                  Realm\n            --gssapi-servicename string                            ServiceName (default \"kafka\")\n            --gssapi-spn-host-mapping stringToString               Mapping of Kafka servers address to SPN hosts (default [])\n            --gssapi-username string                               Username (default \"kafka\")\n        -h, --help                                                 help for server\n            --http-disable                                         Disable HTTP endpoints\n            --http-health-path string                              Path on which to health endpoint (default \"/health\")\n            --http-listen-address string                           Address that kafka-proxy is listening on (default \"0.0.0.0:9080\")\n            --http-metrics-path string                             Path on which to expose metrics (default \"/metrics\")\n            --kafka-client-id string                               An optional identifier to track the source of requests (default \"kafka-proxy\")\n            --kafka-connection-read-buffer-size int                Size of the operating system's receive buffer associated with the connection. If zero, system default is used\n            --kafka-connection-write-buffer-size int               Sets the size of the operating system's transmit buffer associated with the connection. If zero, system default is used\n            --kafka-dial-timeout duration                          How long to wait for the initial connection (default 15s)\n            --kafka-keep-alive duration                            Keep alive period for an active network connection. If zero, keep-alives are disabled (default 1m0s)\n            --kafka-max-open-requests int                          Maximal number of open requests pro tcp connection before sending on it blocks (default 256)\n            --kafka-read-timeout duration                          How long to wait for a response (default 30s)\n            --kafka-write-timeout duration                         How long to wait for a transmit (default 30s)\n            --log-format string                                    Log format text or json (default \"text\")\n            --log-level string                                     Log level trace, debug, info, warning, error, fatal or panic (default \"info\")\n            --log-level-fieldname string                           Log level fieldname for json format (default \"@level\")\n            --log-msg-fieldname string                             Message fieldname for json format (default \"@message\")\n            --log-time-fieldname string                            Time fieldname for json format (default \"@timestamp\")\n            --producer-acks-0-disabled                             Assume fire-and-forget is never sent by the producer. Enabling this parameter will increase performance\n            --proxy-listener-ca-chain-cert-file string             PEM encoded CA's certificate file. If provided, client certificate is required and verified\n            --proxy-listener-cert-file string                      PEM encoded file with server certificate\n            --proxy-listener-cipher-suites strings                 List of supported cipher suites\n            --proxy-listener-crl-file string                       PEM encoded X509 CRLs file\n            --proxy-listener-curve-preferences strings             List of curve preferences\n            --proxy-listener-keep-alive duration                   Keep alive period for an active network connection. If zero, keep-alives are disabled (default 1m0s)\n            --proxy-listener-key-file string                       PEM encoded file with private key for the server certificate\n            --proxy-listener-key-password string                   Password to decrypt rsa private key\n            --proxy-listener-read-buffer-size int                  Size of the operating system's receive buffer associated with the connection. If zero, system default is used\n            --proxy-listener-tls-enable                            Whether or not to use TLS listener\n            --proxy-listener-tls-refresh duration                  Interval for refreshing server TLS certificates. If set to zero, the refresh watch is disabled\n            --proxy-listener-tls-required-client-subject strings   Required client certificate subject common name; example; s:/CN=[value]/C=[state]/C=[DE,PL] or r:/CN=[^val.{2}$]/C=[state]/C=[DE,PL]; check manual for more details\n            --proxy-listener-write-buffer-size int                 Sets the size of the operating system's transmit buffer associated with the connection. If zero, system default is used\n            --proxy-request-buffer-size int                        Request buffer size pro tcp connection (default 4096)\n            --proxy-response-buffer-size int                       Response buffer size pro tcp connection (default 4096)\n            --sasl-aws-identity-lookup                             Verify AWS authentication identity\n            --sasl-aws-profile string                              AWS profile\n            --sasl-aws-region string                               Region for AWS IAM Auth\n            --sasl-aws-role-arn string                             AWS Role ARN to assume\n            --sasl-enable                                          Connect using SASL\n            --sasl-jaas-config-file string                         Location of JAAS config file with SASL username and password\n            --sasl-method string                                   SASL method to use (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, GSSAPI, AWS_MSK_IAM (default \"PLAIN\")\n            --sasl-password string                                 SASL user password\n            --sasl-plugin-command string                           Path to authentication plugin binary\n            --sasl-plugin-enable                                   Use plugin for SASL authentication\n            --sasl-plugin-log-level string                         Log level of the auth plugin (default \"trace\")\n            --sasl-plugin-mechanism string                         SASL mechanism used for proxy authentication: PLAIN or OAUTHBEARER (default \"OAUTHBEARER\")\n            --sasl-plugin-param stringArray                        Authentication plugin parameter\n            --sasl-plugin-timeout duration                         Authentication timeout (default 10s)\n            --sasl-username string                                 SASL user name\n            --tls-ca-chain-cert-file string                        PEM encoded CA's certificate file\n            --tls-client-cert-file string                          PEM encoded file with client certificate\n            --tls-client-key-file string                           PEM encoded file with private key for the client certificate\n            --tls-client-key-password string                       Password to decrypt rsa private key\n            --tls-enable                                           Whether or not to use TLS when connecting to the broker\n            --tls-insecure-skip-verify                             It controls whether a client verifies the server's certificate chain and host name\n            --tls-refresh duration                                 Interval for refreshing client TLS certificates. If set to zero, the refresh watch is disabled\n            --tls-same-client-cert-enable                          Use only when mutual TLS is enabled on proxy and broker. It controls whether a proxy validates if proxy client certificate exactly matches brokers client cert (tls-client-cert-file)\n            --tls-system-cert-pool                                 Use system pool for root CAs\n\n### Usage example\n\t\n\tkafka-proxy server --bootstrap-server-mapping \"192.168.99.100:32400,0.0.0.0:32399\"\n\t\n\tkafka-proxy server --bootstrap-server-mapping \"192.168.99.100:32400,127.0.0.1:32400\" \\\n\t                   --bootstrap-server-mapping \"192.168.99.100:32401,127.0.0.1:32401\" \\\n\t                   --bootstrap-server-mapping \"192.168.99.100:32402,127.0.0.1:32402\" \\\n\t                   --dynamic-listeners-disable\n\n\tkafka-proxy server --bootstrap-server-mapping \"kafka-0.example.com:9092,0.0.0.0:32401,kafka-0.grepplabs.com:9092\" \\\n\t                   --bootstrap-server-mapping \"kafka-1.example.com:9092,0.0.0.0:32402,kafka-1.grepplabs.com:9092\" \\\n\t                   --bootstrap-server-mapping \"kafka-2.example.com:9092,0.0.0.0:32403,kafka-2.grepplabs.com:9092\" \\\n\t                   --dynamic-listeners-disable\n\n\tkafka-proxy server --bootstrap-server-mapping \"192.168.99.100:32400,127.0.0.1:32400\" \\\n\t                   --external-server-mapping \"192.168.99.100:32401,127.0.0.1:32402\" \\\n\t                   --external-server-mapping \"192.168.99.100:32402,127.0.0.1:32403\" \\\n\t                   --forbidden-api-keys 20\n    \n\n    export BOOTSTRAP_SERVER_MAPPING=\"192.168.99.100:32401,0.0.0.0:32402 192.168.99.100:32402,0.0.0.0:32403\" \u0026\u0026 kafka-proxy server\n\n\n### Restrict proxy listener cipher suites\n\n    kafka-proxy server --bootstrap-server-mapping \"localhost:19092,0.0.0.0:30001,localhost:30001\" \\\n                       --bootstrap-server-mapping \"localhost:29092,0.0.0.0:30002,localhost:30002\" \\\n                       --bootstrap-server-mapping \"localhost:39092,0.0.0.0:30003,localhost:30003\" \\\n                       --proxy-listener-cert-file \"tls/ca-cert.pem\" \\\n                       --proxy-listener-key-file \"tls/ca-key.pem\"  \\\n                       --proxy-listener-tls-enable \\\n                       --proxy-listener-cipher-suites TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256\n\n### SASL authentication initiated by proxy example\n\nSASL authentication is initiated by the proxy. SASL authentication is disabled on the clients and enabled on the Kafka brokers.   \n\n    kafka-proxy server --bootstrap-server-mapping \"kafka-0.grepplabs.com:9093,0.0.0.0:32399\" \\\n                       --tls-enable --tls-insecure-skip-verify \\\n                       --sasl-enable --sasl-username myuser --sasl-password mysecret\n\n    kafka-proxy server --bootstrap-server-mapping \"kafka-0.example.com:9092,0.0.0.0:30001\" \\\n                       --bootstrap-server-mapping \"kafka-1.example.com:9092,0.0.0.0:30002\" \\\n                       --bootstrap-server-mapping \"kafka-1.example.com:9093,0.0.0.0:30003\" \\\n                       --sasl-enable \\\n                       --sasl-username \"alice\" \\\n                       --sasl-password \"alice-secret\" \\\n                       --sasl-method \"SCRAM-SHA-512\" \\\n                       --log-level debug\n\n    make clean build plugin.unsecured-jwt-provider \u0026\u0026 build/kafka-proxy server \\\n                             --sasl-enable \\\n                             --sasl-plugin-enable \\\n                             --sasl-plugin-mechanism \"OAUTHBEARER\" \\\n                             --sasl-plugin-command build/unsecured-jwt-provider \\\n                             --sasl-plugin-param \"--claim-sub=alice\" \\\n                             --bootstrap-server-mapping \"192.168.99.100:32400,127.0.0.1:32400\"\n\n\nGSSAPI / Kerberos authentication\n\n\n    kafka-proxy server --bootstrap-server-mapping \"kafka-0.grepplabs.com:9092,127.0.0.1:32500\" \\\n                       --bootstrap-server-mapping \"kafka-1.grepplabs.com:9092,127.0.0.1:32501\" \\\n                       --bootstrap-server-mapping \"kafka-2.grepplabs.com:9092,127.0.0.1:32502\" \\\n                       --sasl-enable \\\n                       --sasl-method \"GSSAPI\" \\\n                       --gssapi-servicename kafka \\\n                       --gssapi-username kafkaclient1 \\\n                       --gssapi-realm EXAMPLE.COM \\\n                       --gssapi-krb5 /etc/krb5.conf \\\n                       --gssapi-keytab /etc/security/keytabs/kafka.keytab\n\nAWS MSK IAM\n\n    kafka-proxy server --bootstrap-server-mapping \"b-1-public.kafkaproxycluster.uls9ao.c4.kafka.eu-central-1.amazonaws.com:9198,0.0.0.0:30001\" \\\n                       --bootstrap-server-mapping \"b-2-public.kafkaproxycluster.uls9ao.c4.kafka.eu-central-1.amazonaws.com:9198,0.0.0.0:30002\" \\\n                       --bootstrap-server-mapping \"b-3-public.kafkaproxycluster.uls9ao.c4.kafka.eu-central-1.amazonaws.com:9198,0.0.0.0:30003\" \\\n                       --tls-enable --tls-insecure-skip-verify \\\n                       --sasl-enable \\\n                       --sasl-method \"AWS_MSK_IAM\" \\\n                       --sasl-aws-region \"eu-central-1\" \\\n                       --log-level debug\n\n\n### Proxy authentication example\n\nSASL authentication is performed by the proxy. SASL authentication is enabled on the clients and disabled on the Kafka brokers.   \n\n    make clean build plugin.auth-user \u0026\u0026 build/kafka-proxy server --proxy-listener-key-file \"server-key.pem\"  \\\n                             --proxy-listener-cert-file \"server-cert.pem\" \\\n                             --proxy-listener-ca-chain-cert-file \"ca.pem\" \\\n                             --proxy-listener-tls-enable \\\n                             --auth-local-enable \\\n                             --auth-local-command build/auth-user \\\n                             --auth-local-param \"--username=my-test-user\" \\\n                             --auth-local-param \"--password=my-test-password\"\n\n    make clean build plugin.auth-ldap \u0026\u0026 build/kafka-proxy server \\\n                             --auth-local-enable \\\n                             --auth-local-command build/auth-ldap \\\n                             --auth-local-param \"--url=ldaps://ldap.example.com:636\" \\\n                             --auth-local-param \"--user-dn=cn=users,dc=exemple,dc=com\" \\\n                             --auth-local-param \"--user-attr=uid\" \\\n                             --bootstrap-server-mapping \"192.168.99.100:32400,127.0.0.1:32400\"\n\n    make clean build plugin.unsecured-jwt-info \u0026\u0026 build/kafka-proxy server \\\n                             --auth-local-enable \\\n                             --auth-local-command build/unsecured-jwt-info \\\n                             --auth-local-mechanism \"OAUTHBEARER\" \\\n                             --auth-local-param \"--claim-sub=alice\" \\\n                             --auth-local-param \"--claim-sub=bob\" \\\n                             --bootstrap-server-mapping \"192.168.99.100:32400,127.0.0.1:32400\"\n                             \n### Same client certificate check enabled example\n\nValidate that client certificate used by proxy client is exactly the same as client certificate in authentication initiated by proxy \n                       \n    kafka-proxy server --bootstrap-server-mapping \"kafka-0.grepplabs.com:9093,0.0.0.0:32399\" \\\n       --tls-enable \\\n       --tls-client-cert-file client.crt \\\n       --tls-client-key-file client.pem \\\n       --tls-client-key-password changeit \\\n       --proxy-listener-tls-enable \\\n       --proxy-listener-key-file server.pem \\\n       --proxy-listener-cert-file server.crt \\\n       --proxy-listener-key-password changeit \\\n       --proxy-listener-ca-chain-cert-file ca.crt \\\n       --tls-same-client-cert-enable\n\n### Kafka Gateway example\n\nAuthentication between Kafka Proxy Client and Kafka Proxy Server with Google-ID (service account JWT)\n\n    kafka-proxy server --bootstrap-server-mapping \"kafka-0.grepplabs.com:9092,127.0.0.1:32500\" \\\n                       --bootstrap-server-mapping \"kafka-1.grepplabs.com:9092,127.0.0.1:32501\" \\\n                       --bootstrap-server-mapping \"kafka-2.grepplabs.com:9092,127.0.0.1:32502\" \\\n                       --dynamic-listeners-disable \\\n                       --http-disable \\\n                       --proxy-listener-tls-enable \\\n                       --proxy-listener-cert-file=/var/run/secret/server.cert.pem \\\n                       --proxy-listener-key-file=/var/run/secret/server.key.pem \\\n                       --auth-gateway-server-enable \\\n                       --auth-gateway-server-method google-id \\\n                       --auth-gateway-server-magic 3285573610483682037 \\\n                       --auth-gateway-server-command google-id-info \\\n                       --auth-gateway-server-param  \"--timeout=10\" \\\n                       --auth-gateway-server-param  \"--audience=tcp://kafka-gateway.grepplabs.com\" \\\n                       --auth-gateway-server-param  \"--email-regex=^kafka-gateway@my-project.iam.gserviceaccount.com$\"\n\n    kafka-proxy server --bootstrap-server-mapping \"127.0.0.1:32500,127.0.0.1:32400\" \\\n                       --bootstrap-server-mapping \"127.0.0.1:32501,127.0.0.1:32401\" \\\n                       --bootstrap-server-mapping \"127.0.0.1:32502,127.0.0.1:32402\" \\\n                       --dynamic-listeners-disable \\\n                       --http-disable \\\n                       --tls-enable \\\n                       --tls-ca-chain-cert-file /var/run/secret/client/ca-chain.cert.pem \\\n                       --auth-gateway-client-enable \\\n                       --auth-gateway-client-method google-id \\\n                       --auth-gateway-client-magic 3285573610483682037 \\\n                       --auth-gateway-client-command google-id-provider \\\n                       --auth-gateway-client-param  \"--credentials-file=/var/run/secret/client/service-account.json\" \\\n                       --auth-gateway-client-param  \"--target-audience=tcp://kafka-gateway.grepplabs.com\" \\\n                       --auth-gateway-client-param  \"--timeout=10\"\n\n### Connect to Kafka through SOCKS5 Proxy example\n\nConnect through test SOCKS5 Proxy server\n\n```\n    kafka-proxy tools socks5-proxy --addr localhost:1080\n\n    kafka-proxy server --bootstrap-server-mapping \"kafka-0.grepplabs.com:9092,127.0.0.1:32500\" \\\n                       --bootstrap-server-mapping \"kafka-1.grepplabs.com:9092,127.0.0.1:32501\" \\\n                       --bootstrap-server-mapping \"kafka-2.grepplabs.com:9092,127.0.0.1:32502\"\n                       --forward-proxy socks5://localhost:1080\n```\n\n```\n    kafka-proxy tools socks5-proxy --addr localhost:1080 --username my-proxy-user --password my-proxy-password\n\n    kafka-proxy server --bootstrap-server-mapping \"kafka-0.grepplabs.com:9092,127.0.0.1:32500\" \\\n                       --bootstrap-server-mapping \"kafka-1.grepplabs.com:9092,127.0.0.1:32501\" \\\n                       --bootstrap-server-mapping \"kafka-2.grepplabs.com:9092,127.0.0.1:32502\" \\\n                       --forward-proxy socks5://my-proxy-user:my-proxy-password@localhost:1080\n```\n\n### Connect to Kafka through HTTP Proxy example\n\nConnect through test HTTP Proxy server using CONNECT method\n\n```\n    kafka-proxy tools http-proxy --addr localhost:3128\n\n    kafka-proxy server --bootstrap-server-mapping \"kafka-0.grepplabs.com:9092,127.0.0.1:32500\" \\\n                       --bootstrap-server-mapping \"kafka-1.grepplabs.com:9092,127.0.0.1:32501\" \\\n                       --bootstrap-server-mapping \"kafka-2.grepplabs.com:9092,127.0.0.1:32502\"\n                       --forward-proxy http://localhost:3128\n```\n\n```\n    kafka-proxy tools http-proxy --addr localhost:3128 --username my-proxy-user --password my-proxy-password\n\n    kafka-proxy server --bootstrap-server-mapping \"kafka-0.grepplabs.com:9092,127.0.0.1:32500\" \\\n                       --bootstrap-server-mapping \"kafka-1.grepplabs.com:9092,127.0.0.1:32501\" \\\n                       --bootstrap-server-mapping \"kafka-2.grepplabs.com:9092,127.0.0.1:32502\" \\\n                       --forward-proxy http://my-proxy-user:my-proxy-password@localhost:3128\n```\n\n### Validating client certificate DN\n\nSometimes it might be necessary to not only validate that the client certificate is valid but also that the client certificate DN is issued for a concrete use case. This can be achieved using the following set of arguments:\n\n```\n--proxy-listener-tls-client-cert-validate-subject bool                        Whether to validate client certificate subject (default false)\n--proxy-listener-tls-required-client-subject-common-name string               Required client certificate subject common name\n--proxy-listener-tls-required-client-subject-country stringArray              Required client certificate subject country\n--proxy-listener-tls-required-client-subject-province stringArray             Required client certificate subject province\n--proxy-listener-tls-required-client-subject-locality stringArray             Required client certificate subject locality\n--proxy-listener-tls-required-client-subject-organization stringArray         Required client certificate subject organization\n--proxy-listener-tls-required-client-subject-organizational-unit stringArray  Required client certificate subject organizational unit\n```\n\nBy setting `--proxy-listener-tls-client-cert-validate-subject true`, Kafka Proxy will inspect client certificate DN fields for the expected values set with the `--proxy-listener-tls-required-client-*` arguments. The matches are always exact and used together, fo all non empty values. For example, to allow a valid certificate for `country=DE` and `organization=grepplabs`, configure Kafka Proxy in the following way:\n\n```\n    kafka-proxy server \\\n      --proxy-listener-tls-client-cert-validate-subject true \\\n      --proxy-listener-tls-required-client-subject-country DE \\\n      --proxy-listener-tls-required-client-subject-organization grepplabs\n```\n\n### Kubernetes sidecar container example\n\n```yaml\n\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n   name: myapp\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      app: myapp\n  template:\n    metadata:\n      labels:\n        app: myapp\n      annotations:\n        prometheus.io/scrape: 'true'\n    spec:\n      containers:\n        - name: kafka-proxy\n          image: grepplabs/kafka-proxy:latest\n          args:\n            - 'server'\n            - '--log-format=json'\n            - '--bootstrap-server-mapping=kafka-0:9093,127.0.0.1:32400'\n            - '--bootstrap-server-mapping=kafka-1:9093,127.0.0.1:32401'\n            - '--bootstrap-server-mapping=kafka-2:9093,127.0.0.1:32402'\n            - '--tls-enable'\n            - '--tls-ca-chain-cert-file=/var/run/secret/kafka-ca-chain-certificate/ca-chain.cert.pem'\n            - '--tls-client-cert-file=/var/run/secret/kafka-client-certificate/client.cert.pem'\n            - '--tls-client-key-file=/var/run/secret/kafka-client-key/client.key.pem'\n            - '--tls-client-key-password=$(TLS_CLIENT_KEY_PASSWORD)'\n            - '--sasl-enable'\n            - '--sasl-jaas-config-file=/var/run/secret/kafka-client-jaas/jaas.config'\n          env:\n          - name: TLS_CLIENT_KEY_PASSWORD\n            valueFrom:\n              secretKeyRef:\n                name: tls-client-key-password\n                key: password\n          volumeMounts:\n          - name: \"sasl-jaas-config-file\"\n            mountPath: \"/var/run/secret/kafka-client-jaas\"\n          - name: \"tls-ca-chain-certificate\"\n            mountPath: \"/var/run/secret/kafka-ca-chain-certificate\"\n          - name: \"tls-client-cert-file\"\n            mountPath: \"/var/run/secret/kafka-client-certificate\"\n          - name: \"tls-client-key-file\"\n            mountPath: \"/var/run/secret/kafka-client-key\"\n          ports:\n          - name: metrics\n            containerPort: 9080\n          securityContext:\n            runAsNonRoot: true\n            runAsUser: 65534\n            allowPrivilegeEscalation: false\n            capabilities:\n              drop:\n                - ALL\n              add:\n                - NET_BIND_SERVICE\n            seccompProfile:\n              type: RuntimeDefault\n          livenessProbe:\n            httpGet:\n              path: /health\n              port: 9080\n            initialDelaySeconds: 5\n            periodSeconds: 3\n          readinessProbe:\n            httpGet:\n              path: /health\n              port: 9080\n            initialDelaySeconds: 5\n            periodSeconds: 10\n            timeoutSeconds: 5\n            successThreshold: 2\n            failureThreshold: 5\n        - name: myapp\n          image: myapp:latest\n          ports:\n          - containerPort: 8080\n            name: metrics\n          env:\n          - name: BOOTSTRAP_SERVERS\n            value: \"127.0.0.1:32400,127.0.0.1:32401,127.0.0.1:32402\"\n      volumes:\n      - name: sasl-jaas-config-file\n        secret:\n          secretName: sasl-jaas-config-file\n      - name: tls-ca-chain-certificate\n        secret:\n          secretName: tls-ca-chain-certificate\n      - name: tls-client-cert-file\n        secret:\n          secretName: tls-client-cert-file\n      - name: tls-client-key-file\n        secret:\n          secretName: tls-client-key-file\n```\n\n### Connect to Kafka running in Kubernetes example (kafka proxy runs in cluster)\n\n```yaml\n\n---\napiVersion: apps/v1\nkind: StatefulSet\nmetadata:\n   name: kafka-proxy\nspec:\n  selector:\n    matchLabels:\n      app: kafka-proxy\n  replicas: 1\n  serviceName: kafka-proxy\n  template:\n    metadata:\n      labels:\n        app: kafka-proxy\n    spec:\n      containers:\n        - name: kafka-proxy\n          image: grepplabs/kafka-proxy:latest\n          args:\n            - 'server'\n            - '--log-format=json'\n            - '--bootstrap-server-mapping=kafka-0:9093,127.0.0.1:32400'\n            - '--bootstrap-server-mapping=kafka-1:9093,127.0.0.1:32401'\n            - '--bootstrap-server-mapping=kafka-2:9093,127.0.0.1:32402'\n            - '--tls-enable'\n            - '--tls-ca-chain-cert-file=/var/run/secret/kafka-ca-chain-certificate/ca-chain.cert.pem'\n            - '--tls-client-cert-file=/var/run/secret/kafka-client-certificate/client.cert.pem'\n            - '--tls-client-key-file=/var/run/secret/kafka-client-key/client.key.pem'\n            - '--tls-client-key-password=$(TLS_CLIENT_KEY_PASSWORD)'\n            - '--sasl-enable'\n            - '--sasl-jaas-config-file=/var/run/secret/kafka-client-jaas/jaas.config'\n            - '--proxy-request-buffer-size=32768'\n            - '--proxy-response-buffer-size=32768'\n            - '--proxy-listener-read-buffer-size=32768'\n            - '--proxy-listener-write-buffer-size=131072'\n            - '--kafka-connection-read-buffer-size=131072'\n            - '--kafka-connection-write-buffer-size=32768'\n          env:\n          - name: TLS_CLIENT_KEY_PASSWORD\n            valueFrom:\n              secretKeyRef:\n                name: tls-client-key-password\n                key: password\n          volumeMounts:\n          - name: \"sasl-jaas-config-file\"\n            mountPath: \"/var/run/secret/kafka-client-jaas\"\n          - name: \"tls-ca-chain-certificate\"\n            mountPath: \"/var/run/secret/kafka-ca-chain-certificate\"\n          - name: \"tls-client-cert-file\"\n            mountPath: \"/var/run/secret/kafka-client-certificate\"\n          - name: \"tls-client-key-file\"\n            mountPath: \"/var/run/secret/kafka-client-key\"\n          securityContext:\n            runAsNonRoot: true\n            runAsUser: 65534\n            allowPrivilegeEscalation: false\n            capabilities:\n              drop:\n                - ALL\n              add:\n                - NET_BIND_SERVICE\n            seccompProfile:\n              type: RuntimeDefault\n          ports:\n          - name: metrics\n            containerPort: 9080\n          - name: kafka-0\n            containerPort: 32400\n          - name: kafka-1\n            containerPort: 32401\n          - name: kafka-2\n            containerPort: 32402\n          livenessProbe:\n            httpGet:\n              path: /health\n              port: 9080\n            initialDelaySeconds: 5\n            periodSeconds: 3\n          readinessProbe:\n            httpGet:\n              path: /health\n              port: 9080\n            initialDelaySeconds: 5\n            periodSeconds: 10\n            timeoutSeconds: 5\n            successThreshold: 2\n            failureThreshold: 5\n          resources:\n            requests:\n              memory: 128Mi\n              cpu: 1000m\n      restartPolicy: Always\n      volumes:\n      - name: sasl-jaas-config-file\n        secret:\n          secretName: sasl-jaas-config-file\n      - name: tls-ca-chain-certificate\n        secret:\n          secretName: tls-ca-chain-certificate\n      - name: tls-client-cert-file\n        secret:\n          secretName: tls-client-cert-file\n      - name: tls-client-key-file\n        secret:\n          secretName: tls-client-key-file\n```\n\n\n```bash\nkubectl port-forward kafka-proxy-0 32400:32400 32401:32401 32402:32402\n```\n\nUse localhost:32400, localhost:32401 and localhost:32402 as bootstrap servers\n\n\n### Connect to Kafka running in Kubernetes example (kafka proxy runs locally)\n####  one node Kafka cluster\nkafka.properties\n\n```\nbroker.id=0\nadvertised.listeners=PLAINTEXT://kafka-0.kafka-headless.kafka:9092\n...\n```\n\n```bash\nkubectl port-forward -n kafka kafka-0 9092:9092\n```\n\n```bash\nkafka-proxy server --bootstrap-server-mapping \"127.0.0.1:9092,0.0.0.0:19092\" --dial-address-mapping \"kafka-0.kafka-headless.kafka:9092,0.0.0.0:9092\"\n```\n\nUse localhost:19092 as bootstrap servers\n\n#### 3 nodes Kafka cluster\n\n[strimzi 0.13.0 CRD](https://strimzi.io/)\n\n```yaml\napiVersion: kafka.strimzi.io/v1beta1\nkind: Kafka\nmetadata:\n  name: test-cluster\n  namespace: kafka\nspec:\n  kafka:\n    version: 2.3.0\n    replicas: 3\n    listeners:\n      plain: {}\n      tls: {}\n    config:\n      offsets.topic.replication.factor: 3\n      transaction.state.log.replication.factor: 3\n      transaction.state.log.min.isr: 2\n      num.partitions: 60\n      default.replication.factor: 3\n    storage:\n      type: jbod\n      volumes:\n        - id: 0\n          type: persistent-claim\n          size: 20Gi\n          deleteClaim: true\n  zookeeper:\n    replicas: 3\n    storage:\n      type: persistent-claim\n      size: 5Gi\n      deleteClaim: true\n  entityOperator:\n    topicOperator: {}\n    userOperator: {}\n```\n\n```bash\nkubectl port-forward -n kafka test-cluster-kafka-0 9092:9092\nkubectl port-forward -n kafka test-cluster-kafka-1 9093:9092\nkubectl port-forward -n kafka test-cluster-kafka-2 9094:9092\n\nkafka-proxy server --log-level debug \\\n  --bootstrap-server-mapping \"127.0.0.1:9092,0.0.0.0:19092\" \\\n  --bootstrap-server-mapping \"127.0.0.1:9093,0.0.0.0:19093\" \\\n  --bootstrap-server-mapping \"127.0.0.1:9094,0.0.0.0:19094\" \\\n  --dial-address-mapping \"test-cluster-kafka-0.test-cluster-kafka-brokers.kafka.svc.cluster.local:9092,0.0.0.0:9092\" \\\n  --dial-address-mapping \"test-cluster-kafka-1.test-cluster-kafka-brokers.kafka.svc.cluster.local:9092,0.0.0.0:9093\" \\\n  --dial-address-mapping \"test-cluster-kafka-2.test-cluster-kafka-brokers.kafka.svc.cluster.local:9092,0.0.0.0:9094\"\n```\n\nUse localhost:19092 as bootstrap servers\n\n### Embedded third-party source code \n\n* [Cloud SQL Proxy](https://github.com/GoogleCloudPlatform/cloudsql-proxy)\n* [Sarama](https://github.com/Shopify/sarama)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrepplabs%2Fkafka-proxy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgrepplabs%2Fkafka-proxy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrepplabs%2Fkafka-proxy/lists"}