{"id":29291116,"url":"https://github.com/networknt/kafka-sidecar","last_synced_at":"2026-05-06T01:06:32.819Z","repository":{"id":37987379,"uuid":"384810311","full_name":"networknt/kafka-sidecar","owner":"networknt","description":"Kafka sidecar of light-mesh to provide cross-cutting concerns for Kubernetes service to interact with Kafka","archived":false,"fork":false,"pushed_at":"2025-09-20T01:43:20.000Z","size":508,"stargazers_count":2,"open_issues_count":7,"forks_count":1,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-09-20T02:34:04.994Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/networknt.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE","maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2021-07-10T22:44:01.000Z","updated_at":"2025-09-20T01:43:24.000Z","dependencies_parsed_at":"2023-10-15T20:34:27.557Z","dependency_job_id":"e801cb63-dcf5-4b3f-88c0-ff8355e14b57","html_url":"https://github.com/networknt/kafka-sidecar","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/networknt/kafka-sidecar","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/networknt%2Fkafka-sidecar","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/networknt%2Fkafka-sidecar/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/networknt%2Fkafka-sidecar/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/networknt%2Fkafka-sidecar/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/networknt","download_url":"https://codeload.github.com/networknt/kafka-sidecar/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/networknt%2Fkafka-sidecar/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29028237,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-03T01:44:09.608Z","status":"online","status_checked_at":"2026-02-03T02:00:06.989Z","response_time":96,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-06T07:34:24.512Z","updated_at":"2026-02-03T02:14:52.978Z","avatar_url":"https://github.com/networknt.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"A fast and light-weight reverse proxy with embedded gateway to wrap third party APIs and bring them to the ecosystem of light-4j frameworks\n\n[Stack Overflow](https://stackoverflow.com/questions/tagged/light-4j) |\n[Google Group](https://groups.google.com/forum/#!forum/light-4j) |\n[Gitter Chat](https://gitter.im/networknt/light-proxy) |\n[Subreddit](https://www.reddit.com/r/lightapi/) |\n[Youtube Channel](https://www.youtube.com/channel/UCHCRMWJVXw8iB7zKxF55Byw) |\n[Documentation](https://doc.networknt.com/service/proxy/) |\n[Contribution Guide](https://doc.networknt.com/contribute/) |\n\n[![Build Status](https://travis-ci.org/networknt/light-proxy.svg?branch=master)](https://travis-ci.org/networknt/light-proxy)\n\n## Why Reverse Proxy\n\nAll the services developed on top of light-4j frameworks support [client side service discovery](http://microservices.io/patterns/client-side-discovery.html),\nload balance and cluster natively. So there is no need to put a reverse proxy instance in front of our\nservices like other API frameworks that support only [server side service discovery](http://microservices.io/patterns/server-side-discovery.html).\n\nAlso, light services embed a distributed gateway to address all the cross-cutting concerns in the\nrequest/response chain and work with the ecosystem that consists:\n\n* [light-oauth2](https://doc.networknt.com/service/oauth/) for security\n* [light-portal](https://github.com/networknt/light-portal) for API management and market place\n* [light-config-server](https://github.com/networknt/light-config-server) for centralized configuration management\n* [light-eventuate-4j](https://doc.networknt.com/style/light-eventuate-4j/) for eventual consistency based on event sourcing, CQRS and Kafka\n* [ELK](https://www.elastic.co/webinars/introduction-elk-stack) for centralized logging with traceabilityId and correlationId\n* [InfluxDB](https://github.com/influxdata/influxdb) and [Grafana](https://github.com/grafana/grafana) for metrics\n* [Consul](https://github.com/hashicorp/consul) or [Zookeeper](http://zookeeper.apache.org/) for service registry\n* [Kubernetes](https://kubernetes.io/) for container orchestration\n\nCurrently, we only support Java language; however, we are planning to support Nodejs and Go in the future\nif there are enough customer demands. For some of our customers, they have some existing RESTful APIs that\nbuilt on top of other Java frameworks or other languages. We've been asked frequently on how to interact\nwith these services to/from light services and how to enable security, metrics, logging, discovery,\nvalidation, sanitization etc. on the existing services.\n\nOur answer is to deploy a reverse proxy built on top of light-4j framework that wraps the existing service.\n\nThe reverse proxy has the following features:\n\n* High throughput, low latency and small footprint.\n* Integrate light-oauth2 to protect un-secured services\n* Built-in load balancer\n* Can be started with Docker or standalone\n* Support HTTP 2.0 protocol on both in/out connections\n* TLS termination\n* Support REST, GraphQL and RPC style of APIs\n* Centralized logging with ELK, traceabilityId and CorrelationId\n* Collect client and service metrics into InfluxDB and view the dashboard on Grafana\n* Service registry and discovery with Consul or Zookeeper\n* Manage configuration with light-config-server\n\n## Local environment test.\n\nkafka sidecar normally is used for deploying as a separate sidecar container for kubernetes cluster. But we still can verify it locally.\n\nClick [here](https://doc.networknt.com/tutorial/kafka-sidecar/local-dev/#reactive-consumer) for local test detil.\n\nIf the evnironment is windows OS, we can start confluent kafka docker-compose for testing:\n\nFrom command line, create a docker network first which will indicate kafka and sidecar running in same network.\n\n```\ncd kafka-sidecar\n\ndocker network create localnet\n\ndocker-compose up -d\n```\n\n---\n\nThe above command starts Confluent Platform with a separate container for each Confluent Platform component. Your output should resemble the following:\n\n```text\nCreating network \"cp-all-in-one_default\" with the default driver\nCreating zookeeper ... done\nCreating broker    ... done\nCreating schema-registry ... done\nCreating rest-proxy      ... done\nCreating connect         ... done\nCreating ksql-datagen    ... done\nCreating ksqldb-server   ... done\nCreating control-center  ... done\nCreating ksqldb-cli      ... done\n```\n\nTo verify that the services are up and running, run the following command:\n\n```text\ndocker-compose ps\n```\nYour output should resemble the following:\n\n```text\n     Name                    Command               State                Ports\n------------------------------------------------------------------------------------------\nbroker            /etc/confluent/docker/run        Up      0.0.0.0:29092-\u003e29092/tcp,\n                                                           0.0.0.0:9092-\u003e9092/tcp\nconnect           /etc/confluent/docker/run        Up      0.0.0.0:8083-\u003e8083/tcp,\n                                                           9092/tcp\ncontrol-center    /etc/confluent/docker/run        Up      0.0.0.0:9021-\u003e9021/tcp\nksqldb-cli        /bin/sh                          Up\nksql-datagen      bash -c echo Waiting for K ...   Up\nksqldb-server     /etc/confluent/docker/run        Up      0.0.0.0:8088-\u003e8088/tcp\nrest-proxy        /etc/confluent/docker/run        Up      0.0.0.0:8082-\u003e8082/tcp\nschema-registry   /etc/confluent/docker/run        Up      0.0.0.0:8081-\u003e8081/tcp\nzookeeper         /etc/confluent/docker/run        Up      0.0.0.0:2181-\u003e2181/tcp,\n                                                           2888/tcp, 3888/tcp\n```\n\nNow we can Navigate to the Control Center web interface to verify:\n\nhttp://localhost:9021.\n\n---\n\n### Verify kafka sidecar  with local kafka:\n\n1. Navigate to the Control Center web interface (http://localhost:9021),in the navigation bar, click Topics to open the topics list, and then click Add a topic.\n\n![create-topic](doc/create-topic.png)\n\nCreate a topic \"test1\" and client the topic add register the topic schema:\n\nkey schema:\n\n```json\n{\n  \"$id\": \"http://example.com/myURI.schema.json\",\n  \"$schema\": \"http://json-schema.org/draft-07/schema#\",\n  \"additionalProperties\": false,\n  \"description\": \"Sample schema to help you get started.\",\n  \"title\": \"key_test1\",\n  \"type\": \"string\"\n}\n```\nvalue schema:\n\n```json\n{\n  \"$id\": \"http://example.com/myURI.schema.json\",\n  \"$schema\": \"http://json-schema.org/draft-07/schema#\",\n  \"additionalProperties\": false,\n  \"description\": \"Sample schema to help you get started.\",\n  \"properties\": {\n    \"count\": {\n      \"description\": \"The integer type is used for integral numbers.\",\n      \"type\": \"integer\"\n    }\n  },\n  \"title\": \"value_test1\",\n  \"type\": \"object\"\n}\n```\n\n2. Start kafka sidecar and backend api:\n\nThere are two options for starting kafka sidecar and backend api:\n\n- by standalone APIs\n\n- by docker-compose\n----\n#### By docker-compose\n\nStart kafka sidecar and backend api docker compose:\n\n```text\n\ndocker-compose -f docker-compose-demo.yml up\n```\n\nIt will start kafka sidecar and backend api by using config in the config/docker folder\n\nNotes: this is API docker container to kafka docker container, if you need change kafka lisener ports, please [refer](https://github.com/bitnami/bitnami-docker-kafka/blob/master/README.md#accessing-kafka-with-internal-and-external-clients) here\n\n\n#### By standalone APIs\n\nOr start kafka sidecar from IDE with VM options:\n\n-Dlight-4j-config-dir=config\\local\n\nStart Backend service (optional, we can still test connect without backend service)\n\nStart below the backend API from IDE:\n\nhttps://github.com/networknt/light-example-4j/tree/release/kafka/sidecar-backend\n\n----\n\n3. Produce messages to the test1 topic\n\nTo produce some messages to the test1 topic, we can issue a curl command.\n\n```json\ncurl --location --request POST 'http://localhost:8084/producers/test1' \\\n--header 'X-Traceability-Id: 111111' \\\n--header 'Content-Type: application/json' \\\n--data-raw '{\n    \"records\": [\n        {\n            \"key\": \"alice\",\n            \"value\": {\n                \"count\": 2\n            }\n        },\n        {\n            \"key\": \"john\",\n            \"value\": {\n                \"count\": 1\n            }\n        },\n        {\n            \"key\": \"alex\",\n            \"value\": {\n                \"count\": 2\n            }\n        }\n    ]\n}'\n```\n\n4. Verify the message in kafka test1 topic\n\n![create-topic](doc/test1-topic.png)\n\n5. Check the log, and we can see the result from backend API call:\n\n```text\n16:25:08.075 [XNIO-1 task-1]  mXFjPCfGSMiYfq_F3YrXSw INFO  c.s.e.e.m.k.h.ProducersTopicPostHandler:154 handleRequest - ProducerTopicPostHandler handleRequest start with topic test1\n16:25:09.851 [pool-3-thread-1]   INFO  c.s.e.e.m.k.ReactiveConsumerStartupHook$1:172 onCompletion - Send a batch to the backend API\n16:25:09.868 [pool-3-thread-1]   INFO  c.s.e.e.m.k.ReactiveConsumerStartupHook$1:186 onCompletion - Got successful response from the backend API\n```\n\n\n### Verify Reactive Consumer Dead Letter Queue (DLQ) feature:\n\nReactive Consumer can set DLQ for those messgage process failed. The default DLQ name is the topic name + \".dlq\". For our local test case, we can add a DQL: test1.dlq on kafka control center.\n\nAnd the enable DLQ on local_config/values.yml:\n\n```text\nkafka-consumer.deadLetterEnabled: true\n```\n\nIn the backend-api, the first event for each message will mark as process failed:\n\n```text\n RecordProcessedResult rpr = new RecordProcessedResult(record, false, sw.toString());\n```\n\nThe Reactive Consumer will send those message to DLQ if deadLetterEnabled.\n\n### Using the sidecar endpoint to producer error message to DLQ:\n\n- endpoint:\n\n  /consumers/deadLetter/active\n\n- method\n\n   POST\n\nSample request payload:\n\n```json\n[\n    {\n        \"record\": {\n            \"topic\": \"test6\",\n            \"key\": \"YWxleA==\",\n            \"value\": \"xyz\",\n            \"partition\": 0,\n            \"offset\": 0\n        },\n        \"processed\": false,\n        \"stacktrace\": \"error happened\",\n        \"correlationId\": \"cccc-1111111\",\n        \"traceabilityId\": \"tttt-1111111\",\n        \"key\": \"YWxleA==\"\n    },\n        {\n        \"record\": {\n            \"topic\": \"test7\",\n            \"key\": \"YWxleA==\",\n            \"value\": \"xyz\",\n            \"partition\": 0,\n            \"offset\": 0\n        },\n        \"processed\": false,\n        \"stacktrace\": \"error happened\",\n        \"correlationId\": \"cccc-1111111\",\n        \"traceabilityId\": \"tttt-1111111\",\n        \"key\": \"YWxleA==\"\n    }\n]\n```\n\n### Verify KsqlDB query:\n\n[ksqlDB query](doc/ksql.md)\n\n\n### Active Consumer Workflow:\n\nKafka sidecar provide the end-to-end workflow for actively consumer the records from kafka topic(s).\n\nWhen user try  following workflow to consumer records:\n\n- enable ActiveConsumerStartupHook on service.yml (or values.yml)\n\n   This will create kafka consumer manager for active consumer on kafka sidecar server startup\n\n\n-  create consumer group\n\n    endpoint: /consumers/{group}\n\n    method: POST\n\n\n-  subscribe the topics for consumer group created above\n\n   endpoint: /consumers/{group}/instances/{instance}/subscriptions\n\n   method: POST\n\n\n- consumer records from kafka topics subscribed above (default will start from 0 offset, user move the offset by calling different endpoints)\n\n  endpoint: /consumers/{group}/instances/{instance}/records\n\n  method: GET\n\n\n- submit offset after consumer\n\n  endpoint: /consumers/{group}/instances/{instance}/offsets\n\n  method: POST\n\n\n- add the records process detail to audit\n\n  endpoint: /consumers/active/audit\n\n  method: POST\n\n### To learn how to use this proxy, pleases refer to\n\n* [Getting Started](https://doc.networknt.com/getting-started/light-proxy/) to learn core concepts\n* [Tutorial](https://doc.networknt.com/tutorial/proxy/) with step by step guide for RESTful proxy\n* [Configuration](https://doc.networknt.com/service/proxy/configuration/) for different configurations based on your situations\n* [Artifact](https://doc.networknt.com/service/proxy/artifact/) to guide customer to choose the right artifact to deploy light-proxy.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnetworknt%2Fkafka-sidecar","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnetworknt%2Fkafka-sidecar","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnetworknt%2Fkafka-sidecar/lists"}