{"id":22389469,"url":"https://github.com/adrien-ben/kstream-connections-aggregation-example","last_synced_at":"2026-05-01T12:32:28.230Z","repository":{"id":91820169,"uuid":"195267470","full_name":"adrien-ben/kstream-connections-aggregation-example","owner":"adrien-ben","description":"This is a simple Kafka Stream project that aggregates connections to servers.","archived":false,"fork":false,"pushed_at":"2019-07-12T14:18:58.000Z","size":5499,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-10-05T19:59:23.751Z","etag":null,"topics":["java11","kafka","kafka-streams","kotlin","spring-boot"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/adrien-ben.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-07-04T15:43:08.000Z","updated_at":"2020-09-20T19:05:16.000Z","dependencies_parsed_at":null,"dependency_job_id":"5f9fbdd9-5d0d-4bd5-85e2-a387a4a97e95","html_url":"https://github.com/adrien-ben/kstream-connections-aggregation-example","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/adrien-ben/kstream-connections-aggregation-example","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adrien-ben%2Fkstream-connections-aggregation-example","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adrien-ben%2Fkstream-connections-aggregation-example/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adrien-ben%2Fkstream-connections-aggregation-example/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adrien-ben%2Fkstream-connections-aggregation-example/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/adrien-ben","download_url":"https://codeload.github.com/adrien-ben/kstream-connections-aggregation-example/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adrien-ben%2Fkstream-connections-aggregation-example/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32497812,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-30T13:12:12.517Z","status":"online","status_checked_at":"2026-05-01T02:00:05.856Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["java11","kafka","kafka-streams","kotlin","spring-boot"],"created_at":"2024-12-05T03:12:01.346Z","updated_at":"2026-05-01T12:32:28.199Z","avatar_url":"https://github.com/adrien-ben.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# KStream connection aggregation example\n\n[![Build Status](https://travis-ci.com/adrien-ben/kstream-sales-aggregation-example.svg?branch=master)](https://travis-ci.com/adrien-ben/kstream-sales-aggregation-example)\n\nThis is a simple Kafka Stream project that aggregates connections to servers.\n\nThis is written in java 11 and uses maven. You can find the equivalent in Kotlin with Gradle on [this][0] branch.\n\n## Input \n\nThe application consumes 2 topics:\n\n- connections: represents a connection to a server\n\nA connection to a server is done by providing either the server ip address.\n\n```json\n{\n  \"sourceIp\":\"147.154.12.54\",\n  \"serverIp\":\"127.0.0.1\"\n}\n```\n\nOr the server name.\n\n```json\n{\n  \"sourceIp\":\"147.154.12.54\",\n  \"serverName\":\"localhost\"\n}\n``` \n\nIf none is specified, the connection is considered invalid. The key the messages are ignored.\n\n- dns: contains the server name to ip address mapping\n\nThe key is the server name and the value the ip address.\n\n## Output\n\nThe application pushes aggregated connection by servers.\n\n```json\n{\n  \"ip\": \"127.0.0.1\",\n  \"sourceIps\": [\n    {\n      \"ip\": \"147.154.12.54\",\n      \"timestamp\": \"2019-07-04T16:06:49.0458772\"\n    },\n    {\n      \"ip\": \"147.154.12.54\",\n      \"timestamp\": \"2019-07-04T16:07:15.5910779\"\n    }\n  ]\n}\n``` \n\n## Implementation\n\nThe `dns` topic is consumed as a KTable.\n\nThe `connections` topic is consumed as a KStream and split in 3:\n\n- The invalid messages (no server id/name)\n\nThese will just be logged as errors.\n\n- The connections by ip\n\nThese will by re-keyed so the key of the messages will now be the server id and sent to the `connections_by_ip_rekey`.\n\n- The connections by address\n\nThese will be joined with the dns KTable so the server id will be resolved from its name.\nThen the messages will be treated just like the messages from the previous branch (re-key + push).\n\nThen all connection messages from `connections_by_ip_rekey` will be grouped by key and aggregated into a message containing \nthe server ip and all connection events.\n\n## Topology\n\n![Topology](topology.png)\n\n## Run the app \n\n```sh\n# Run app\n./mvnw spring-boot:run\n\n#Run tests (note the clean to ensure the state store is cleaned up)\n./mvnw test\n```\n\nYou will also need a running Kafka instance. If you have docker installed this project has a\ndocker-compose file that starts a zookeeper, a kafka broker and kafkahq which is a nice ui\nthat ease the interaction with the kafka broker and that you can access at localhost:8090.\n\n```sh\ndocker-compose up -d\n```\n\n[0]: https://github.com/adrien-ben/kstream-connections-aggregation-example/tree/kotlin_gradle\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadrien-ben%2Fkstream-connections-aggregation-example","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fadrien-ben%2Fkstream-connections-aggregation-example","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadrien-ben%2Fkstream-connections-aggregation-example/lists"}