{"id":16380327,"url":"https://github.com/aramperes/kafka-denormalization","last_synced_at":"2025-07-29T22:34:28.986Z","repository":{"id":78211622,"uuid":"527352206","full_name":"aramperes/kafka-denormalization","owner":"aramperes","description":"Denormalizing Kafka topics using Kafka Streams","archived":false,"fork":false,"pushed_at":"2022-08-27T00:53:06.000Z","size":129,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-21T14:49:41.495Z","etag":null,"topics":["analytics","denormalization","kafka","kafka-streams","spring-kafka","streaming"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aramperes.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-21T23:36:20.000Z","updated_at":"2022-08-22T02:51:56.000Z","dependencies_parsed_at":"2023-03-22T23:01:26.168Z","dependency_job_id":null,"html_url":"https://github.com/aramperes/kafka-denormalization","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/aramperes/kafka-denormalization","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aramperes%2Fkafka-denormalization","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aramperes%2Fkafka-denormalization/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aramperes%2Fkafka-denormalization/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aramperes%2Fkafka-denormalization/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aramperes","download_url":"https://codeload.github.com/aramperes/kafka-denormalization/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aramperes%2Fkafka-denormalization/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267772331,"owners_count":24142087,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-29T02:00:12.549Z","response_time":2574,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","denormalization","kafka","kafka-streams","spring-kafka","streaming"],"created_at":"2024-10-11T03:51:10.746Z","updated_at":"2025-07-29T22:34:28.938Z","avatar_url":"https://github.com/aramperes.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# kafka-denormalization\n\nThis is a sample project to denormalize two Kafka topics into one. In other words,\nit performs a many-to-one join between two topics based on a foreign key, and emits\nthe joined data to a third topic.\n\nThe basic use-case is for when your data is already being produced on some topics, and you need\nto combine them over time as updates come in (and updating existing records from either side).\n\n## Example\n\nThis repository contains an example using Hacker News comments and stories. The `services` directory\ncontains 2 microservices, one that polls for new stories and produces them on a topic, and the other for comments.\n\n`hn.comments` (left)\n```json\n{\"by\":\"zinekeller\",\"id\":32546427,\"parent\":32546388,\"text\":\"...\",\"time\":1661132891,\"type\":\"comment\",\"story\":32545513}\n```\n\n`hn.stories` (right)\n```json\n{\"by\":\"thesuperbigfrog\",\"descendants\":40,\"id\":32545513,\"score\":50,\"time\":1661124181,\"title\":\"The Google Pixel 6a highlights everything wrong with the U.S. phone market\",\"type\":\"story\",\"url\":\"https://www.xda-developers.com/google-pixel-6a-us-market-editorial/\"}\n```\n\nOur objective is to join these 2 topics into one. Each message will contain the comment object, as well as the inflated story object.\n\n`hn.comments-with-story`\n```json\n{\n    \"comment\": {\"by\":\"zinekeller\",\"id\":32546427,\"parent\":32546388,\"text\":\"...\",\"time\":1661132891,\"type\":\"comment\",\"story\":32545513},\n    \"story\": {\"by\":\"thesuperbigfrog\",\"descendants\":40,\"id\":32545513,\"score\":50,\"time\":1661124181,\"title\":\"The Google Pixel 6a highlights everything wrong with the U.S. phone market\",\"type\":\"story\",\"url\":\"https://www.xda-developers.com/google-pixel-6a-us-market-editorial/\"}\n}\n```\n\nUsing the DSL I made for this project, it can be represented like this:\n\n```java\n@Autowired\npublic void buildPipeline(StreamsBuilder builder) {\n    var indexStore = Stores.inMemoryKeyValueStore(\"index\");\n\n    StreamDenormalize.\u003cString, Comment, String, Story, String, JoinedCommentStoryEvent\u003ebuilder()\n        .keySchema(JoinKeySchemas.Blake2b(8, Serdes.String(), Serdes.String()))\n        .indexTopic(\"hn.index\")\n            .indexStore(indexStore)\n        .leftTopic(\"hn.comments\")\n            .leftSerde(Comment.serde)\n        .rightTopic(\"hn.stories\")\n            .rightSerde(Story.serde)\n        .joinOn(comment -\u003e comment.story().toString())\n            .joiner((comment, story) -\u003e new JoinedCommentStoryEvent(comment, story))\n            .keyMapper((k, joined) -\u003e joined.comment().id().toString())\n        .build()\n        .innerJoin(builder)\n            .to(\"hn.comments-with-story\", Produced.with(Serdes.String(), JoinedCommentStoryEvent.serde));\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faramperes%2Fkafka-denormalization","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faramperes%2Fkafka-denormalization","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faramperes%2Fkafka-denormalization/lists"}