{"id":34834568,"url":"https://github.com/cerchie/notes-for-gh-video","last_synced_at":"2026-05-21T20:09:12.743Z","repository":{"id":229674509,"uuid":"777342844","full_name":"Cerchie/notes-for-gh-video","owner":"Cerchie","description":"some notes on what I'll be typing for my coding video ","archived":false,"fork":false,"pushed_at":"2024-04-11T18:46:20.000Z","size":117,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-11T02:08:34.242Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Cerchie.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-03-25T17:10:50.000Z","updated_at":"2024-06-30T15:53:26.000Z","dependencies_parsed_at":"2024-03-26T20:24:48.279Z","dependency_job_id":"9c5f9c7e-c132-4e07-8c09-82989e3c8d83","html_url":"https://github.com/Cerchie/notes-for-gh-video","commit_stats":null,"previous_names":["cerchie/notes-for-gh-video"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Cerchie/notes-for-gh-video","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Cerchie%2Fnotes-for-gh-video","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Cerchie%2Fnotes-for-gh-video/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Cerchie%2Fnotes-for-gh-video/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Cerchie%2Fnotes-for-gh-video/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Cerchie","download_url":"https://codeload.github.com/Cerchie/notes-for-gh-video/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Cerchie%2Fnotes-for-gh-video/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28032293,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-25T02:00:05.988Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-12-25T16:00:05.549Z","updated_at":"2025-12-25T16:00:08.304Z","avatar_url":"https://github.com/Cerchie.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Getting started with Kafka Streams: The Processing Process\n\n\nNow, I'll pivot to building a Kafka Streams application that processes the 'github-pull_requests' events, creating an up-to-date ratio of open/closed pull requests.\n\nIf you want to follow along with this part, check out the GitHub repo in the links below. I’m going to focus on the domain logic given our timeframe, so if you really want to dive in, I suggest consulting the codebase. \n\n\n\n```java\nstatic class MyProcessorSupplier implements ProcessorSupplier\u003cString, GitHubPRInfo, String, GitHubPRStateCounter\u003e {\n}\n```\n\nNow I'll  write the ProcessorSupplier, using the class provided by KafkaStreams. I'll use the record shape supplied by GitHubPRInfo as well as the object provided by GitHubPrStateCounter in our models folder.\n\n---\n\n```java\nstatic class MyProcessorSupplier implements ProcessorSupplier\u003cString, GitHubPRInfo, String, GitHubPRStateCounter\u003e { \n\tstatic final String STORE_KEY = \"state-counter\";\n}\n```\n\nOur store key will be named 'state_counter'. \n\n---\n\n```java\nstatic class MyProcessorSupplier implements ProcessorSupplier\u003cString, GitHubPRInfo, String, GitHubPRStateCounter\u003e { \n\tstatic final String STORE_KEY = \"state-counter\";\n   @Override\n   public Processor\u003cString, GitHubPRInfo, String, GitHubPRStateCounter\u003e get() {\n\t\treturn new Processor\u003c\u003e() {\n\t}\n}\n```\n\nNow, let's create a new processor instance to return. \n\n```java\nstatic class MyProcessorSupplier implements ProcessorSupplier\u003cString, GitHubPRInfo, String, GitHubPRStateCounter\u003e { \n\tstatic final String STORE_KEY = \"state-counter\";\n   @Override\n   public Processor\u003cString, GitHubPRInfo, String, GitHubPRStateCounter\u003e get() {\n\t\treturn new Processor\u003c\u003e() {\n\n    private KeyValueStore\u003cString, GitHubPRStateCounter\u003e kvStore;\n                @Override\n                public void init(final ProcessorContext\u003cString, GitHubPRStateCounter\u003e context) {\n                    this.kvStore = context.getStateStore(STORE_NAME);\n\n                    context.schedule(Duration.ofSeconds(1), PunctuationType.STREAM_TIME, timestamp -\u003e {\n                        GitHubPRStateCounter entry = kvStore.get(STORE_KEY);\n                        System.out.printf(\"Store value %s%n\", entry);\n                        context.forward(new Record\u003c\u003e(\"pr\", entry, timestamp));\n                    });\n\t}\n}\n\n```\n\nThen, I’ll initialize it with our state store. The init method schedules a punctuation to fire every second, printing the ratio in the store. \n\n```java\nstatic class MyProcessorSupplier implements ProcessorSupplier\u003cString, GitHubPRInfo, String, GitHubPRStateCounter\u003e { \n\tstatic final String STORE_KEY = \"state-counter\";\n   @Override\n   public Processor\u003cString, GitHubPRInfo, String, GitHubPRStateCounter\u003e get() {\n\t\treturn new Processor\u003c\u003e() {\n\n    private KeyValueStore\u003cString, GitHubPRStateCounter\u003e kvStore;\n                @Override\n                public void init(final ProcessorContext\u003cString, GitHubPRStateCounter\u003e context) {\n                    this.kvStore = context.getStateStore(STORE_NAME);\n\n                    context.schedule(Duration.ofSeconds(1), PunctuationType.STREAM_TIME, timestamp -\u003e {\n                        GitHubPRStateCounter entry = kvStore.get(STORE_KEY);\n                        System.out.printf(\"Store value %s%n\", entry);\n                        context.forward(new Record\u003c\u003e(\"pr\", entry, timestamp));\n                    });\n\t}\n              @Override\n                public void process(final Record\u003cString, GitHubPRInfo\u003e record) {\n\n                }\n            };\n        }\n}\n```\n\nNext, I’ll write the process method to  take in the events, mark them as open or closed, increment the count, and stash them in the state store.\n\n```java\nstatic class MyProcessorSupplier implements ProcessorSupplier\u003cString, GitHubPRInfo, String, GitHubPRStateCounter\u003e { \n\tstatic final String STORE_KEY = \"state-counter\";\n   @Override\n   public Processor\u003cString, GitHubPRInfo, String, GitHubPRStateCounter\u003e get() {\n\t\treturn new Processor\u003c\u003e() {\n\n    private KeyValueStore\u003cString, GitHubPRStateCounter\u003e kvStore;\n                @Override\n                public void init(final ProcessorContext\u003cString, GitHubPRStateCounter\u003e context) {\n                    this.kvStore = context.getStateStore(STORE_NAME);\n\n                    context.schedule(Duration.ofSeconds(1), PunctuationType.STREAM_TIME, timestamp -\u003e {\n                        GitHubPRStateCounter entry = kvStore.get(STORE_KEY);\n                        System.out.printf(\"Store value %s%n\", entry);\n                        context.forward(new Record\u003c\u003e(\"pr\", entry, timestamp));\n                    });\n\t}\n              @Override\n                public void process(final Record\u003cString, GitHubPRInfo\u003e record) {\n                    GitHubPRInfo prInfo = record.value();\n\n                }\n            };\n        }\n}\n```\nWe’ll get the value of the record and start a chain of logic. \n\n```java\nSHOULD I ZOOM IN HERE I WONDER\n                    if(!prInfo.state().equals(NO_STATE)) {\n                        GitHubPRStateCounter stateCounter = kvStore.get(STORE_KEY);\n                        if (stateCounter == null) {\n                            stateCounter = new GitHubPRStateCounter();\n                        }\n                        if (prInfo.state().equalsIgnoreCase(\"open\")) {\n                            stateCounter.setOpen(stateCounter.getOpen() + 1);\n                        } else {\n                            stateCounter.setClosed(stateCounter.getClosed() + 1);\n                        }\n                        kvStore.put(STORE_KEY, stateCounter);\n                    }\n```\n\nIf there’s no state and the state counter is null, we instantiate a new state counter. \n\n```java\n                    if(!prInfo.state().equals(NO_STATE)) {\n                        GitHubPRStateCounter stateCounter = kvStore.get(STORE_KEY);\n                        if (stateCounter == null) {\n                            stateCounter = new GitHubPRStateCounter();\n                        }\n                        if (prInfo.state().equalsIgnoreCase(\"open\")) {\n                            stateCounter.setOpen(stateCounter.getOpen() + 1);\n                        } else {\n                            stateCounter.setClosed(stateCounter.getClosed() + 1);\n                        }\n                    }\n```\n\nIf the pr state is either open or closed, we increment the open or closed counts. \n\n```java\n                    if(!prInfo.state().equals(NO_STATE)) {\n                        GitHubPRStateCounter stateCounter = kvStore.get(STORE_KEY);\n                        if (stateCounter == null) {\n                            stateCounter = new GitHubPRStateCounter();\n                        }\n                        if (prInfo.state().equalsIgnoreCase(\"open\")) {\n                            stateCounter.setOpen(stateCounter.getOpen() + 1);\n                        } else {\n                            stateCounter.setClosed(stateCounter.getClosed() + 1);\n                        }\n                        kvStore.put(STORE_KEY, stateCounter);\n                    }\n```\n\nThen, we update the store.\n\n```java\n\n    static class MyProcessorSupplier implements ProcessorSupplier\u003cString, GitHubPRInfo, String, GitHubPRStateCounter\u003e {\n\n        static final String STORE_KEY = \"state-counter\";\n\n        @Override\n        public Processor\u003cString, GitHubPRInfo, String, GitHubPRStateCounter\u003e get() {\n            return new Processor\u003c\u003e() {\n                private KeyValueStore\u003cString, GitHubPRStateCounter\u003e kvStore;\n\n                @Override\n                public void init(final ProcessorContext\u003cString, GitHubPRStateCounter\u003e context) {\n                    this.kvStore = context.getStateStore(STORE_NAME);\n\n                    context.schedule(Duration.ofSeconds(1), PunctuationType.STREAM_TIME, timestamp -\u003e {\n                        GitHubPRStateCounter entry = kvStore.get(STORE_KEY);\n                        System.out.printf(\"Store value %s%n\", entry);\n                        context.forward(new Record\u003c\u003e(\"pr\", entry, timestamp));\n                    });\n                }\n\n                @Override\n                public void process(final Record\u003cString, GitHubPRInfo\u003e record) {\n                    GitHubPRInfo prInfo = record.value();\n                    if(!prInfo.state().equals(NO_STATE)) {\n                        GitHubPRStateCounter stateCounter = kvStore.get(STORE_KEY);\n                        if (stateCounter == null) {\n                            stateCounter = new GitHubPRStateCounter();\n                        }\n                        if (prInfo.state().equalsIgnoreCase(\"open\")) {\n                            stateCounter.setOpen(stateCounter.getOpen() + 1);\n                        } else {\n                            stateCounter.setClosed(stateCounter.getClosed() + 1);\n                        }\n                        kvStore.put(STORE_KEY, stateCounter);\n                    }\n                }\n            };\n        }\n\n        @Override\n        public Set\u003cStoreBuilder\u003c?\u003e\u003e stores() {\n            return Collections.singleton(Stores.keyValueStoreBuilder(Stores.inMemoryKeyValueStore(STORE_NAME), Serdes.String(), prStateCounterSerde));\n        }\n    }\n\n}\n```\n\nLastly, we use the processor API to create our state store. We return a singleton with the store values built from our store builder. \n\n\n## The GitHub PR Ratio class\n\n Next, I’m going to create a GitHubPRRatio class to hold this processing… process!. I’ll need some variables:\n\nA logger for verifying that the data’s coming in, \nVariables working towards creating my custom serde,\nAnd some strings – my storename, a no state variable, and some input and output topic names. \nThe main method\n\nLet’s wrap all this up in our main method. \n\nI’ll load the properties – and hold on, let’s make the method that loads them. This method points at the path to our properties file and returns the contents.\n\nLet’s add a couple things to the properties down here– consumer group id, set the offset to earliest, and a max cache bytes config. \n\nI’ll call the  GitHubPrRatio method, to get the ratio coming in. \n\nCreate a new Streams instance with our topology. What’s a topology? Hold on a sec, I’ll show you, but first, let’s tie this up.\n\nThen we’ll set up a countdown latch so it waits for other threads before it starts… \n\nThen, in our try block, we’ll actually start the stream.\n\nOur catch block will print the error and the stack trace. \n\n\n## The topology\nOk, so,  what’s a topology? It’s  a definition of the computational logic we need to process our data stream. What does it look like? Let’s make one. \n\nFirst, I’ll instantiate the builder, then I’ll create the stream. We specify the input topic in the Consumed.with method, as well as the deserializer we’ll use for the key and value.\n\n I’ll add a peek method so we can log what’s coming in, then chain on a valueMapper, which we’ll define in a second. \n\nNext, we’ll need to process the data. I’ll make that processor in a minute as well.\n\n I’ll chain on another peek method to log if our data is processed successfully, then we’ll send it to the output topic with the .to method.  \n\nNow, I want to make the data coming in a more convenient shape, and I’ll use  ValueMapper for that. It’ll take the data coming in from the connector and give it a more convenient shape. I’ll use the GitHubPRInfo model for that.  \n\nThe 4 pieces of information we’re taking in as JsonNodes are the PR’s state, when it was created, and when it was closed, as well as the PR number. We then transform those to strings and ints, making sure to provide a case if the value happens to be null. \n\n\n## Wind the frog! \nHere’s the exciting bit! Let’s run the command which will start up our Kafka Streams instance… and we can see the data coming in! \n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcerchie%2Fnotes-for-gh-video","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcerchie%2Fnotes-for-gh-video","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcerchie%2Fnotes-for-gh-video/lists"}