{"id":18602125,"url":"https://github.com/build-on-aws/prioritizing-event-processing-with-apache-kafka","last_synced_at":"2025-05-06T15:29:04.626Z","repository":{"id":173458639,"uuid":"650794560","full_name":"build-on-aws/prioritizing-event-processing-with-apache-kafka","owner":"build-on-aws","description":"Technical solution to implement event processing prioritization with Apache Kafke using the concept of buckets.","archived":false,"fork":false,"pushed_at":"2024-11-19T18:10:46.000Z","size":599,"stargazers_count":28,"open_issues_count":3,"forks_count":8,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-03-31T01:31:40.264Z","etag":null,"topics":["amazon-msk","amazon-msk-connect","apache-flink","apache-kafka","event-processing","kafka-connect","kafka-streams"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit-0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/build-on-aws.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-07T20:29:46.000Z","updated_at":"2025-03-15T05:53:08.000Z","dependencies_parsed_at":null,"dependency_job_id":"b3a55469-e44f-4533-a7f7-55fea5395688","html_url":"https://github.com/build-on-aws/prioritizing-event-processing-with-apache-kafka","commit_stats":null,"previous_names":["build-on-aws/prioritizing-event-processing-with-apache-kafka"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/build-on-aws%2Fprioritizing-event-processing-with-apache-kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/build-on-aws%2Fprioritizing-event-processing-with-apache-kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/build-on-aws%2Fprioritizing-event-processing-with-apache-kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/build-on-aws%2Fprioritizing-event-processing-with-apache-kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/build-on-aws","download_url":"https://codeload.github.com/build-on-aws/prioritizing-event-processing-with-apache-kafka/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252712628,"owners_count":21792347,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon-msk","amazon-msk-connect","apache-flink","apache-kafka","event-processing","kafka-connect","kafka-streams"],"created_at":"2024-11-07T02:10:24.536Z","updated_at":"2025-05-06T15:29:04.609Z","avatar_url":"https://github.com/build-on-aws.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Prioritizing Event Processing with Apache Kafka\n\nImplement event processing prioritization in [Apache Kafka](https://kafka.apache.org) is often a hard task because Kafka doesn't support broker-level reordering of messages like some messaging technologies do. This is not necessarily a limitation, since Kafka is a [distributed commit log](https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying). With this data structure, messages are immutable, and so their ordering is within partitions. But this doesn't change the fact the developers may need to implement event processing prioritization with Kafka, anyway.\n\nThis project addresses event processing prioritization via the bucket pattern. It groups partitions into simpler abstractions called buckets. Bigger buckets mean a higher priority, and smaller buckets mean less priority. The number of partitions associated with each bucket defines their size. The bucket pattern also addresses code simplicity by providing a way to do all of this without forcing developers to handle low-level code related to event partitioning and consumer assignment.\n\nLet's understand how this works with an example.\n\n![Partitioner Overview](images/partitioner-overview.png)\n\nHere we can see that the partitions were grouped into the buckets `Platinum` and `Gold`. The Platinum bucket has a higher priority and therefore was configured to have `70%` of the allocation, whereas the Gold bucket has lower priority and therefore was configured to have only `30%`. This means that for a topic that contains `6` partitions, `4` of them will be associated with the Platinum bucket and `2` will be associated with the Gold bucket. To implement the prioritization, there has to be a process that ensures that messages with higher priority will end up in one the partitions from the Platinum bucket and messages with lower priority will end up in one the partitions from the Gold bucket. Consumers need to subscribe to the topic knowing which buckets they need to be associated with. This means that developers can decide to execute more consumers for the Platinum bucket and fewer consumers for the Gold bucket to ensure that they process high priority messages faster.\n\nTo ensure that each message will end up in their respective bucket, use the `BucketPriorityPartitioner`. This partitioner uses data in the message key to decide which bucket to use and therefore which partition from the bucket the message should be written. This partitioner distributes the messages within the bucket using a round robin algorithm to maximize consumption parallelism. On the consumer side, use the `BucketPriorityAssignor` to ensure that the consumer will be assigned only to the partitions that represent the bucket they want to process.\n\n![Assignor Overview](images/assignor-overview.png)\n\nWith the bucket priority, you can implement event processing prioritization by having more consumers working on buckets with higher priorities, while buckets with less priority can have fewer consumers. Event processing prioritization can also be obtained by executing these consumers in an order that gives preference to processing high priority buckets before the less priority ones. While coordinating this execution may involve some extra coding from you (perhaps using some sort of scheduler) you don't have to implement low-level code to manage partition assignment and keep your consumers simple by leveraging the standard `subscribe()` and `poll()` methods.\n\nYou can read more about the bucket priority pattern in this blog post: https://www.buildon.aws/posts/prioritizing-event-processing-with-apache-kafka\n\n## Building the project\n\nThe first thing you need to do to start using this partitioner is building it. In order to do that, you need to install the following dependencies:\n\n- [Java 11+](https://openjdk.java.net/)\n- [Apache Maven](https://maven.apache.org/)\n\nAfter installing these dependencies, execute the following command:\n\n```bash\nmvn clean package\n```\n\n## Using the partitioner\n\nTo use the `BucketPriorityPartitioner` in your producer you need to register it in the configuration.\n\n```bash\nProperties configs = new Properties();\n\nconfigs.setProperty(ProducerConfig.PARTITIONER_CLASS_CONFIG,\n   BucketPriorityPartitioner.class.getName());\n\nKafkaProducer\u003cK, V\u003e producer = new KafkaProducer\u003c\u003e(configs);\n```\n\nTo work properly, you need to specify in the configuration which topic will have its partitions grouped into buckets. This is important because, in Kafka, topics are specified at a message level and not at a producer level. This means that the same producer can write messages on different topics, so the partitioner needs to know which topic will have their partitions grouped into buckets.\n\n```bash\nconfigs.setProperty(BucketPriorityConfig.TOPIC_CONFIG, \"orders\");\n```\n\nFinally, specify in the configuration which buckets will be configured and what is the partition allocation for each one of them. The partition allocation is specified in terms of percentage. Note that the usage of the symbol `%` is optional.\n\n```bash\nconfigs.setProperty(BucketPriorityConfig.BUCKETS_CONFIG, \"Platinum, Gold\");\nconfigs.setProperty(BucketPriorityConfig.ALLOCATION_CONFIG, \"70%, 30%\");\n```\n\nThe partitioner ensures that all partitions from the topic will be assigned to the buckets.\nIn case of the allocation result in some partitions being left behind because the distribution is not even, the remaining partitions will be assigned to the buckets using a round robin algorithm over the buckets sorted by allocation.\n\n### Messages and buckets\n\nIn order to specify which bucket should be used, your producer need to provide this information on the message key. The partitioner will inspect each key in the attempt to understand in which bucket the message should be written. For this reason, the key must be an instance of a [java.lang.String](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/String.html) and it needs to contain the bucket name either as one literal string or as the first part of a string separated by a delimiter. For example, to specify that the bucket is `Platinum` then following examples are valid:\n\n* Key = `\"Platinum\"`\n* Key = `\"Platinum-001\"`\n* Key = `\"Platinum-Group01-001\"`\n\nThe default delimiter is `-` but you can change to something else:\n\n```bash\nconfigs.setProperty(BucketPriorityConfig.DELIMITER_CONFIG, \"|\");\n```\n\n### Discarding messages\n\nDiscarding any message that can't be sent to any of the buckets is also possible:\n\n```bash\nconfigs.setProperty(BucketPriorityConfig.FALLBACK_PARTITIONER_CONFIG,\n   \"code.buildon.aws.streaming.kafka.DiscardPartitioner\");\n```\n\n## Using the assignor\n\nTo use the `BucketPriorityAssignor` in your consumer you need to register it in the configuration.\n\n```bash\nProperties configs = new Properties();\n\nconfigs.setProperty(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,\n   BucketPriorityAssignor.class.getName());\n\nKafkaConsumer\u003cK, V\u003e consumer = new KafkaConsumer\u003c\u003e(configs);\n```\n\nTo work properly, you need to specify in the configuration which topic will have its partitions grouped into buckets. This is important because, in Kafka, consumers can subscribe to multiple topics. This means that the same consumer can read messages from different topics, so the assignor needs to know which topic will have their partitions grouped into buckets.\n\n```bash\nconfigs.setProperty(BucketPriorityConfig.TOPIC_CONFIG, \"orders\");\n```\n\nYou also have to specify in the configuration which buckets will be configured and what is the partition allocation for each one of them.\nThe partition allocation is specified in terms of percentage. Note that the usage of the symbol `%` is optional. Ideally, the partition allocation configuration needs to be the same used in the producer.\n\n\n```bash\nconfigs.setProperty(BucketPriorityConfig.BUCKETS_CONFIG, \"Platinum, Gold\");\nconfigs.setProperty(BucketPriorityConfig.ALLOCATION_CONFIG, \"70%, 30%\");\n```\n\nThe assignor ensures that all partitions from the topic will be assigned to the buckets.\nIn case of the allocation result in some partitions being left behind because the distribution is not even, the remaining partitions will be assigned to the buckets using a round robin algorithm over the buckets sorted by allocation.\n\nFinally you need to specify in the configuration which bucket the consumer will be associated.\n\n```bash\nconfigs.setProperty(BucketPriorityConfig.BUCKET_CONFIG, \"Platinum\");\n```\n\n### What about the other topics?\n\nIn Kafka, a consumer can subscribe to multiple topics, allowing the same consumer to read messages from partitions belonging to different topics. Because of this, the assignor ensures that only the topic specified in the configuration will have its partitions assigned to the consumers using the bucket priority logic. The other topics will have their partitions assigned to consumers using a fallback assignor.\n\nHere is an example of configuring the fallback assignor to round-robin:\n\n```bash\nconfigs.setProperty(BucketPriorityConfig.FALLBACK_ASSIGNOR_CONFIG,\n   \"org.apache.kafka.clients.consumer.RoundRobinAssignor\");\n```\n\nIf you don't configure a fallback assignor explicitly, Kafka's default assignor will be used.\n\n## Security\n\nSee [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information.\n\n## License\n\nThis project is licensed under the MIT-0 License. See the [LICENSE](./LICENSE) file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbuild-on-aws%2Fprioritizing-event-processing-with-apache-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbuild-on-aws%2Fprioritizing-event-processing-with-apache-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbuild-on-aws%2Fprioritizing-event-processing-with-apache-kafka/lists"}