{"id":18336571,"url":"https://github.com/kumuluz/kumuluzee-streaming","last_synced_at":"2025-04-06T04:35:41.782Z","repository":{"id":45979598,"uuid":"96937266","full_name":"kumuluz/kumuluzee-streaming","owner":"kumuluz","description":"KumuluzEE Event Streaming extension for developing event-based microservices using Apache Kafka.","archived":false,"fork":false,"pushed_at":"2021-11-23T08:53:03.000Z","size":142,"stargazers_count":2,"open_issues_count":1,"forks_count":2,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-03-21T17:25:07.000Z","etag":null,"topics":["apache-kafka","cloud-native","event-stream","java","javaee","kumuluzee","microservices"],"latest_commit_sha":null,"homepage":"https://ee.kumuluz.com/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kumuluz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-07-11T21:07:42.000Z","updated_at":"2023-03-10T13:15:35.000Z","dependencies_parsed_at":"2022-07-20T04:18:05.895Z","dependency_job_id":null,"html_url":"https://github.com/kumuluz/kumuluzee-streaming","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kumuluz%2Fkumuluzee-streaming","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kumuluz%2Fkumuluzee-streaming/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kumuluz%2Fkumuluzee-streaming/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kumuluz%2Fkumuluzee-streaming/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kumuluz","download_url":"https://codeload.github.com/kumuluz/kumuluzee-streaming/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247435041,"owners_count":20938530,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-kafka","cloud-native","event-stream","java","javaee","kumuluzee","microservices"],"created_at":"2024-11-05T20:08:17.633Z","updated_at":"2025-04-06T04:35:41.348Z","avatar_url":"https://github.com/kumuluz.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# KumuluzEE Event Streaming\n![KumuluzEE CI](https://github.com/kumuluz/kumuluzee-streaming/workflows/KumuluzEE%20CI/badge.svg)\n\n\u003e KumuluzEE Event Streaming project for developing event-based microservices using Apache Kafka.\n\nKumuluzEE Event Streaming project for the KumuluzEE microservice framework provides easy-to-use annotations for\ndeveloping microservices that produce or consume event streams. KumuluzEE Event Streaming has been designed to support\nmodularity with pluggable streaming platforms. Currently, [Apache Kafka](https://kafka.apache.org/) is supported. In\nthe future, other event streaming platforms will be supported too (contributions are welcome).\n\n## Usage\n\nYou can enable KumuluzEE Event Streaming with Kafka by adding the following dependency:\n\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.kumuluz.ee.streaming\u003c/groupId\u003e\n    \u003cartifactId\u003ekumuluzee-streaming-kafka\u003c/artifactId\u003e\n    \u003cversion\u003e${kumuluzee-streaming.version}\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nIf you would like to collect Kafka related logs through the KumuluzEE Logs, you have to include the `kumuluzee-logs`\nimplementation and slf4j-log4j adapter dependencies:\n\n```xml\n\u003cdependency\u003e\n    \u003cartifactId\u003ekumuluzee-logs-log4j2\u003c/artifactId\u003e\n    \u003cgroupId\u003ecom.kumuluz.ee.logs\u003c/groupId\u003e\n    \u003cversion\u003e${kumuluzee-logs.version}\u003c/version\u003e\n\u003c/dependency\u003e\n\n\u003cdependency\u003e\n    \u003cgroupId\u003eorg.apache.logging.log4j\u003c/groupId\u003e\n    \u003cartifactId\u003elog4j-slf4j-impl\u003c/artifactId\u003e\n    \u003cversion\u003e${log4j-slf4j-impl.version}\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nYou also need to include a Log4j2 configuration, which should be in a file named `log4j2.xml`, located in\n`src/main/resources`. For more information about KumuluzEE Logs visit the\n[KumuluzEE Logs Github page](https://github.com/kumuluz/kumuluzee-logs).\n\n#### Configuring Kafka Producers and Consumers\n\nKafka Consumers and Producers are configured with the common KumuluzEE configuration framework. Configuration properties\ncan be defined with the environment variables or with the configuration files. Alternatively, they can also be stored in\na configuration server, such as etcd or Consul (for which the KumuluzEE Config project is required). For more details\nsee the [KumuluzEE configuration wiki page](https://github.com/kumuluz/kumuluzee/wiki/Configuration) and\n[KumuluzEE Config](https://github.com/kumuluz/kumuluzee-config).\nThe default configuration prefix for consumers is `consumer`, for producers is `producer`, but you can assign your\ncustom configuration prefix. This way you can configure several different producers and/or consumers at the same time.\n\nThe example below shows a sample configuration for the Kafka producer and consumer using default prefix.\n\n```yaml\n# producer config\nkumuluzee:\n  streaming:\n    kafka:\n      producer:\n        bootstrap-servers: localhost:9092\n        acks: all\n        retries: 0\n        batch-size: 16384\n        linger-ms: 1\n        buffer-memory: 33554432\n        key-serializer: org.apache.kafka.common.serialization.StringSerializer\n        value-serializer: org.apache.kafka.common.serialization.StringSerializer\n        . . .\n\n# consumer config\nkumuluzee:\n  streaming:\n    kafka:\n      consumer:\n        bootstrap-servers: localhost:9092\n        group-id: group1\n        enable-auto-commit: true\n        auto-commit-interval-ms: 1000\n        auto-offset-reset: earliest\n        key-deserializer: org.apache.kafka.common.serialization.StringDeserializer\n        value-deserializer: org.apache.kafka.common.serialization.StringDeserializer\n        . . .\n```\n\nTo use custom prefix, the configuration could look like this:\n\n```yaml\n# custom producer config\nkumuluzee:\n  streaming:\n    kafka:\n      custom-producer:\n        bootstrap-servers: localhost:9092\n        acks: all\n        retries: 0\n        batch-size: 16384\n        linger-ms: 1\n        buffer-memory: 33554432\n        key-serializer: org.apache.kafka.common.serialization.StringSerializer\n        value-serializer: org.apache.kafka.common.serialization.StringSerializer\n        . . .\n\n# custom consumer config\nkumuluzee:\n  streaming:\n    kafka:\n      custom-consumer:\n        bootstrap-servers: localhost:9092\n        group-id: group1\n        enable-auto-commit: true\n        auto-commit-interval-ms: 1000\n        auto-offset-reset: earliest\n        key-deserializer: org.apache.kafka.common.serialization.StringDeserializer\n        value-deserializer: org.apache.kafka.common.serialization.StringDeserializer\n        . . .\n```\n\nYou can also configure the Kafka Consumer poll parameter timeout, which is the time, in milliseconds, spent waiting in\npoll if data is not available in the buffer. If 0, it returns immediately with any records that are available currently\nin the buffer. Otherwise it returns empty. Must not be negative. In the KumuluzEE configuration you define the timeout\nparameter like this:\n\n```yaml\nkumuluzee:\n  streaming:\n    kafka:\n      poll-timeout: 1000\n```\n\n### Stream Producer annotation\n\nFor injecting the Kafka `Producer`, the KumuluzEE Kafka provides a `@StreamProducer` annotation, which will inject the\nproducer reference. We have to use it in conjunction with the `@Inject` annotation, as shown on the example below.  \nThe example bellow shows an example `@StreamProducer` code excerpt:\n\n```java\n@Inject\n@StreamProducer\nprivate Producer\u003cString, String\u003e producer;\n``` \n\nThe annotation has two parameter, both of which are optional. The `config` parameter is used for assigning the custom\nproducer configuration prefix, used in the KumuluzEE configuration. If not specified, the default value is `producer`.\nThe next example shows how to specify a custom producer configuration prefix within the annotation:\n\n```java\n@Inject\n@StreamProducer(config = \"custom-producer\")\nprivate Producer\u003cString, String\u003e producer;\n```\n\nThe second parameter is `configOverrides` and is covered in the section _Overriding configuration_ below.\n\n### Stream Consumer annotation\n\nFor consuming Kafka messages, KumuluzEE Event Streaming with Kafka provides the `@StreamListener` annotation. It is\nused to annotate the method that will be invoked when a message is received. It works similarly as a classic JMS\nlistener or a MDB. Please pay attention to the fact that you can only use application scoped beans in the\n`@StreamListener` annotated method.\n\nThe annotation takes four parameters: \n\n- `topics` an array of topics names, if none is defined the name of the annotated method will be used as a topic name. \n- `config` is the configuration prefix name for the KumuluzEE configuration. The default value is `consumer`.\n- `batchListener` a boolean value, for enabling batch message consuming. The default value is `false`.\n- `configOverrides` covered in the section _overriding configuration_ below. The default value is empty array (`{}`).\n\nThe example shows a `@StreamListener` annotated _topicName_ method with default configuration prefix name:\n\n```java\n@StreamListener\npublic void topicName(ConsumerRecord\u003cString, String\u003e record) {\n\t// process the message record\n}\n``` \n\nIf you like to add custom configuration prefix name and specify topic names in the annotation, you can do it like this:\n\n```java\n@StreamListener(topics = {\"topic1\", \"topic2\"}, config = \"custom-consumer\")\npublic void onMessage(ConsumerRecord\u003cString, String\u003e record) {\n\t// process the message record\n}\n``` \n\nYou can also consume a batch of messages, with the `batchListener` parameter set to `true`. In this case the annotated\nmethod parameter must be a List of ConsumerRecords, like in the example below:\n\n```java\n@StreamListener(topics = {\"topic\"}, batchListener = true)\npublic void onMessage(List\u003cConsumerRecord\u003cString, String\u003e\u003e records) {\n\t// process the message records\n}\n``` \n\nThe `@StreamListener` annotation also allows manual message committing. First you have to set the property of \n`enable.auto.commit` in the consumer configuration to `false`. Then add another parameter `Acknowledgement` to the \nannotated method, which has two methods for committing the message offsets:\n\n- `acknowledge()` that commits the last consumed message for all the subscribed list of topics and partitions and\n- `acknowledge(java.util.Map\u003cTopicPartition,OffsetAndMetadata\u003e offsets)` that commits the specified offsets for the \n  specified list of topics and partitions\n\nExample of manual message committing:\n\n```java\n@StreamListener(topics = {\"topic\"})\npublic void onMessage(ConsumerRecord\u003cString, String\u003e record, Acknowledgement ack) {\n\t// process the message record\n\t\n\t// commit the message record\n\tack.acknowledge();\n}\n```\n\n### Stream processing\n\nKumuluzEE Event Streaming with Kafka supports stream processors. `@StreamProcessor` annotation is used for building a\nstream processor.\n\nExample of stream processor:\n\n```java\n@StreamProcessor(id = \"word-count\", autoStart = false)\npublic StreamsBuilder wordCountBuilder() {\n\n    StreamsBuilder builder = new StreamsBuilder();\n\n    // configure the builder\n\n    return builder;\n\n}\n```\n\n`@StreamProcessor` annotation has several parameters. The `config` parameter specifies the prefix used for configuration\nlookup, similar to the one used by `@StreamProducer` and `@StreamListener` annotations described above. The `autoStart`\nparameter allows automatic or manual initiation of the stream processor.\n\nIf the `autoStart` parameter is set to `false`, a `StreamsController` can be used to control the lifecycle of\nthe stream processor. `@StreamProcessorController` is used to obtain an instance of `StreamsController`.\n\nExample usage of `StreamsController`:\n\n```java\n@StreamProcessorController(id=\"word-count\")\nStreamsController wordCountStreams;\n\npublic void startStream(@Observes @Initialized(ApplicationScoped.class) Object init) {\n    wordCountStreams.start();\n}\n```\n\n### Overriding configuration\n\nThe annotations `@StreamProducer`, `@StreamListener` and `@StreamProcessor` support the parameter `configOverrides`,\nwhich enables user to override or supply additional configuration from the code. For example:\n\n```java\n@Inject\n@StreamProducer(configOverrides = {@ConfigurationOverride(key = \"bootstrap-servers\", value = \"localhost:1234\")})\nprivate Producer\u003cString, String\u003e overriddenProducer;\n```\n\n### JSON serializer/deserializer\n\nThe KumuluzEE Streaming Kafka library includes convenient JSON serializer/deserializer implementations. The\nimplementations are:\n\n- `com.kumuluz.ee.streaming.kafka.serdes.JsonSerializer`\n- `com.kumuluz.ee.streaming.kafka.serdes.JsonDeserializer`\n- `com.kumuluz.ee.streaming.kafka.serdes.JsonSerde`\n\nExample configuration for producer with JSON serializer and consumer with JSON deserializer:\n\n```yaml\nkumuluzee:\n  streaming:\n    kafka:\n      producer:\n        bootstrap-servers: localhost:9092\n        key-serializer: org.apache.kafka.common.serialization.UUIDSerializer\n        value-serializer: com.kumuluz.ee.streaming.kafka.serdes.JsonSerializer\n      consumer:\n        bootstrap-servers: localhost:9092\n        key-deserializer: org.apache.kafka.common.serialization.UUIDDeserializer\n        value-deserializer: com.kumuluz.ee.streaming.kafka.serdes.JsonDeserializer\n        value-deserializer-type: com.example.test.models.Order\n```\n\nNote that the class that JSON representation should be deserialized into must be provided with the\n`\u003ckey/value\u003e-deserializer-type` property.\n\nWhen using stream processors the JSON SerDe can be programmatically obtained with `KumuluzSerdes.JsonSerde()` method.\nFor example:\n\n```java\nSerde\u003cMyModel\u003e myModelSerde = KumuluzSerdes.JsonSerde(MyModel.class);\n```\n\n#### Providing ObjectMapper\n\nJSON serializers/deserializers can use a custom instance of `ObjectMapper` to perform the conversion. In order to supply\na custom instance implement the `KafkaObjectMapperProvider` interface and register the implementation in a service file.\nFor example:\n\n```java\npublic class KafkaMapperProvider implements KafkaObjectMapperProvider {\n\n    @Override\n    public ObjectMapper provideObjectMapper(Map\u003cString, ?\u003e configs, boolean isKey) {\n        ObjectMapper om = new ObjectMapper();\n        om.registerModule(new JavaTimeModule());\n        return om;\n    }\n}\n```\n\nDo not forget to register implementation in a service file named\n`com.kumuluz.ee.streaming.kafka.utils.KafkaObjectMapperProvider`.\n\n## Schema Registry Support\nYou can configure schema registry for Serialization and Deserialization simply by adding the relavant configuration properties to the consumer and producer:\n\n```yaml\nkumuluzee:\n  streaming:\n    kafka:\n      consumer-avro:\n        bootstrap-servers: localhost:29092\n        group-id: group1\n        enable-auto-commit: true\n        auto-offset-reset: latest\n        key-deserializer: org.apache.kafka.common.serialization.StringDeserializer\n        value-deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer\n        schema-registry-url: http://localhost:8081\n        specific-avro-reader: true\n      producer-avro:\n        bootstrap-servers: localhost:29092\n        key-serializer: org.apache.kafka.common.serialization.StringSerializer\n        value-serializer: io.confluent.kafka.serializers.KafkaAvroSerializer\n        schema-registry-url: http://localhost:8081\n        auto-register-schemas: false\n```\n\nFor full sample with Kafka and Schema Registry you should check out [kumuluzee-samples](https://github.com/kumuluz/kumuluzee-samples) repository, module `kumuluzee-streaming-kafka-registry`.\n\n__NOTE: Json Serializer and Deserializer provided by this extension do not support Schema Registry! Use the Confluent or other 3rd party provided SerDes.__\n\n### Disabling extension\n\nThe extension can be disabled by setting the `kumuluzee.streaming.kafka.enabled` configuration property to `false`. This\ndisables the consumer and stream processor initialization and makes injection of `Producer` and `StreamsController`\nalways return `null`.\n\n## Changelog\n\nRecent changes can be viewed on Github on the [Releases Page](https://github.com/kumuluz/kumuluzee-streaming/releases)\n\n## Contribute\n\nSee the [contributing docs](https://github.com/kumuluz/kumuluzee-streaming/blob/master/CONTRIBUTING.md)\n\nWhen submitting an issue, please follow the \n[guidelines](https://github.com/kumuluz/kumuluzee-streaming/blob/master/CONTRIBUTING.md#bugs).\n\nWhen submitting a bugfix, write a test that exposes the bug and fails before applying your fix. Submit the test \nalongside the fix.\n\nWhen submitting a new feature, add tests that cover the feature.\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkumuluz%2Fkumuluzee-streaming","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkumuluz%2Fkumuluzee-streaming","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkumuluz%2Fkumuluzee-streaming/lists"}