{"id":27248889,"url":"https://github.com/bakdata/streams-bootstrap","last_synced_at":"2025-04-10T23:48:26.044Z","repository":{"id":38368182,"uuid":"174344296","full_name":"bakdata/streams-bootstrap","owner":"bakdata","description":"Utility functions and base classes for Kafka Streams applications","archived":false,"fork":false,"pushed_at":"2025-04-10T19:03:28.000Z","size":16572,"stargazers_count":32,"open_issues_count":20,"forks_count":3,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-04-10T23:48:09.250Z","etag":null,"topics":["helm","kafka","kafka-streams","kubernetes","stream-processing"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bakdata.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-03-07T12:57:41.000Z","updated_at":"2025-04-04T12:15:17.000Z","dependencies_parsed_at":"2023-02-13T19:35:40.480Z","dependency_job_id":"ff3080e7-dd5e-4d7d-b09e-f9f398a37e35","html_url":"https://github.com/bakdata/streams-bootstrap","commit_stats":null,"previous_names":["bakdata/common-kafka-streams"],"tags_count":88,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fstreams-bootstrap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fstreams-bootstrap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fstreams-bootstrap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fstreams-bootstrap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bakdata","download_url":"https://codeload.github.com/bakdata/streams-bootstrap/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248317722,"owners_count":21083528,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["helm","kafka","kafka-streams","kubernetes","stream-processing"],"created_at":"2025-04-10T23:48:25.551Z","updated_at":"2025-04-10T23:48:26.029Z","avatar_url":"https://github.com/bakdata.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://github.com/bakdata/streams-bootstrap/actions/workflows/build-and-publish.yaml/badge.svg?event=push)](https://github.com/bakdata/streams-bootstrap/actions/workflows/build-and-publish.yaml/badge.svg?event=push)\n[![Sonarcloud status](https://sonarcloud.io/api/project_badges/measure?project=com.bakdata.kafka%3Astreams-bootstrap\u0026metric=alert_status)](https://sonarcloud.io/dashboard?id=com.bakdata.kafka%3Astreams-bootstrap)\n[![Code coverage](https://sonarcloud.io/api/project_badges/measure?project=com.bakdata.kafka%3Astreams-bootstrap\u0026metric=coverage)](https://sonarcloud.io/dashboard?id=com.bakdata.kafka%3Astreams-bootstrap)\n[![Maven](https://img.shields.io/maven-central/v/com.bakdata.kafka/streams-bootstrap-cli.svg)](https://search.maven.org/search?q=g:com.bakdata.kafka%20AND%20a:streams-bootstrap-cli\u0026core=gav)\n\n# streams-bootstrap\n\n`streams-bootstrap` provides base classes and utility functions for Kafka Streams applications.\n\nIt provides a common way to\n\n- configure Kafka Streams applications\n- deploy streaming applications on Kubernetes via Helm charts\n- reprocess data\n\nVisit our [blogpost](https://medium.com/bakdata/continuous-nlp-pipelines-with-python-java-and-apache-kafka-f6903e7e429d)\nand [demo](https://github.com/bakdata/common-kafka-streams-demo) for an overview and a demo application.  \nThe common configuration and deployments on Kubernetes are supported by\nthe [Streams Explorer](https://github.com/bakdata/streams-explorer), which makes it possible to explore and monitor data\npipelines in Apache Kafka.\n\n## Getting Started\n\nYou can add streams-bootstrap via Maven Central.\n\n#### Gradle\n\n```gradle\nimplementation group: 'com.bakdata.kafka', name: 'streams-bootstrap-cli', version: '3.1.0'\n```\n\nWith Kotlin DSL\n\n```gradle\nimplementation(group = \"com.bakdata.kafka\", name = \"streams-bootstrap-cli\", version = \"3.1.0\")\n```\n\n#### Maven\n\n```xml\n\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.bakdata.kafka\u003c/groupId\u003e\n  \u003cartifactId\u003estreams-bootstrap-cli\u003c/artifactId\u003e\n  \u003cversion\u003e3.1.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nFor other build tools or versions, refer to\nthe [latest version in MvnRepository](https://mvnrepository.com/artifact/com.bakdata.kafka/streams-bootstrap/latest).\n\n### Usage\n\n#### Kafka Streams\n\nCreate a subclass of `KafkaStreamsApplication` and implement the abstract methods `buildTopology()`\nand `getUniqueAppId()`. You can define the topology of your application in `buildTopology()`.\n\n```java\nimport com.bakdata.kafka.KStreamX;\nimport com.bakdata.kafka.KafkaStreamsApplication;\nimport com.bakdata.kafka.SerdeConfig;\nimport com.bakdata.kafka.StreamsApp;\nimport com.bakdata.kafka.StreamsTopicConfig;\nimport com.bakdata.kafka.StreamsBuilderX;\nimport java.util.Map;\nimport org.apache.kafka.common.serialization.Serdes.StringSerde;\n\npublic class MyStreamsApplication extends KafkaStreamsApplication\u003cStreamsApp\u003e {\n    public static void main(final String[] args) {\n      startApplication(new MyStreamsApplication(), args);\n    }\n\n    @Override\n    public StreamsApp createApp() {\n      return new StreamsApp() {\n        @Override\n        public void buildTopology(final StreamsBuilderX builder) {\n          final KStreamX\u003cString, String\u003e input = builder.streamInput();\n\n          // your topology\n\n          input.toOutputTopic();\n        }\n\n        @Override\n        public String getUniqueAppId(final StreamsTopicConfig topics) {\n          return \"streams-bootstrap-app-\" + topics.getOutputTopic();\n        }\n\n        @Override\n        public SerdeConfig defaultSerializationConfig() {\n          return new SerdeConfig(StringSerde.class, StringSerde.class);\n        }\n\n        // Optionally you can define custom Kafka properties\n        @Override\n        public Map\u003cString, Object\u003e createKafkaProperties() {\n          return Map.of(\n                  // your config\n          );\n        }\n      };\n    }\n}\n```\n\nThe following configuration options are available:\n\n- `--bootstrap-servers`, `--bootstrap-server`: List of Kafka bootstrap servers (comma-separated) (**required**)\n\n- `--schema-registry-url`: The URL of the Schema Registry\n\n- `--kafka-config`: Kafka Streams configuration (`\u003cString=String\u003e[,\u003cString=String\u003e...]`)\n\n- `--input-topics`: List of input topics (comma-separated)\n\n- `--input-pattern`: Pattern of input topics\n\n- `--output-topic`: The output topic\n\n- `--error-topic`: A topic to write errors to\n\n- `--labeled-input-topics`: Additional labeled input topics if you need to specify multiple topics with different\n  message types (`\u003cString=String\u003e[,\u003cString=String\u003e...]`)\n\n- `--labeled-input-patterns`: Additional labeled input patterns if you need to specify multiple topics with different\n  message types (`\u003cString=String\u003e[,\u003cString=String\u003e...]`)\n\n- `--labeled-output-topics`: Additional labeled output topics if you need to specify multiple topics with different\n  message types (`String=String\u003e[,\u003cString=String\u003e...]`)\n\n- `--application-id`: Unique application ID to use for Kafka Streams. Can also be provided by\n  implementing `StreamsApp#getUniqueAppId()`\n\n- `--volatile-group-instance-id`: Whether the group instance id is volatile, i.e., it will change on a Streams shutdown.\n\nAdditionally, the following commands are available:\n\n- `clean`: Reset the Kafka Streams application. Additionally, delete the consumer group and all output and intermediate\n  topics associated with the Kafka Streams application.\n\n- `reset`: Clear all state stores, consumer group offsets, and internal topics associated with the Kafka Streams\n  application.\n\n#### Kafka producer\n\nCreate a subclass of `KafkaProducerApplication`.\n\n```java\nimport com.bakdata.kafka.KafkaProducerApplication;\nimport com.bakdata.kafka.ProducerApp;\nimport com.bakdata.kafka.ProducerBuilder;\nimport com.bakdata.kafka.ProducerRunnable;\nimport com.bakdata.kafka.SerializerConfig;\nimport java.util.Map;\nimport org.apache.kafka.clients.producer.Producer;\nimport org.apache.kafka.common.serialization.StringSerializer;\n\npublic class MyProducerApplication extends KafkaProducerApplication\u003cProducerApp\u003e {\n    public static void main(final String[] args) {\n      startApplication(new MyProducerApplication(), args);\n    }\n\n    @Override\n    public ProducerApp createApp() {\n      return new ProducerApp() {\n        @Override\n        public ProducerRunnable buildRunnable(final ProducerBuilder builder) {\n          return () -\u003e {\n            try (final Producer\u003cObject, Object\u003e producer = builder.createProducer()) {\n              // your producer\n            }\n          };\n        }\n\n        @Override\n        public SerializerConfig defaultSerializationConfig() {\n          return new SerializerConfig(StringSerializer.class, StringSerializer.class);\n        }\n\n        // Optionally you can define custom Kafka properties\n        @Override\n        public Map\u003cString, Object\u003e createKafkaProperties() {\n          return Map.of(\n                  // your config\n          );\n        }\n      };\n    }\n}\n```\n\nThe following configuration options are available:\n\n- `--bootstrap-servers`, `--bootstrap-server`: List of Kafka bootstrap servers (comma-separated) (**required**)\n\n- `--schema-registry-url`: The URL of the Schema Registry\n\n- `--kafka-config`: Kafka producer configuration (`\u003cString=String\u003e[,\u003cString=String\u003e...]`)\n\n- `--output-topic`: The output topic\n\n- `--labeled-output-topics`: Additional labeled output topics (`String=String\u003e[,\u003cString=String\u003e...]`)\n\nAdditionally, the following commands are available:\n\n- `clean`: Delete all output topics associated with the Kafka Producer application.\n\n### Helm Charts\n\nFor the configuration and deployment to Kubernetes, you can use\nthe [Helm Charts](https://github.com/bakdata/streams-bootstrap/tree/master/charts).\n\nTo configure your streams app, you can use\nthe [`values.yaml`](https://github.com/bakdata/streams-bootstrap/blob/master/charts/streams-app/values.yaml) as a\nstarting point.\nWe also provide a chart\nto [clean](https://github.com/bakdata/streams-bootstrap/tree/master/charts/streams-app-cleanup-job) your streams app.\n\nTo configure your producer app, you can use\nthe [`values.yaml`](https://github.com/bakdata/streams-bootstrap/blob/master/charts/producer-app/values.yaml) as a\nstarting point.\nWe also provide a chart\nto [clean](https://github.com/bakdata/streams-bootstrap/tree/master/charts/producer-app-cleanup-job) your producer app.\n\n## Development\n\nIf you want to contribute to this project, you can simply clone the repository and build it via Gradle.\nAll dependencies should be included in the Gradle files, there are no external prerequisites.\n\n```bash\n\u003e git clone git@github.com:bakdata/streams-bootstrap.git\n\u003e cd streams-bootstrap \u0026\u0026 ./gradlew build\n```\n\nPlease note, that we have [code styles](https://github.com/bakdata/bakdata-code-styles) for Java.\nThey are basically the Google style guide, with some small modifications.\n\n## Contributing\n\nWe are happy if you want to contribute to this project.\nIf you find any bugs or have suggestions for improvements, please open an issue.\nWe are also happy to accept your PRs.\nJust open an issue beforehand and let us know what you want to do and why.\n\n## License\n\nThis project is licensed under the MIT license.\nHave a look at the [LICENSE](https://github.com/bakdata/streams-bootstrap/blob/master/LICENSE) for more details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbakdata%2Fstreams-bootstrap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbakdata%2Fstreams-bootstrap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbakdata%2Fstreams-bootstrap/lists"}