{"id":37026268,"url":"https://github.com/vspiliop/embedded-kafka-cluster","last_synced_at":"2026-01-14T03:03:19.001Z","repository":{"id":57735150,"uuid":"179857803","full_name":"vspiliop/embedded-kafka-cluster","owner":"vspiliop","description":"Start a fully configurable docker based Kafka cluster as part of your tests just by adding a Spring annotation.","archived":false,"fork":false,"pushed_at":"2021-07-24T13:01:47.000Z","size":88,"stargazers_count":6,"open_issues_count":1,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-08-18T03:26:56.675Z","etag":null,"topics":["cucumber-jvm","docker","junit","kafka","kafka-streams","kafka-streams-test","kafka-testing","spring-boot","spring-kafka","testcontainers"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vspiliop.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-04-06T16:25:09.000Z","updated_at":"2024-08-29T06:09:16.000Z","dependencies_parsed_at":"2022-09-26T22:11:28.949Z","dependency_job_id":null,"html_url":"https://github.com/vspiliop/embedded-kafka-cluster","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/vspiliop/embedded-kafka-cluster","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vspiliop%2Fembedded-kafka-cluster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vspiliop%2Fembedded-kafka-cluster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vspiliop%2Fembedded-kafka-cluster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vspiliop%2Fembedded-kafka-cluster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vspiliop","download_url":"https://codeload.github.com/vspiliop/embedded-kafka-cluster/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vspiliop%2Fembedded-kafka-cluster/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28408800,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T01:52:23.358Z","status":"online","status_checked_at":"2026-01-14T02:00:06.678Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cucumber-jvm","docker","junit","kafka","kafka-streams","kafka-streams-test","kafka-testing","spring-boot","spring-kafka","testcontainers"],"created_at":"2026-01-14T03:03:18.252Z","updated_at":"2026-01-14T03:03:18.993Z","avatar_url":"https://github.com/vspiliop.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![License](https://img.shields.io/:license-apache-brightgreen.svg)](http://www.apache.org/licenses/LICENSE-2.0.html)\r\n[![Maven Central](https://maven-badges.herokuapp.com/maven-central/io.github.vspiliop.testing/embedded-kafka-cluster/badge.svg)](https://maven-badges.herokuapp.com/maven-central/io.github.vspiliop.testing/embedded-kafka-cluster)\r\n\r\n# embedded-kafka-cluster\r\n\r\nStart a fully configurable docker based Kafka cluster as part of your tests by just adding @EmbeddedKafkaCluster to your test class.\r\n\r\n## Build/ Install\r\n\r\nRequires a running docker service.\r\n\r\n### Build and run tests\r\n\r\n```\r\nmvn clean package\r\n```\r\n\r\n### Run tests and install to local maven repo \r\n\r\n```\r\nmvn clean install\r\n```\r\n\r\n### Release to Sonatype Nexus Repository Manager and Maven Central\r\n\r\n```\r\nmvn release:clean release:prepare\r\n```\r\n\r\nfollowed by\r\n\r\n```\r\nmvn release:perform\r\n```\r\n\r\n## @EmbeddedKafkaCluster\r\n\r\n### Features\r\n\r\n * Start a docker based Kafka cluster that includes one or more Kafka brokers, Zookeepers and Confluent Schema Registries.\r\n * Parameterize the cluster (see Parameters section).\r\n * The cluster starts before the test spring context.\r\n * Reconfigure your spring test context to point to the embedded Kafka cluster (e.g point to the proper IPs and ports).\r\n * Run more than one unit test or Cucumber feature scenario in parallel and reduce the total execution time.\r\n\r\n### Parameters\r\n\r\n| Parameter | Description | Example | Default Value |\r\n| --- | --- | --- | --- |\r\n| topics | List of topics to be created | topics = {\"topic1.t\", \"topic2.t\"} or topics = {\"${my.spring.property}\"} | |\r\n| brokersCount or value | # of brokers of the cluster | brokersCount = 3 | 1 |\r\n| zookeepersCount | # of ZKs of the cluster | zookeepersCount = 3 | 1 |\r\n| schemaRegistriesCount | # of Confluent Schema Registries of the cluster | schemaRegistriesCount = 3 | 1 |\r\n| kafkaServersProperty | The spring test context property to reconfigure, so that it points to the embedded Kafka broker  | kafkaServersProperty = \"my.broker.url\" | config.kafka.bootstrap |\r\n| schemaRegistryServersProperty | The spring test context property to reconfigure, so that it points to the embedded Schema Registry | schemaRegistryServersProperty = \"my.registry.url\" | config.kafka.registry |\r\n| platformVersion | Confluent platform version | 4.1.2 | 4.1.2 |\r\n| minInSyncReplicas | Minimum in sync replicas for all topics, apart from transaction internal ones. | 1 | 1 |\r\n| minTransactionInSynceReplicas |  Minimum in sync replicas for transaction internal topics.  | 1 | 1 |\r\n| defaultReplicationFactor | Default replication factor for all topics, apart from internal ones. | 1 | 1 |\r\n| transactionReplicationFactor | Default replication factor for transaction topic. | 1 | 1 |\r\n| offsetsReplicationFactor | Default replication factor for offsets internal topic. | 1 | 1 |\r\n| schemaRegistryReplicationFactor | Replication factor for _schemas topic used by Confluent Schema Registry. | 1 | 1 |\r\n\r\n### Usage Examples\r\n\r\n\u003ca href=\"https://www.testcontainers.org/\"\u003eTestcontainers\u003c/a\u003e based.\r\n\r\nThe typical usage of this annotation is like (see also tests): \r\n\r\n```\r\n @RunWith(SpringRunner.class)\r\n @EmbeddedKafkaCluster(topics = {\"test.t\"})\r\n public class MyKafkaTests {\r\n    \r\n    // optionally autowire if needed\r\n    @Autowired\r\n    private EmbeddedMultiNodeKafkaCluster embeddedMultiNodeKafkaCluster;\r\n\r\n }\r\n```\r\n\r\nControl the number of services that form the cluster (see also tests):\r\n\r\n```\r\n@EmbeddedKafkaCluster(topics = {\"test.t\"}, brokersCount = 1, zookeepersCount = 1, schemaRegistriesCount = 1)\r\n```\r\n\r\nAlways use \"topics\" parameter to pre-create all topics that you use, as it speeds up the consumption of events!\r\n\r\n## Running multiples tests in parallel\r\n\r\n\u003ca href=\"https://www.testcontainers.org/\"\u003eTestcontainers\u003c/a\u003e framework is used to spin up docker containers to form a proper Kafka cluster (broker, zookeeper, schema registry). Each service has its own container, while the test (e.g. junit or cucumber feature) runs locally (not in a container).\r\n\r\n```\r\n46141b0a712b        confluentinc/cp-kafka:4.1.2             \"/etc/confluent/dock…\"   32 seconds ago      Up 28 seconds       0.0.0.0:33158-\u003e9092/tcp, 0.0.0.0:33157-\u003e9093/tcp                            confident_williams\r\n24685e0dd16a        alpine/socat:latest                     \"/bin/sh -c 'socat T…\"   41 seconds ago      Up 35 seconds       0.0.0.0:33156-\u003e2181/tcp, 0.0.0.0:33155-\u003e9093/tcp                            testcontainers-socat-Vsk8MbBx\r\n2e975fe6b3fc        confluentinc/cp-schema-registry:4.1.2   \"/etc/confluent/dock…\"   41 seconds ago      Up 36 seconds       0.0.0.0:33151-\u003e8081/tcp                                                     fervent_neumann\r\n6ead242bca82        confluentinc/cp-zookeeper:4.1.2         \"/etc/confluent/dock…\"   41 seconds ago      Up 36 seconds       0.0.0.0:33154-\u003e2181/tcp, 0.0.0.0:33153-\u003e2888/tcp, 0.0.0.0:33152-\u003e3888/tcp   friendly_heyrovsky\r\n42bc258afbd2        quay.io/testcontainers/ryuk:0.2.2       \"/app\"                   46 seconds ago      Up 43 seconds       0.0.0.0:33150-\u003e8080/tcp        \r\n```\r\n\r\nDocker deals with all port allocations and there is no chance of port conflicts.\r\n\r\nTests (junit ones or cucumber features) may run in parallel. Select proper maven profile to control the level of parallelism (rule of thump is the # of cores). If you use too many jvms your laptop will be unresponsive and random tests will fail.\r\n\r\n```\r\nmvn package -Pparallel_2_jvms # default profile if none is set\r\n\r\nmvn package -Pparallel_3_jvms\r\n\r\nmvn package -Pparallel_4_jvms\r\n\r\nmvn package -Pserial\r\n```\r\n  \r\n#### alpine/socat:latest\r\n\r\nA \u003ca href=\"https://www.testcontainers.org/\"\u003etestcontainer\u003c/a\u003e specific instance so that the kafka clients (of the unit test that runs outside of the docker network) may access the Kafka container.\r\n\r\n#### quay.io/testcontainers/ryuk:0.2.2 \r\n\r\nA \u003ca href=\"https://www.testcontainers.org/\"\u003etestcontainer\u003c/a\u003e specific instance used by the framework to start and stop containers.\r\n\r\n## Add as a test dependency to your project\r\n\r\nFrom Maven Central as follows:\r\n\r\n```\r\n\u003cdependency\u003e\r\n  \u003cgroupId\u003eio.github.vspiliop.testing\u003c/groupId\u003e\r\n  \u003cartifactId\u003eembedded-kafka-cluster\u003c/artifactId\u003e\r\n  \u003cversion\u003e...\u003c/version\u003e\r\n  \u003cscope\u003etest\u003c/scope\u003e\r\n\u003c/dependency\u003e\r\n```\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvspiliop%2Fembedded-kafka-cluster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvspiliop%2Fembedded-kafka-cluster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvspiliop%2Fembedded-kafka-cluster/lists"}