{"id":20772388,"url":"https://github.com/rcardin/kafkaesque","last_synced_at":"2025-06-20T04:08:00.194Z","repository":{"id":45205763,"uuid":"280867814","full_name":"rcardin/kafkaesque","owner":"rcardin","description":"A testing 🧰 library for Kafka-based applications","archived":false,"fork":false,"pushed_at":"2025-06-19T02:10:39.000Z","size":291,"stargazers_count":26,"open_issues_count":14,"forks_count":4,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-20T04:07:54.850Z","etag":null,"topics":["hacktoberfest","kafka","testcontainers","testing-tools"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rcardin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-07-19T13:08:41.000Z","updated_at":"2025-04-26T12:49:35.000Z","dependencies_parsed_at":"2023-12-06T06:26:52.274Z","dependency_job_id":"1e89b10a-eb1f-4274-9592-714caeb18865","html_url":"https://github.com/rcardin/kafkaesque","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/rcardin/kafkaesque","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rcardin%2Fkafkaesque","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rcardin%2Fkafkaesque/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rcardin%2Fkafkaesque/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rcardin%2Fkafkaesque/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rcardin","download_url":"https://codeload.github.com/rcardin/kafkaesque/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rcardin%2Fkafkaesque/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260878463,"owners_count":23075962,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hacktoberfest","kafka","testcontainers","testing-tools"],"created_at":"2024-11-17T12:20:51.624Z","updated_at":"2025-06-20T04:07:55.165Z","avatar_url":"https://github.com/rcardin.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Java](https://img.shields.io/badge/java-11-%23ED8B00.svg?logo=openjdk)\n![](https://github.com/rcardin/kafkaesque/workflows/Kafkaesque/badge.svg)\n![Maven Central](https://img.shields.io/maven-central/v/in.rcard/kafkaesque-core)\n[![GitHub Package Registry version](https://img.shields.io/github/v/release/rcardin/kafkaesque)]()\n\n# 🐛 Kafkaesque\n_Kafkaesque_ is a test library whose aim is to make the experience in testing Kafka application less painful. By now, the project is in its early stage, defining the API that we will implement in the near future.\n\nEvery help will be very useful :)\n\nThe library allows to test the following use cases:\n\n## Use Case 1: The Application Produces Some Messages on a Topic\n\nThe first use case tests the messages produced by an application, reading them from the topic. The code, producing the messages, is external to _Kafkaesque_. Through _Kafkaesque_, it is possible to assert some properties on the messages generated by the application.\n\n```java\nKafkaesque\n  .at(\"broker:port\")\n  .\u003cKey, Value\u003econsume()\n  .fromTopic(\"topic-name\")\n  .withDeserializers(keyDeserializer, valueDeserializer)\n  .waitingAtMost(10, SECONDS)\n  .waitingEmptyPolls(2, 50L, MILLISECONDS)\n  .expectingConsumed()\n  .havingRecordsSize(3) // \u003c-- from here we use a ConsumedResult\n  .havingHeaders(headers -\u003e {\n    // Assertions on headers\n  })\n  .havingKeys(keys -\u003e {\n    // Assertions on keys\n  })\n  .havingPayloads(payloads -\u003e {\n    // Asserions on payloads\n  })\n  .havingConsumerRecords(records -\u003e {\n    // Assertions on the full list of ConsumerRecord\u003cKey, Value\u003e\n  })\n  .assertingThatPayloads(contains(\"42\")) // Uses Hamcrest.Matchers on collections :)\n  .andCloseConsumer();\n```\n\n## Use Case 2: The Application Consumes Some Messages from a Topic\n\nThe second use case tests an application that reads messages from a topic. _Kafkaesque_ is responsible to produce such messages to trigger the execution of the application. It is also possible to assert conditions on the system state after the consumption of the messages.\n\n```java\nKafkaesque\n  .at(\"broker:port\")\n  .\u003cKey, Value\u003eproduce()\n  .toTopic(\"topic-name\")\n  .withDeserializers(keyDeserializer, valueDeserializer)\n  .messages( /* Some list of messages, eventually with headers */)\n  .waitingAtMostForEachAck(100, MILLISECONDS) // Waiting time for each ack from the broker\n  .waitingForTheConsumerAtMost(10, SECONDS) // Waiting time for the consumer to read one / all the messages\n  .andAfterAll()\n  .asserting(messages -\u003e {\n    // Assertions on the consumer process after the sending of all the messages\n  });\n```\n\nAn equivalent method pipeline is available to test assertions after the consumption of each message:\n\n```java\nKafkaesque\n  .at(\"broker:port\")\n  .\u003cKey, Value\u003eproduce()\n  .toTopic(\"topic-name\")\n  .withDeserializers(keyDeserializer, valueDeserializer)\n  .messages( /* Some list of messages, eventually with headers */)\n  .waitingAtMostForEachAck(100, MILLISECONDS) // Waiting time for each ack from the broker\n  .waitingForTheConsumerAtMost(10, SECONDS) // Waiting time for the consumer to read one / all the messages\n  .andAfterEach()\n  .asserting(message -\u003e {\n    // Assertions on the consumer process after the sending of each message\n  });\n```\n\n## Use Case 3: Synchronize on Produced or Consumed Messages and Test Them Outside Kafkaesque\n\nThe [`kafka-streams-test-utils`](https://kafka.apache.org/documentation/streams/developer-guide/testing.html) testing library offers to developers some useful and powerful abstractions. Indeed, the `TestInputTopic` and the `TestOutputTopic` let developers manage asynchronous communication with a broker as it is fully synchronous. In this case, the library does not start any broker, not even embedded.\n\nKafkaesque offers to developers the same abstractions, trying to achieve the same synchronous behavior, using the `yolo.Kfksq` class.\n\n```java\nvar kfksq = Kfksq.at(\"broker:port\");\nvar inputTopic = kfksq.createInputTopic(\"inputTopic\", keySerializer, valueSerializer);\ninputTopic.pipeInput(\"key\", \"value\");\n\nvar outputTopic = kfksq.createOutputTopic(\"outputTopic\", keyDeserializer, valueDeserializer);\nvar records = outputTopic.readRecordsToList();\n```\n\n## Modules\n\n### Core module\n\nThe _Kafkaesque_ library contains many submodules. The `kafkaesque-core` module contains the interfaces and agnostic concrete classes offering the above fluid API. Add the following dependency to your `pom.xml` file to use module:\n\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003ein.rcard\u003c/groupId\u003e\n  \u003cartifactId\u003ekafkaesque-core\u003c/artifactId\u003e\n  \u003cversion\u003e0.2.0\u003c/version\u003e\n  \u003cscope\u003etest\u003c/scope\u003e\n\u003c/dependency\u003e\n```\n\nIn detail, the `kafkaesque-core` module uses the [Awaitility](http://www.awaitility.org/) Java library to deal with the asynchronicity nature of each of the above use cases.\n\n## Configuration\n\n_Kafkaesque_ also supports internal producers and consumers configuration via an external configuration file. Kafkaesque can read multiple file formats. The available ones are the [HOCON](https://github.com/lightbend/config/blob/main/HOCON.md) file format, JSON format, and Java properties format.\n\nThe configurations must be prefixed with `kafkaesque.consumer` for consumers. The available configuration are:\n\n* `group-id`\n* `auto-offset-reset`\n* `enable-auto-commit`\n* `auto-commit-interval`\n* `client-id`\n* `fetch-max-wait`\n* `fetch-min-size`\n* `isolation-level`\n* `max-poll-records`\n\nThe configurations must be prefixed with `kafkaesque.producer` for producers, instead. The available configuration are:\n\n* `client-id`\n* `retries`\n* `acks`\n* `batch-size`\n* `buffer-memory`\n* `compression-type`\n\nYou can pass the path to the file using the `withConfiguration` method available both for consumers and producers. Here is an example:\n\n```java\nKafkaesque\n  .at(\"broker:port\")\n  .\u003cKey, Value\u003eproduce()\n  .toTopic(\"topic-name\")\n  .withDeserializers(keyDeserializer, valueDeserializer)\n  .withConfiguration(\"path-to-the-file.conf\")\n  .messages( /* Some list of messages */)\n  .waitingAtMostForEachAck(100, MILLISECONDS) // Waiting time for each ack from the broker\n  .waitingForTheConsumerAtMost(10, SECONDS) // Waiting time for the consumer to read one / all the messages\n  .andAfterAll()\n  .asserting(messages -\u003e {\n    // Assertions on the consumer process after the sending of all the messages\n  });\n```\n\nThe path is relative to the `/src/test/resources` folder.\n\nAn example of configuration file could be the following. The file contains the configurations for both producers and consumers:\n\n```hocon\nkafkaesque {\n  consumer {\n    group-id: \"kfksq-test-consumer\"\n    client-id: \"kfksq-client-id\"\n    auto-commit-interval: 5000\n    auto-offset-reset: \"earliest\"\n    enable-auto-commit: false\n    fetch-max-wait: 500\n    fetch-min-size: 1\n    heartbeat-interval: 3000\n    isolation-level: \"read_uncommitted\"\n    max-poll-records: 500\n  }\n\n  producer {\n    acks: \"all\"\n    batch-size: 16384\n    buffer-memory: 33554432\n    client-id: \"kfksq-client-id\"\n    compression-type: \"none\"\n    retries: 2147483647\n  }\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frcardin%2Fkafkaesque","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frcardin%2Fkafkaesque","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frcardin%2Fkafkaesque/lists"}