{"id":19168386,"url":"https://github.com/bloomberg/corokafka","last_synced_at":"2025-05-07T14:41:10.778Z","repository":{"id":35501646,"uuid":"214511973","full_name":"bloomberg/corokafka","owner":"bloomberg","description":"C++ Kafka coroutine library using Quantum dispatcher and wrapping CppKafka","archived":false,"fork":false,"pushed_at":"2022-10-04T21:08:18.000Z","size":58399,"stargazers_count":51,"open_issues_count":0,"forks_count":12,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-19T23:31:57.903Z","etag":null,"topics":["coroutine","cplusplus","cppkafka","kafka","quantum","rdkafka"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bloomberg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-10-11T19:13:37.000Z","updated_at":"2025-03-07T01:29:47.000Z","dependencies_parsed_at":"2023-01-15T22:18:12.370Z","dependency_job_id":null,"html_url":"https://github.com/bloomberg/corokafka","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bloomberg%2Fcorokafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bloomberg%2Fcorokafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bloomberg%2Fcorokafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bloomberg%2Fcorokafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bloomberg","download_url":"https://codeload.github.com/bloomberg/corokafka/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252897330,"owners_count":21821425,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["coroutine","cplusplus","cppkafka","kafka","quantum","rdkafka"],"created_at":"2024-11-09T09:42:29.592Z","updated_at":"2025-05-07T14:41:10.752Z","avatar_url":"https://github.com/bloomberg.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CoroKafka library [![Build Status](https://travis-ci.com/bloomberg/corokafka.svg?branch=master)](https://travis-ci.com/bloomberg/corokafka)\n\nCoroutine-based Kafka messaging library! **CoroKafka** is a scalable and simple to use C++ library built on top of [cppkafka](https://github.com/mfontanini/cppkafka) and [quantum](https://github.com/bloomberg/quantum), supporting any number of parallel producers and consumers.\nProducing and consuming of messages is simplified by allowing applications to use their own native message formats which are automatically serialized and de-serialized by the connector. Message processing spreads to optimal number of parallel coroutines and threads. Integrated support for **RdKafka** headers. Currently the library only supports static topics and admin API is not yet available. \n\nSupports C++14 and later.\n\n## Menu\n\n- [Rationale](#rationale)\n    - [Salient Features](#salient-features)\n- [Quick start](#quick-start)\n    - [Serializers and Deserializers](#serializers-and-de-serializers)\n    - [Producer Example](#producer-example)\n    - [Consumer Example](#consumer-example)\n- [Building](#building)\n    - [CMake options](#cmake-options)\n- [Installation](#installation)\n- [Linking and Using](#linking-and-using)\n- [Contributions](#contributions)\n- [License](#license)\n- [Code of Conduct](#code-of-conduct)\n- [Security Vulnerability Reporting](#security-vulnerability-reporting)\n\n## Rationale\n\n**RdKafka** is a popular high-performance C-based Kafka library, but using it requires a lot of extra code to be written which often leads to unwieldy bugs, misinterpreted API usage, and a lengthy development cycle. **CppKafka** was written with this in mind and provides a clean C++ raw-buffer wrapper on top of **RdKafka** with integrated object lifetime management as well as introducing various powerful utilities for producing and consuming, such as the `BufferedProducer` and the `BackoffCommitter`. Despite all this, **CppKafka** remains a thin library on top of **RdKafka** and the current library, **CoroKafka** extends its functionality into a rich-featured API.\nProducing and consuming messages in a multi-threaded environment can easily be enhanced by using _coroutines_, which makes this a great integration scenario for Bloomberg's **Quantum** library.\n\n**CoroKafka** offers a multitude of [configuration options](https://github.com/bloomberg/corokafka/blob/master/CONFIGURATION.md) besides those [offered](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md) natively by **RdKafka**. **CoroKafka** does not pretend to be the fastest or best performing Kafka implementation, its main goal being ease-of-use, convenience, as well as abstracting lots of boiler-plate code which all applications must write.\n\n### Salient Features\n\n* Application publishes and consumes using native types. Serialization and deserialization is handled internally by **CoroKafka** and provides strict compile-time type-safety on both ends.\n* Full support for any number  of **RdKafka** headers for all message types.\n* All **RdKafka** callbacks are supported as well as additional ones as provided by **CppKafka**.\n* Producer and consumer polling is handled internally, so applications need not worry about blocking or missing heartbeats.\n* Support for message retries, buffering, strict order guarantee delivery for all producers.\n* Support for commit retries, auto-commit management as well as batching and pre-fetching of messages for increased performance. \n* Support for balanced round-robin partition reading.\n* Serialization and de-serialization is highly parallelized on coroutines. Various configuration options allow tweaking this behavior and how messages are delivered to the end application.\n* Support for throttling consumers and producers as well as pausing and resuming streams.\n* Consumers can either use dynamic partition assignment or static via config files. \n* Processing received messages on the `quantum::Sequencer`, can easily create thousands of parallel streams of data yielding tremendous performance benefits.\n* Pre-processing of messages prior to de-serializing can allow for early poison-message detection.\n* **CoroKafka's** `OffsetManager` allows proper ordering of commits for parallel streams, preventing higher offsets from being committed before lower ones on the same partition.\n* Synchronous and asynchronous producers.\n\n## Quick Start\n\nThe following code snippet shows how to setup a basic consumer and producer. \n\n### Serializers and De-serializers\n\nIn **CoroKafka** applications must define serializers and de-serializer for each key, payload and header(s) they wish to send and receive. These specializations must be defined in either the global namespace or the `corokafka` namespace.\n\nThe _serializer_ **must** have the following signature: \n\n```c++\ntemplate \u003ctypename T\u003e\nstruct Serialize\n{\n    std::vector\u003cuint8_t\u003e operator()(const T\u0026);\n}\n```\nSimilarly the _de-serializer_ **must** have the following signature:\n\n```c++\ntemplate \u003ctypename T\u003e\nstruct Deserialize\n{\n    T operator()(const cppkafka::TopicPartition\u0026, const cppkafka::Buffer\u0026);\n}\n```\nThe first parameter indicates the partition on which this message arrived and can be used for logging, stats, anti-poison message prevention, etc.\n\n```c++\n//==========================================================================\n//                       Serializers/De-serializers\n//==========================================================================\n//------------------------ serializers.h -----------------------------------\n#include \u003ccorokafka/corokafka.h\u003e\n\n// Declare functions inside corokafka namespace...\nnamespace Bloomberg { namespace corokafka {\n    // This serializer is used for the key\n    template \u003c\u003e\n    struct Serialize\u003csize_t\u003e {\n        std::vector\u003cuint8_t\u003e operator()(const size_t\u0026 value) {\n            return {reinterpret_cast\u003cconst uint8_t*\u003e(\u0026value),\n                    reinterpret_cast\u003cconst uint8_t*\u003e(\u0026value) + sizeof(size_t)};\n        }\n    };\n    \n    // This serializer is used for the headers and the payload\n    template \u003c\u003e\n    struct Serialize\u003cstd::string\u003e {\n        std::vector\u003cuint8_t\u003e operator()(const std::string\u0026 value) {\n            return {value.begin(), value.end()};\n        }\n    };\n    \n    // This is the deserializer for the key\n    template \u003c\u003e\n    struct Deserialize\u003csize_t\u003e {\n        size_t operator()(const cppkafka::TopicPartition\u0026, \n                          const cppkafka::Buffer\u0026 buffer) {\n            return *static_cast\u003cconst size_t*\u003e((void*)buffer.get_data());\n        }\n    };\n    \n    // This is the deserializer for the header and payload\n    template \u003c\u003e\n    struct Deserialize\u003cstd::string\u003e {\n        std::string operator()(const cppkafka::TopicPartition\u0026, \n                               const cppkafka::Buffer\u0026 buffer) {\n            return {buffer.begin(), buffer.end()};\n        }\n    };\n}} //namespace Bloomberg::corokafka\n\n//==========================================================================\n//                                Topic\n//==========================================================================\n//------------------------------ mytopic.h ---------------------------------\n#include \u003cserializers.h\u003e\n\nusing MyTopic = corokafka::Topic\u003csize_t, std::string, coroakafka::Headers\u003cstd::string\u003e\u003e;\n\n// Create a topic which will be shared between producer and consumer.\nstatic MyTopic myTopic(\"my-topic\", corokafka::Header\u003cstd::string\u003e(\"Header1\");\n```\n\n### Producer example\n\nIn the following example we will be producing some messages with a key of type `size_t`, a payload of type\n`std::string` and a simple header called _Header1_. For full producer API see [here](https://github.com/bloomberg/corokafka/blob/master/corokafka/corokafka_producer_manager.h).\n```c++\n//==========================================================================\n//                          Producer setup\n//==========================================================================\n#include \u003cmytopic.h\u003e\n\n// Define a delivery report callback (optional - see documentation for all available callbacks)\nvoid deliveryReportCallback(const corokafka::ProducerMetadata\u0026 metadata,\n                            const corokafka::SentMessage\u0026 msg)\n{\n    std::cout \u003c\u003c \"Produced message for [\" \u003c\u003c metadata.getInternalName()\n              \u003c\u003c \":\" \u003c\u003c msg.getTopic() \u003c\u003c \"]\"\n              \u003c\u003c std::endl;\n    if (msg) {\n        if (msg.getError()) {\n            return;\n        }\n        //print message contents\n        std::cout \u003c\u003c \"Message key: \" \u003c\u003c *reinterpret_cast\u003cconst size_t*\u003e(msg.getKeyBuffer().get_data())\n                  \u003c\u003c \" Message content: [\" \u003c\u003c (std::string)msg.getPayloadBuffer() \u003c\u003c \"]\"\n                  \u003c\u003c \" Message partition: \" \u003c\u003c msg.getPartition()\n                  \u003c\u003c \" Message offset: \" \u003c\u003c msg.getOffset()\n                  \u003c\u003c std::endl;\n    }\n}\n\n// Define a log callback (optional - see documentation for all available callbacks)\nvoid logCallback(const corokafka::Metadata\u0026 metadata,\n                 cppkafka::LogLevel level,\n                 const std::string\u0026 facility,\n                 const std::string\u0026 message)\n{\n    std::cout \u003c\u003c \"Log from: \" \u003c\u003c metadata.getInternalName()\n              \u003c\u003c \" facility: \" \u003c\u003c facility\n              \u003c\u003c \" text: \" \u003c\u003c message \u003c\u003c std::endl;\n}\n\n// Create a topic configuration (optional)\nstd::initializer_list\u003ccppkafka::ConfigurationOption \u003e topicOptions = {\n    { \"produce.offset.report\",  true },\n    { \"request.required.acks\", -1 }\n};\n\n// Create a producer configuration (only 'metadata.broker.list' setting is mandatory)\nstd::initializer_list\u003ccppkafka::ConfigurationOption \u003e producerOptions = {\n    { \"metadata.broker.list\", \"broker_url:port\" },\n    { \"api.version.request\", true },\n    { \"internal.producer.retries\", 5 }\n};\n\n// Associate the topic and producer configuration with a topic name.\n// Note: any number of producer configs/topics can be created.\ncorokafka::ProducerConfiguration config(myTopic, producerOptions, topicOptions);\n\n// Add the callbacks\nconfig.setDeliveryReportCallback(deliveryReportCallback);\nconfig.setLogCallback(logCallback);\n\n// Create the connector config (optional)\ncorokafka::ConnectorConfiguration connectorConfiguration(\n    {{ \"internal.connector.poll.interval.ms\", 100 }}\n);\n\n// Combine all the configs together\ncorokafka::ConfigurationBuilder builder;\nbuilder(config).(connectorConfiguration);\n\n// Create the connector\ncorokafka::Connector connector(std::move(builder));\n\n// Produce 10 messages\ncorokafka::ProducerManager\u0026 producer = connector.producer(); //get handle on producer\n\n// Produce 10 messages. 'i' represents the 'key'\nfor (size_t i = 0; i \u003c 10; ++i) { \n    //produce a message synchronously (async production also available)\n    producer.send(myTopic, 0, i, std::string(\"Hello world\"), std::string(\"This is some header\"));\n}\n\n```\n\n### Consumer example\n\nIn the following example we will be consuming messages with a key of type `size_t`, a payload of type\n`std::string` and a simple header called _Header1_. For full consumer API see [here](https://github.com/bloomberg/corokafka/blob/master/corokafka/corokafka_consumer_manager.h).\n```c++\n//==========================================================================\n//                 Message worker queue and processor\n//==========================================================================\n#include \u003cmytopic.h\u003e\n\nstd::mutex messageMutex;\nstd::deque\u003ccorokafka::ReceivedMessage\u003csize_t, std::string\u003e\u003e messages;\n\nvoid messageProcessor()\n{\n    while (1)\n    {\n        if (!messages.empty()) {\n            std::lock_guard\u003cstd::mutex\u003e lock(messageMutex);\n            corokafka::ReceivedMessage\u003csize_t, std::string\u003e message = std::move(messages.front());\n            messages.pop_front();\n                       \n            //get the headers\n            const ckf::HeaderPack \u0026headers = message.getHeaders();\n            std::string headerName;\n            std::string headerValue;\n            if (headers) {\n                headerName = headers.getAt\u003cstd::string\u003e(0).name();\n                //get header value (type unsafe)\n                headerValue = headers.getAt\u003cstd::string\u003e(0).value();\n                //get header value (type-safe)\n                //headerValue = message.getHeaderAt\u003c0\u003e();\n            }\n            std::ostringstream oss;\n            oss \u003c\u003c \"Received message from topic: \" \u003c\u003c message.getTopic()\n                \u003c\u003c \" partition: \" \u003c\u003c message.getPartition()\n                \u003c\u003c \" offset: \" \u003c\u003c message.getOffset()\n                \u003c\u003c \" key: \" \u003c\u003c message.getKey()\n                \u003c\u003c \" header: \" \u003c\u003c headerName \u003c\u003c \" :: \" \u003c\u003c headerValue\n                \u003c\u003c \" payload: \" \u003c\u003c message.getPayload()\n                \u003c\u003c std::endl;\n            std::cout \u003c\u003c oss.str() \u003c\u003c std::endl;\n            \n            //commit the message. This can also be done automatically (see consumer configuration).\n            message.commit();\n        }\n        // Pause a little. Typically this could be implemented with a condition variable being signalled\n        // when a message gets enqueued.\n        std::this_thread::sleep_for(std::chrono::milliseconds(10));\n    }\n}\n\n//==========================================================================\n//                         Consumer setup\n//==========================================================================\n#include \u003ccorokafka/corokafka.h\u003e\n\n// Define a log callback (optional - see documentation for all available callbacks)\nvoid logCallback(const corokafka::Metadata\u0026 metadata,\n                 cppkafka::LogLevel level,\n                 const std::string\u0026 facility,\n                 const std::string\u0026 message)\n{\n    std::cout \u003c\u003c \"Log from: \" \u003c\u003c metadata.getInternalName()\n              \u003c\u003c \" facility: \" \u003c\u003c facility\n              \u003c\u003c \" text: \" \u003c\u003c message \u003c\u003c std::endl;\n}\n\n// Define a receiver callback (mandatory)\nvoid receiverCallback(MyTopic::receivedMessage message)\n{\n    if (!message) return; //invalid message\n     \n    if (message.getError()) {\n        std::cout \u003c\u003c \"Enqueued message from topic: \" \u003c\u003c message.getTopic()\n                  \u003c\u003c \" with error: \" \u003c\u003c message.getError().to_string()\n                  \u003c\u003c std::endl;\n    }\n    else {\n        std::cout \u003c\u003c \"Enqueued message from topic: \" \u003c\u003c message.getTopic()\n                  \u003c\u003c \" partition: \" \u003c\u003c message.getPartition()\n                  \u003c\u003c \" offset: \" \u003c\u003c message.getOffset()\n                  \u003c\u003c \" key: \" \u003c\u003c message.getKey()\n                  \u003c\u003c \" payload: \" \u003c\u003c (std::string)message.getPayload()\n                  \u003c\u003c std::endl;\n    }\n\n    // Post message unto a worker queue...\n    std::lock_guard\u003cstd::mutex\u003e lock(messageMutex);\n    messages.emplace_back(std::move(message));\n}\n\n// Create the producer and topic config\nstd::initializer_list\u003ccppkafka::ConfigurationOption \u003e consumerOptions = {\n    { \"metadata.broker.list\", \"broker_url:port\" },\n    { \"group.id\", \"my-group\" },\n    { \"api.version.request\", true },\n    { \"enable.partition.eof\", true },\n    { \"enable.auto.commit\", true },\n    { \"auto.commit.interval.ms\", 1000 },\n    { \"auto.offset.reset\", \"earliest\" },\n    { \"partition.assignment.strategy\", \"roundrobin\" },\n    { \"internal.consumer.pause.on.start\", false }\n};\n\n//create the consumer configuration\ncorokafka::ConsumerConfiguration config(myTopic, consumerOptions, {}, receiverCallback);\n\n//add the callbacks\nconfig.setLogCallback(logCallback);\n\n// Optionally set initial partition assignment (4 partitions per topic)\nconfig.assignInitialPartitions(corokafka::PartitionStrategy::Dynamic,\n    {\n    {topic, 0, cppkafka::TopicPartition::OFFSET_BEGINNING},\n    {topic, 1, cppkafka::TopicPartition::OFFSET_BEGINNING},\n    {topic, 2, cppkafka::TopicPartition::OFFSET_BEGINNING},\n    {topic, 3, cppkafka::TopicPartition::OFFSET_BEGINNING}\n    });\n\n// Create the connector (this will subscribe all consumers and start receiving messages)\ncorokafka::Connector connector(corokafka::ConfigurationBuilder(config));\n\n// Start processing messages in the background...\nstd::thread t(messageProcessor);\n\n```\n\n## Building\n\nCreate a `build` directory and call **CMake** with various options below. Then simply call `make`. \n```shell\n\u003e cmake -B build \u003ccmake_options\u003e .\n\u003e cd build \u0026\u0026 make\n```\n\n### Cmake options\n\nVarious **CMake** options can be used to configure the output:\n* `COROKAFKA_BUILD_SHARED`     : Build **CoroKafka** as a shared library. Default `OFF`.\n* `COROKAFKA_ENABLE_PIC`       : Enable position independent code for shared library builds. Default `ON`.\n* `COROKAFKA_CPPKAFKA_STATIC_LIB` : Link with **CppKafka** static library. Default `ON`.\n* `COROKAFKA_RDKAFKA_STATIC_LIB`  : Link with **RdKafka** static library. Default `ON`.\n* `COROKAFKA_BUILD_DOC`        : Build **Doxygen** documentation. Default `OFF`.\n* `COROKAFKA_ENABLE_DOT`       : Enable generation of DOT viewer files. Default `OFF`.\n* `COROKAFKA_VERBOSE_MAKEFILE` : Enable verbose cmake output. Default `ON`.\n* `COROKAFKA_ENABLE_TESTS`     : Builds the `tests` target. Default `OFF`.\n* `COROKAFKA_BOOST_STATIC_LIBS`: Link with **Boost** static libraries. Default `ON`.\n* `COROKAFKA_BOOST_USE_MULTITHREADED` : Use **Boost** multi-threaded libraries. Default `ON`.\n* `COROKAFKA_BOOST_USE_VALGRIND` : Enable **Valgrind** on **Boost**. Default `OFF`.\n* `COROKAFKA_INSTALL_ROOT`     : Specify custom install path. Default is `/usr/local/include` for Linux or `c:/Program Files` for Windows.\n* `RDKAFKA_ROOT`               : Specify a different **RdKafka** install directory.\n* `CPPKAFKA_ROOT`              : Specify a different **CppKafka** install directory.\n* `QUANTUM_ROOT`               : Specify a different **Quantum** install directory.\n* `BOOST_ROOT`                 : Specify a different **Boost** install directory.\n* `GTEST_ROOT`                 : Specify a different **GTest** install directory.\n* `COROKAFKA_PKGCONFIG_DIR`    : Install location of the **.pc** file. Default is `share/pkgconfig`.\n* `COROKAFKA_EXPORT_PKGCONFIG` : Generate `corokafka.pc` file. Default `ON`.\n* `COROKAFKA_CMAKE_CONFIG_DIR` : Install location of the package config file and exports. Default is `share/cmake/CoroKafka`.\n* `COROKAFKA_EXPORT_CMAKE_CONFIG` : Generate **CMake** config, target and version files. Default `ON`.\n\nNote: options must be preceded with `-D` when passed as arguments to **CMake**.\n\n## Installation\n\nTo install, simply run `make install` in the build directory\n```shell\n\u003e cd build \u0026\u0026 make install\n```\n\n## Linking and Using\n\nTo use the library simply include `\u003ccorokafka/corokafka.h\u003e` in your application. \n\nIn you application **CMakeLists.txt** you can load the libraries simply by calling:\n```cmake\nfind_package(CoroKafka REQUIRED)\ntarget_link_libraries(\u003cyour_target\u003e CoroKafka::corokafka \u003cother_dependencies\u003e)\n```\n\n## Contributions\n\nWe :heart: contributions.\n\nHave you had a good experience with this project? Why not share some love and contribute code, or just let us know about any issues you had with it?\n\nWe welcome issue reports [here](../../issues); be sure to choose the proper issue template for your issue, so that we can be sure you're providing the necessary information.\n\nBefore sending a [Pull Request](../../pulls), please make sure you read our\n[Contribution Guidelines](https://github.com/bloomberg/.github/blob/master/CONTRIBUTING.md).\n\n## License\n\nPlease read the [LICENSE](LICENSE) file.\n\n## Code of Conduct\n\nThis project has adopted a [Code of Conduct](https://github.com/bloomberg/.github/blob/master/CODE_OF_CONDUCT.md).\nIf you have any concerns about the Code, or behavior which you have experienced in the project, please\ncontact us at opensource@bloomberg.net.\n\n## Security Vulnerability Reporting\n\nIf you believe you have identified a security vulnerability in this project, please send email to the project\nteam at opensource@bloomberg.net, detailing the suspected issue and any methods you've found to reproduce it.\n\nPlease do NOT open an issue in the GitHub repository, as we'd prefer to keep vulnerability reports private until\nwe've had an opportunity to review and address them.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbloomberg%2Fcorokafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbloomberg%2Fcorokafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbloomberg%2Fcorokafka/lists"}