{"id":13439182,"url":"https://github.com/fede1024/rust-rdkafka","last_synced_at":"2025-05-14T22:06:45.123Z","repository":{"id":37602187,"uuid":"72286208","full_name":"fede1024/rust-rdkafka","owner":"fede1024","description":"A fully asynchronous, futures-based Kafka client library for Rust based on librdkafka","archived":false,"fork":false,"pushed_at":"2025-02-03T20:36:13.000Z","size":15019,"stargazers_count":1755,"open_issues_count":129,"forks_count":301,"subscribers_count":29,"default_branch":"master","last_synced_at":"2025-05-07T21:59:25.112Z","etag":null,"topics":["futures","kafka","kafka-client","librdkafka","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fede1024.png","metadata":{"files":{"readme":"README.md","changelog":"changelog.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-10-29T13:23:35.000Z","updated_at":"2025-05-07T08:33:24.000Z","dependencies_parsed_at":"2023-09-21T22:24:10.851Z","dependency_job_id":"85931606-a35e-4090-818b-5b36039fbb16","html_url":"https://github.com/fede1024/rust-rdkafka","commit_stats":{"total_commits":637,"total_committers":71,"mean_commits":8.971830985915492,"dds":0.6059654631083202,"last_synced_commit":"611aeea5012787ad0f1a3ff74af32b64d41142f8"},"previous_names":[],"tags_count":45,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fede1024%2Frust-rdkafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fede1024%2Frust-rdkafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fede1024%2Frust-rdkafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fede1024%2Frust-rdkafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fede1024","download_url":"https://codeload.github.com/fede1024/rust-rdkafka/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254235695,"owners_count":22036963,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["futures","kafka","kafka-client","librdkafka","rust"],"created_at":"2024-07-31T03:01:11.793Z","updated_at":"2025-05-14T22:06:40.098Z","avatar_url":"https://github.com/fede1024.png","language":"Rust","funding_links":[],"categories":["Libraries","Rust","库 Libraries","库","虚拟化","Clients"],"sub_categories":["Distributed systems","分布式系统 Distributed systems","分布式系统","Rust"],"readme":"# rust-rdkafka\n\n[![crates.io](https://img.shields.io/crates/v/rdkafka.svg)](https://crates.io/crates/rdkafka)\n[![docs.rs](https://docs.rs/rdkafka/badge.svg)](https://docs.rs/rdkafka/)\n[![Build Status](https://travis-ci.org/fede1024/rust-rdkafka.svg?branch=master)](https://travis-ci.org/fede1024/rust-rdkafka)\n[![coverate](https://codecov.io/gh/fede1024/rust-rdkafka/graphs/badge.svg?branch=master)](https://codecov.io/gh/fede1024/rust-rdkafka/)\n[![Join the chat at https://gitter.im/rust-rdkafka/Lobby](https://badges.gitter.im/rust-rdkafka/Lobby.svg)](https://gitter.im/rust-rdkafka/Lobby?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge)\n\nA fully asynchronous, [futures]-enabled [Apache Kafka] client\nlibrary for Rust based on [librdkafka].\n\n## The library\n\n`rust-rdkafka` provides a safe Rust interface to librdkafka. This version\nis compatible with librdkafka v1.9.2+.\n\n### Documentation\n\n- [Current master branch](https://fede1024.github.io/rust-rdkafka/)\n- [Latest release](https://docs.rs/rdkafka/)\n- [Changelog](https://github.com/fede1024/rust-rdkafka/blob/master/changelog.md)\n\n### Features\n\nThe main features provided at the moment are:\n\n- Support for all Kafka versions since 0.8.x. For more information about\n  broker compatibility options, check the [librdkafka\n  documentation][broker-compat].\n- Consume from single or multiple topics.\n- Automatic consumer rebalancing.\n- Customizable rebalance, with pre and post rebalance callbacks.\n- Synchronous or asynchronous message production.\n- Customizable offset commit.\n- Create and delete topics and add and edit partitions.\n- Alter broker and topic configurations.\n- Access to cluster metadata (list of topic-partitions, replicas, active\n  brokers etc).\n- Access to group metadata (list groups, list members of groups, hostnames,\n  etc.).\n- Access to producer and consumer metrics, errors and callbacks.\n- Exactly-once semantics (EOS) via idempotent and transactional producers\n  and read-committed consumers.\n\n### One million messages per second\n\n`rust-rdkafka` is designed to be easy and safe to use thanks to the\nabstraction layer written in Rust, while at the same time being extremely\nfast thanks to the librdkafka C library.\n\nHere are some benchmark results using the [`BaseProducer`],\nsending data to a single Kafka 0.11 process running in localhost (default\nconfiguration, 3 partitions). Hardware: Dell laptop, with Intel Core\ni7-4712HQ @ 2.30GHz.\n\n- Scenario: produce 5 million messages, 10 bytes each, wait for all of them to be acked\n  - 1045413 messages/s, 9.970 MB/s  (average over 5 runs)\n\n- Scenario: produce 100000 messages, 10 KB each, wait for all of them to be acked\n  - 24623 messages/s, 234.826 MB/s  (average over 5 runs)\n\nFor more numbers, check out the [kafka-benchmark] project.\n\n### Client types\n\n`rust-rdkafka` provides low level and high level consumers and producers.\n\nLow level:\n\n* [`BaseConsumer`]: a simple wrapper around the librdkafka consumer. It\n  must be periodically `poll()`ed in order to execute callbacks, rebalances\n  and to receive messages.\n* [`BaseProducer`]: a simple wrapper around the librdkafka producer. As in\n  the consumer case, the user must call `poll()` periodically to execute\n  delivery callbacks.\n* [`ThreadedProducer`]: a `BaseProducer` with a separate thread dedicated to\n  polling the producer.\n\nHigh level:\n\n * [`StreamConsumer`]: a [`Stream`] of messages that takes care of\n   polling the consumer automatically.\n * [`FutureProducer`]: a [`Future`] that will be completed once\n   the message is delivered to Kafka (or failed).\n\nFor more information about consumers and producers, refer to their\nmodule-level documentation.\n\n*Warning*: the library is under active development and the APIs are likely\nto change.\n\n### Asynchronous data processing with Tokio\n\n[Tokio] is a platform for fast processing of asynchronous events in Rust.\nThe interfaces exposed by the [`StreamConsumer`] and the [`FutureProducer`]\nallow rust-rdkafka users to easily integrate Kafka consumers and producers\nwithin the Tokio platform, and write asynchronous message processing code.\nNote that rust-rdkafka can be used without Tokio.\n\nTo see rust-rdkafka in action with Tokio, check out the\n[asynchronous processing example] in the examples folder.\n\n### At-least-once delivery\n\nAt-least-once delivery semantics are common in many streaming applications:\nevery message is guaranteed to be processed at least once; in case of\ntemporary failure, the message can be re-processed and/or re-delivered,\nbut no message will be lost.\n\nIn order to implement at-least-once delivery the stream processing\napplication has to carefully commit the offset only once the message has\nbeen processed. Committing the offset too early, instead, might cause\nmessage loss, since upon recovery the consumer will start from the next\nmessage, skipping the one where the failure occurred.\n\nTo see how to implement at-least-once delivery with `rdkafka`, check out the\n[at-least-once delivery example] in the examples folder. To know more about\ndelivery semantics, check the [message delivery semantics] chapter in the\nKafka documentation.\n\n### Exactly-once semantics\n\nExactly-once semantics (EOS) can be achieved using transactional producers,\nwhich allow produced records and consumer offsets to be committed or aborted\natomically. Consumers that set their `isolation.level` to `read_committed`\nwill only observe committed messages.\n\nEOS is useful in read-process-write scenarios that require messages to be\nprocessed exactly once.\n\nTo learn more about using transactions in rust-rdkafka, see the\n[Transactions](producer-transactions) section of the producer documentation.\n\n### Users\n\nHere are some of the projects using rust-rdkafka:\n\n- [kafka-view]: a web interface for Kafka clusters.\n- [kafka-benchmark]: a high performance benchmarking tool for Kafka.\n- [callysto]: Stream processing framework in Rust.\n- [bytewax]: Python stream processing framework using Timely Dataflow.\n\n*If you are using rust-rdkafka, please let us know!*\n\n## Installation\n\nAdd this to your `Cargo.toml`:\n\n```toml\n[dependencies]\nrdkafka = { version = \"0.25\", features = [\"cmake-build\"] }\n```\n\nThis crate will compile librdkafka from sources and link it statically to\nyour executable. To compile librdkafka you'll need:\n\n* the GNU toolchain\n* GNU `make`\n* `pthreads`\n* `zlib`: optional, but included by default (feature: `libz`)\n* `cmake`: optional, *not* included by default (feature: `cmake-build`)\n* `libssl-dev`: optional, *not* included by default (feature: `ssl`)\n* `libsasl2-dev`: optional, *not* included by default (feature: `gssapi`)\n* `libzstd-dev`: optional, *not* included by default (feature: `zstd-pkg-config`)\n\nNote that using the CMake build system, via the `cmake-build` feature, is\nencouraged if you can take the dependency on CMake.\n\nBy default a submodule with the librdkafka sources pinned to a specific\ncommit will be used to compile and statically link the library. The\n`dynamic-linking` feature can be used to instead dynamically link rdkafka to\nthe system's version of librdkafka. Example:\n\n```toml\n[dependencies]\nrdkafka = { version = \"0.25\", features = [\"dynamic-linking\"] }\n```\n\nFor a full listing of features, consult the [rdkafka-sys crate's\ndocumentation][rdkafka-sys-features]. All of rdkafka-sys features are\nre-exported as rdkafka features.\n\n### Minimum supported Rust version (MSRV)\n\nThe current minimum supported Rust version (MSRV) is 1.70.0. Note that\nbumping the MSRV is not considered a breaking change. Any release of\nrust-rdkafka may bump the MSRV.\n\n### Asynchronous runtimes\n\nSome features of the [`StreamConsumer`] and [`FutureProducer`] depend on\nTokio, which can be a heavyweight dependency for users who only intend to\nuse the low-level consumers and producers. The Tokio integration is\nenabled by default, but can be disabled by turning off default features:\n\n```toml\n[dependencies]\nrdkafka = { version = \"0.25\", default-features = false }\n```\n\nIf you would like to use an asynchronous runtime besides Tokio, you can\nintegrate it with rust-rdkafka by providing a shim that implements the\n[`AsyncRuntime`] trait. See the following examples for details:\n\n  * [smol][runtime-smol]\n  * [async-std][runtime-async-std]\n\n## Examples\n\nYou can find examples in the [`examples`] folder. To run them:\n\n```bash\ncargo run --example \u003cexample_name\u003e -- \u003cexample_args\u003e\n```\n\n## Debugging\n\nrust-rdkafka uses the [`log`] crate to handle logging.\nOptionally, enable the `tracing` feature to emit [`tracing`]\nevents as opposed to [`log`] records.\n\nIn test and examples, rust-rdkafka uses the  [`env_logger`] crate\nto format logs. In those contexts, logging can be enabled\nusing the `RUST_LOG` environment variable, for example:\n\n```bash\nRUST_LOG=\"librdkafka=trace,rdkafka::client=debug\" cargo test\n```\n\nThis will configure the logging level of librdkafka to trace, and the level\nof the client module of the Rust client to debug. To actually receive logs\nfrom librdkafka, you also have to set the `debug` option in the producer or\nconsumer configuration (see librdkafka\n[configuration][librdkafka-config]).\n\nTo enable debugging in your project, make sure you initialize the logger\nwith `env_logger::init()`, or the equivalent for any `log`-compatible\nlogging framework.\n\n[`AsyncRuntime`]: https://docs.rs/rdkafka/*/rdkafka/util/trait.AsyncRuntime.html\n[`BaseConsumer`]: https://docs.rs/rdkafka/*/rdkafka/consumer/base_consumer/struct.BaseConsumer.html\n[`BaseProducer`]: https://docs.rs/rdkafka/*/rdkafka/producer/base_producer/struct.BaseProducer.html\n[`Future`]: https://doc.rust-lang.org/stable/std/future/trait.Future.html\n[`FutureProducer`]: https://docs.rs/rdkafka/*/rdkafka/producer/future_producer/struct.FutureProducer.html\n[`Stream`]: https://docs.rs/futures/*/futures/stream/trait.Stream.html\n[`StreamConsumer`]: https://docs.rs/rdkafka/*/rdkafka/consumer/stream_consumer/struct.StreamConsumer.html\n[`ThreadedProducer`]: https://docs.rs/rdkafka/*/rdkafka/producer/base_producer/struct.ThreadedProducer.html\n[`log`]: https://docs.rs/log\n[`tracing`]: https://docs.rs/tracing\n[`env_logger`]: https://docs.rs/env_logger\n[Apache Kafka]: https://kafka.apache.org\n[asynchronous processing example]: https://github.com/fede1024/rust-rdkafka/blob/master/examples/asynchronous_processing.rs\n[at-least-once delivery example]: https://github.com/fede1024/rust-rdkafka/blob/master/examples/at_least_once.rs\n[runtime-smol]: https://github.com/fede1024/rust-rdkafka/blob/master/examples/runtime_smol.rs\n[runtime-async-std]: https://github.com/fede1024/rust-rdkafka/blob/master/examples/runtime_async_std.rs\n[broker-compat]: https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#broker-version-compatibility\n[bytewax]: https://github.com/bytewax/bytewax\n[callysto]: https://github.com/vertexclique/callysto\n[`examples`]: https://github.com/fede1024/rust-rdkafka/blob/master/examples/\n[futures]: https://github.com/rust-lang/futures-rs\n[kafka-benchmark]: https://github.com/fede1024/kafka-benchmark\n[kafka-view]: https://github.com/fede1024/kafka-view\n[librdkafka]: https://github.com/edenhill/librdkafka\n[librdkafka-config]: https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md\n[message delivery semantics]: https://kafka.apache.org/0101/documentation.html#semantics\n[producer-transactions]: https://docs.rs/rdkafka/*/rdkafka/producer/#transactions\n[rdkafka-sys-features]: https://github.com/fede1024/rust-rdkafka/tree/master/rdkafka-sys/README.md#features\n[rdkafka-sys-known-issues]: https://github.com/fede1024/rust-rdkafka/tree/master/rdkafka-sys/README.md#known-issues\n[smol]: https://docs.rs/smol\n[Tokio]: https://tokio.rs/\n\n## rdkafka-sys\n\nSee [rdkafka-sys](https://github.com/fede1024/rust-rdkafka/tree/master/rdkafka-sys).\n\n## Contributors\n\nThanks to:\n* Thijs Cadier - [thijsc](https://github.com/thijsc)\n\n## Alternatives\n\n* [kafka-rust]: a pure Rust implementation of the Kafka client.\n\n[kafka-rust]: https://github.com/spicavigo/kafka-rust\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffede1024%2Frust-rdkafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffede1024%2Frust-rdkafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffede1024%2Frust-rdkafka/lists"}