{"id":18123275,"url":"https://github.com/jay-johnson/rust-kafka-threadpool","last_synced_at":"2025-07-20T00:33:50.098Z","repository":{"id":59845367,"uuid":"539608994","full_name":"jay-johnson/rust-kafka-threadpool","owner":"jay-johnson","description":"An async rust threadpool for publishing messages to kafka using SSL (mTLS) or PLAINTEXT protocols.","archived":false,"fork":false,"pushed_at":"2022-09-22T23:38:32.000Z","size":244,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-11T12:18:01.672Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jay-johnson.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-09-21T17:31:48.000Z","updated_at":"2023-03-05T00:57:25.000Z","dependencies_parsed_at":"2022-09-23T03:30:53.295Z","dependency_job_id":null,"html_url":"https://github.com/jay-johnson/rust-kafka-threadpool","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jay-johnson/rust-kafka-threadpool","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jay-johnson%2Frust-kafka-threadpool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jay-johnson%2Frust-kafka-threadpool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jay-johnson%2Frust-kafka-threadpool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jay-johnson%2Frust-kafka-threadpool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jay-johnson","download_url":"https://codeload.github.com/jay-johnson/rust-kafka-threadpool/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jay-johnson%2Frust-kafka-threadpool/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265212426,"owners_count":23728533,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-01T07:08:56.107Z","updated_at":"2025-07-20T00:33:50.070Z","avatar_url":"https://github.com/jay-johnson.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka Threadpool for Rust with mTLS Support\n\nAn async rust threadpool for publishing messages to kafka using ``SSL`` (mTLS) or ``PLAINTEXT`` protocols.\n\n## Architecture\n\nThis is a work in progress. The architecture will likely change over time. For now here's the latest reference architecture:\n\n![kafka-threadpool Reference Architecture](./images/kafka_threadpool_design_v1.png)\n\n## Background\n\nPlease refer to the [blog post](https://jaypjohnson.com/2022-09-19-designing-a-high-performance-rust-threadpool-for-kafka-with-mtls.html) for more information on this repo.\n\n## Configuration\n\n### Supported Environment Variables\n\n| Environment Variable Name        | Purpose / Value                                |\n| -------------------------------- | ---------------------------------------------- |\n| KAFKA_ENABLED                    | toggle the kafka_threadpool on with: ``true`` or ``1`` anything else disables the threadpool | \n| KAFKA_LOG_LABEL                  | tracking label that shows up in all crate logs | \n| KAFKA_BROKERS                    | comma-delimited list of brokers (``host1:port,host2:port,host3:port``) |\n| KAFKA_TOPICS                     | comma-delimited list of supported topics |\n| KAFKA_PUBLISH_RETRY_INTERVAL_SEC | number of seconds to sleep before each publish retry |\n| KAFKA_PUBLISH_IDLE_INTERVAL_SEC  | number of seconds to sleep if there are no message to process |\n| KAFKA_NUM_THREADS                | number of threads for the threadpool |\n| KAFKA_TLS_CLIENT_KEY             | optional - path to the kafka mTLS key |\n| KAFKA_TLS_CLIENT_CERT            | optional - path to the kafka mTLS certificate |\n| KAFKA_TLS_CLIENT_CA              | optional - path to the kafka mTLS certificate authority (CA) |\n| KAFKA_METADATA_COUNT_MSG_OFFSETS | optional - set to anything but ``true`` to bypass counting the offsets |\n\n## Getting Started\n\nPlease ensure your kafka cluster is running before starting. If you need help running a kafka cluster please refer to the [rust-with-strimzi-kafka-tls repo](https://github.com/jay-johnson/rust-with-strimzi-kafka-and-tls) for more details.\n\n### Set up the Environment Variables\n\nYou can create an ``./env/kafka.env`` file storing the environment variables to make your producer and consumer consistent (and ready for podman/docker or kubernetes):\n\n```bash\nexport KAFKA_ENABLED=1\nexport KAFKA_LOG_LABEL=\"ktp\"\nexport KAFKA_BROKERS=\"host1:port,host2:port,host3:port\"\nexport KAFKA_TOPICS=\"testing\"\nexport KAFKA_PUBLISH_RETRY_INTERVAL_SEC=\"1.0\"\nexport KAFKA_NUM_THREADS=\"5\"\nexport KAFKA_TLS_CLIENT_CA=\"PATH_TO_TLS_CA_FILE\"\nexport KAFKA_TLS_CLIENT_CERT=\"PATH_TO_TLS_CERT_FILE\"\nexport KAFKA_TLS_CLIENT_KEY=\"PATH_TO_TLS_KEY_FILE\"\nexport KAFKA_METADATA_COUNT_MSG_OFFSETS=\"true\"\n```\n\n#### Load the Environment\n\n```bash\nsource ./env/kafka.env\n```\n\n### Start the Kafka Threadpool and Publish 100 Messages\n\nThe included [./examples/start-threadpool.rs](https://github.com/jay-johnson/rust-kafka-threadpool/blob/main/examples/start-threadpool.rs) example will connect to the kafka cluster based off the environment configuration and publish 100 messages into the kafka ``testing`` topic.\n\n```bash\ncargo build --example start-threadpool\nexport RUST_BACKTRACE=1\nexport RUST_LOG=info,kafka_threadpool=info,rdkafka=info\n./target/debug/examples/start-threadpool\n```\n\n### Consume Messages\n\nTo consume the newly-published test messages from the ``testing`` topic, you can use your own consumer or the [rust-with-strimzi-kafka-and-tls/examples/run-consumer.rs](https://github.com/jay-johnson/rust-with-strimzi-kafka-and-tls/blob/main/examples/run-consumer.rs) example:\n\n```bash\n# from the rust-with-strimzi-kafka-and-tls directory:\ncargo build --example run-consumer\nexport RUST_BACKTRACE=1\nexport RUST_LOG=info,rdkafka=info\n./target/debug/examples/run-consumer -g rust-consumer-testing -t testing\n```\n\n### Get Kafka Cluster Metadata for All Topics, Partitions, ISR, and Offsets\n\nRun the [./examples/get-all-metadata.rs](https://github.com/jay-johnson/rust-kafka-threadpool/blob/main/examples/get-all-metadata.rs) example:\n\n```bash\ncargo build --example get-all-metadata\nexport RUST_BACKTRACE=1\nexport RUST_LOG=info,kafka_threadpool=info,rdkafka=info\n./target/debug/examples/get-all-metadata\n```\n\n### Get Kafka Cluster Metadata for a Single Topic including Partitions, ISR and Offsets\n\n1.  Set the Topic Name as an Environment Variable\n\n    ```bash\n    export KAFKA_TOPIC=testing\n    ```\n\n1.  Run the [./examples/get-metadata-for-topic.rs](https://github.com/jay-johnson/rust-kafka-threadpool/blob/main/examples/get-metadata-for-topic.rs) example:\n\n    ```bash\n    cargo build --example get-metadata-for-topic\n    export RUST_BACKTRACE=1\n    export RUST_LOG=info,kafka_threadpool=info,rdkafka=info\n    ./target/debug/examples/get-metadata-for-topic\n    ```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjay-johnson%2Frust-kafka-threadpool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjay-johnson%2Frust-kafka-threadpool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjay-johnson%2Frust-kafka-threadpool/lists"}