{"id":26058031,"url":"https://github.com/gapitio/tiny_kafka_gapit","last_synced_at":"2025-03-08T11:59:59.959Z","repository":{"id":246212207,"uuid":"820423369","full_name":"gapitio/tiny_kafka_gapit","owner":"gapitio","description":"tiny_kafka but better","archived":false,"fork":false,"pushed_at":"2025-01-14T10:33:18.000Z","size":24,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-01-14T11:47:42.783Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gapitio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-26T12:45:14.000Z","updated_at":"2025-01-14T10:33:21.000Z","dependencies_parsed_at":"2024-06-26T15:55:45.966Z","dependency_job_id":"32087f38-6c94-4d51-ba2a-ac906a1daa27","html_url":"https://github.com/gapitio/tiny_kafka_gapit","commit_stats":null,"previous_names":["gapitio/tiny_kafka_gapit"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gapitio%2Ftiny_kafka_gapit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gapitio%2Ftiny_kafka_gapit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gapitio%2Ftiny_kafka_gapit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gapitio%2Ftiny_kafka_gapit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gapitio","download_url":"https://codeload.github.com/gapitio/tiny_kafka_gapit/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242546034,"owners_count":20147096,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-08T11:59:59.291Z","updated_at":"2025-03-08T11:59:59.953Z","avatar_url":"https://github.com/gapitio.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka Producer and Consumer in Rust\n\nThis repository contains Rust-based implementations of a Kafka producer and a Kafka consumer. It leverages the `rdkafka` library to communicate with Kafka and manage the production and consumption of messages.\n\n## Prerequisites\n\n- **Rust**: Ensure you have Rust and Cargo installed on your machine.\n- **Kafka**: A running Kafka cluster that you can connect to.\n\n## Setup\n\n1. **Clone the Repository**:\n   ```bash\n   git clone github.com/cploutarchou/tiny_kafka\n   ```\n\n2. **Navigate to the Repository**:\n   ```bash\n   cd path-to-repository\n   ```\n\n3. **Build the Code**:\n   ```bash\n   cargo build\n   ```\n\n## Producer\n\nThe Kafka producer is a higher-level interface for producing messages to Kafka. It encapsulates the process of initializing a connection to the Kafka broker and sending messages to a specified topic.\n\n### Usage\n\nTo initialize the producer:\n\n```rust\nlet producer = KafkaProducer::new(\"localhost:9092\", None);\n```\n\nTo send a message to a Kafka topic:\n\n```rust\nlet msg = Message::new(\"key1\", \"value1\");\nproducer.send_message(\"my-topic\", msg).await;\n```\n\n### Custom Configurations\n\nYou can provide custom configurations while initializing the producer:\n\n```rust\nlet mut configs = HashMap::new();\nconfigs.insert(\"max.in.flight.requests.per.connection\", \"5\");\nlet producer_with_configs = KafkaProducer::new(\"localhost:9092\", Some(configs));\n```\n\n## Consumer\n\nThe Kafka consumer provides functionality to consume messages from a Kafka topic.\n\n### Usage\n\n#### To initialize the consumer:\n\n```rust\nlet consumer = KafkaConsumer::new(\"localhost:9092\", \"my_group\", \"my_topic\");\n```\n\n#### To consume messages:\n\n```rust\nif let Some(msg) = consumer.poll().await {\n    println!(\"Received: {} -\u003e {}\", msg.key, msg.value);\n}\n```\n#### Full Main Function Example with Tokio and Async\n\nBelow is a detailed example of how to utilize both the Kafka producer and consumer within an asynchronous context, leveraging Tokio.\n\n```rust\nuse log::info;\nuse std::sync::Arc;\nuse tiny_kafka::consumer::KafkaConsumer;\nuse tiny_kafka::producer::{KafkaProducer, Message};\n\n#[tokio::main]\nasync fn main() -\u003e Result\u003c(), Box\u003cdyn std::error::Error\u003e\u003e {\n\n   let rt = tokio::runtime::Runtime::new().unwrap();\n   // Assuming kafka_bootstrap_servers is of type String\n   let brokers = Arc::new(\"localhost:9092\".to_string());\n   let topics = Arc::new(vec![\"test\".to_string()]);\n\n   // Consumer task\n   let brokers_for_task1 = brokers.clone();\n   let topics_for_task1 = topics.clone();\n   let task1 = async move {\n      let consumer = KafkaConsumer::new(\n         brokers_for_task1.as_str(),\n         \"kafka-to-elastic\",\n         topics_for_task1.get(0).unwrap(),\n      );\n      loop {\n         if let Some(msg) = consumer.poll().await {\n            info!(\n                    \"Consumed message with key: {} and value: {}\",\n                    msg.key, msg.value\n                );\n         }\n      }\n   };\n   rt.spawn(task1);\n\n   // Producer task\n   let brokers_for_task2 = brokers.clone();\n   let topics_for_task2 = topics.clone();\n   let task2 = async move {\n      let producer = KafkaProducer::new(brokers_for_task2.as_str(), Option::None);\n\n      for i in 0..100 {\n         let key = format!(\"test_key_{}\", i);\n         let value = format!(\"test_value_{}\", i);\n         let message = Message::new(\u0026key, \u0026value);\n\n         producer\n                 .send_message(topics_for_task2.get(0).unwrap(), message)\n                 .await;\n         tokio::time::sleep(tokio::time::Duration::from_millis(100)).await;\n      }\n   };\n   rt.spawn(task2);\n\n   // Wait for a ctrl-c signal\n   tokio::signal::ctrl_c().await?;\n   println!(\"ctrl-c received!\");\n\n   Ok(())\n}\n\n```\n\n### Custom Configurations\n\nJust like the producer, the consumer also supports custom configurations:\n\n```rust\nlet mut new_configs = HashMap::new();\nnew_configs.insert(\"auto.offset.reset\".to_string(), \"earliest\".to_string());\nconsumer.set_client_config(\"localhost:9092\", \"my_group\", \"my_topic\", new_configs);\n```\n\n## Running Tests\n\nTo run tests:\n\n```bash\ncargo test\n```\n\nEnsure your Kafka broker is running and the test topic exists.\n\n## Contribution\n\nWe welcome contributions! Feel free to open issues, submit pull requests, or just spread the word.\n\n## License\n\nThis project is licensed under the [MIT License](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgapitio%2Ftiny_kafka_gapit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgapitio%2Ftiny_kafka_gapit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgapitio%2Ftiny_kafka_gapit/lists"}