{"id":28164065,"url":"https://github.com/seedyrom/batched-queue","last_synced_at":"2025-05-15T11:15:35.529Z","repository":{"id":292464171,"uuid":"980989974","full_name":"SeedyROM/batched-queue","owner":"SeedyROM","description":"A high-performance, lock-free batched queue implementation for Rust.","archived":false,"fork":false,"pushed_at":"2025-05-10T06:13:58.000Z","size":20,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-10T06:22:55.629Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SeedyROM.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-10T05:09:28.000Z","updated_at":"2025-05-10T06:14:01.000Z","dependencies_parsed_at":"2025-05-10T06:23:00.182Z","dependency_job_id":"b3ef4a71-34ab-430e-afe7-76629d45d557","html_url":"https://github.com/SeedyROM/batched-queue","commit_stats":null,"previous_names":["seedyrom/batched-queue"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SeedyROM%2Fbatched-queue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SeedyROM%2Fbatched-queue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SeedyROM%2Fbatched-queue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SeedyROM%2Fbatched-queue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SeedyROM","download_url":"https://codeload.github.com/SeedyROM/batched-queue/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254328384,"owners_count":22052634,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-15T11:15:06.327Z","updated_at":"2025-05-15T11:15:35.511Z","avatar_url":"https://github.com/SeedyROM.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# `batched-queue`\n\nA high-performance, highly-concurrent batched queue implementation for Rust.\n\n[![GitHub](https://img.shields.io/badge/github-batched--queue-8da0cb?logo=github)](https://github.com/SeedyROM/batched-queue)\n[![Crates.io](https://img.shields.io/crates/v/batched-queue.svg)](https://crates.io/crates/batched-queue)\n[![Documentation](https://docs.rs/batched-queue/badge.svg)](https://docs.rs/batched-queue)\n[![crates.io version](https://img.shields.io/crates/l/batched-queue.svg)](https://github.com/SeedyROM/batched-queue/blob/main/LICENSE)\n\n## Overview\n\n`batched-queue` provides an efficient way to collect individual items into batches for processing, which can significantly improve throughput in high-volume systems. The library offers both synchronous and asynchronous implementations, making it suitable for a wide range of applications.\n\n## Features\n\n- **Automatic Batching**: Collects individual items into batches of configurable size\n- **Multiple Implementations**:\n  - Synchronous (default) using `parking_lot` and `crossbeam-channel`\n  - Asynchronous using `tokio` (via feature flag)\n- **Thread-safe**: Designed for concurrent usage with multiple producers and consumers\n- **Backpressure Control**: Optional bounded queue to manage memory usage\n- **Flexible Retrieval**: Blocking, non-blocking, and timeout-based batch retrieval methods\n- **High Performance**: Optimized for low contention and high throughput\n\n## Installation\n\nAdd `batched-queue` to your `Cargo.toml`:\n\n```toml\n[dependencies]\nbatched-queue = \"0.1.0\"\n```\n\n### Feature Flags\n\n- `sync` (default): Enables the synchronous implementation using `parking_lot` and `crossbeam-channel`\n- `async`: Enables the asynchronous implementation using `tokio`\n\nTo use the async implementation:\n\n```toml\n[dependencies]\nbatched-queue = { version = \"0.1.0\", default-features = false, features = [\"async\"] }\n```\n\n## Usage Examples\n\n### Basic Usage\n\n```rust\nuse batched_queue::{BatchedQueue, BatchedQueueTrait};\n\nfn main() {\n    // Create a queue with batch size of 10\n    let queue = BatchedQueue::new(10).expect(\"Failed to create queue\");\n    \n    // Create a sender that can be shared across threads\n    let sender = queue.create_sender();\n    \n    // Push items to the queue\n    for i in 0..25 {\n        sender.push(i).expect(\"Failed to push item\");\n    }\n    \n    // Flush any remaining items that haven't formed a complete batch\n    sender.flush().expect(\"Failed to flush queue\");\n    \n    // Process batches\n    while let Ok(batch) = queue.try_next_batch() {\n        println!(\"Processing batch of {} items\", batch.len());\n        for item in batch {\n            println!(\"  Item: {}\", item);\n        }\n    }\n}\n```\n\n### Multi-threaded Usage\n\n```rust\nuse batched_queue::{BatchedQueue, BatchedQueueTrait};\nuse std::thread;\nuse std::time::Duration;\n\nfn main() {\n    // Create a queue with batch size of 5\n    let queue = BatchedQueue::new(5).expect(\"Failed to create queue\");\n    \n    // Create a sender that can be shared across threads\n    let sender = queue.create_sender();\n    \n    // Producer thread\n    let producer = thread::spawn(move || {\n        for i in 0..100 {\n            sender.push(i).expect(\"Failed to push item\");\n            thread::sleep(Duration::from_millis(5));\n        }\n        sender.flush().expect(\"Failed to flush queue\"); // Send any remaining items\n    });\n    \n    // Consumer thread\n    let consumer = thread::spawn(move || {\n        let mut all_items = Vec::new();\n        \n        // Process batches as they become available\n        while all_items.len() \u003c 100 {\n            if let Some(batch) = queue.next_batch_timeout(Duration::from_millis(100)) {\n                println!(\"Received batch of {} items\", batch.len());\n                all_items.extend(batch);\n            }\n        }\n        \n        all_items\n    });\n    \n    // Wait for threads to complete\n    producer.join().unwrap();\n    let result = consumer.join().unwrap();\n    \n    println!(\"Processed {} items in total\", result.len());\n}\n```\n\n### Bounded Queue with Backpressure\n\n```rust\nuse batched_queue::{BatchedQueue, BatchedQueueTrait};\n\nfn main() {\n    // Create a queue with batch size 10 and at most 5 batches in the channel\n    let queue = BatchedQueue::new_bounded(10, 5).expect(\"Failed to create bounded queue\");\n    \n    // When the channel is full, producers will block when attempting to send a full batch\n    // This provides automatic backpressure to control memory usage\n    \n    let sender = queue.create_sender();\n    \n    // ... use queue as normal\n}\n```\n\n## Performance\n\n`batched-queue` is designed for high performance in concurrent environments:\n\n- Optimized for minimal lock contention\n- Uses efficient lock-free algorithms where possible, highly concurrent otherwise\n- Can achieve millions of items per second on modern hardware\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fseedyrom%2Fbatched-queue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fseedyrom%2Fbatched-queue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fseedyrom%2Fbatched-queue/lists"}