{"id":37065279,"url":"https://github.com/arthur78/nque","last_synced_at":"2026-01-14T07:38:31.049Z","repository":{"id":294099769,"uuid":"861044205","full_name":"arthur78/nque","owner":"arthur78","description":"High-performance persistent FIFO queues with LMDB backend supporting both single-queue and multi-queue broadcast patterns.","archived":false,"fork":false,"pushed_at":"2025-06-07T16:03:14.000Z","size":47,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-08-19T04:10:46.075Z","etag":null,"topics":["fifo","lmdb","persistent","queues"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/arthur78.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-09-21T21:07:18.000Z","updated_at":"2025-08-11T06:57:16.000Z","dependencies_parsed_at":null,"dependency_job_id":"9c9c959d-6cdf-44a7-a636-d649e21dd3a4","html_url":"https://github.com/arthur78/nque","commit_stats":null,"previous_names":["arthur78/nque"],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/arthur78/nque","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arthur78%2Fnque","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arthur78%2Fnque/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arthur78%2Fnque/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arthur78%2Fnque/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/arthur78","download_url":"https://codeload.github.com/arthur78/nque/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arthur78%2Fnque/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28413463,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T05:26:33.345Z","status":"ssl_error","status_checked_at":"2026-01-14T05:21:57.251Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fifo","lmdb","persistent","queues"],"created_at":"2026-01-14T07:38:29.656Z","updated_at":"2026-01-14T07:38:31.034Z","avatar_url":"https://github.com/arthur78.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# nque - Persistent FIFO Queue Implementation in Python\n![Version](https://img.shields.io/badge/version-1.0.2-blue)\n![Python](https://img.shields.io/badge/python-3.10%2B-blue)\n![Coverage](https://img.shields.io/badge/coverage-92%25-blue)\n![License](https://img.shields.io/badge/license-MIT-blue)\n\n\n`nque` is a Python library that implements persistent FIFO queues backed by [LMDB](https://www.symas.com/mdb) (Lightning\nMemory-Mapped Database). Suitable for building data pipelines and handling data exchange between producer and consumer \nprocesses. The queues operate in memory for performance while persisting data to disk for reliability. For detailed API \ninformation and usage examples, please refer to the class and method docstrings.\n\n## Features\n\n- Transaction-safe operations\n- Two queue implementations:\n  - Basic FIFO queue (`FifoBasicQueueLmdb`) for single queue scenarios\n  - Multi-queue system (`FifoMultiQueueLmdb`) supporting multiple named queues with broadcast capability\n- Configurable queue size and item size limits\n\n## Requirements\n\n- Python 3.10+\n- LMDB\n\n## Installation\n\n```bash\npip install nque\n```\n\n## Usage Examples\n\n### Basic FIFO Queue\n\n```python\nfrom nque import FifoBasicQueueLmdb\n\n# Initialize queue with max 1000 items of max 20 KB each\nqueue = FifoBasicQueueLmdb(\n    db_path=\"queue\", \n    items_count_max=1000,\n    item_bytes_max=20 * 1024\n)\n\n# Add items\nqueue.put([b\"item1\", b\"item2\"])\n\n# Get items without removing (peek)\nitems = queue.get(2)\n\n# Remove items after successful processing \nqueue.remove(2)\n\n# Get and remove items atomically\nitems = queue.pop(2)\n```\n\n### Multi-Queue System\n\n```python\nfrom nque import FifoMultiQueueLmdb\n\n# Create producer that broadcasts to all internal named queues\nproducer = FifoMultiQueueLmdb(\n    db_path=\"multiqueue\",\n    queues=(b\"queue1\", b\"queue2\")\n)\n\n# Create consumer for specific internal queue\nconsumer = FifoMultiQueueLmdb(\n    db_path=\"multiqueue\",\n    queues=(b\"queue1\", b\"queue2\"),\n    use=b\"queue1\"\n)\n\n# Broadcast to all queues atomically\nproducer.put([b\"broadcast_message\"])\n\n# Consume from specific queue\nitems = consumer.pop(1)\n```\n\n## Error Handling\n\nThe library provides three exception types:\n\n- `QueueError`: Base exception for general queue operations\n- `ArgumentError`: Invalid argument errors \n- `TryLater`: Temporary condition preventing operation completion\n\n## Performance Considerations\n\n- Queue max size is pre-defined using max items and item size\n- Write operations (both producers and consumers) are serialized through LMDB's internal locking\n\n## API Reference\n\n### Common Queue Parameters\n\n- `db_path`: Path to the LMDB database storing the queue data\n- `items_count_max`: Maximum number of items in the queue (default: 1000)\n- `item_bytes_max`: Maximum size of each item in bytes (default: 20KB)\n\n### Multi-Queue Additional Parameters\n\n- `queues`: Names of internal queues producers broadcast to\n- `use`: Name of queue to consume from (for consumers only)\n\n### Common Methods\n\n- `put(items)`: Add items to the queue\n- `get(items_count)`: Retrieve items without removing\n- `remove(items_count)`: Remove items from the queue\n- `pop(items_count)`: Atomic get and remove operation\n\n## Best Practices and Error Prevention\n\n1. Use `get()` to retrieve items first, then call `remove()` after successful processing\n2. Use `pop()` for atomic operations, particularly when concurrent consumers are needed\n3. Implement retry logic for `TryLater` exceptions\n\n## Limitations\n\n- Write operations (including from consumers) are serialized\n- Get/remove operation pattern requires single consumer per queue\n- Multi-queue supports maximum of 10 internal queues\n- LMDB constraints apply to overall performance\n\n## Future Development\n\nThe `nque` library is designed with extensibility in mind. Future development areas may include:\n\n- Support for monitoring and metrics capabilities\n- Additional queue patterns and implementations\n- Support for additional backend storage systems beyond LMDB\n\nSuggestions and contributions are welcome.\n\n## Author\n\nArtur Khakimau\n\n## License\n\n[MIT](https://opensource.org/licenses/MIT)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farthur78%2Fnque","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farthur78%2Fnque","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farthur78%2Fnque/lists"}