{"id":47840337,"url":"https://github.com/rbw/pyfiq","last_synced_at":"2026-04-03T20:37:59.103Z","repository":{"id":302327938,"uuid":"1011956053","full_name":"rbw/pyfiq","owner":"rbw","description":"FIFO Microqueue for Python","archived":false,"fork":false,"pushed_at":"2025-07-15T12:00:53.000Z","size":174,"stargazers_count":6,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-15T17:31:44.478Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rbw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-01T15:27:28.000Z","updated_at":"2025-07-15T12:00:57.000Z","dependencies_parsed_at":"2025-07-24T16:36:35.708Z","dependency_job_id":"bc68fd65-5d35-4818-9972-f03768d6aa49","html_url":"https://github.com/rbw/pyfiq","commit_stats":null,"previous_names":["rbw/pyfiq"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/rbw/pyfiq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rbw%2Fpyfiq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rbw%2Fpyfiq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rbw%2Fpyfiq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rbw%2Fpyfiq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rbw","download_url":"https://codeload.github.com/rbw/pyfiq/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rbw%2Fpyfiq/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31375772,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-03T17:53:18.093Z","status":"ssl_error","status_checked_at":"2026-04-03T17:53:17.617Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-03T20:37:57.762Z","updated_at":"2026-04-03T20:37:59.080Z","avatar_url":"https://github.com/rbw.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"extra/logo.png\" alt=\"pyfiq Logo\" width=\"592\"\u003e\n\u003c/p\u003e\n\n---\n`pyfiq` is a MIT-licensed, lightweight, [Redis Streams](https://redis.io/docs/latest/develop/data-types/streams/) backed [FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)) task queue for Python that works seamlessly with Redis-compatible forks like [Valkey](https://valkey.io/), [KeyDB](https://docs.keydb.dev/), and others.\nIt lets you decorate functions with `@pyfiq.fifo(...)`, enqueue them for execution, and ensures those functions run **in strict order**, even across multiple application instances. \n\nYou can think of `pyfiq` as an **embedded, Python-native alternative to AWS Lambda + SQS FIFO**: no external infrastructure, no vendor lock-in--just drop it into your app.\n\n\n---\n\n### Why pyfiq?  \n\n- **Strict ordering**: tasks on the same queue are always executed in the order they were enqueued.  \n- **Portable**: runs anywhere Python and Redis are available.  \n- **Embedded**: workers run inside your application process--no external workers needed.  \n- **Distributed**: automatically scales across multiple app instances, providing redundancy and load balancing.  \n- **Parallel where it matters**: one worker per queue, with multiple queues processed concurrently. \n- **Lightweight and scalable**: ideal for both small apps and large distributed backends.  \n- **Non-breaking API**: decorate any function with `@pyfiq.fifo(...)` and call it as usual, queued transparently.  \n- **Zero-config**: no brokers, orchestrators, or external services required.  \n\nDecorated functions are invoked like normal Python functions, but instead of executing immediately, they're placed into a FIFO queue for asynchronous processing by background workers.  \n\npyfiq is designed for workflows where **ordering matters more than raw throughput**, such as event-driven, state-changing operations.\n\n---\n\n### When does pyfiq make sense?  \n\nIn Python, offloading CPU-bound work to external systems (like Celery) is often necessary because of the Global Interpreter Lock (GIL). But for I/O-bound workloads, the story is different:  \n- the GIL isn't a bottleneck.  \n- asyncio or multithreading often provide great concurrency.\n\nSo why use a queue at all for I/O tasks?  \n\nBecause some workloads require **more than just concurrency**:  \n\n- In distributed deployments, **concurrent requests** from multiple app instances can apply changes **out of order**, causing inconsistent state downstream.\n- **Directly calling external systems** ties up app threads and pushes upstream errors down to clients\n- **Traffic spikes** can flood external APIs and trigger rate limits, leading to outages.  \n\n## Project status\n\nThis project is in its early stages of development.\n\n## Usage\n\n### Installing\n\n`pyfiq` is on PyPI, but for now install directly from the Git master branch to try it out:\n\n```bash\n$ pip install git+https://github.com/rbw/pyfiq.git\n```\n\n---\n\n### Start the worker\n\nStart a background worker once at application startup, typically in your main thread or service entrypoint:\n\n```python\nimport pyfiq\n\n# Start a threaded worker that processes queued tasks\nworker = pyfiq.ThreadedWorker(redis_url=\"redis://localhost\")\nworker.start()\n```\n\nThis spins up a lightweight background thread that consumes tasks (i.e., functions decorated with `@pyfiq.fifo`) from Redis.\n\n\n### Decorate your functions\n\nMark any function you want to run asynchronously and in strict FIFO order:\n\n```python\nimport logging\nimport pyfiq\n\nlog = logging.getLogger(__name__)\n\ndef handle_success(retval, task, binding):\n    log.info(f\"Task succeeded ({task}): {retval}\")\n\ndef handle_error(exc, task, _):\n    log.exception(f\"Task failed ({task})\", exc_info=exc)\n\n    \n@pyfiq.fifo(queue=\"queue1\")  # Shared\ndef sync_user_subscription():\n    ...\n\n@pyfiq.fifo(queue=\"queue1\", max_retries=-1, retry_wait=3, on_error=handle_error)  # Shared, retry forever (block consumption)\ndef sync_user_metadata():\n    ...\n\n@pyfiq.fifo(queue=\"queue2\", on_success=handle_success)\ndef update_payment_transaction():\n    ...\n```\n\n### Call functions as usual\n\nNo need to manage queues, spin up workers, or handle consumer lifecycles--just call your decorated functions:\n\n```python\nsync_user_metadata()\n```\n\nInstead of running immediately, the call is enqueued for background execution in the order it was made.\n\n\n---\n\n## 📖 Examples\n\nSee [extra/examples](./extra/examples) for ready-to-run code samples.  \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frbw%2Fpyfiq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frbw%2Fpyfiq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frbw%2Fpyfiq/lists"}