{"id":43813026,"url":"https://github.com/not-empty/omniq","last_synced_at":"2026-04-05T14:03:53.676Z","repository":{"id":336733669,"uuid":"1149846049","full_name":"not-empty/omniq","owner":"not-empty","description":"Language agnostic queue ecosystem with parallel limits, draining and much more. Publish and consume in your favorite language like Python, Go, Node (TS or Vanilla), PHP and any other you want to.","archived":false,"fork":false,"pushed_at":"2026-04-03T22:51:59.000Z","size":204,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-04T00:31:22.881Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Lua","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/not-empty.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-04T15:36:02.000Z","updated_at":"2026-03-19T11:38:01.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/not-empty/omniq","commit_stats":null,"previous_names":["not-empty/omniq"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/not-empty/omniq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/not-empty%2Fomniq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/not-empty%2Fomniq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/not-empty%2Fomniq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/not-empty%2Fomniq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/not-empty","download_url":"https://codeload.github.com/not-empty/omniq/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/not-empty%2Fomniq/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31437928,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-05T13:13:19.330Z","status":"ssl_error","status_checked_at":"2026-04-05T13:13:17.778Z","response_time":75,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-02-06T00:00:26.150Z","updated_at":"2026-04-05T14:03:53.667Z","avatar_url":"https://github.com/not-empty.png","language":"Lua","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OmniQ\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"img/OmniQ.png\" alt=\"Omniq Logo\" width=\"200\"/\u003e\n\u003c/p\u003e\n\nOmniQ is a job queue built around a single idea: **keep all queue semantics centralized, deterministic, and atomic**.\n\nInstead of reimplementing behavior in each client, OmniQ defines its core operations in **Lua**, so **every language behaves exactly the same**. This makes execution **predictable**, avoids **race conditions**, and keeps the system **consistent under concurrency**.\n\nAt the same time, it supports practical needs such as **controlled parallelism (via groups)**, **fair scheduling (round-robin)**, and **dependent workloads (parent/child jobs)**, without adding unnecessary complexity.\n\n------------------------------------------------------------------------------\n\n## Description\n\nOmniQ provides a consistent model for background job processing across different runtimes. Instead of embedding queue logic in each language implementation, the core behavior is defined through Lua scripts, ensuring that all clients behave identically.\n\nThe system is built around a few key principles:\n\n- Deterministic and atomic operations  \n- Fair job scheduling across independent workloads  \n- Controlled parallelism  \n- Support for dependent (child) jobs  \n\n------------------------------------------------------------------------------\n\n## Implementations\n\nYou can run OmniQ using one of the existing implementations:\n\n- **Python:** https://github.com/not-empty/omniq-python  \n- **Go:** https://github.com/not-empty/omniq-go  \n- **Node.js:** https://github.com/not-empty/omniq-node  \n\nOr implement it yourself in any language capable of executing Lua scripts against the backing datastore (typically Redis).\n\n------------------------------------------------------------------------------\n\n## Core Concepts\n\n### Lua (Atomic \u0026 Agnostic Operations)\n\nAll queue operations are executed through Lua scripts.\n\nThis guarantees:\n\n- **Atomicity:** each operation runs as a single, indivisible unit  \n- **Consistency:** no race conditions between consumers  \n- **Agnostic behavior:** every language implementation shares the same logic  \n\nThis approach removes discrepancies between clients and ensures that queue semantics remain identical regardless of the runtime.\n\n------------------------------------------------------------------------------\n\n## Group × Parallel Limit\n\nJobs can be assigned to groups, where each group defines its own maximum number of concurrent executions.\n\nThis allows:\n\n- Isolation between different workloads  \n- Protection against resource exhaustion  \n- Fine-grained concurrency control  \n\nConsumers will only reserve jobs from a group if its parallel limit has not been reached.\n\n------------------------------------------------------------------------------\n\n## Round Robin (Fairness)\n\nOmniQ uses a round-robin strategy when selecting jobs across groups.\n\nInstead of draining one group entirely, the scheduler cycles between groups, ensuring:\n\n- Fair distribution of processing time  \n- No starvation of smaller or slower queues  \n- Balanced throughput across all workloads  \n\n------------------------------------------------------------------------------\n\n## Child Jobs\n\nThe diagram below illustrates how OmniQ handles parent/child relationships between jobs:\n\n![Parent / Child Flow](img/Child.png)\n\nOmniQ allows a parent job to spawn multiple child jobs and remain dependent on their completion. Instead of treating jobs as isolated units, this model enables coordinated execution across related tasks.\n\nThis is particularly useful for:\n\n- Multi-step workflows  \n- Parallel execution of sub-tasks  \n- Explicit synchronization through child acknowledgements  \n\nThe child job lifecycle is managed through dedicated operations such as `child_init` (to initialize children) and `child_ack` (to track their completion), allowing the parent job to progress only when all required child jobs are resolved.\n\n------------------------------------------------------------------------------\n\n## Lua Actions\n\nThe following operations define the full behavior of the queue. All of them are executed atomically via Lua:\n\n| Action             | Description                                   |\n|--------------------|-----------------------------------------------|\n| ack_fail           | Mark a job as failed and update its state     |\n| ack_success        | Mark a job as successfully completed          |\n| child_ack          | Register completion of a child job            |\n| child_init         | Initialize child jobs for a parent            |\n| enqueue            | Insert a new job into the queue               |\n| heartbeat          | Update job execution heartbeat                |\n| pause              | Pause processing for a queue or group         |\n| promoted_delay     | Move delayed jobs into the ready state        |\n| reap_expired       | Clean up expired or abandoned jobs            |\n| remove_job         | Remove a specific job from the system         |\n| remove_jobs_batch  | Remove multiple jobs in a single operation    |\n| reserve            | Reserve the next available job for processing |\n| resume             | Resume a paused queue or group                |\n| retry_failed       | Retry a previously failed job                 |\n| retry_failed_batch | Retry multiple failed jobs                    |\n\n------------------------------------------------------------------------------\n\n## Notes\n\n- OmniQ does not enforce a specific transport or consumer model, it only defines queue semantics.  \n- The backing datastore is typically Redis due to native Lua support and atomic execution guarantees.  \n- Each language implementation is expected to be a thin layer over the Lua scripts.  \n\n------------------------------------------------------------------------------\n\n## License\n\nSee LICENSE file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnot-empty%2Fomniq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnot-empty%2Fomniq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnot-empty%2Fomniq/lists"}