{"id":38862131,"url":"https://github.com/shiblon/entroq","last_synced_at":"2026-01-17T14:22:52.676Z","repository":{"id":34034105,"uuid":"134464022","full_name":"shiblon/entroq","owner":"shiblon","description":"Competing-consumer fault-tolerant task queues. Written in Go.","archived":false,"fork":false,"pushed_at":"2023-08-19T13:41:17.000Z","size":9689,"stargazers_count":22,"open_issues_count":6,"forks_count":4,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-10-05T14:51:45.618Z","etag":null,"topics":["asynchronous-tasks","competing-consumers","golang","grpc","message-queue","postgresql","task-manager","task-queue","taskmaster","workflow-engine"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shiblon.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-05-22T19:13:53.000Z","updated_at":"2024-06-09T05:14:29.000Z","dependencies_parsed_at":"2024-06-19T02:58:03.851Z","dependency_job_id":"7309e604-5101-4709-9ce5-ec331f2cc5ab","html_url":"https://github.com/shiblon/entroq","commit_stats":null,"previous_names":[],"tags_count":68,"template":false,"template_full_name":null,"purl":"pkg:github/shiblon/entroq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shiblon%2Fentroq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shiblon%2Fentroq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shiblon%2Fentroq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shiblon%2Fentroq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shiblon","download_url":"https://codeload.github.com/shiblon/entroq/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shiblon%2Fentroq/sbom","scorecard":{"id":818195,"data":{"date":"2025-08-11","repo":{"name":"github.com/shiblon/entroq","commit":"22d49e67203d2524fd473141ff6c92ea5b23e2cb"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2,"checks":[{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Code-Review","score":0,"reason":"Found 1/19 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Pinned-Dependencies","score":1,"reason":"dependency not pinned by hash detected -- score normalized to 1","details":["Warn: containerImage not pinned by hash: Dockerfile:4","Warn: containerImage not pinned by hash: Dockerfile:19: pin your Docker image by updating alpine:latest to alpine:latest@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1","Warn: containerImage not pinned by hash: contrib/js/protoc/Dockerfile:1: pin your Docker image by updating node:lts-buster-slim to node:lts-buster-slim@sha256:b5c14e85662c43b8c86a3a88259a34f0071474ad0a029ecb4ec39bbea588b030","Warn: containerImage not pinned by hash: contrib/py/protoc/Dockerfile:1: pin your Docker image by updating python:3-slim-buster to python:3-slim-buster@sha256:c46b0ae5728c2247b99903098ade3176a58e274d9c7d2efeaaab3e0621a53935","Warn: containerImage not pinned by hash: proto/Dockerfile:1","Warn: containerImage not pinned by hash: proto/Dockerfile:19: pin your Docker image by updating debian:buster-slim to debian:buster-slim@sha256:bb3dc79fddbca7e8903248ab916bb775c96ec61014b3d02b4f06043b604726dc","Warn: npmCommand not pinned by hash: contrib/js/protoc/Dockerfile:4-15","Warn: pipCommand not pinned by hash: contrib/py/protoc/Dockerfile:9-14","Warn: pipCommand not pinned by hash: contrib/py/protoc/Dockerfile:9-14","Info:   0 out of   6 containerImage dependencies pinned","Info:   1 out of   1 goCommand dependencies pinned","Info:   0 out of   1 npmCommand dependencies pinned","Info:   0 out of   2 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 13 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":3,"reason":"7 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GO-2024-3250 / GHSA-29wx-vh33-7x7r","Warn: Project is vulnerable to: GO-2025-3553 / GHSA-mh63-6h87-95cp","Warn: Project is vulnerable to: GO-2022-0978","Warn: Project is vulnerable to: GO-2024-3141 / GHSA-c77r-fh37-x2px","Warn: Project is vulnerable to: GO-2025-3660 / GHSA-6m8w-jc87-6cr7","Warn: Project is vulnerable to: GO-2023-2153 / GHSA-m425-mq94-257g / GHSA-qppj-fm5r-hxr3","Warn: Project is vulnerable to: GO-2024-2611 / GHSA-8r3f-844c-mc37"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-23T14:42:53.383Z","repository_id":34034105,"created_at":"2025-08-23T14:42:53.383Z","updated_at":"2025-08-23T14:42:53.383Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28509945,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T13:38:16.342Z","status":"ssl_error","status_checked_at":"2026-01-17T13:37:44.060Z","response_time":85,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asynchronous-tasks","competing-consumers","golang","grpc","message-queue","postgresql","task-manager","task-queue","taskmaster","workflow-engine"],"created_at":"2026-01-17T14:22:45.604Z","updated_at":"2026-01-17T14:22:52.666Z","avatar_url":"https://github.com/shiblon.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# EntroQ\n\nA task queue with strong competing-consumer semantics and transactional updates.\n\nSee here for an article explaining how this might fit into your system:\n\n[Asynchronous Thinking for Microservice System Design](https://github.com/shiblon/entroq/wiki/Asynchronous-Thinking-for-Microservice-System-Design)\n\nThe Go components of this package can be found in online documenation for the\n[github.com/shiblon/entroq](https://pkg.go.dev/github.com/shiblon/entroq) Go package.\n\nPronounced \"Entro-Q\" (\"Entro-Queue\"), as in the letter than comes after\n\"Entro-P\". We aim to take the next step away from parallel systems chaos. It is\nalso the descendent of `github.com/shiblon/taskstore`, an earlier and less\nrobust attempt at the same idea.\n\nIt is designed to be as simple and modular as possible, doing one thing well.\nIt is not a pubsub system, a database, or a generic RPC mechanism. It is only a\ncompeting-consumer unordered work queue manager, and will only ever be that.\nAs such, it has also been designed to do that one thing really well.\n\n## Use\n\nA Docker container is available on Docker Hub as `shiblon/entroq`. You can use\nthis to start an EntroQ service, then talk to it using the provided Go or\nPython libraries. It exposes port `37706` by default.\n\nThe default service that runs uses an in-memory work queue, coupled with an\noptional write-ahead log for persistence and fault tolerance. There is also a\nPostgreSQL-backed version that can be chosen.\n\nIf you merely want an in-process work queue for Go, you can just use the\nin-memory implementation of the library without any server at all. For other\nlanguages, you should use the service and the gRPC-based language-specific\nlibrary to talk to it.\n\nThere is also a command-line client that you can use to talk to the EntroQ\nservice:\n\n    go install github.com/shiblon/entroq/cmd/eqc@latest\n    eqc --help\n\nYou can then run `eqc` to talk to an EntroQ service (such as one started in the\n`shiblon/entroq` container from Docker Hub).\n\nThere is also a Python-based command line, installable via pip:\n\n    python3 -m pip install git+https://github.com/shiblon/entroq\n    python3 -m entroq --help\n\n## Concepts\n\nEntroQ supports precisely two atomic mutating operations:\n\n- Claim an available task from one of possibly several queues, or\n- Update a set of tasks (delete, change, insert), optionally depending on the passive existence of other task versions.\n\nThere are a few read-only accessors, as well, such as a way to list tasks\nwithin a queue, a way to list queues with their sizes, etc. These read-only\noperations do not have any transactional properties, and are best-effort\nsnapshots of queue state. Every effort has been made to ensure that these\nread-only operations do not cause starvation of fundamental operations.\n\nBoth `Claim` and `Modify` change the version number of every task they affect.\nAny time any task is mutated, its version increments. Thus, if one process\nmanages to mutate a task, any other process that was working on it will find\nthat it is not available, and will fail. This is the key concept behind the\n\"commit once\" semantics of EntroQ.\n\nUnlike many pub/sub systems that are used as competing consumer queues, this\neliminates the possibility of work getting dropped after delivery, or work\nbeing committed more than once. Some additional detail on this approach is\ngiven below, but the fundamental principle is this:\n\n    Work commits should never be lost or duplicated.\n\nNote that, perhaps counterintuivitely because of naming, *queues are not ordered*.\nOr more accurately stated, they are *not strictly ordered*. There is a loose\nordering imposed on all queues, but it is basically binary: there are tasks that\nare ready to be claimed, and tasks that are not. Among the tasks that are claimed,\nsome implementations may choose to favor older tasks first, but it is important\nthat a strict ordering *not* be imposed, as doing so would open the door to\npersistent-task-error system starvation, which is another principle of this\napproach:\n\n    Workers should continue making progress even when some tasks are bad.\n\nConsider a small set of tasks in a queue that cause workers to crash every time\ndue, say, to an error in the worker's parser that causes it to die on particular\ncharacters that are in a few tasks' data.\n\nIf you only pull the oldest task first, then the tasks that cause problems will\neventually be the oldest: newer tasks are coming after them, and because the\ncrashy tasks never complete and are never deleted, the worker starts to only\npick up tasks that cause a crash.\n\nInstead, the EntroQ system selects a *random* available task when claiming from a\nqueue, which doesn't *stop* crashes, but does allow the system to *make progress*\neven when a set of tasks cause persistent worker crashes.\n\nThus, if you are looking for a strict ordering of task execution, EntroQ isn't\ngoing to help you by itself. It is very typical to couple EntroQ with a\ndatabase of some kind to maintain complex state or to retain order information,\nand there are common idioms and recipes that can make this very simple while\nretaining the important principles of fault tolerance that EntroQ provides.\n\nDoing things in this way also contributes to a principle that is both crucial\nand possibly the most easily understood of all of them:\n\n    You should be able to scale workers without communication.\n\nThe entire idea is that once a task is in a queue, that's the only communication\nmechanism needed. If you want to process things faster, just make more workers.\nIf you have to scale things down, just kill workers without worry: whatever\nthey were working on will be picked up again when one of your now-depleted worker\nfleet is free to work. Scaling is just a matter of creating or killing jobs,\nand no other concurrency primitives are needed.\n\nThis opens up very powerful and simple idioms for running systems at all kinds\nof scales.\n\nObviously, scaling extremely high in a system with a bottleneck process like\nEntroQ can raise some important questions around task granularity or\ntask system hierarchy. Check out Google's \"Workflow\" chapter in the\n\"Site Reliability Engineering\" book, available online for free, for thoughts\naround massive scaling using a system like EntroQ.\n\n### Tasks\n\nA task, at its most basic, is defined by these traits:\n\n- Queue Name\n- Globally Unique ID\n- Version\n- Arrival Time\n- Value\n\nTasks also hold other information, such as the ID of the current client that\nholds a claim on it, when it was created, and when it was modified, and how\nmany times it has been claimed, but these are implementation details.\n\n### Queues\n\nEntroQ can hold multiple \"queues\", though these do not exist as first-class\nconcepts in the system. A queue only exists so long as there are tasks that\nwant to be in it. If no tasks are in a particular queue, that queue does not\nexist. In database terms, you can think of the queue as being a property of the\ntask, not the other way around. Indeed, in the PostgreSQL backend, it is simply\na column in a `tasks` table.\n\nBecause task IDs are durable, and only the version increments during mutation,\nit is possible to track a single task through multiple modifications. Any\nattempted mutation, however, will fail if both the ID *and* version don't match\nthe task to be mutated. The ID of the task causing a modification failure is\nreturned with the error.\n\nThis allows for a robust competing-consumer worker setup, where workers are\nguaranteed to not accidentally clobber each other's tasks.\n\n### Claims\n\nA `Claim` occurs on one or more queues. A randomly chosen task among those that\nhave an Arrival Time (spelled `at`) in the past is returned if any are\navailable. Otherwise this call blocks. There is a similar call `TryClaim` that\nis non-blocking (and returns a nil value if no task could be claimed\nimmediately), if that is desired.\n\nWhen more than one queue is specified in a `Claim` operation, only one task\nwill be returned from one of those queues. This allows a worker to be a fan-in\nconsumer, fairly pulling tasks from multiple queues. If a \"fast lane\" is\ndesired for a particular worker, this can be achieved by simply having more\nthan one queue that it claims from. Tasks will be pulled fairly from multiple\nqueues, and thus the shortest will be consumed earlier than any longer ones.\nThis is how things tend to work in amusement parks, for example. More complex\npriority schemes have been considered, but tend to be fraught with peril and\nunintended consequences.\n\nWhen successfully claiming a task, the following happen atomically:\n\n- Its version is incremented, and\n- Its arrival time is advanced to a point in the future.\n\nIncrementing the version ensures that any previous claimants will fail to\nupdate that task: the version number will not match. Setting the arrival time\nin the future gives the new claimant time to do the work, or to establish a\ncycle of \"at updates\", where it renews the lease on the task until it is\nfinished. It also allows the network time to return the task to the claimant.\n\nNote: best practice is to keep the initial claim time relatively low, then rely\non the claimant to update the arrival time periodically. This allows for\nsituations like network partitions to be recovered from relatively quickly:\na new worker will more quickly pick up a task that missed a 30-second renewal\nwindow than one that was reserved for 15 minutes with a claimant that died\nearly in its lease.\n\n### Modify\n\nThe `Modify` call can do multiple modifications at once, in an atomic\ntransaction. There are four kinds of modifications that can be made:\n\n- Insert: add one or more tasks to desired queues.\n- Delete: delete one or more tasks.\n- Change: change the value, queue, or arrival time of a given task.\n- Depend: require passive presence of one or more tasks and versions.\n\nIf any of the dependencies are not present (tasks to be deleted, changed, or\ndepended upon, or tasks to be inserted when a colliding ID is specified), then\nthe entire operation fails with an error indicating exactly which tasks were\nnot present in the requested version or had colliding identifiers in the case\nof insertion.\n\nBecause work is not lost until explicitly acknowledged (deleted), it is usually\nsafe to simply abandon work when receiving a deependency error, and grab\nanother task to work on. EntroQ has been designed to avoid starving any queue\nwith tasks that might have inherent data that causes crashes or bugs in\nworkers. These tasks will stick around and be retried periodically, but\nmeanwhile others will go ahead of them because ready tasks are selected at\nrandom from each queue.\n\n### Workers\n\nOnce you create an EntroQ client instance, you can use it to create what is\ncalled a \"worker\". A worker is essentially a claim-renew-modify loop on one or\nmore queues. These can be run in goroutines (or threads, or your language's\nequivalent). Creating a worker using the standard library allows you to focus\non writing only the logic that happens once a task has been acquired. In the\nbackground the claimed task is renewed for as long as the worker code is\nrunning. Good worker code is available in Go, and less-good-but-reasonable code\nfor workers is provided in `contrib/py`. The principles are straightforward to\nimplement in any language that can speak gRPC.\n\nThe code that a worker runs is responsible for doing something with the claimed\ntask, then returning the intended modifications that should happen when it is\nsuccessful. For modification of the claimed task, the standard worker code\nhandles updating the version number in case that task has been renewed in the\nbackground (and thus had its version increment while the work was being done).\n\nThis is a very common way of using EntroQ: stand up an EntroQ service, then\nstart up one or more workers responsible for handling the flow of tasks through\nyour system. Initial injection of tasks can easily be done with the provided\nlibraries or command-line clients.\n\nRather than design a pipeline, it makes sense to have workers that are\nresponsible for doing small tasks, and one or more worker types that are\nresponsible for implementing a pipeline state machine. In its simplest form,\nyou can create a \"trampoline\" worker that handles responses to a single global\nqueue and pushes them into individual task queues depending on contents and\ndisposition.\n\nPipelines are very brittle ideas and should generally be avoided. In a pipeline\nthat grows over time, the complexity of each component increases exponentially\nwith the number of possible input and output types. A trampoline, on the other\nhand, allows every worker to be \"single-purpose\", encoding state transitions in\none place instead of spreading them across the entirety of a microservice\narchitecture.\n\n### Commit Once, Maybe Work Twice\n\nTasks in EntroQ can only be *acknowledged* (deleted or moved) once. It is\npossible for more than one claimant (or \"worker\") to be performing the same\ntasks at the same time, but only one of them will succeed in deleting that\ntask. Because the other will fail to mutate its task, it will also know to\ndiscard any work that it did.\n\nThus, it's possible for the actual work to be done more than once, but that\nwork will only be durably *recorded* once. Because of this, idempotent worker\ndesign is still important, and some helpful principles are described below.\n\nMeanwhile, to give some more detail about how two workers might end up working\non the same task, consider an \"Early and Slow\" (ES) worker and a \"Late and\nQuick\" (LQ) worker. \"Early and Slow\" is the first to claim a particular task,\nbut then takes a long time getting it done. This delay may be due to a network\npartition, or a slow disk on a machine, memory pressure, or process restarts.\nWhatever the reason, ES claims a task, then doesn't acknowledge it before the\ndeadline.\n\nWhile ES is busy working on its task, but not acknowleding or renewing it,\n\"Late and Quick\" (LQ) successfully claims the task after its arrival time is\nreached. If, while it is working on it, ES tries to commit its task, it has an\nearlier version of the task and the commit fails. LQ then finishes and\nsucceeds, because it holds the most recent version of the task, which is the\none represented in the queue.\n\nThis also works if LQ finishes before ES does, in which case ES tries to\nfinally commit its work, but the task that is in the queue is no longer there:\nit has been deleted because LQ finished it and requested deletion.\n\nThese semantics, where a claim is a mutating event, and every alteration to a\ntask changes its version, make it safe for multiple workers to attempt to do\nthe same work without the danger of it being committed (reported) twice.\n\n### Safe Work\n\nIt is possible to abuse the ID/version distinction, by asking EntroQ to tell\nyou about a given task ID, then overriding the claimant ID and task version.\nThis is, in fact, how \"force delete\" works in the provided command-line client.\nIf you wish to mutate a task, you should have first _claimed_ it. Otherwise you\nhave no guarantee that the operation is safe. Merely reading it (e.g., using\nthe `Tasks` call) is not sufficient to guarantee safe mutation.\n\nIf you feel the need to use these facilities in normal worker code, however,\nthat should be a sign that the *design is wrong*. Only in extreme cases, like\nmanual intervention, should these ever be used. As a general rule,\n\n    Only work on claimed tasks, and never override the claimant or version.\n\nIf you need to force something, you probably want to use the command-line\nclient so that a human with human judgement is involved, not in a worker. Then\nyou should be sure of the potential consequences to your system.\n\nTo further ensure safety when using a competing-consumer work queue like\nEntroQ, it is important to adhere to a few simple principles:\n\n- All outside mutations should be idempotent, and\n- Any output files should be uniquely named *every time*.\n\nThe first principle of idempotence allows things like database writes to be\ndone safely by multiple workers (remembering the ES vs. LQ concept above). As\nan example, instead of *incrementing* a value in a database, it should simply\nbe *set to its final value*. Sometimes an increment is really what you want, in\nwhich case you can make that operation idempotent by storing the final value\n*in the task itself* so that the worker simply records that. That way, no\nmatter how many workers make the change, they make it to the same value. The\nprinciple is this:\n\n    Use stable, absolute values in mutations, not relative updates.\n\nThe second principle of unique file names applies to every worker that attempts\nto write anything. Each worker, even those working on the same task, should\ngenerate a random or time-based token in the file name for anything that it\nwrites to the file system. While this can generate garbage that needs to be\ncollected later, it also guarantees that partial writes do not corrupt complete\nones. File system semantics are quite different from database semantics, and\nhaving uniquely-named outputs for every write helps to guarantee that\ncorruption is avoided.\n\nIn short, when there is any likelihood of a file being written by more than one\nprocess,\n\n    Choose garbage collection over write efficiency.\n\nThankfully, adhering to these safety principles is usually pretty easy,\nresulting in great benefits to system stability and trustworthiness.\n\n## Backends\n\nTo create a Go program that makes use of EntroQ, use the `entroq.New` function\nand hand it a `BackendOpener`. An \"opener\" is a function that can, given a\ncontext and suitable parameters, create a backend for the client to use as its\nimplementation.\n\nTo create a Python client, you can use the `entroq` package, which *always*\nspeaks gRPC to a backend EntroQ service.\n\nIn Go, there are three primary backend implementations:\n\n- In-memory,\n- PostgreSQL, and\n- A gRPC client.\n\n### In-Memory Backend\n\nThe `eqmem` backend allows your `EntroQ` instance to work completely\nin-process. You can use exactly the same library calls to talk to it as you\nwould if it were somewhere on the network, making it easy to start in memory\nand progress to database or networked implementations later as needed.\n\nThe following is a short example of how to create an in-memory work queue:\n\n```go\npackage main\n\nimport (\n  \"context\"\n\n  \"github.com/shiblon/entroq\"\n  \"github.com/shiblon/entroq/backend/eqmem\"\n)\n\nfunc main() {\n  ctx := context.Background()\n  eq := entroq.New(ctx, eqmem.Opener())\n\n  // Use eq.Modify, eq.Insert, eq.Claim, etc., probably in goroutines.\n}\n```\n\nThe memory backend contains a write-ahead log implementation for persistence.\nSee the code documentation for how to set parameters to specify the journal\ndirectory and when journals should be rotated.\n\nThe [EntroQ Docker Hub Image](https://hub.docker.com/r/shiblon/entroq) defaults\nto using an in-memory implementation backed by a journal with periodic\nsnapshots. See the volume mounts in the relevant `Dockerfile` to know how to\nmount your own data directories into a running container. The default container\nstarts a gRPC service using this journal-backed in-memory implementation.\n\n### gRPC Backend\n\nThe `grpc` backend is somewhat special. It converts an `entroq.EntroQ` client\ninto a gRPC *client* that can talk to the provided `qsvc` implementation,\ndescribed below.\n\nThis allows you to stand up a gRPC endpoint in front of your \"real\" persistent\nbackend, giving you authentication management and all of the other goodies that\ngRPC provides on the server side, all exposed via protocol buffers and the\nstandard gRPC service interface.\n\nAll clients would then use the `grpc` backend to connect to this service, again\nwith gRPC authentication and all else that it provides. This is the preferred\nway to use the EntroQ client library. In fact, the `eqc` command-line client is\nreally just a gRPC client that can be used to speak to the default Docker\ncontainer mentioned earlier.\n\nAs a basic example of how to set up a gRPC-based EntroQ client:\n\n```go\npackage main\n\nimport (\n  \"context\"\n\n  \"github.com/shiblon/entroq\"\n  \"github.com/shiblon/entroq/backend/eqgrpc\"\n)\n\nfunc main() {\n  ctx := context.Background()\n  eq := entroq.New(ctx, eqgrpc.Opener(\":37706\"))\n\n  // Use eq.Modify, eq.Insert, eq.Claim, etc., probably in goroutines.\n}\n```\n\nThe opener accepts a host name and a number of other gRPC-related optional\nparameters, including mTLS parameters and other familiar gRPC controls.\n\n### PostgreSQL Backend\n\nThe `pg` backend uses a PostgreSQL database. This is a performant, persistent\nbackend option that is suitable for heavy loads (though if your load on this\nsystem is truly heavy, you might have gotten your task granularity wrong).\n\n```go\npackage main\n\nimport (\n  \"context\"\n\n  \"github.com/shiblon/entroq\"\n  \"github.com/shiblon/entroq/backend/eqpg\"\n)\n\nfunc main() {\n  ctx := context.Background()\n  eq := entroq.New(ctx, eqpg.Opener(\":5432\", eqpg.WithDB(\"postgres\"), eqpg.WithUsername(\"myuser\")))\n  // The above supports other postgres-related parameters, as well.\n\n  // Use eq.Modify, eq.Insert, eq.Claim, etc., probably in goroutines.\n}\n```\n\nThis backend is highly PostgreSQL-specific, as it requires the ability to issue\na `SELECT ... FOR UPDATE SKIP LOCKED` query in order to performantly claim\ntasks. MySQL has similar support, so a similar backend could be written for it\nif desired.\n\nUnfortunately, CockroachDB does *not* contain support for the necessary SQL\nstatements, even though it speaks the PostgreSQL wire format. It cannot be used\nin place of PostgreSQL without implementing an entirely new backend (not\nimpossible, just not done).\n\n### Starting a PostgreSQL Instance\n\nIf you wish to start an EntroQ service backed by PostgreSQL, you can run the\ndatabase and EntroQ service in containers on the same Docker network fairly\neasily.\n\nNote that no matter how you run things, there is no need to create any tables\nin your database. The EntroQ service checks for the existence of a `tasks`\ntable and creates it if it is not present. It is also carefully set up to\nupdate older tables when newer versions are deployed.\n\nThe `eqc` command-line utility is included in the `entroq` container, so you\ncan play around with it using `docker exec`.\n\nIf you are using version 0.7, for example, you can run\n\n```\ndocker exec shiblon/entroq:v0.7 eqc --help\n```\n\nAn example `docker-compose` file is shown below that should give\nyou the idea of how they interoperate.\n\nNote that we use `/tmp` for the example below. This is not recommended in\nproduction for obvious reasons, but should illuminate the way things fit\ntogether.\n\n```yaml\nversion: \"3\"\nservices:\n  database:\n    image: \"postgres:12\"\n    deploy:\n      restart_policy:\n        condition: any\n    restart: always\n    volumes:\n      - /tmp/postgres/data:/var/lib/postgresql/data\n\n  queue:\n    image: \"shiblon/entroq:v0.7\"\n    depends_on:\n      - database\n    deploy:\n      restart_policy:\n        condition: any\n    restart: always\n    ports:\n      - 37706:37706\n    command:\n      - \"pg\"\n      - \"--dbaddr=database:5432\"\n      - \"--attempts=10\"\n```\n\nThis starts up PostgreSQL and EntroQ, where EntroQ will make multiple attempts\nto connect to the database before giving up, allowing PostgreSQL some time to\nget its act together.\n\n## QSvc\n\nThe qsvc directory contains the gRPC service implementation that is found in\nthe Docker container `shiblon/entroq`. This service exposes the endpoints found\nin `proto`. Working services using various backends are found in the cmd\ndirectories, e.g.,\n\n- `cmd/eqmemsvc`\n- `cmd/eqpgsvc`\n\nYou can build any of these and start them on desired ports and with desired\nbackend connections based on flag settings.\n\nThere is no *service* backend for `grpc`, though one could conceivably make\nsense as a sort of proxy. But in that case you should really just use a\nstandard gRPC proxy. There are very good reasons to not build your own gRPC\nproxy, no matter how convenient it might seem given the architecture.\n\nWhen using one of these services, this is your main server and should be\ntreated as a singleton. Clients should use the `grpc` backend to connect to it.\n\nDoes making the server a singleton affect performance? Yes, of course, you can't\nscale a singleton, but in practice if you are hitting a work queue that hard you\nlikely have a granularity problem in your design. Additionally, a single\nprocess like this can easily handle thousands of workers.\n\n## Python Implementation\n\nIn `contrib/py` there is an implementation of a gRPC client in Python,\nincluding a basic CLI. You can use this to talk to a gRPC EntroQ service, and\nit includes a basic worker implementation so that you can relatively easily\nreplace uses of Celery in Python.\n\nThe Python implementation is made to be pip-installable directly from github:\n\n```\n$ python3 -m pip install git+https://github.com/shiblon/entroq\n```\n\nThis creates the `entroq` module and supporting protocol buffers beneath it.\nUse of these can be seen in the command-line client, implemented in\n`__main__.py`.\n\n## Examples\n\nOne of the most complete Go examples that is also used as a stress test is a\n**very naive** implementation of MapReduce, in `contrib/mr`. If you look in\nthere, you will see numerous idiomatic uses of the competing queue concept,\ncomplete with workers, using queue empty tests a part of system semantics,\nqueues assigned to specific processes, and others.\n\n## Authorization\n\nEntroQ, when run by itself, doesn't do any authorization. If you simply include\nthe library into a process, and access the backends directly (not the gRPC\nbackend), then authorization is, in fact, not possible: you just have access to\neverything.\n\nIf you do want to include authorization, however, there's good news: the gRPC\nservice *does* allow authorization, and there is an\n[OPA](https://openpolicyagent.org)-based implementation of it ready to go and\navailable for both in-memory and postgres backends.\n\nTo use the OPA-HTTP strategy (where gRPC service request authorization is sent\nto the Open Policy Agent to get authorization approval or failure messages),\nyou can specify the `--authz=opahttp` flag on the command line for the various\nservices you can run.\n\nNote that this means that you would need to have a working OPA instance with\nappropriate packages running at a location that you can specify.\n\n### How it Works\n\nThe `eqc` client has the ability to accept an authorization token, which it\npasses through gRPC in the standard `Authorization: Bearer \u003ctoken\u003e` HTTP\nheader. If the OPA HTTP authorization strategy is enabled in the service flags,\nthe server then packages up this header into a request, along with the desired\nactions on the desired queues (e.g., a claim on an inbox queue), and sends that\nrequest along to OPA.\n\nThe authorization token is passed in two places: within the request itself, and\nin the standard `Authorization` HTTP header. This gives you some flexibility:\nyou can use the OPA system authorization to get an `input.identity` created for\nyou, or you can just unpack the input fields and do that by hand, bypassing the\nOPA internal authorization and just focusing on getting answers about your\nspecific query.\n\nOPA must then have an `entroq.authz` package that is shapaed like an\n`authz.AuthzError` type, defined in [the authz package](pkg/authz/authz.go).\n\nThe [opadata](pkg/authz/opadata) directory contains configurations that work in\nprecisely this way, but it is important to understand the delineation of\nresponsibilities first:\n\n- EntroQ client:\n  - Sends the authorization token in an `Authorization` header when asked.\n\n- EntroQ service:\n  - Forwards the `Authorization` header to OPA with a [request](pkg/authz/authz.go) representing desired queues and actions.\n  - Unpackes the OPA response and allows or disallows the request, accordingly.\n  - Packages up any unauthorized responses into structured errors for the client, if structure is desired.\n\n- OPA:\n  - Unpack the authorization token to get user information, if any.\n  - Produce a set of \"permitted queues and actions\" that can be matched against the request.\n  - Compare and produce either \"allow\" or a set of \"failed\" queues and actions, with error messages.\n\n- Some other system:\n  - Do authentication, generate tokens.\n\nOf note: there are two critical responsibilities that EntroQ *does not\nparticipate in at all*:\n\n- Generation of authorization tokens (from an authentication process), and\n- Interpretation of authorization tokens.\n\nAnother system must be used for authentication and production of valid tokens\nfor a user. EntroQ has zero opinions on that matter.\n\nFurthermore, OPA only *inspects* the authorization token, it does not produce\none.\n\nBecause EntroQ is particular about what it sends as \"input\" and what it\nreceives as a \"document\" (in OPA parlance), some core OPA packages are already\nprovided for you, under [authz/opadata](pkg/authz/opadata/conf/core). These\nfiles should be used without alteration in any OPA configuration that you\nultimately use. They contain mehods for comparing queue specs, and the\n`entroq.authz` package in particular ensures that data is both properly shaped\nand has proper error semantics for a reply.\n\nThe system user (deployer) is responsible for providing the following values:\n\n- `entroq.permissions.allowed_queues`: a set of queue specifications shaped like `Queue` in [authz](pkg/authz), and\n- `entroq.user.username`: a string containing a username, can be empty or undefined.\n\nExample configurations that are not terribly secure in how users are determined\n(e.g., no JWT validation) are found in\n[authz/opadata/conf/example](pkg/authz/opadata/conf/example), and policy data\nin the shape understood by those example files is found in\n[authz/opadata/policy/example](pkg/authz/opadata/policy/example).\n\nAll of these have associated tests that can be run in the standard way, or you\ncan invoke them using `go test` inside the `authz` direcory.\n\nThese examples are used in [contrib/opa-compose](contrib/opa-compose), where a\n`docker-compose.yaml` file shows an example of how you might set up an EntroQ\nand OPA instance side by side, using simple JWT tokens to hold `sub` claims\nwith usernames.\n\nThe basic idea is this:\n\n- Define `entroq.user.username` such that the username is safely pulled from\n  whatever kind of token *your system* needs.\n- Define `entroq.permissions.allowed_queues` to contain all queue\n  specifications that are relevant for the user you get from `entroq.user`.\n- Define policy in whatever way you prefer (there are many possibilities of how\n  to provide \"data\" to OPA - we chose, for our example, to provide it as an\n  `entroq.policy` package, but you may choose to use a data service, push\n  documents directly into OPA, etc.).\n\nAfter that, the core files and EntroQ itself do the rest. You just have to have\nvalid tokens, which you will need to get from somewhere, and OPA will need to\nknow enough to unpack and validate them (e.g., it might need the signing key).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshiblon%2Fentroq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshiblon%2Fentroq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshiblon%2Fentroq/lists"}