{"id":16584496,"url":"https://github.com/brettlangdon/qw","last_synced_at":"2025-10-29T08:30:32.721Z","repository":{"id":21668441,"uuid":"24989421","full_name":"brettlangdon/qw","owner":"brettlangdon","description":"qw (QueueWorker) - python library for processing a redis list as a work queue","archived":false,"fork":false,"pushed_at":"2023-03-31T14:25:53.000Z","size":18,"stargazers_count":7,"open_issues_count":1,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-09-14T03:34:28.130Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/brettlangdon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-10-09T13:05:27.000Z","updated_at":"2018-05-24T20:46:25.000Z","dependencies_parsed_at":"2022-08-17T16:11:13.466Z","dependency_job_id":null,"html_url":"https://github.com/brettlangdon/qw","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brettlangdon%2Fqw","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brettlangdon%2Fqw/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brettlangdon%2Fqw/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brettlangdon%2Fqw/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/brettlangdon","download_url":"https://codeload.github.com/brettlangdon/qw/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219858067,"owners_count":16556048,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T22:44:47.748Z","updated_at":"2025-10-29T08:30:27.444Z","avatar_url":"https://github.com/brettlangdon.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"qw\n==\n\nqw (or QueueWorker) is used to run worker processes which listen on a redis list for jobs to process.\n\n## Setup\n### pip\n\n`pip install qw`\n\n### git\n\n```\ngit clone git://github.com/brettlangdon/qw.git\ncd ./qw\npython setup.py install\n```\n\n## Design\n### Manager\nThe manager is simply a process manager. It's job is to start/stop worker sub-processes.\n\n### Worker\nThe workers are processes which sit and listen for jobs on a few queues and then process\nthose jobs.\n\n### Target\nThe worker/manager take a `target` which can be either a function or a string (importable function).\n\n```python\ndef target(job_id, job_data):\n    pass\n\nmanager = Manager(target)\n# OR\nmanager = Manager('__main__.target')\n```\n### Queues\nThere are a few different queues that are used. The job queues are just redis lists, manager/worker lists are sets and jobs are hashes.\n\nA worker picks up a job from either `all:jobs`, `\u003cmanager\u003e:jobs` or `\u003cworker\u003e:jobs`, pulls the corresponding `job:\u003cjob_id\u003e` key and\nprocesses it with the provided `target`, after processing it will then remove the `job:\u003cjob_id\u003e` key as well as the job id from\nthe `\u003cworker\u003e:jobs` queue.\n\n* `all:managers` - a set of all managers\n* `all:jobs` - a queue that all workers can pull jobs from, the values are just the job ids\n* `job:\u003cjob_id\u003e` - a hash of the job data\n* `\u003cmanager\u003e:workers` - a set of all workers belonging to a given manager\n* `\u003cmanager\u003e:jobs` - a queue of jobs for a specific manager, workers will try to pull from here before `all:jobs`, the values are just the job ids\n* `\u003cworker\u003e:jobs` - a queue of jobs for a specific worker, this is meant as a in progress queue for each worker, the workers will pull jobs into this queue from either `\u003cmanager\u003e:jobs` or `all:jobs`, the values are just the job ids\n\n### Results\n`qw` Workers make no assumptions about the result of processing each job. It does not place finished job into a queue or database\nor anything else. Once a job has been successfully processed by a Worker, that job is removed completely from queues and from redis.\nThe worker itself must properly store the results into a finished queue or database if that is what is required.\n\n## Basic Usage\n\n```python\nfrom qw.manager import Manager\n\n\ndef job_printer(job_id, job_data):\n    print job_id\n    print job_data\n\n\nmanager = Manager(job_printer)\nmanager.start()\nmanager.join()\n```\n\n## API\n### Manager(object)\n* `__init__(self, target, host=\"localhost\", port=6379, db=0, num_workers=None, name=None)`\n* `start(self)`\n* `stop(self)`\n* `join(self)`\n\n### Worker(multiprocess.Process)\n* `__init__(self, client, target, manager_name=None, timeout=10)`\n* `run(self)`\n* `shutdown(self)`\n\n### Client(redis.StrictRedis)\n* `__init__(self, host=\"localhost\", port=6379, db=0)`\n* `register_manager(self, name)`\n* `deregister_manager(self, name)`\n* `register_worker(self, manager, name)`\n* `deregister_worker(self, manager, name)`\n* `queue_job(self, job_data, manager=None, worker=None)`\n* `fetch_next_job(self, manager, worker, timeout=10)`\n* `finish_job(self, job_id, worker_name)`\n* `get_all_managers(self)`\n* `get_manager_workers(self, manager_name)`\n* `get_worker_pending_jobs(self, worker_name)`\n* `get_manager_queued_jobs(self, manager_name)`\n* `get_all_queued_jobs(self)`\n* `get_all_pending_jobs(self)`\n\n## CLI Tools\n### qw-manager\nThe `qw-manager` tool is used to start a new manager process with the provided `target` string, which gets run\nfor every job processed by a worker.\n```\n$ qw-manager --help\nUsage:\n  qw-manager [--level=\u003clog-level\u003e] [--workers=\u003cnum-workers\u003e] [--name=\u003cname\u003e] [--host=\u003chost\u003e] [--port=\u003cport\u003e] [--db=\u003cdb\u003e] \u003ctarget\u003e\n  qw-manager (--help | --version)\n\nOptions:\n  --help                       Show this help message\n  --version                    Show version information\n  -l --level=\u003clog-level\u003e       Set the log level (debug,info,warn,error) [default: info]\n  -w --workers=\u003cnum-workers\u003e   Set the number of workers to start, defaults to number of cpus\n  -n --name=\u003cname\u003e             Set the manager name, defaults to hostname\n  -h --host=\u003chost\u003e             Set the redis host to use [default: localhost]\n  -p --port=\u003cport\u003e             Set the redis port to use [default: 6379]\n  -d --db=\u003cdb\u003e                 Set the redis db number to use [default: 0]\n```\n### qw-client\nThe `qw-client` command is useful to look at basic stats of running managers, workers and job queues\nas well as to push json data in the form of a string or a file to the main queue or a manager specific queue.\n```\n$ qw-client --help\nUsage:\n  qw-client [--host=\u003chost\u003e] [--port=\u003cport\u003e] [--db=\u003cdb\u003e] managers\n  qw-client [--host=\u003chost\u003e] [--port=\u003cport\u003e] [--db=\u003cdb\u003e] workers [\u003cmanager\u003e]\n  qw-client [--host=\u003chost\u003e] [--port=\u003cport\u003e] [--db=\u003cdb\u003e] jobs [\u003cmanager\u003e]\n  qw-client [--host=\u003chost\u003e] [--port=\u003cport\u003e] [--db=\u003cdb\u003e] queue string \u003cdata\u003e [\u003cmanager\u003e]\n  qw-client [--host=\u003chost\u003e] [--port=\u003cport\u003e] [--db=\u003cdb\u003e] queue file \u003cfile\u003e [\u003cmanager\u003e]\n  qw-client (--help | --version)\n\nOptions:\n  --help                       Show this help message\n  --version                    Show version information\n  -h --host=\u003chost\u003e             Set the redis host to use [default: localhost]\n  -p --port=\u003cport\u003e             Set the redis port to use [default: 6379]\n  -d --db=\u003cdb\u003e                 Set the redis db number to use [default: 0]\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrettlangdon%2Fqw","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrettlangdon%2Fqw","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrettlangdon%2Fqw/lists"}