{"id":21925289,"url":"https://github.com/fabienarcellier/spike-rq-task-distribution-pattern","last_synced_at":"2026-05-16T18:06:06.270Z","repository":{"id":50162936,"uuid":"282718981","full_name":"FabienArcellier/spike-rq-task-distribution-pattern","owner":"FabienArcellier","description":"``RQ`` is a library designed to implement a simple task distribution system based on Redis as distribution broker.","archived":false,"fork":false,"pushed_at":"2022-12-08T11:20:21.000Z","size":172,"stargazers_count":0,"open_issues_count":7,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-07-11T22:25:23.592Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FabienArcellier.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-07-26T19:29:05.000Z","updated_at":"2020-07-26T19:32:33.000Z","dependencies_parsed_at":"2023-01-25T12:30:50.518Z","dependency_job_id":null,"html_url":"https://github.com/FabienArcellier/spike-rq-task-distribution-pattern","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/FabienArcellier/spike-rq-task-distribution-pattern","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FabienArcellier%2Fspike-rq-task-distribution-pattern","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FabienArcellier%2Fspike-rq-task-distribution-pattern/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FabienArcellier%2Fspike-rq-task-distribution-pattern/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FabienArcellier%2Fspike-rq-task-distribution-pattern/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FabienArcellier","download_url":"https://codeload.github.com/FabienArcellier/spike-rq-task-distribution-pattern/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FabienArcellier%2Fspike-rq-task-distribution-pattern/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275772859,"owners_count":25525909,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-18T02:00:09.552Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-28T21:17:54.496Z","updated_at":"2025-09-18T13:23:24.916Z","avatar_url":"https://github.com/FabienArcellier.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Spike - RQ as task distribution pattern\n\n![continuous_integration](https://github.com/FabienArcellier/spike-rq-task-distribution-pattern/workflows/continuous_integration/badge.svg)\n\n``RQ`` is a library designed to implement a simple task distribution system based\non Redis as distribution broker.\n\n* [RQ : Easy queues job for python](https://python-rq.org/)\n\nA task distribution pattern, also called producer / consumer is an easy way to create\ndistributed architecture. It allow to scale independently the producer and the consumer\ninstance thanks to a job broker in the middle. In the case of RQ, the job broker is\n`Redis`\n\n![](docs/diagrams.png)\n\nA task distribution pattern allow to queue jobs and process them in a background task.\nIn this example, we will design an architecture to fetch website url and count the number of words\nin background workers.\n\nA distribution pattern offers advantages as :\n\n* the producer is working even if the worker stop working\n* if a worker stop working, the job can be retry after a specified timeout\n* the number of worker can scale independently of the number of producer\n* the number of producer can scale independently of the number of worker\n* the job broker persist message if it has to restart\n\nAlternatives to RQ are ``celery``, ``dramatiq``, ...\n\n## Getting started\n\nThis spike implements [word counter example](https://python-rq.org/) described in ``rq``.\nYou need two terminal to run it.\n\n1. Terminal 1 - InRun the system (job broker and one worker)\n\n```bash\nhoncho start\n```\n\n2. Terminal 2 - Submit a job through command line\n\n* ``trigger`` :\n\n```bash\n$ pipenv run mycommand trigger --url www.google.fr\n```\n\n* ``result`` :\n\n```bash\npipenv run mycommand result --jobid {{ uuid4 }}\n```\n## The latest version\n\nYou can find the latest version to ...\n\n```bash\ngit clone https://github.com/FabienArcellier/blueprint-cli-multicommands-python.git\n```\n\n## Contributors\n\n* Fabien Arcellier\n\n## License\n\nMIT License\n\nCopyright (c) 2018 Fabien Arcellier\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffabienarcellier%2Fspike-rq-task-distribution-pattern","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffabienarcellier%2Fspike-rq-task-distribution-pattern","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffabienarcellier%2Fspike-rq-task-distribution-pattern/lists"}