{"id":13502246,"url":"https://github.com/pricingassistant/mrq","last_synced_at":"2025-09-17T23:04:46.336Z","repository":{"id":14094056,"uuid":"16798102","full_name":"pricingassistant/mrq","owner":"pricingassistant","description":"Mr. Queue - A distributed worker task queue in Python using Redis \u0026 gevent","archived":false,"fork":false,"pushed_at":"2023-06-13T19:16:57.000Z","size":2001,"stargazers_count":883,"open_issues_count":64,"forks_count":114,"subscribers_count":58,"default_branch":"master","last_synced_at":"2025-09-14T02:47:49.187Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pricingassistant.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/contributing.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2014-02-13T09:32:40.000Z","updated_at":"2025-08-28T14:11:00.000Z","dependencies_parsed_at":"2022-07-16T16:47:04.999Z","dependency_job_id":"f58d4502-56b4-4645-9266-0fec15cf417b","html_url":"https://github.com/pricingassistant/mrq","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/pricingassistant/mrq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pricingassistant%2Fmrq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pricingassistant%2Fmrq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pricingassistant%2Fmrq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pricingassistant%2Fmrq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pricingassistant","download_url":"https://codeload.github.com/pricingassistant/mrq/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pricingassistant%2Fmrq/sbom","scorecard":{"id":744940,"data":{"date":"2025-08-11","repo":{"name":"github.com/pricingassistant/mrq","commit":"5a1154dc6a60d8b6d6423937f6569ce8fb0347b6"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":1.8,"checks":[{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Code-Review","score":2,"reason":"Found 8/28 approved changesets -- score normalized to 2","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: containerImage not pinned by hash: Dockerfile:1: pin your Docker image by updating debian:buster-slim to debian:buster-slim@sha256:bb3dc79fddbca7e8903248ab916bb775c96ec61014b3d02b4f06043b604726dc","Warn: containerImage not pinned by hash: Dockerfile-with-code:1: pin your Docker image by updating pricingassistant/mrq-env:latest to pricingassistant/mrq-env:latest@sha256:f182f519f6b32fb668ddd5daf21b0ed5e2a636a24984c2afa696d7a0350183e2","Warn: downloadThenRun not pinned by hash: Dockerfile:34","Warn: pipCommand not pinned by hash: Dockerfile:45","Warn: pipCommand not pinned by hash: Dockerfile:46","Warn: pipCommand not pinned by hash: Dockerfile:54-58","Warn: pipCommand not pinned by hash: Dockerfile:54-58","Warn: pipCommand not pinned by hash: Dockerfile:54-58","Warn: pipCommand not pinned by hash: Dockerfile:54-58","Warn: pipCommand not pinned by hash: Dockerfile:60-64","Warn: pipCommand not pinned by hash: Dockerfile:60-64","Warn: pipCommand not pinned by hash: Dockerfile:60-64","Warn: pipCommand not pinned by hash: Dockerfile:60-64","Warn: pipCommand not pinned by hash: Dockerfile:66-70","Warn: pipCommand not pinned by hash: Dockerfile:66-70","Warn: pipCommand not pinned by hash: Dockerfile:66-70","Warn: pipCommand not pinned by hash: Dockerfile:66-70","Info:   0 out of  14 pipCommand dependencies pinned","Info:   0 out of   2 containerImage dependencies pinned","Info:   0 out of   1 downloadThenRun dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Vulnerabilities","score":0,"reason":"19 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GHSA-55x5-fj6c-h6m8","Warn: Project is vulnerable to: PYSEC-2021-19 / GHSA-jq4v-f5q6-mjqq","Warn: Project is vulnerable to: PYSEC-2020-62 / GHSA-pgww-xf46-h92r","Warn: Project is vulnerable to: PYSEC-2022-230 / GHSA-wrxv-2j5q-m38w","Warn: Project is vulnerable to: PYSEC-2018-12 / GHSA-xp26-p53h-6h2p","Warn: Project is vulnerable to: GHSA-9hjg-9r4m-mvj7","Warn: Project is vulnerable to: GHSA-9wx4-h78v-vm56","Warn: Project is vulnerable to: PYSEC-2023-74 / GHSA-j8r2-6x86-q33q","Warn: Project is vulnerable to: PYSEC-2022-42991 / GHSA-v3c5-jqr6-7qm8","Warn: Project is vulnerable to: PYSEC-2023-177 / GHSA-x7m3-jprg-wc5g","Warn: Project is vulnerable to: PYSEC-2019-41 / GHSA-qfc5-mcwq-26q8","Warn: Project is vulnerable to: GHSA-m87m-mmvp-v9qm","Warn: Project is vulnerable to: GHSA-fm67-cv37-96ff","Warn: Project is vulnerable to: GHSA-wpqr-jcpx-745r","Warn: Project is vulnerable to: PYSEC-2018-66 / GHSA-562c-5r94-xh97","Warn: Project is vulnerable to: PYSEC-2019-179 / GHSA-5wv5-4vpf-pj6m","Warn: Project is vulnerable to: PYSEC-2023-62 / GHSA-m2qf-hxjv-5gpq","Warn: Project is vulnerable to: PYSEC-2018-78 / GHSA-h2vm-c85r-5vh5","Warn: Project is vulnerable to: GHSA-vcph-37mh-fqrh"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 10 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-22T18:27:01.600Z","repository_id":14094056,"created_at":"2025-08-22T18:27:01.601Z","updated_at":"2025-08-22T18:27:01.601Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275094407,"owners_count":25404452,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-14T02:00:10.474Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T22:02:07.549Z","updated_at":"2025-09-17T23:04:46.316Z","avatar_url":"https://github.com/pricingassistant.png","language":"Python","funding_links":[],"categories":["Queue","资源列表","Python","任务队列","Task Queues","数据管道和流处理","HarmonyOS","队列","Task Queues [🔝](#readme)","Awesome Python"],"sub_categories":["队列","Windows Manager","Queue","ASGI Servers"],"readme":"# MRQ\n\n[![Build Status](https://travis-ci.org/pricingassistant/mrq.svg?branch=master)](https://travis-ci.org/pricingassistant/mrq) [![MIT License](https://img.shields.io/github/license/pricingassistant/mrq.svg)](LICENSE)\n\n[MRQ](http://pricingassistant.github.io/mrq) is a distributed task queue for python built on top of mongo, redis and gevent.\n\nFull documentation is available on [readthedocs](http://mrq.readthedocs.org/en/latest/)\n\n# Why?\n\nMRQ is an opinionated task queue. It aims to be simple and beautiful like [RQ](http://python-rq.org) while having performances close to [Celery](http://celeryproject.org)\n\nMRQ was first developed at [Pricing Assistant](http://pricingassistant.com) and its initial feature set matches the needs of worker queues with heterogenous jobs (IO-bound \u0026 CPU-bound, lots of small tasks \u0026 a few large ones).\n\n# Main Features\n\n * **Simple code:** We originally switched from Celery to RQ because Celery's code was incredibly complex and obscure ([Slides](http://www.slideshare.net/sylvinus/why-and-how-pricing-assistant-migrated-from-celery-to-rq-parispy-2)). MRQ should be as easy to understand as RQ and even easier to extend.\n * **Great [dashboard](http://mrq.readthedocs.org/en/latest/dashboard/):** Have visibility and control on everything: queued jobs, current jobs, worker status, ...\n * **Per-job logs:** Get the log output of each task separately in the dashboard\n * **Gevent worker:** IO-bound tasks can be done in parallel in the same UNIX process for maximum throughput\n * **Supervisord integration:** CPU-bound tasks can be split across several UNIX processes with a single command-line flag\n * **Job management:** You can retry, requeue, cancel jobs from the code or the dashboard.\n * **Performance:** Bulk job queueing, easy job profiling\n * **Easy [configuration](http://mrq.readthedocs.org/en/latest/configuration):** Every aspect of MRQ is configurable through command-line flags or a configuration file\n * **Job routing:** Like Celery, jobs can have default queues, timeout and ttl values.\n * **Builtin scheduler:** Schedule tasks by interval or by time of the day\n * **Strategies:** Sequential or parallel dequeue order, also a burst mode for one-time or periodic batch jobs.\n * **Subqueues:** Simple command-line pattern for dequeuing multiple sub queues, using auto discovery from worker side.\n * **Thorough [testing](http://mrq.readthedocs.org/en/latest/tests):** Edge-cases like worker interrupts, Redis failures, ... are tested inside a Docker container.\n * **Greenlet tracing:** See how much time was spent in each greenlet to debug CPU-intensive jobs.\n * **Integrated memory leak debugger:** Track down jobs leaking memory and find the leaks with objgraph.\n\n# Dashboard Screenshots\n\n![Job view](http://i.imgur.com/xaXmrvX.png)\n\n![Worker view](http://i.imgur.com/yYUMCbm.png)\n\n# Get Started\n\nThis 5-minute tutorial will show you how to run your first jobs with MRQ.\n\n## Installation\n\n - Make sure you have installed the [dependencies](dependencies.md) : Redis and MongoDB\n - Install MRQ with `pip install mrq`\n - Start a mongo server with `mongod \u0026`\n - Start a redis server with `redis-server \u0026`\n\n\n## Write your first task\n\nCreate a new directory and write a simple task in a file called `tasks.py` :\n\n```makefile\n$ mkdir test-mrq \u0026\u0026 cd test-mrq\n$ touch __init__.py\n$ vim tasks.py\n```\n\n```python\nfrom mrq.task import Task\nimport urllib2\n\n\nclass Fetch(Task):\n\n    def run(self, params):\n\n        with urllib2.urlopen(params[\"url\"]) as f:\n          t = f.read()\n          return len(t)\n```\n\n## Run it synchronously\n\nYou can now run it from the command line using `mrq-run`:\n\n```makefile\n$ mrq-run tasks.Fetch url http://www.google.com\n\n2014-12-18 15:44:37.869029 [DEBUG] mongodb_jobs: Connecting to MongoDB at 127.0.0.1:27017/mrq...\n2014-12-18 15:44:37.880115 [DEBUG] mongodb_jobs: ... connected.\n2014-12-18 15:44:37.880305 [DEBUG] Starting tasks.Fetch({'url': 'http://www.google.com'})\n2014-12-18 15:44:38.158572 [DEBUG] Job None success: 0.278229s total\n17655\n```\n\n## Run it asynchronously\n\nLet's schedule the same task 3 times with different parameters:\n\n```makefile\n$ mrq-run --queue fetches tasks.Fetch url http://www.google.com \u0026\u0026\n  mrq-run --queue fetches tasks.Fetch url http://www.yahoo.com \u0026\u0026\n  mrq-run --queue fetches tasks.Fetch url http://www.wordpress.com\n\n2014-12-18 15:49:05.688627 [DEBUG] mongodb_jobs: Connecting to MongoDB at 127.0.0.1:27017/mrq...\n2014-12-18 15:49:05.705400 [DEBUG] mongodb_jobs: ... connected.\n2014-12-18 15:49:05.729364 [INFO] redis: Connecting to Redis at 127.0.0.1...\n5492f771520d1887bfdf4b0f\n2014-12-18 15:49:05.957912 [DEBUG] mongodb_jobs: Connecting to MongoDB at 127.0.0.1:27017/mrq...\n2014-12-18 15:49:05.967419 [DEBUG] mongodb_jobs: ... connected.\n2014-12-18 15:49:05.983925 [INFO] redis: Connecting to Redis at 127.0.0.1...\n5492f771520d1887c2d7d2db\n2014-12-18 15:49:06.182351 [DEBUG] mongodb_jobs: Connecting to MongoDB at 127.0.0.1:27017/mrq...\n2014-12-18 15:49:06.193314 [DEBUG] mongodb_jobs: ... connected.\n2014-12-18 15:49:06.209336 [INFO] redis: Connecting to Redis at 127.0.0.1...\n5492f772520d1887c5b32881\n```\n\nYou can see that instead of executing the tasks and returning their results right away, `mrq-run` has added them to the queue named `fetches` and printed their IDs.\n\nNow start MRQ's dasbhoard with `mrq-dashboard \u0026` and go check your newly created queue and jobs on [localhost:5555](http://localhost:5555/#jobs)\n\nThey are ready to be dequeued by a worker. Start one with `mrq-worker` and follow it on the dashboard as it executes the queued jobs in parallel.\n\n```makefile\n$ mrq-worker fetches\n\n2014-12-18 15:52:57.362209 [INFO] Starting Gevent pool with 10 worker greenlets (+ report, logs, adminhttp)\n2014-12-18 15:52:57.388033 [INFO] redis: Connecting to Redis at 127.0.0.1...\n2014-12-18 15:52:57.389488 [DEBUG] mongodb_jobs: Connecting to MongoDB at 127.0.0.1:27017/mrq...\n2014-12-18 15:52:57.390996 [DEBUG] mongodb_jobs: ... connected.\n2014-12-18 15:52:57.391336 [DEBUG] mongodb_logs: Connecting to MongoDB at 127.0.0.1:27017/mrq...\n2014-12-18 15:52:57.392430 [DEBUG] mongodb_logs: ... connected.\n2014-12-18 15:52:57.523329 [INFO] Fetching 1 jobs from ['fetches']\n2014-12-18 15:52:57.567311 [DEBUG] Starting tasks.Fetch({u'url': u'http://www.google.com'})\n2014-12-18 15:52:58.670492 [DEBUG] Job 5492f771520d1887bfdf4b0f success: 1.135268s total\n2014-12-18 15:52:57.523329 [INFO] Fetching 1 jobs from ['fetches']\n2014-12-18 15:52:57.567747 [DEBUG] Starting tasks.Fetch({u'url': u'http://www.yahoo.com'})\n2014-12-18 15:53:01.897873 [DEBUG] Job 5492f771520d1887c2d7d2db success: 4.361895s total\n2014-12-18 15:52:57.523329 [INFO] Fetching 1 jobs from ['fetches']\n2014-12-18 15:52:57.568080 [DEBUG] Starting tasks.Fetch({u'url': u'http://www.wordpress.com'})\n2014-12-18 15:53:00.685727 [DEBUG] Job 5492f772520d1887c5b32881 success: 3.149119s total\n2014-12-18 15:52:57.523329 [INFO] Fetching 1 jobs from ['fetches']\n2014-12-18 15:52:57.523329 [INFO] Fetching 1 jobs from ['fetches']\n```\n\nYou can interrupt the worker with Ctrl-C once it is finished.\n\n## Going further\n\nThis was a preview on the very basic features of MRQ. What makes it actually useful is that:\n\n* You can run multiple workers in parallel. Each worker can also run multiple greenlets in parallel.\n* Workers can dequeue from multiple queues\n* You can queue jobs from your Python code to avoid using `mrq-run` from the command-line.\n\nThese features will be demonstrated in a future example of a simple web crawler.\n\n\n# More\n\nFull documentation is available on [readthedocs](http://mrq.readthedocs.org/en/latest/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpricingassistant%2Fmrq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpricingassistant%2Fmrq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpricingassistant%2Fmrq/lists"}