{"id":13416028,"url":"https://github.com/Bogdanp/django_dramatiq","last_synced_at":"2025-03-14T23:31:14.753Z","repository":{"id":26468801,"uuid":"108956841","full_name":"Bogdanp/django_dramatiq","owner":"Bogdanp","description":"A Django app that integrates with Dramatiq.","archived":false,"fork":false,"pushed_at":"2024-06-03T07:17:41.000Z","size":222,"stargazers_count":349,"open_issues_count":36,"forks_count":76,"subscribers_count":14,"default_branch":"master","last_synced_at":"2024-10-29T15:47:59.587Z","etag":null,"topics":["django","python","queue"],"latest_commit_sha":null,"homepage":"https://dramatiq.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Bogdanp.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-10-31T06:51:36.000Z","updated_at":"2024-10-26T20:24:49.000Z","dependencies_parsed_at":"2024-06-03T09:06:26.621Z","dependency_job_id":"125e6b84-3635-4e9d-b94a-c04d08d86b51","html_url":"https://github.com/Bogdanp/django_dramatiq","commit_stats":null,"previous_names":[],"tags_count":30,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bogdanp%2Fdjango_dramatiq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bogdanp%2Fdjango_dramatiq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bogdanp%2Fdjango_dramatiq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bogdanp%2Fdjango_dramatiq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Bogdanp","download_url":"https://codeload.github.com/Bogdanp/django_dramatiq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242939969,"owners_count":20209881,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["django","python","queue"],"created_at":"2024-07-30T21:00:53.768Z","updated_at":"2025-03-14T23:31:14.747Z","avatar_url":"https://github.com/Bogdanp.png","language":"Python","funding_links":[],"categories":["Third-Party Packages","Python","Tasks","Django"],"sub_categories":["Task Queues","Tools","Repositories"],"readme":"# Django Dramatiq\n\n![Python Version](https://img.shields.io/pypi/pyversions/django-dramatiq)\n![Django Versions](https://img.shields.io/pypi/frameworkversions/django/django-dramatiq)\n[![Build Status](https://github.com/Bogdanp/django_dramatiq/actions/workflows/ci.yml/badge.svg)](https://github.com/Bogdanp/django_dramatiq/actions/workflows/ci.yml)\n[![PyPI version](https://badge.fury.io/py/django-dramatiq.svg)](https://badge.fury.io/py/django-dramatiq)\n[![License](https://img.shields.io/badge/License-Apache_2.0-orange.svg)](https://opensource.org/licenses/Apache-2.0)\n\nSeemlessly integrate [Dramatiq][dramatiq] with your Django project!\n\n# Contents\n\n- [Installation](#installation)\n- [Getting Started](#getting-started)\n- [Testing](#testing)\n- [Middleware](#middleware)\n- [Advanced Usage](#advanced-usage)\n- [Third-Party Support](#third-party-support)\n- [Example App](#example)\n\n## Installation\n\nTo install, ensure both Django Dramatiq and Dramatiq are installed, along with RabbitMQ:\n\n    pip install django-dramatiq 'dramatiq[rabbitmq]'\n\nOr with Redis:\n\n    pip install django-dramatiq 'dramatiq[redis]'\n\nIf you would like to install with `watch`:\n\n    pip install django-dramatiq 'dramatiq[rabbitmq, watch]'\n\n\nAdd `django_dramatiq` to installed apps *before* any of your custom\napps:\n\n``` python\nINSTALLED_APPS = [\n    \"django_dramatiq\",\n\n    \"myprojectapp1\",\n    \"myprojectapp2\",\n    # etc...\n]\n```\n\nConfigure your broker in `settings.py`:\n\n``` python\nDRAMATIQ_BROKER = {\n    \"BROKER\": \"dramatiq.brokers.rabbitmq.RabbitmqBroker\", \n    \"OPTIONS\": {\n        \"url\": \"amqp://localhost:5672\",\n    },\n    \"MIDDLEWARE\": [\n        \"dramatiq.middleware.Prometheus\",\n        \"dramatiq.middleware.AgeLimit\",\n        \"dramatiq.middleware.TimeLimit\",\n        \"dramatiq.middleware.Callbacks\",\n        \"dramatiq.middleware.Retries\",\n        \"django_dramatiq.middleware.DbConnectionsMiddleware\",\n        \"django_dramatiq.middleware.AdminMiddleware\",\n    ]\n}\n\n# Defines which database should be used to persist Task objects when the\n# AdminMiddleware is enabled.  The default value is \"default\".\nDRAMATIQ_TASKS_DATABASE = \"default\"\n```\n\n## Getting Started\n\n### Declaring tasks\n\nDjango Dramatiq will auto-discover tasks defined in `tasks` modules in\neach of your installed apps.  For example, if you have an app named\n`customers`, your tasks for that app should live in a module called\n`customers.tasks`:\n\n``` python\nimport dramatiq\n\nfrom django.core.mail import send_mail\n\nfrom .models import Customer\n\n@dramatiq.actor\ndef email_customer(customer_id, subject, message):\n    customer = Customer.get(pk=customer_id)\n    send_mail(subject, message, \"webmaster@example.com\", [customer.email])\n```\n\nYou can override the name of the `tasks` module by setting one or more\nnames in settings:\n\n``` python\nDRAMATIQ_AUTODISCOVER_MODULES = [\"tasks\", \"services\"]\n```\n\n### Running workers\n\nDjango Dramatiq comes with a management command you can use to\nauto-discover task modules and run workers:\n\n```sh\n    python manage.py rundramatiq\n```\n\nBy default, `rundramatiq` will adjust the number of processes/threads used\nby Dramatiq based on the number of detected CPUs: one process will be launched\nper CPU, and each process will have 8 worker threads.\n\nThe default number of processes, threads per process can be overridden through\nenvironment variables, which take precedence over the defaults:\n\n```sh\n    export DRAMATIQ_NPROCS=2 DRAMATIQ_NTHREADS=2\n    python manage.py rundramatiq\n```\n\nOr alternatively through command line arguments, which take precedence over the\ndefaults and any environment variables:\n\n```sh\n    python manage.py rundramatiq -p 2 -t 2\n```\n\nThis is useful e.g. to facilitate faster Dramatiq restarts in your development\nenvironment.\n\nIf your project for some reason has apps with modules named `tasks` that\nare not intended for use with Dramatiq, you can ignore them:\n\n``` python\nDRAMATIQ_IGNORED_MODULES = (\n    'app1.tasks',\n    'app2.tasks',\n    'app3.tasks.utils',\n    'app3.tasks.utils.*',\n    ...\n)\n```\n\nThe wildcard detection will ignore all sub modules from that point on. You\nwill need to ignore the module itself if you don't want the `__init__.py` to\nbe processed.\n\n### Results Backend\n\nYou may also configure a result backend:\n\n``` python\nDRAMATIQ_RESULT_BACKEND = {\n    \"BACKEND\": \"dramatiq.results.backends.redis.RedisBackend\",\n    \"BACKEND_OPTIONS\": {\n        \"url\": \"redis://localhost:6379\",\n    },\n    \"MIDDLEWARE_OPTIONS\": {\n        \"result_ttl\": 1000 * 60 * 10\n    }\n}\n```\n\n## Testing\n\nYou should have a separate settings file for test.  In that file,\noverwrite the broker to use Dramatiq's [StubBroker][stubbroker]:\n\n``` python\nDRAMATIQ_BROKER = {\n    \"BROKER\": \"dramatiq.brokers.stub.StubBroker\",\n    \"OPTIONS\": {},\n    \"MIDDLEWARE\": [\n        \"dramatiq.middleware.AgeLimit\",\n        \"dramatiq.middleware.TimeLimit\",\n        \"dramatiq.middleware.Callbacks\",\n        \"dramatiq.middleware.Pipelines\",\n        \"dramatiq.middleware.Retries\",\n        \"django_dramatiq.middleware.DbConnectionsMiddleware\",\n        \"django_dramatiq.middleware.AdminMiddleware\",\n    ]\n}\n```\n\n#### Using [pytest-django][pytest-django]\n\nIn your `conftest` module set up fixtures for your broker and a\nworker:\n\n``` python\nimport dramatiq\nimport pytest\n\n@pytest.fixture\ndef broker():\n    broker = dramatiq.get_broker()\n    broker.flush_all()\n    return broker\n\n@pytest.fixture\ndef worker(broker):\n    worker = dramatiq.Worker(broker, worker_timeout=100)\n    worker.start()\n    yield worker\n    worker.stop()\n```\n\nIn your tests, use those fixtures whenever you want background tasks\nto be executed:\n\n``` python\ndef test_customers_can_be_emailed(transactional_db, broker, worker, mailoutbox):\n    customer = Customer(email=\"jim@gcpd.gov\")\n    # Assuming \"send_welcome_email\" enqueues an \"email_customer\" task\n    customer.send_welcome_email()\n\n    # Wait for all the tasks to be processed\n    broker.join(\"default\")\n    worker.join()\n\n    assert len(mailoutbox) == 1\n    assert mailoutbox[0].subject == \"Welcome Jim!\"\n```\n\n\n\u003e [!NOTE]  \n\u003e If your tests rely on the results of the actor, you may experience inconsistent results. Due to the nature of the worker and test running in seperate threads, the test DB state may be different. \n\u003e \n\u003e To solve this you need to add the addtional `@pytest.mark.django_db(transaction=True)` decorator. \n\n#### Using unittest\n\nA simple test case has been provided that will automatically set up the\nbroker and worker for each test, which are accessible as attributes on\nthe test case. Note that `DramatiqTestCase` inherits\n[`django.test.TransactionTestCase`][transactiontestcase].\n\n\n```python\nfrom django.core import mail\nfrom django.test import override_settings\nfrom django_dramatiq.test import DramatiqTestCase\n\n\nclass CustomerTestCase(DramatiqTestCase):\n\n    @override_settings(EMAIL_BACKEND='django.core.mail.backends.locmem.EmailBackend')\n    def test_customers_can_be_emailed(self):\n        customer = Customer(email=\"jim@gcpd.gov\")\n        # Assuming \"send_welcome_email\" enqueues an \"email_customer\" task\n        customer.send_welcome_email()\n\n        # Wait for all the tasks to be processed\n        self.broker.join(customer.queue_name)\n        self.worker.join()\n\n        self.assertEqual(len(mail.outbox), 1)\n        self.assertEqual(mail.outbox[0].subject, \"Welcome Jim!\")\n```\n\n\n## Advanced Usage\n\n### Cleaning up old tasks\n\nThe `AdminMiddleware` stores task metadata in a relational DB so it's\na good idea to garbage collect that data every once in a while.  You\ncan use the `delete_old_tasks` actor to achieve this on a cron:\n\n``` python\nfrom django_dramatiq.tasks import delete_old_tasks\n\ndelete_old_tasks.send(max_task_age=60 * 60 * 24)\n```\n\n\n## Middleware\n\n\u003cdl\u003e\n  \u003cdt\u003edjango_dramatiq.middleware.DbConnectionsMiddleware\u003c/dt\u003e\n  \u003cdd\u003e\n    This middleware is vital in taking care of closing expired\n    connections after each message is processed.\n  \u003c/dd\u003e\n\n  \u003cdt\u003edjango_dramatiq.middleware.AdminMiddleware\u003c/dt\u003e\n  \u003cdd\u003e\n    This middleware stores metadata about tasks in flight to a\n    database and exposes them via the Django admin.\n  \u003c/dd\u003e\n\u003c/dl\u003e\n\n### Custom keyword arguments to Middleware\n\nSome middleware classes require dynamic arguments.  An example of this\nwould be the backend argument to `dramatiq.middleware.GroupCallbacks`.\n\nTo do this, you might add the middleware to your `settings.py`:\n\n```python\nDRAMATIQ_BROKER = {\n    ...\n    \"MIDDLEWARE\": [\n        ...\n        \"dramatiq.middleware.GroupCallbacks\",\n        ...\n    ]\n    ...\n}\n```\n\nNext, you need to extend `DjangoDramatiqConfig` to provide the\narguments for this middleware:\n\n```python\nfrom django_dramatiq.apps import DjangoDramatiqConfig\n\n\nclass CustomDjangoDramatiqConfig(DjangoDramatiqConfig):\n    @classmethod\n    def middleware_groupcallbacks_kwargs(cls):\n        return {\"rate_limiter_backend\": cls.get_rate_limiter_backend()}\n```\n\nNotice the naming convention, to provide arguments to\n`dramatiq.middleware.GroupCallbacks` you need to add a `@classmethod`\nwith the name `middleware_\u003cmiddleware_name\u003e_kwargs`, where\n`\u003cmiddleware_name\u003e` is the lowercase name of the middleware.\n\nFinally, add the custom app config to your `settings.py`, replacing\nthe existing `django_dramatiq` app config:\n\n```python\nINSTALLED_APPS = [\n    ...\n    \"yourapp.apps.CustomDjangoDramatiqConfig\",\n    ...\n]\n```\n\n\n## Third-Party Support\n\n#### Usage with [django-configurations]\n\nTo use django_dramatiq together with [django-configurations] you need\nto define your own `rundramatiq` command as a subclass of the one in\nthis package.\n\nIn `YOURPACKAGE/management/commands/rundramatiq.py`:\n\n``` python\nfrom django_dramatiq.management.commands.rundramatiq import Command as RunDramatiqCommand\n\n\nclass Command(RunDramatiqCommand):\n    def discover_tasks_modules(self):\n        tasks_modules = super().discover_tasks_modules()\n        tasks_modules[0] = \"YOURPACKAGE.dramatiq_setup\"\n        return tasks_modules\n```\n\nAnd in `YOURPACKAGE/dramatiq_setup.py`:\n\n``` python\nimport django\n\nfrom configurations.importer import install\n\ninstall(check_options=True)\ndjango.setup()\n```\n\n## Example\n\nYou can find an example application built with Django Dramatiq [here](/examples/basic/README.md).\n\n\n[dramatiq]: https://github.com/Bogdanp/dramatiq\n[pytest-django]: https://pytest-django.readthedocs.io/en/latest/index.html\n[stubbroker]: https://dramatiq.io/reference.html#dramatiq.brokers.stub.StubBroker\n[django-configurations]: https://github.com/jazzband/django-configurations/\n[transactiontestcase]: https://docs.djangoproject.com/en/dev/topics/testing/tools/#django.test.TransactionTestCase\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FBogdanp%2Fdjango_dramatiq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FBogdanp%2Fdjango_dramatiq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FBogdanp%2Fdjango_dramatiq/lists"}