Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/maciej-gol/tenant-schemas-celery
https://github.com/maciej-gol/tenant-schemas-celery
Last synced: 16 days ago
JSON representation
- Host: GitHub
- URL: https://github.com/maciej-gol/tenant-schemas-celery
- Owner: maciej-gol
- License: mit
- Created: 2014-07-10T19:34:22.000Z (over 10 years ago)
- Default Branch: master
- Last Pushed: 2024-05-21T06:32:43.000Z (6 months ago)
- Last Synced: 2024-05-23T10:03:20.197Z (6 months ago)
- Language: Python
- Size: 141 KB
- Stars: 166
- Watchers: 8
- Forks: 36
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Funding: .github/FUNDING.yml
- License: LICENSE
Awesome Lists containing this project
README
tenant-schemas-celery
=====================
Celery application implementation that allows celery tasks to cooperate with
multi-tenancy provided by [django-tenant-schemas](https://github.com/bernardopires/django-tenant-schemas) and
[django-tenants](https://github.com/tomturner/django-tenants) packages.
This project might not seem frequently updated, but it just has all the functionality needed. Issues and questions are answered quickly.
Installation
------------
```bash
$ pip install tenant-schemas-celery
$ pip install django-tenants
```
Usage
-----
* Define a celery app using given `CeleryApp` class.
```python
import os
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'app.settings')
from django.conf import settings
from tenant_schemas_celery.app import CeleryApp as TenantAwareCeleryApp
app = TenantAwareCeleryApp()
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
```
This assumes a fresh Celery 5.2.0 application. For previous versions, the key is to create a new `CeleryApp` instance that will be used to access task decorator from.
* Replace your `@task` decorator with `@app.task`
```python
from celery import shared_task
from django.db import connection
from myproject.celery import app
from tenant_schemas_celery.task import TenantTask
@app.task
def my_task():
print(connection.schema_name)
@shared_task(base=TenantTask, bind=True)
def my_shared_task():
print("foo")
```
* Run celery worker (`myproject.celery` is where you've defined the `app` variable)
```bash
$ celery worker -A myproject.celery
```
* Post registered task. The schema name will get automatically added to the task's arguments.
```python
from myproject.tasks import my_task
my_task.delay()
```
The `TenantTask` class transparently inserts current connection's schema into
the task's kwargs. The schema name is then popped from the task's kwargs in
`task_prerun` signal handler, and the connection's schema is changed
accordingly.
### Multiple databases support
New in `2.0.0`.
Inside your celery tasks you might be working with multiple databases. You might want to change the schema for
all of the connections, or just a subset of them.
You can now use the `CELERY_TASK_TENANT_CACHE_SECONDS` django setting, or `TASK_TENANT_CACHE_SECONDS` celery setting, or
the `tenant_databases` attribute of the `TenantTask` to a list of database names (the key in the `settings.DATABASES` dictionary).
If not set, the settings defaults to `["default"]`.
### Tenant objects cache
New in `0.3.0`.
Every time a celery task is executed, the tenant object of the `connection` object is being refetched.
For some use cases, this can introduce significant performance hit.
In such scenarios, you can pass `tenant_cache_seconds` argument to the `@app.task()` decorator. This will
cause the tenant objects to be cached for given period of time. `0` turns this off. You can also enable cache globally
by setting celery's `TASK_TENANT_CACHE_SECONDS` (app-specific, usually it's `CELERY_TASK_TENANT_CACHE_SECONDS`).
```python
@app.task(tenant_cache_seconds=30)
def some_task():
...
```
Celery beat integration
-----------------------
In order to run celery beat tasks in a multi-tenant environment, you've got two options:
- Use a dispatching task that will send a task for each tenant
- Use a custom scheduler
i.e: Let's say that you would like to run a `reset_remaining_jobs` tasks periodically, for every tenant that you have.
### Dispatcher task pattern
You can schedule one dispatcher task that will iterate over all schemas and send that task within the schema's context:
```python
from django_tenants.utils import get_tenant_model, tenant_context
from django_tenant_schemas.utils import get_tenant_model, tenant_context
@app.task
def reset_remaining_jobs_in_all_schemas():
for tenant in get_tenant_model().objects.exclude(schema_name='public'):
with tenant_context(tenant):
reset_remaining_jobs_in_schema.delay()
@app.task
def reset_remaining_jobs_in_schema():
```
The `reset_remaining_jobs_in_all_schemas` task (called the dispatch task) should be registered in your celery beat schedule. The `reset_remaining_jobs_in_schema` task should be called from the dispatch task.
That way you have full control over which schemas the task should be scheduled in.
### Custom scheduler
If you are using the standard `Scheduler` or `PersistentScheduler` classes provided by `celery`, you can transition to using this package's `TenantAwareScheduler` or `TenantAwarePersistentScheduler` classes. You should then specify the scheduler you want to use in the celery beat config or your invocation to `beat`. i.e:
```bash
celery -A proj beat --scheduler=tenant_schemas_celery.scheduler.TenantAwareScheduler
```
#### Caveats
- There's a chance that celery beat will try to run a task for a newly created tenant before its migrations are ready, which could potentially lead to deadlocks. This is specially true with big projects with a lot of migrations and very frequent tasks (i.e: every minute). In order to mitigate it, one could do the following:
1. Subclass any of `TenantAwareScheduler` or `TenantAwarePersistentScheduler` and override the `get_queryset` method to match your definition of "ready" tenants. For example, imagine that you had a `ready` flag in your tenant model. You could do the following:
```python
# tenants_app/scheduler.py
class MyTenantAwareScheduler(TenantAwareScheduler):
@classmethod
def get_queryset(cls):
return super().get_queryset().filter(ready=True)
```
2. Use the new scheduler in your celery beat config or invocation:
```bash
celery -A proj beat --scheduler=tenants_app.scheduler.MyTenantAwareScheduler
```
- `TenantAwareSchedulerMixin` uses a subclass of `SchedulerEntry` that allows the user to provide specific schemas to run a task on. This might prove useful if you have a task you only want to run in the `public` schema or to a subset of your tenants. In order to use set that, you must configure `tenant_schemas` in the tasks definition as such:
```python
app.conf.beat_schedule = {
"my-task": {
"task": "myapp.tasks.my_task",
"schedule": schedules.crontab(minute="*/5"),
"tenant_schemas": ["public"]
}
}
```
Compatibility changes
=====================
The `>=3.0` series drop support for Python 3.7.
The `>=2.1` series drop support for `tenant-schemas`. It hasn't been maintainted for
a long time.
The `2.x` series support Python>=3.7.
The `1.x` series support Python>=3.6. Python 3.6 reached EOL 2021-12.
The `0.x` series are the last one to support Python<3.6.