{"id":20743098,"url":"https://github.com/datagreed/django-eb-sqs-worker","last_synced_at":"2025-06-10T16:11:11.558Z","repository":{"id":40050646,"uuid":"273086697","full_name":"DataGreed/django-eb-sqs-worker","owner":"DataGreed","description":"Django Background Tasks for Amazon Elastic Beanstalk","archived":false,"fork":false,"pushed_at":"2022-04-06T01:42:20.000Z","size":237,"stargazers_count":33,"open_issues_count":7,"forks_count":8,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-05-28T08:09:41.925Z","etag":null,"topics":["amazon","amazon-elastic-beanstalk","amazon-web-services","aws","beanstalk","beanstalk-worker","django","elastic","elastic-beanstalk","elasticbeanstalk","sqs","sqs-client","sqs-consumer","sqs-queue","worker"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DataGreed.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"github":["datagreed"]}},"created_at":"2020-06-17T22:01:18.000Z","updated_at":"2024-07-26T19:14:45.000Z","dependencies_parsed_at":"2022-06-26T10:10:20.716Z","dependency_job_id":null,"html_url":"https://github.com/DataGreed/django-eb-sqs-worker","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataGreed%2Fdjango-eb-sqs-worker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataGreed%2Fdjango-eb-sqs-worker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataGreed%2Fdjango-eb-sqs-worker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataGreed%2Fdjango-eb-sqs-worker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DataGreed","download_url":"https://codeload.github.com/DataGreed/django-eb-sqs-worker/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataGreed%2Fdjango-eb-sqs-worker/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259106728,"owners_count":22805941,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon","amazon-elastic-beanstalk","amazon-web-services","aws","beanstalk","beanstalk-worker","django","elastic","elastic-beanstalk","elasticbeanstalk","sqs","sqs-client","sqs-consumer","sqs-queue","worker"],"created_at":"2024-11-17T07:09:06.910Z","updated_at":"2025-06-10T16:11:11.519Z","avatar_url":"https://github.com/DataGreed.png","language":"Python","funding_links":["https://github.com/sponsors/datagreed"],"categories":[],"sub_categories":[],"readme":"[![Tests](https://github.com/DataGreed/django-eb-sqs-worker/workflows/Tests/badge.svg)](https://github.com/DataGreed/django-eb-sqs-worker/actions?query=workflow%3ATests)\n[![PyPI version](https://badge.fury.io/py/django-eb-sqs-worker.svg)](https://badge.fury.io/py/django-eb-sqs-worker)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n# django-eb-sqs-worker\n\nDjango Background Tasks for Amazon Elastic Beanstalk.\n\nCreated by Alexey \"DataGreed\" Strelkov.\n\n## Overview\n\n_django-eb-sqs-worker_ lets you handle background jobs on \n[Elastic Beanstalk Worker Environment](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html) \nsent via SQS and provides methods to send tasks to worker.\n\nYou can use the same Django codebase for both your Web Tier and Worker Tier environments and send tasks \nfrom Web environment to Worker environment. Amazon fully manages autoscaling for you. \n\nTasks are sent via [Amazon Simple Queue Service](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html) \nand are delivered to your worker with [Elastic Beanstalk's SQS daemon](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html#worker-daemon). \nPeriodic tasks are also supported. \n\nHere's the diagram of how tasks move through the system, tasks movement is represented by arrows:\n\n![](./docs/img/ebsqs_diag.png)\n\n## Installation\n\nInstall using pip (only python3.x+ is supported):\n\n```\npip install django-eb-sqs-worker\n```\n\nAdd `eb_sqs_worker` to `settings.INSTALLED_APPS`:\n```python\nINSTALLED_APPS = [\n    # ...\n    \"eb_sqs_worker\",\n]\n```\n\nAdd `localhost` to `settings.ALLOWED_HOSTS` so SQS Daemon can post tasks from \nthe queue to your worker:\n```python\nALLOWED_HOSTS = [\n    # ...\n    \"localhost\",\n]\n```\n\n\nUpdate your `settings.py` for both Worker and Web EB environments:\n```python\n# region where your elastic beanstalk environments are deployed, e.g. \"us-west-1\"\nAWS_EB_DEFAULT_REGION = \"your default region\"    \n# your aws access key id\nAWS_ACCESS_KEY_ID = \"insert your key id here\"\n# your aws access key \nAWS_SECRET_ACCESS_KEY = \"insert your key here\"\n# queue name to use - queues that don't exist will be created automatically\nAWS_EB_DEFAULT_QUEUE_NAME = \"any_queue_name_to_use\"\n```\n\nIn the settings file for your **Web** tier environment add the following setting \n(this is important due to possible security problems if you don't set this):\n```python\n# never set to True on Web environment. Use True only on Worker env and local development env\nAWS_EB_HANDLE_SQS_TASKS=False   \n```\n\nIn the setting files used by your **Worker** environments add the following setting:\n```python\n# never set to True on Web environment. Use True only on Worker env and local development env\nAWS_EB_HANDLE_SQS_TASKS=True\n```\n\nAdd eb-sqs-worker urls to your project's main `urls.py` module:\n```python\n# urls.py\n\nurlpatterns = [\n    # your url patterns\n    # ... \n]\n\nfrom eb_sqs_worker.urls import urlpatterns as eb_sqs_urlpatterns\nurlpatterns += eb_sqs_urlpatterns\n```\n\nNavigate to your Worker environment in Elastic Beanstalk Web console, then go to _Configuration \u003e Worker_\nand set HTTP path to `/sqs/`.\n\nYou should also select the queue to use here corresponding to your `AWS_EB_DEFAULT_QUEUE_NAME` \nor, if you prefer to use the autogenerated one, you can copy its name and set as your `AWS_EB_DEFAULT_QUEUE_NAME`.\nIf you don't see your `AWS_EB_DEFAULT_QUEUE_NAME` here, try sending first task to it (see *\"Queueing tasks\"* section)\nand it will be automatically created for you (you may need to reload the page for it to appear here). \n\nApply changes. \n\n\n\n## Usage\n\n### Simple way\n\n#### Defining Background Tasks\n\nTo define a job create a function decorated by `task` decorator:\n\n```python\nfrom eb_sqs_worker.decorators import task\n@task\ndef some_task(**kwargs):\n    # define your task here\n    print(f\"The decorated test task is being run with kwargs {kwargs} and will echo them back\")\n\n    return kwargs\n``` \n\nMake sure the module with your tasks is imported so they will register correctly. \n\nThe best practice is to do it as soon as django loads, e.g. in your app's `models.py` or in corresponding `AppConfig`. \n\n#### Queueing tasks\n\nIf the task was defined using `@task` decorator, you can send it to background queue like this:\n\n```python\n# sends the task to SQS queue where it will be automatically picked up and executed by worker\nsome_task(foo=\"bar\")    \n```\n\nYou can set `settings.AWS_EB_RUN_TASKS_LOCALLY` to `True` in development – this will force all tasks to execute\nlocally in sync mode without sending them to the queue. This is useful for testing.\n\nIf you need to execute the function synchronously just one time somewhere in your code without changing this setting, \nyou can do it like this: \n\n```python\n# runs the task function synchronously without sending it to the queue and returns result\nresult = some_task.execute(foo=\"bar\")   \n```\n\n**Note:** don't supply positional arguments to the task, always use keyword arguments.\n\n#### Periodic tasks\n\nPeriodic tasks are defined the same way as regular task, but it's better to supply a custom name for them:\n\n```python\nfrom eb_sqs_worker.decorators import task\n@task(task_name=\"some_periodic_task\")\ndef periodic_task():\n    # define your periodic task here\n    print(f\"Periodic test task is being run \")\n\n    return True\n``` \n\nAdd `cron.yaml` to the root of the project:\n\n```yaml\nversion: 1\ncron:\n - name: \"some_periodic_task\"\n   url: \"/sqs/\"\n   schedule: \"0 23 * * *\"\n```\n\nDeploy your project to elastic beanstalk and your task will run every day at 23:00. \n\nRefer to [the documentation](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html#worker-periodictasks) for more info on periodic tasks.\n\n**Note**: periodic tasks don't support arguments passing\n\n\n`#TODO describe` (add link to https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html#worker-periodictasks), explain configuration\n\n\n### Advanced way\n\n#### Defining Background Tasks\n\n`#TODO describe`\n#### Queueing tasks\n`#TODO describe`\n\n\n#### Periodic tasks\n\n`#TODO describe`\n\n### Interface reference\n\n## @task decorator\n\n`#TODO: add info on arguments`\n\n## send_task\n\n`#TODO: add info on arguments`\n\n## Settings\n\n### AWS_EB_HANDLE_SQS_TASKS\nIf set to `True`, tasks will be accepted and handled on this instance. If set to `False`, the URL for handling \ntasks will return 404. Defaults to `False`.\n\n**Important:** set this to `True` _only_ on your [Worker environment](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html)\n\n### AWS_EB_ENABLED_TASKS\nDictionary of enabled tasks. Routes task names to actual task methods.\n\nIf you register your tasks using the `task` decorator, you don't need to worry about this setting at all,\nit will be set automatically by the decorator. \n\nE.g.:\n\n```python\nAWS_EB_ENABLED_TASKS = {\n    # name used in serialization   # path to actual method that does the job\n    \"accounts_confirmation_email\": \"accounts.tasks.send_confirmation_email\",\n    \"analytics_track_event\": \"analytics.tasks.track_event\"\n}\n``` \n\n### AWS_EB_DEFAULT_REGION\n\nDefault Elastic Beanstalk Region. Use the one that your app id deployed in. \n\n### AWS_EB_DEFAULT_QUEUE_NAME\n\nName of the queue used by default. If the queue with specified name does not exist, it will be created\nautomatically when the first task is queued.\n\n### AWS_ACCESS_KEY_ID\n\nAmazon Access Key Id, refer to [the docs](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys)\n\n### AWS_SECRET_ACCESS_KEY\n\nAmazon Secret Access Key, refer to [the docs](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys)\n\n### AWS_EB_RUN_TASKS_LOCALLY\n\nIf set to true, all tasks will be run locally and synchronnously instead of being sent to SQS Queue. Defaults to `False`\n\n### AWS_EB_ALERT_WHEN_EXECUTES_LONGER_THAN_SECONDS\n\nSet this to the maximum number of seconds the job is supposed to run. If the job finishes requires more time to finish\nADMINS will be notified by email.\n\n## Security\n\nAlways set `AWS_EB_HANDLE_SQS_TASKS=False` on Web Tier Environment so the tasks could not be spoofed! \nWeb Tier environments are typically used for hosting publici websites and can be accessed by anoyone on the Internet, \nmeaning that anyone can send any jobs to your site if you leave this option on on Web environment.\n\nWorker environments can only be accessed internally, e.g. via SQS Daemon that POSTs, so `AWS_EB_HANDLE_SQS_TASKS=True` \nshould be set only on worker environments.\n\nUse [Elastic Beanstalk Environment properties](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/environments-cfg-softwaresettings.html#environments-cfg-softwaresettings-console) \nto supply different setting files for Web and Worker environments. See also: [docs on designating the Django settings](https://docs.djangoproject.com/en/3.0/topics/settings/#designating-the-settings)\n\n## Tips\n\n`#TODO`\n\n### Accessing Web Tier Database from Worker\n\nYou will probably want your worker environment to have access to the same database as your web tier environment.\n\nAssuming you have a web tier environment and a worker environment with the same Django apps deployed \n(if you don't have a worker environment, yet, you can create it using `eb create -t worker \u003cenvironment name\u003e`)\nand the web tier environment has an \n[attached database](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.managing.db.html) \nset up via Elastic Beanstalk with database connection settings populated from environmantal variables, \ndo the following:\n\n1. Open Elastic Beanstalk Web Console \n2. Navigate to your Web Tier environment \u003e Configuration \u003e Database\n3. Copy database connection settings. Note that the database password will not be shown here. If you don't remember it, \nyou can connect to the Web environment using `eb ssh` and getting it using `cat /opt/python/current/env`\n4. Navigate to your Worker environment \u003e  Configuration \u003e Software \u003e Edit\n5. Add environmental variables for DB connection that you've copied (`RDS_PORT`,`RDS_PASSWORD`,`RDS_USERNAME`, \n`RDS_DB_NAME`, `RDS_HOSTNAME`) and hit \"Apply\"\n6. Navigate to your Worker environment \u003e  Configuration \u003e Instances \u003e Edit\n7. Add security group corresponding to your Web Tier environment and hit \"Apply\", confirm changes. (If you are using a single-instance environment instead of a load-balanced one, take a look at [#5](https://github.com/DataGreed/django-eb-sqs-worker/issues/5) to make sure you don't run in security issues)\n8. Re-deploy the application using `eb deploy` to make sure that everything works as expected.\n\n### Delay abstraction\n\n`#TODO`\n\n### Using different cron files for different environments\n\n`#TODO`\n\n## Testing\n\n### Synchronous mode\n\nWhen developing on local machine it might be a good idea to set `AWS_EB_RUN_TASKS_LOCALLY=True`, so all the tasks \nthat should normally be sent to queue will be executed locally on the same machine in sync mode. This lets you test\nyour actual task methods in integration tests.\n\n### Testing django-eb-sqs-worker itself\n\nClone the repository.\n\n```\ngit clone https://github.com/DataGreed/django-eb-sqs-worker.git\n```\n\nInstall requirements (use python virtual environment)\n```\ncd django-eb-sqs-worker\npip install -r requirements.txt\n```\n\nRun tests\n```\nsh test.sh\n```\n\n\n## Contributing\n\nIf you would like to contribute, please make a Pull Request with the description of changes and add tests to cover\nthese changes.\n\nFeel free to open issues if you have any problems or questions with this package.\n\n\n# TODOs\n\n- take advantage of the new [environment link feature](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/environment-cfg-links.html)\n- add pickle serialization\n- finish readme\n- parse GET-parameters for periodic tasks? \n\n---\nSearch tags \n\nDjango Elastic Beanstalk Worker Web Tier Asynchronous celery async django-q Jobs Background Tasks SQS \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatagreed%2Fdjango-eb-sqs-worker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatagreed%2Fdjango-eb-sqs-worker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatagreed%2Fdjango-eb-sqs-worker/lists"}