{"id":18123268,"url":"https://github.com/jay-johnson/celery-loaders","last_synced_at":"2025-10-20T06:28:11.819Z","repository":{"id":62561281,"uuid":"122780371","full_name":"jay-johnson/celery-loaders","owner":"jay-johnson","description":"Examples for Celery applications and task loading","archived":false,"fork":false,"pushed_at":"2018-11-02T19:53:46.000Z","size":22,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-28T11:05:27.672Z","etag":null,"topics":["celery","distributed-computing","docker","python","redis"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jay-johnson.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-02-24T21:14:53.000Z","updated_at":"2019-07-19T14:23:10.000Z","dependencies_parsed_at":"2022-11-03T15:16:11.756Z","dependency_job_id":null,"html_url":"https://github.com/jay-johnson/celery-loaders","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jay-johnson%2Fcelery-loaders","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jay-johnson%2Fcelery-loaders/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jay-johnson%2Fcelery-loaders/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jay-johnson%2Fcelery-loaders/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jay-johnson","download_url":"https://codeload.github.com/jay-johnson/celery-loaders/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246301963,"owners_count":20755512,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["celery","distributed-computing","docker","python","redis"],"created_at":"2024-11-01T07:08:55.076Z","updated_at":"2025-10-20T06:28:11.733Z","avatar_url":"https://github.com/jay-johnson.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Examples for Celery Applications and Task Loading\n=================================================\n\nThis is an example for running Celery applications that demonstrates how to load multiple task modules from different files using environment variables. It also shows how to utilize a ``get celery config`` approach that allows for decoupling inclusion of tasks from the initialization of the Celery application. It also includes a derived Celery task called ``CustomTask`` for showing how to handle ``on_success`` and ``on_failure`` task control events. I find this easier to manage my Celery workers and the applications that publish tasks to them.\n\nThis is not an official Celery project, it is just examples I use to test my workers and tasks.\n\nI hope you find it valuable.\n\nInstall\n-------\n\n::\n\n    pip install celery-loaders\n\nStart Redis\n-----------\n\nThis is only needed if redis is not running already on port 6739. It also requires having docker and docker-compose available to your user.\n\n::\n\n    ./run-redis.sh\n\nStart Celery Worker\n-------------------\n\nPlease run this from the base repository directory or the module paths to the Celery tasks will fail.\n\n::\n\n    ./run-celery-loaders-worker.sh \n    Starting Worker=celery_worker\n    celery worker -A celery_worker -c 1 -l INFO -n default@%h\n    2018-02-24 13:01:49,843 - worker - INFO - start - worker\n    2018-02-24 13:01:49,843 - worker - INFO - broker=redis://localhost:6379/9 backend=redis://localhost:6379/10 include_tasks=['celery_loaders.work_tasks.tasks', 'celery_loaders.work_tasks.always_fails_tasks']\n    2018-02-24 13:01:49,844 - get_celery_app - INFO - creating celery app=worker tasks=['celery_loaders.work_tasks.tasks', 'celery_loaders.work_tasks.always_fails_tasks']\n    2018-02-24 13:01:49,845 - worker - INFO - starting celery\n    \n    -------------- default@dev v4.1.0 (latentcall)\n    ---- **** ----- \n    --- * ***  * -- Linux-4.13.0-16-generic-x86_64-with-Ubuntu-17.10-artful 2018-02-24 13:01:49\n    -- * - **** --- \n    - ** ---------- [config]\n    - ** ---------- .\u003e app:         worker:0x7f384346e550\n    - ** ---------- .\u003e transport:   redis://localhost:6379/9\n    - ** ---------- .\u003e results:     redis://localhost:6379/10\n    - *** --- * --- .\u003e concurrency: 1 (prefork)\n    -- ******* ---- .\u003e task events: OFF (enable -E to monitor tasks in this worker)\n    --- ***** ----- \n    -------------- [queues]\n                    .\u003e celery           exchange=celery(direct) key=celery\n                    \n\n    [tasks]\n    . celery_loaders.work_tasks.always_fails_tasks.always_fails\n    . celery_loaders.work_tasks.tasks.do_some_work\n\n    2018-02-24 13:01:49,956 - celery.worker.consumer.connection - INFO - Connected to redis://localhost:6379/9\n    2018-02-24 13:01:49,963 - celery.worker.consumer.mingle - INFO - mingle: searching for neighbors\n    2018-02-24 13:01:50,981 - celery.worker.consumer.mingle - INFO - mingle: all alone\n    2018-02-24 13:01:51,013 - celery.apps.worker - INFO - default@dev ready.\n\nRun Success and Failure Tests\n-----------------------------\n\nRun this in another terminal with the ``celery_loaders`` pip available to the python 3 runtime.\n\n::\n\n    ./run-celery-loaders-tests.py \n    2018-02-24 13:01:56,768 - worker - INFO - start - worker\n    2018-02-24 13:01:56,768 - worker - INFO - broker=redis://localhost:6379/9 backend=redis://localhost:6379/10 include_tasks=['celery_loaders.work_tasks.tasks', 'celery_loaders.work_tasks.always_fails_tasks']\n    2018-02-24 13:01:56,768 - worker - INFO - broker=redis://localhost:6379/9 backend=redis://localhost:6379/10\n    2018-02-24 13:01:56,769 - get_celery_app - INFO - creating celery app=worker tasks=['celery_loaders.work_tasks.tasks', 'celery_loaders.work_tasks.always_fails_tasks']\n    2018-02-24 13:01:56,781 - worker - INFO - calling task - success testing\n    2018-02-24 13:01:56,822 - worker - INFO - calling task=celery_loaders.work_tasks.tasks.do_some_work - success job_id=3dcd5066-46fe-43cc-b0c6-8cff5499a7b1\n    2018-02-24 13:01:56,822 - worker - INFO - calling task - failure testing\n    2018-02-24 13:01:56,823 - worker - INFO - calling task=celery_loaders.work_tasks.always_fails_tasks.always_fails - failure job_id=c6f91e65-f541-40ad-9226-eaf97b223723\n    2018-02-24 13:01:56,823 - worker - INFO - end - worker\n\nConfirm the Celery Worker Processed the Tasks\n---------------------------------------------\n\n::\n\n    2018-02-24 13:01:56,822 - celery.worker.strategy - INFO - Received task: celery_loaders.work_tasks.tasks.do_some_work[3dcd5066-46fe-43cc-b0c6-8cff5499a7b1]  \n    2018-02-24 13:01:56,824 - tasks - INFO - task - do_some_work - start work_dict={'user_id': 1}\n    2018-02-24 13:01:56,824 - tasks - INFO - task - do_some_work - done\n    2018-02-24 13:01:56,828 - custom_task - INFO - custom_task SUCCESS - retval=True task_id=3dcd5066-46fe-43cc-b0c6-8cff5499a7b1 args=[{'user_id': 1}] kwargs={}\n    2018-02-24 13:01:56,828 - celery.app.trace - INFO - Task celery_loaders.work_tasks.tasks.do_some_work[3dcd5066-46fe-43cc-b0c6-8cff5499a7b1] succeeded in 0.004043873999762582s: True\n    2018-02-24 13:01:57,007 - celery.worker.strategy - INFO - Received task: celery_loaders.work_tasks.always_fails_tasks.always_fails[c6f91e65-f541-40ad-9226-eaf97b223723]  \n    2018-02-24 13:01:57,009 - always_fails_tasks - INFO - task - always_fails - start work_dict={'test_failure': 'Should fail now 2018-02-24T13:01:56.781481'}\n    2018-02-24 13:01:57,010 - custom_task - ERROR - custom_task FAIL - exc=Should fail now 2018-02-24T13:01:56.781481 args=[{'test_failure': 'Should fail now 2018-02-24T13:01:56.781481'}] kwargs={}\n    2018-02-24 13:01:57,010 - celery.app.trace - ERROR - Task celery_loaders.work_tasks.always_fails_tasks.always_fails[c6f91e65-f541-40ad-9226-eaf97b223723] raised unexpected: Exception('Should fail now 2018-02-24T13:01:56.781481',)\n    Traceback (most recent call last):\n    File \"/home/jay/.venvs/celeryloaders/lib/python3.6/site-packages/celery/app/trace.py\", line 374, in trace_task\n        R = retval = fun(*args, **kwargs)\n    File \"/home/jay/.venvs/celeryloaders/lib/python3.6/site-packages/celery/app/trace.py\", line 629, in __protected_call__\n        return self.run(*args, **kwargs)\n    File \"/opt/redten/stack/jwt/webapp/celery_examples/build/lib/celery_loaders/work_tasks/always_fails_tasks.py\", line 32, in always_fails\n        \"simulating a failure\"))\n    Exception: Should fail now 2018-02-24T13:01:56.781481\n\nDebug a Celery Task from the Command Line\n-----------------------------------------\n\n#.  Start a named Celery Task and pass in data from a file containing a testing JSON payload\n\n    ::\n\n        ./run-celery-task.py -f tests/data/user_lookup_test.json -t celery_loaders.work_tasks.tasks.do_some_work -w 5.0\n        2018-02-26 00:11:50,192 - run-celery-task - INFO - start - run-celery-task\n        2018-02-26 00:11:50,192 - run-celery-task - INFO - connecting Celery=run-celery-task broker=redis://localhost:6379/9 backend=redis://localhost:6379/10 tasks=['celery_loaders.work_tasks.tasks']\n        2018-02-26 00:11:50,192 - get_celery_app - INFO - creating celery app=run-celery-task tasks=['celery_loaders.work_tasks.tasks']\n        2018-02-26 00:11:50,207 - run-celery-task - INFO - app.broker_url=redis://localhost:6379/9 calling task=celery_loaders.work_tasks.tasks.do_some_work data={'user_id': 1}\n        2018-02-26 00:11:50,264 - run-celery-task - INFO - calling task=celery_loaders.work_tasks.tasks.do_some_work - started job_id=561c80bc-0bab-4387-a842-4372d11c8291\n        2018-02-26 00:11:50,264 - run-celery-task - INFO - task=561c80bc-0bab-4387-a842-4372d11c8291 - waiting seconds=5.0 for results\n        2018-02-26 00:11:50,268 - run-celery-task - INFO - task=celery_loaders.work_tasks.tasks.do_some_work - success job_id=561c80bc-0bab-4387-a842-4372d11c8291 task_result={'job_results': 'some response key=c60adfdc-e27c-41f4-8440-666be599ab4a'}\n        2018-02-26 00:11:50,268 - run-celery-task - INFO - end - run-celery-task\n\n#.  Verify the Worker Processed the Task\n\n    ::\n\n        2018-02-26 00:11:50,265 - celery.worker.strategy - INFO - Received task: celery_loaders.work_tasks.tasks.do_some_work[561c80bc-0bab-4387-a842-4372d11c8291]  \n        2018-02-26 00:11:50,266 - tasks - INFO - task - do_some_work - start work_dict={'user_id': 1}\n        2018-02-26 00:11:50,266 - tasks - INFO - task - {'job_results': 'some response key=c60adfdc-e27c-41f4-8440-666be599ab4a'} - result=do_some_work done\n        2018-02-26 00:11:50,267 - custom_task - INFO - custom_task SUCCESS - retval={'job_results': 'some response key=c60adfdc-e27c-41f4-8440-666be599ab4a'} task_id=561c80bc-0bab-4387-a842-4372d11c8291 args=[{'user_id': 1}] kwargs={}\n        2018-02-26 00:11:50,268 - celery.app.trace - INFO - Task celery_loaders.work_tasks.tasks.do_some_work[561c80bc-0bab-4387-a842-4372d11c8291] succeeded in 0.001375271000142675s: {'job_results': 'some response key=c60adfdc-e27c-41f4-8440-666be599ab4a'}\n\nDevelopment\n-----------\n\n::\n\n    virtualenv -p python3 ~/.venvs/celeryloaders \u0026\u0026 source ~/.venvs/celeryloaders/bin/activate \u0026\u0026 pip install -e .\n    \nRun tests\n\n::\n\n    python setup.py test    \n\nLinting\n-------\n\n::\n\n    flake8 .\n    pycodestyle .\n\nLicense\n=======\n\nApache 2.0\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjay-johnson%2Fcelery-loaders","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjay-johnson%2Fcelery-loaders","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjay-johnson%2Fcelery-loaders/lists"}