{"id":20540472,"url":"https://github.com/eduzen/task_executor","last_synced_at":"2026-04-06T21:31:18.861Z","repository":{"id":45123739,"uuid":"148678171","full_name":"eduzen/task_executor","owner":"eduzen","description":"Task runner","archived":false,"fork":false,"pushed_at":"2022-07-15T18:40:58.000Z","size":16,"stargazers_count":2,"open_issues_count":2,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-06T04:17:41.698Z","etag":null,"topics":["celery","docker","docker-compose","python3","rabbitmq","redis"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eduzen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-09-13T18:07:56.000Z","updated_at":"2023-03-05T05:00:48.000Z","dependencies_parsed_at":"2022-09-22T17:30:55.902Z","dependency_job_id":null,"html_url":"https://github.com/eduzen/task_executor","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/eduzen/task_executor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eduzen%2Ftask_executor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eduzen%2Ftask_executor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eduzen%2Ftask_executor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eduzen%2Ftask_executor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eduzen","download_url":"https://codeload.github.com/eduzen/task_executor/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eduzen%2Ftask_executor/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31491096,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-06T17:22:55.647Z","status":"ssl_error","status_checked_at":"2026-04-06T17:22:54.741Z","response_time":112,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["celery","docker","docker-compose","python3","rabbitmq","redis"],"created_at":"2024-11-16T01:15:31.437Z","updated_at":"2026-04-06T21:31:18.830Z","avatar_url":"https://github.com/eduzen.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Task executor [![Build Status](https://travis-ci.org/eduzen/task_executor.svg?branch=master)](https://travis-ci.org/eduzen/task_executor) [![codecov](https://codecov.io/gh/eduzen/task_executor/branch/master/graph/badge.svg)](https://codecov.io/gh/eduzen/task_executor)\n\nFor this exercise, we chose [Celery](http://www.celeryproject.org/) that is an asynchronous\ntask queue/job queue based on distributed message passing. Tasks can execute asynchronously\n(in the background) or synchronously (wait until ready). Celery requires a message transport\nto send and receive message. We chose [Rabbitmq](https://www.rabbitmq.com/) because it works well with celery.\nOther broker could be [Redis](https://redis.io/), but for this exercise we used it as a memcachedb, a\nkind of persistent key-value store, for managing locks through all the tasks. We can use this\ndistributed lock to have our tasks try to acquire a non-blocking lock, and exit if the lock isn’t acquired.\n\n## Installation\n\nThis project runs with `docker` (you can use traditional `virtualenv` but it's prepared out of the box for `docker`).\nWe choose `python 3.6.4` and not `python 3.7` because `Celery` doesn't support it yet.\nAlso to manage docker, we are using [docker-compose](https://docs.docker.com/compose/).\n\n## Usage\n\nIf you already have `docker` and `docker-compose`, just run:\n\n```bash\nmake start\n\n# only test and flake8\nmake test\n```\n\nThis command will download the images and build them in a container.\n\n## Workers\n\nRight now the celery worker is configure to execute __1) no more than 3 tasks in parallel__.\nWe can change this changing the number of concurrency of the worker:\n\n```bash\ncelery worker --app=task_manager.celeryapp:app --concurrency=3 --loglevel=info\n```\n\nOther option is to run several workers:\n\n```bash\ndocker-compose scale worker=5\n```\n\n## Tasks\n\nCelery tasks have a custom decorator in order to achieve: __2) each target (dave for example) can only execute one task at once__.\nTo do this, we choose Redis instead of django cache though this last option is recommended [here](http://docs.celeryproject.org/en/latest/tutorials/task-cookbook.html#ensuring-a-task-is-only-executed-one-at-a-time)\nbecause if memcached (or some other non- persistent cache) is used and (1) the cache daemon crashes or\n(2) the cache key is culled before the appropriate expiration time / lock release,\nthen you have a race condition where two or more tasks could simultaneously acquire the task lock. We follow this recommendation:\n[this](http://loose-bits.com/2010/10/distributed-task-locking-in-celery.html)) but with some changes.\nSo, in case that the target is busy (locked), we will retry the task with a custom delay. Also the task has a rate limit configured.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feduzen%2Ftask_executor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feduzen%2Ftask_executor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feduzen%2Ftask_executor/lists"}