{"id":16543172,"url":"https://github.com/codemation/easyjobs","last_synced_at":"2025-03-21T10:31:33.986Z","repository":{"id":39831861,"uuid":"324572567","full_name":"codemation/easyjobs","owner":"codemation","description":"A celery like jobs framework for managing and  distributing  async / non-async tasks ","archived":false,"fork":false,"pushed_at":"2021-05-23T11:40:15.000Z","size":5832,"stargazers_count":35,"open_issues_count":1,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-10-12T18:59:28.102Z","etag":null,"topics":["aiojobs","ampq","ampq-consumer","asyncio","distributed-computing","job-manager","job-scheduler","message-queue","rabbitmq"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/easyjobs/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/codemation.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-12-26T14:32:44.000Z","updated_at":"2024-04-17T04:50:56.000Z","dependencies_parsed_at":"2022-08-29T22:22:38.319Z","dependency_job_id":null,"html_url":"https://github.com/codemation/easyjobs","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codemation%2Feasyjobs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codemation%2Feasyjobs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codemation%2Feasyjobs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codemation%2Feasyjobs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/codemation","download_url":"https://codeload.github.com/codemation/easyjobs/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221814057,"owners_count":16884971,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aiojobs","ampq","ampq-consumer","asyncio","distributed-computing","job-manager","job-scheduler","message-queue","rabbitmq"],"created_at":"2024-10-11T18:59:33.437Z","updated_at":"2024-10-28T09:50:01.063Z","avatar_url":"https://github.com/codemation.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![](./images/logo.png)\n\n\u003ch3\u003e A jobs framework for managing and  distributing  async / non-async tasks \u003c/h3\u003e\n\n[![Documentation Status](https://readthedocs.org/projects/easyjobs/badge/?version=latest)](https://easyjobs.readthedocs.io/en/latest/?badge=latest) [![PyPI version](https://badge.fury.io/py/easyjobs.svg)](https://pypi.org/project/easyjobs/)\n\n\n## Quick Start\n\n```bash\n$ virtualenv -p python3.7 easy-job-env\n\n$ source easy-jobs-env/bin/activate\n\n(easy-rpc-env)$ pip install easyjobs\n```\n#\n\u003cb\u003eDocumentation\u003c/b\u003e: [easyjobs.readthedocs.io](https://easyjobs.readthedocs.io/)\n\n#\n\n## Usage - Jobs Manager\n```python\nimport asyncio, os\nfrom easyjobs.manager import EasyJobsManager\nfrom fastapi import FastAPI\n\nserver = FastAPI()\n\nos.environ['DB_PATH'] = '/mnt/jobs_database/'\n\n@server.on_event('startup')\nasync def startup():\n\n    job_manager = await EasyJobsManager.create(\n        server,\n        server_secret='abcd1234'\n    )\n```\n```bash\n$ uvicorn --host 0.0.0.0 --port 8220 job_manager:server\n```\n## Basic Usage - Worker\n\n```python\nimport asyncio, os\nfrom fastapi import FastAPI\nfrom easyjobs.workers.worker import EasyJobsWorker\n\nserver = FastAPI()\n\n@server.on_event('startup')\nasync def setup():\n    worker = await EasyJobsWorker.create(\n        server,\n        server_secret='abcd1234',\n        manager_host='0.0.0.0',\n        manager_port=8220,\n        manager_secret='abcd1234',\n        jobs_queue='ETL',\n        max_tasks_per_worker=5\n    )\n\n    every_minute = '* * * * *'\n    default_args = {'args': ['http://stats']}\n\n    async def get_data(url):\n        return {'a': 1, 'b': 2, 'c': 3}\n    async def load_db(data: dict):\n        await db.tables['transformed'].insert(**data)\n        return f\"data {data} loaded to db\"\n    async def send_email(address: str, message: str):\n        return f\"email sent to {address}\"\n\n    @worker.task(run_after=['transform'], schedule=every_minute, default_args=default_args)\n    async def extract(url: str):\n        print(f\"extract started\")\n        data = await get_data(url)\n        return {'data': data}\n    \n    @worker.task(run_after=['load'])\n    async def transform(data: dict):\n        print(f\"transform started\")\n        for k in data.copy():\n            data[k] = int(data[k]) + 2\n        return {'data': data}\n\n    @worker.task(on_failure='failure_notify', run_after=['compute'])\n    async def load(data):\n        print(f\"load started\")\n        await load_db(data)\n        return {'data': data}\n\n    @worker.task()\n    async def failure_notify(job_failed):\n        await send_email('admin@company.io', job_failed)\n        return job_failed\n\n    @worker.task()\n    async def deploy_environment():\n        print(f\"deploy_environment - started\")\n        await asyncio.sleep(5)\n        print(f\"deploy_environment - completed\")\n        return f\"deploy_environment - completed\"\n\n    @worker.task()\n    async def prepare_db():\n        print(f\"prepare_db - started\")\n        await asyncio.sleep(5)\n        print(f\"prepare_db - completed\")\n        return f\"deploy_environment - completed\"\n\n    @worker.task(run_before=['deploy_environment', 'prepare_db'])\n    async def configure_environment():\n        print(f\"pre_compute - starting\")\n        await asyncio.sleep(5)\n        print(f\"pre_compute - finished\")\n        return f\"pre_compute - finished\"\n\n    os.environ['WORKER_TASK_DIR'] = '/home/codemation/subprocesses'\n\n    @worker.task(subprocess=True, run_before=['configure_environment'])\n    async def compute(data: dict):\n        pass\n```\n```python\n#/home/codemation/subprocessescompute.py\nimport time\nfrom easyjobs.workers.task import subprocess\n\n@subprocess\ndef work(data: dict):\n    print(f\"starting heavy (blocking) computation on {data}\")\n    \"\"\"\n    insert work here\n    \"\"\"\n    time.sleep(5) # Blocking\n    return {'result': f'I slept for 5 seconds - blocking - with data', **data}\n    \nif __name__ == '__main__':\n    work()\n\n```\n\nStart Worker - With 5 Workers\n\n```Bash\n$ uvicorn --host 0.0.0.0 --port 8221 job_worker:server --workers=5\n```\n### Try it out\"\n    Visit Job Manager uri: \n    http://0.0.0.0:8220/docs\n\u003cbr\u003e\n\n### Task Flow\n\n![](./docs/images/task-flow.png)\n\n```bash\nextract started\nget_data: ['http://stats']\nextract finished\ntransform started\ntransform finished\nload started\nload finished\nprepare_db - started\ndeploy_environment - started\nprepare_db - completed\ndeploy_environment - completed\nconfigure_environment - starting\nconfigure_environment - finished\ntask subprocess called with ['/home/josh/Documents/python/easyjobs/compute.py', '0.0.0.0', '8220', 'abcd1234', 'c3076f7f-8b5c-11eb-9cba-6f9cd5406680', '{\"args\": [], \"kwargs\": {\"data\": {\"a\": 3, \"b\": 4, \"c\": 5}}}']\nstarting heavy computation on {'a': 3, 'b': 4, 'c': 5}\n```\n\n### pipeline\n\n![](./docs/images/pipeline.png)\n\n```bash\npipline started\ndeploy_environment - started\nprepare_db - started\ndeploy_environment - completed\nprepare_db - completed\nconfigure_environment - starting\nconfigure_environment - finished\ntask subprocess called with ['/home/josh/Documents/python/easyjobs/compute.py', '0.0.0.0', '8220', 'abcd1234', '23970547-8b5d-11eb-9cba-6f9cd5406680', '{\"args\": [], \"kwargs\": {\"data\": {\"test\": \"data\"}}}']\nstarting heavy computation on {'test': 'data'}\npipline - result is {'result': 'I slept for 5 seconds - blocking - with data', 'test': 'data'} - finished\n\n```\n## EasyJobs - Manager API\n![](./docs/images/ETL_API.png)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodemation%2Feasyjobs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcodemation%2Feasyjobs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodemation%2Feasyjobs/lists"}