{"id":20105490,"url":"https://github.com/nnseva/atasks","last_synced_at":"2025-03-02T17:45:55.739Z","repository":{"id":146501730,"uuid":"211362270","full_name":"nnseva/atasks","owner":"nnseva","description":"Asynchronous Distributed Task Queue","archived":false,"fork":false,"pushed_at":"2019-10-15T21:01:10.000Z","size":34,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-13T05:07:07.645Z","etag":null,"topics":["amqp","async","asynchronous","await","distributed","python","python-library","python3","queue-tasks","queue-workers","server","task-manager","task-queue","task-runner","task-schedule"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nnseva.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-27T16:41:09.000Z","updated_at":"2019-10-15T21:01:12.000Z","dependencies_parsed_at":null,"dependency_job_id":"1e516ae8-ab46-48d4-a28c-63d47c46b74a","html_url":"https://github.com/nnseva/atasks","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nnseva%2Fatasks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nnseva%2Fatasks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nnseva%2Fatasks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nnseva%2Fatasks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nnseva","download_url":"https://codeload.github.com/nnseva/atasks/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241549088,"owners_count":19980475,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amqp","async","asynchronous","await","distributed","python","python-library","python3","queue-tasks","queue-workers","server","task-manager","task-queue","task-runner","task-schedule"],"created_at":"2024-11-13T17:47:10.448Z","updated_at":"2025-03-02T17:45:55.719Z","avatar_url":"https://github.com/nnseva.png","language":"Python","readme":"# ATasks\n\nATasks is an asynchronous distributed task queue system.\n\nEvery task is defined as an asynchronous coroutine. We call such a task `atask`:\na(synchronous) task.\n\n`atask` looks like a usual asynchronous coroutine. It may be awaited using\n`await` syntax, and controlled by the `asyncio` package.\n\nThe `atask` may await other coroutines and `atask`s. Because of asynchronous\nnature of `atask` it doesn't block a thread evaluating `atask` and\nallows easy and transparent task decomposition as usual asynchronous\nprocedure, including sequential and parallel awaiting of other `atask`s.\n\n## Installation\n\n*Stable version* from the PyPi package repository\n\n```bash\npip install atasks\n```\n\n*Last development version* from the GitHub source version control system\n```\npip install git+git://github.com/nnseva/atasks.git\n```\n\n## Initializiation\n\nBefore execution some number of core objects should be constructed and initialized.\n\n```python\nfrom atasks.transport.backends.amqp import AMQPTransport\nfrom atasks.router import get_router\nfrom atasks.codecs import PickleCodec\n\n...\n    PickleCodec()\n    transport = AMQPTransport()\n    await transport.connect()\n\n    if mode == 'server':\n        router = get_router()\n        await router.activate(transport)\n```\n\n### Codec\n\nCodec determines a way to encode and decode objects passed through the network.\nIt should support as many types as it can.\n\nThe `atasks.codecs.PickleCodec` provided by the package uses standard python `pickle` package.\nIt is universal but not always safe solution.\n\n```python\nfrom atasks.codecs import PickleCodec\n\n...\n    PickleCodec()\n```\n\nUser can inherit `atasks.codecs.Codec` as a base class and create an own codec implementation.\nJust replace all methods generating `NotImplementedError`. Note that most of methods are asynchronous.\n```python\nfrom atasks.codecs import Codec\n\nclass MyCodec(Codec):\n    async def encode(self, obj):\n        ...\n    async def decode(self, content):\n        ...\n```\n\nTo activate a codec, yu need just create an instance of it. The codec is installed\ninto the system while construction.\n\n### Transport\n\nTransport determines the method of sending requests and returning results\nfrom awaiter to the performing coroutine and back to support awaiting\n`atask`s among a network.\n\nThe `atasks.transport.base.LoopbackTransport` provided by the package passes\nall requests back to the awaiter thread only. It doesn't allow `atask`s\nperforming distribution among several processes or even threads. You can\nuse it for the testing purposes.\n\nThe `atasks.transport.backends.amqp.AMQPTransport` provided by the package passes\nrequests through the RabbitMQ or other AMQP broker to any ATasks worker started\non the same or another host.\n\nAfter creation a transport instance, the asynchronous `connect()` method of just\ncreated instance should be awaited.\n\n```python\n    from atasks.transport/base import LoopbackTransport\n    from atasks.transport.backends.amqp import AMQPTransport\n\n    ...\n    if transport == 'loopback':\n        LoopbackTransport()\n    elif transport == 'amqp':\n        AMQPTransport()\n\n    await transport.connect()\n```\n\nOther transport kinds may be implemented later.\n\nUser can inherit `atasks.transport.base.Transport` as a base class and create an own\ntransport implementation. Just replace all methods generating `NotImplementedError`. Note that\nmost of methods are asynchronous.\n\n```python\nfrom atasks.transport.base import Transport\n\nclass MyTransport(Transport):\n\n    async def connect(self):\n        ...\n\n    async def disconnect(self):\n        ...\n\n    async def send_request(self, name, content):\n        ...\n```\n\n### Router\n\nRouter determines a way how the reference looks like, how it is awaited,\nwhat data are passed over the network etc. Router is a core of the ATasks package.\n\nThe `atasks.router.Router` is an only default router implementation.\n\nUser can inherit `atasks.router.Router` and create an own\nrouter implementation if necessary.\n\nAs a rule, you don't need to do it. In this case, you can just\nuse `get_router()` function to get a default router instance.\n\n```python\nfrom atasks.router import get_router\n\n...\n    router = get_router()\n```\n\n### Client and Server\n\nIf your application should send requests only, no any\nother actions required on the initialization stage.\n\nServer application which listens to events should\nalso activate a transport to receive requests:\n\n```python\n    server = AMQPTransport()\n\n    ...\n    router = get_router()\n    await router.activate(server)\n```\n\n\n\n## Markup an asynchronous distributed task\n\nDecorator `atasks.tasks.atask` is used to markup the asynchronous coroutine\n(or even synchronous returning `future` object) as an asynchronous distributed\ntask.\n\nNote that the first call to the wrapper creates a default router. You should\ncreate your own Router (or ancestor) instance before the first call\nto the wrapper if necessary.\n\n```python\n@atask\nasync def some_task(a):\n    ...\n```\n\nClient and server should use the same module defining `atask`s as a rule.\n\nIn order to await `atask` the `atask` name is used. Default name is determined\nby the coroutine name and containing module. You can replace a default name\nusing additional `name` parameter of the decorator:\n\n```python\n@atask(name=\"some_other_name\")\nasync def some_task(a):\n    ...\n```\n\n## Awaiting evaluation of the asynchronous distributed task\n\nThe `atask` is awaited as a usual coroutine. You can use `await` keyword, or\nget a `future` calling `atask` synchronously and control future using `asyncio` module.\n\n\n```python\n@atask\nasync def some_task(a):\n    ret = await some_other_task(a)\n\n@atask\nasync def some_other_task(a):\n    ...\n\nasync def not_a_task_just_coro():\n    a = await some_task(42)\n    ...\n```\n\n## Namespaces\n\nObjects may be instantiated in separate namespaces. Just\npass an additional `namespace=...` parameter to:\n\n- constructor of codec, transport, or route object\n- atask decorator\n- `get_route`, `get_transport`, or `get_codec` function\n\nOne namespace is completely separated from anoher. Every\nnamespace uses it's own set of router, transport, and codec,\nso init them separately for every namespace which is used\nin your application.\n\nThe default namespace has a name `default`.\n\nYou can await task from one namespace in another.\n\n```python\n@atask(namespace='one')\nasync def some_task():\n    await some_other_task()\n    ...\n\n@atask(namespace='other')\nasync def some_other_task():\n    ....\n```\n\n## Commands\n\nThe package uses Django management subsystem to provide command-line interface.\n\nDjango project using `django_atasks` application has the following command:\n\n```bash\npython manage.py run_atask file-or-module [options here]\n```\n\nThe command runs any `file-or-module` referenced in the command line which contains\n`@atask` definitions and optional `aiomain` asynchronous coroutine. The\noptional `aiomain` coroutine is evaluated when the file is running.\nAll options passed to the command are passed then to the `aiomain` keyword parameters.\n\nThe `run_atask` management command initializes all necessary objects (as\ndescribed above) to run module in three available modes: `server`, `client`,\nand `loopback`. The `loopback` mode allows to use the same process instance\nas server and client simultaneously.\n\nNote that if you use dedicated `server` process instance, you should not use\n`loopback` transport (which is not appropriate to reach the dedicated server\nin this case). Use `amqp` (or other interprocess transport) instead.\n\nThe module naming is different slightly depending on what you use in command line,\neither file name, or module name. Use the same module naming starting server\nand client to avoid misnaming of `atask`s.\n\nYou can start several modules simultaneously in one process instance\nenlisting them all in the command line.\n\nYou can start several server process instances, the client will then request them\nin arbitrary order.\n\nCall the `help` command to see the command details.\n\nSee `dev/tests/scenarios.py` file as an example of the file which can be called\nby the `run_atask` management command.\n\n## Inspiration\n\nThe idea of ATasks has been inspired by `asyncio`, [Celery](https://docs.celeryproject.org/en/latest)\nand [aiotasks](https://github.com/cr0hn/aiotasks) packages.\n\nThe main advantages of the ATasks comparing with Celery:\n- asynchronous task evaluation instead of synchronous tasks\n- free combining of `atask` awaits inside another `atask` using `await`\n- easy awaiting an `atask` and getting a result\n- parallelization using standard asynchronous syntax\n- no any restriction for recurrent `await`s\n\nThe main advantages of the ATasks comparing with aiotasks:\n- easier getting a result (`await` instead of `async with`)\n- full transparency - the only difference from usual\n  coroutine `await` is distributing `atasks` evaluation\n  among a network\n- actual development\n\nThe main disadvantages comparing with Celery and aiotasks:\n- `delay()`, `send()`, `async_call()`, `s()` etc. syntax is not available,\n  and will never be implemented\n\nUsual scenarios see in the [scenarios.py](dev/tests/scenarios.py) file.\n\n## Where the atask is evaluated\n\nAfter the `atask` is started, it is running in one thread from the beginning\nto the end. Other `atask`s may share the same thread in an asynchronous manner.\n\nOn the other side, another `atask` called from the first one may be\nrunning on any ATasks worker, on the same as the first one, or another\nworker and host, depending on the decision taken on the transport layer,\nand present ATask workers connected to the same transport layer.\n\nThe point where the `atask` is `await`ed is the only point of taking\na decision, where the `await`ed `atask` should run. The\ntransport layer takes this decision.\n\nThe ATasks application can issue remote `await`s immediately after\ntransport `connect()`. The ATask application receives remote\n`await`s after the `activate()` call of the `Router`.\n\nThe `LoopbackTransport` always passes all `await`s immediately to\ncoroutines in the same thread. It may be used for testing purposes.\n\nOther `Transport`s may allow remote `await`s inside a process,\nor a host, or passed among a network.\n\nThe `AMQPTransport` allows using RabbitMQ (or analogue) to\npass remote `await`s among a network to any number\nof instances.\n\n## How to track `atask`\n\n???\n\n## When the `atask` is crashed\n\nThe awaiting coroutine will take an exception if the `atask` is crashed\nwith exception. The exception should be serializable using codec.\n\n## When the worker evaluating `atask` is crashed\n\n???\n\n## How to await `atask` in synchronous program\n\n???\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnnseva%2Fatasks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnnseva%2Fatasks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnnseva%2Fatasks/lists"}