{"id":13936156,"url":"https://github.com/codepr/tasq","last_synced_at":"2026-01-29T12:08:07.902Z","repository":{"id":55888110,"uuid":"131074081","full_name":"codepr/tasq","owner":"codepr","description":"A simple task queue implementation to enqeue jobs on local or remote processes.","archived":false,"fork":false,"pushed_at":"2020-12-09T11:12:04.000Z","size":422,"stargazers_count":96,"open_issues_count":2,"forks_count":5,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-11-27T04:30:41.573Z","etag":null,"topics":["distributed-computing","distributed-job","distributed-systems","job-scheduler","queue","redis","remote-workers","scheduled-jobs","task-queue","zmq"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/codepr.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-04-25T23:23:49.000Z","updated_at":"2024-11-14T08:48:23.000Z","dependencies_parsed_at":"2022-08-15T08:40:36.905Z","dependency_job_id":null,"html_url":"https://github.com/codepr/tasq","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/codepr/tasq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codepr%2Ftasq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codepr%2Ftasq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codepr%2Ftasq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codepr%2Ftasq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/codepr","download_url":"https://codeload.github.com/codepr/tasq/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codepr%2Ftasq/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266019657,"owners_count":23864916,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-computing","distributed-job","distributed-systems","job-scheduler","queue","redis","remote-workers","scheduled-jobs","task-queue","zmq"],"created_at":"2024-08-07T23:02:25.349Z","updated_at":"2026-01-29T12:08:02.865Z","avatar_url":"https://github.com/codepr.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"Tasq\n====\n\n![tasq](https://github.com/codepr/tasq/workflows/tasq/badge.svg)\n\nVery simple distributed Task queue that allow the scheduling of job functions\nto be executed on local or remote workers. Can be seen as a Proof of Concept\nleveraging ZMQ sockets and cloudpickle serialization capabilities as well as a\nvery basic actor system to handle different loads of work from connecting\nclients. Originally it was meant to be just a brokerless job queue, recently\nI dove deeper on the topic and decided to add support for job persistence and\nextensions for Redis/RabbitMQ middlewares as well.\n\nThe main advantage of using a brokerless task queue, beside latencies is the\nlower level of complexity of the system. Additionally Tasq offer the\npossibility of launching and forget some workers on a network and schedule jobs\nto them without having them to know nothing about the code that they will run,\nallowing to define tasks dinamically, without stopping the workers. Obviously\nthis approach open up more risks of malicious code to be injected to the\nworkers, currently the only security measure is to sign serialized data passed\nto workers, but the entire system is meant to be used in a safe environment.\n\n**NOTE:** The project is still in development stage and it's not advisable to\ntry it in production enviroments.\n\nFeatures:\n\n- Redis, RabbitMQ or ZMQ (brokerless) as backend\n- Delayed tasks and scheduled cron tasks\n- Configuration on disk\n- Actor-based workers (I/O bound tasks)\n- Process queue workers (CPU bound tasks)\n\nTodo:\n\n- Check for pynacl for security on pickled data\n- Refactoring of bad parts\n- More debug (constant debugging)\n\n## Quickstart\n\nStarting a worker on a node using Redis as backend\n\n```sh\n$ tq redis-runner --log-level DEBUG\n2019-04-26 23:15:28 - tasq.remote.supervisor-17903: Worker type: Actor\n```\n\nIn a python shell\n\n**Using a queue object**\n\n```python\nPython 3.7.3 (default, Apr 26 2019, 21:43:19)\nType 'copyright', 'credits' or 'license' for more information\nIPython 7.4.0 -- An enhanced Interactive Python. Type '?' for help.\nWarning: disable autoreload in ipython_config.py to improve performance.\n\nIn [1]: import tasq\n\nIn [2]: tq = tasq.queue('redis://localhost:6379')\n\nIn [3]: def fib(n):\n   ...:     if n == 0:\n   ...:         return 0\n   ...:     a, b = 0, 1\n   ...:     for _ in range(n - 1):\n   ...:         a, b = b, a + b\n   ...:     return b\n   ...:\n\nIn [4]: # Asynchronous execution\nIn [5]: fut = tq.put(fib, 50, name='fib-async')\n\nIn [6]: fut\nOut[6]: \u003cTasqFuture at 0x7f2851826518 state=finished returned JobResult\u003e\n\nIn [7]: fut.unwrap()\nOut[7]: 12586269025\n\nIn [8]: res = tq.put_blocking(fib, 50, name='fib-sync')\n\nIn [9]: res.unwrap()\nOut[9]: 12586269025\n```\n\nScheduling jobs after a delay\n```python\n\nIn [10]: fut = tq.put(fib, 5, name='fib-delayed', delay=5)\n\nIn [11]: fut\nOut[11]: \u003cTasqFuture at 0x7f2951856418 state=pending\u003e\n\nIn [12]: # wait 5 seconds\n\nIn [13]: fut.unwrap()\nOut[13]: 5\n\nIn [14] tq.results\nOut[14] {'fib-async': \u003cTasqFuture at 0x7f2851826518 state=finished returned JobResult\u003e,\nOut[14]  'fib-sync': \u003cTasqFuture at 0x7f7d6e047268 state=finished returned JobResult\u003e\nOut[14]  'fib-delayed': \u003cTasqFuture at 0x7f2951856418 state=finished returned JobResult\u003e}\n```\n\nScheduling a task to be executed continously in a defined interval\n\n```python\nIn [15] tq.put(fib, 5, name='8_seconds_interval_fib', eta='8s')\n\nIn [16] tq.put(fib, 5, name='2_hours_interval_fib', eta='2h')\n```\n\nDelayed and interval tasks are supported even in blocking scheduling manner.\n\nTasq also supports an optional static configuration file, in the\n`tasq.settings.py` module is defined a configuration class with some default\nfields. By setting the environment variable `TASQ_CONF` it is possible to\nconfigure the location of the json configuration file on the filesystem.\n\nBy setting the `-c` flag it is possible to also set a location of a\nconfiguration to follow on the filesystem\n\n```sh\n$ tq worker -c path/to/conf/conf.json\n```\n\nA worker can be started by specifying the type of sub worker we want:\n\n```sh\n$ tq rabbitmq-worker --worker-type process\n```\nUsing `process` type subworker it is possible to use a distributed queue for\nparallel execution, usefull when the majority of the jobs are CPU bound instead\nof I/O bound (actors are preferable in that case).\n\nIf jobs are scheduled for execution on a disconnected client, or remote workers\nare not up at the time of the scheduling, all jobs will be enqeued for later\nexecution. This means that there's no need to actually start workers before job\nscheduling, at the first worker up all jobs will be sent and executed.\n\n### Security\n\nCurrently tasq gives the option to send pickled functions using digital sign in\norder to prevent manipulations of the sent payloads, being dependency-free it\nuses `hmac` and `hashlib` to generate digests and to verify integrity of\npayloads, planning to move to a better implementation probably using `pynacl`\nor something similar.\n\n## Behind the scenes\n\nEssentially it is possible to start workers across the nodes of a network\nwithout forming a cluster and every single node can host multiple workers by\nsetting differents ports for the communication.  Each worker, once started,\nsupport multiple connections from clients and is ready to accept tasks.\n\nOnce a worker receive a job from a client, it demand its execution to dedicated\nactor or process, usually selected from a pool according to a defined routing\nstrategy in the case of actor (e.g.  Round robin, Random routing or Smallest\nmailbox which should give a trivial indication of the workload of each actor\nand select the one with minimum pending tasks to execute) or using a simple\ndistributed queue across a pool of process in producer-consumer way.\n\n![Tasq master-workers arch](static/worker_model_2.png)\n\nAnother (pool of) actor(s) is dedicated to answering the clients with the\nresult once it is ready, this way it is possible to make the worker listening\npart unblocking and as fast as possible.\n\nThe reception of jobs from clients is handled by `ZMQ.PULL` socket while the\nresponse transmission handled by `ResponseActor` is served by `ZMQ.PUSH`\nsocket, effectively forming a dual channel of communication, separating ingoing\nfrom outgoing traffic.\n\n## Installation\n\nBeing a didactical project it is not released on Pypi yet, just clone the\nrepository and install it locally or play with it using `python -i` or\n`ipython`.\n\n```sh\n$ git clone https://github.com/codepr/tasq.git\n$ cd tasq\n$ pip install .\n```\n\nor, to skip cloning part\n\n```sh\n$ pip install git+https://github.com/codepr/tasq.git@master#egg=tasq\n```\n\n## Changelog\n\nSee the [CHANGES](CHANGES.md) file.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodepr%2Ftasq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcodepr%2Ftasq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodepr%2Ftasq/lists"}