{"id":23293720,"url":"https://github.com/jaypyles/pytask","last_synced_at":"2025-08-22T00:33:13.053Z","repository":{"id":268933803,"uuid":"905486163","full_name":"jaypyles/pytask","owner":"jaypyles","description":"Simple worker and task queue implementation in python. ","archived":false,"fork":false,"pushed_at":"2024-12-19T19:07:06.000Z","size":0,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-12-19T19:44:14.743Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jaypyles.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-18T23:35:22.000Z","updated_at":"2024-12-19T19:22:34.000Z","dependencies_parsed_at":"2024-12-19T19:44:18.267Z","dependency_job_id":"9813e8c6-3a49-4c11-bc3d-bb04b5794ca6","html_url":"https://github.com/jaypyles/pytask","commit_stats":null,"previous_names":["jaypyles/pytask"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaypyles%2Fpytask","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaypyles%2Fpytask/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaypyles%2Fpytask/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaypyles%2Fpytask/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jaypyles","download_url":"https://codeload.github.com/jaypyles/pytask/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230542252,"owners_count":18242333,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-20T06:16:49.337Z","updated_at":"2025-08-22T00:33:13.015Z","avatar_url":"https://github.com/jaypyles.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Logo Picture](https://github.com/jaypyles/pytask/blob/main/docs/worker.png)\n\n# pytask\n\nA simple sqlite3-based job queue with a worker. Main purpose is to run jobs in a queue. Jobs are not popped from the queue, which means the queue can act as a history.\n\n## Installation\n\n`pip install pytask-queue`\n\n## Usage\n\nThe worker will run the function `func` for each job. The function will be passed a `Job` object. Which means that you can alter the job object in the function, and the newly updated job will be saved to the queue. \n\n```python\n# python process 1\nfrom pytask import Queue, Job, SQLDataType, SQLColumnConditions\n\nqueue = Queue(schema=[\n    (\"foo\", SQLDataType.INTEGER, [SQLColumnConditions.NOT_NULL]), \n    (\"bar\", SQLDataType.TEXT, [SQLColumnConditions.NOT_NULL]), \n    (\"baz\", SQLDataType.JSON, [SQLColumnConditions.NOT_NULL])\n])\nqueue.insert(Job(data={\"foo\": 1, \"bar\": \"test\", \"baz\": {\"foo\": \"bar\"}}))\n```\n\n```python\n# python process 2\nfrom \u003crelative_file\u003e import queue\nfrom pytask import Job\n\ndef func(job: Job):\n    # Do something with job\n    job.data[\"foo\"] += 1\n\nworker = Worker(queue, func)\nworker.run()\n```\n\nCreating multiple queues or multiple workers is possible. Creating a new queue object won't actually create a new queue, it just creates a new connection to the queue. Which means you can have multiple queue objects pointing to the same queue, or you can use the same queue object for multiple workers.\n\nBe careful to avoid race conditions when using the same queue object for multiple workers.\n\n## Flags\n\nFlags are used to configure the behavior of the queue and worker.\n\nCurrent flags:\n\n- `auto_convert_json_keys`: If True, the queue will automatically convert JSON keys to strings. Useful for retrieving and manipulating JSON data.\n- `pop_after_processing`: If True, the job will be popped from the queue after processing.\n\n```python\nfrom pytask import Queue, Worker, Job, SQLDataType, SQLColumnConditions, Flags\n\nflags = Flags(auto_convert_json_keys=True, pop_after_processing=True)\nqueue = Queue(schema=[(\"foo\", SQLDataType.INTEGER, [SQLColumnConditions.NOT_NULL])], flags=flags)\n\nworker = Worker(queue, func, logger=logger)\nworker.run()\n```\n\n## Concurrent Worker\n\nThe concurrent worker is a worker that runs jobs in parallel. It uses a thread pool to run the jobs.\n\n```python\nfrom pytask import Queue, ConcurrentWorker, Job, SQLDataType, SQLColumnConditions\n\nworker = ConcurrentWorker(queue, func, logger=logger, interval=1, max_workers=16)\nworker.run()\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaypyles%2Fpytask","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjaypyles%2Fpytask","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaypyles%2Fpytask/lists"}