{"id":29134788,"url":"https://github.com/drprettyman/jobmaster","last_synced_at":"2026-02-16T22:05:09.297Z","repository":{"id":234503249,"uuid":"789029677","full_name":"DrPrettyman/jobmaster","owner":"DrPrettyman","description":"Job queue system for backend systems.","archived":false,"fork":false,"pushed_at":"2025-03-14T22:49:14.000Z","size":120,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-30T09:02:17.869Z","etag":null,"topics":["job-queue","job-scheduler","postgresql","python","sql"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DrPrettyman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-04-19T15:09:20.000Z","updated_at":"2025-03-14T22:49:18.000Z","dependencies_parsed_at":"2025-03-14T23:23:10.638Z","dependency_job_id":"c7887417-10cc-4e9d-bdbe-51b8821f20df","html_url":"https://github.com/DrPrettyman/jobmaster","commit_stats":null,"previous_names":["drprettyman/jobmaster"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/DrPrettyman/jobmaster","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DrPrettyman%2Fjobmaster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DrPrettyman%2Fjobmaster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DrPrettyman%2Fjobmaster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DrPrettyman%2Fjobmaster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DrPrettyman","download_url":"https://codeload.github.com/DrPrettyman/jobmaster/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DrPrettyman%2Fjobmaster/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262744804,"owners_count":23357481,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["job-queue","job-scheduler","postgresql","python","sql"],"created_at":"2025-06-30T09:01:59.846Z","updated_at":"2026-02-16T22:05:09.264Z","avatar_url":"https://github.com/DrPrettyman.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# JobMaster\n\nJobMaster is a simple job-queue system which allows Python back-end tasks to be triggered or scheduled using a webhook. \nIt works with any PostgreSQL database, and is designed to be simple to use and easy to integrate into your existing codebase.\n\nThe `deploy()` method creates a schema with all necessary tables and procedures in any PostgreSQL database. \nThen, add the `@task` decorator to any python functions — for example a function named `foo()` in module named `my_tasks`.\nIn your web app, you might have a button which passes the following procedure call to your PostgreSQL database:\n```postgresql\ncall jobmaster.insert_job('my_tasks', 'foo', 10, '{\"a\": 1, \"b\": 2}'::json)\n```\nWhenever your python script runs (for example, on a cron) the job will be retrived from the database and will run `my_tasks.foo(a=1, b=2)`. \n\nThere is a lot of additional functionality to explore using optional arguments of the `@task()` decorator. \n\n\n## Installation\n\nPublished to PyPi at https://pypi.org/project/jobmaster/.\n\nSimply use pip:\n\n```shell\npip install jobmaster\n```\n\n## Usage\n\nAny function in your project can become a JobMaster Task by using the `@task` decorator.\nFor example, you may have a function like this:\n```python\n# jmtests/awesome_things/things1.py\n\nfrom jobmaster import task\n\n@task\ndef foo(file_path: str, number: [1, 2, 3]):\n    \"\"\"\n    Write a number to a file.\n\n    :param file_path: the file to write to\n    :param number: the number to write\n    \"\"\"\n    with open(file_path, 'w') as f:\n        f.write(str(number))\n```\nThink of \"tasks\" as functions and \"jobs\" as instances of those functions with specific arguments.\n\nOnce you have properly configured JobMaster, this task will be registered with \n- `type_key='things1'` (the name of the module) \n- `task_key='foo'` (the name of the function)\n\nYou can add a job to the queue by calling the procedure\n```postgresql\ncall jobmaster.insert_job('things1', 'foo', 10, '{\"file_path\": \"/tmp/sum.txt\", \"number\": 2}'::json)\n```\nfrom wherever you have a connection to your database: from a different python script, from a web application, etc..\n`jobmaster.insert_job` takes 4 arguments:\n1. The type key of the task\n2. The task key of the task\n3. The priority of the job\n4. The arguments to pass to the task, json formatted\n\nSomewhere in your python project, you will have a script that pops jobs from the queue and executes them:\n\n```python\n# jmtests/__init__.py\nimport sqlalchemy\n\ndb_engine = sqlalchemy.create_engine(\"postgresql+pg8000://\", ...)\n```\n\n```python\n# jmtests/utils/jmutils.py\n\nfrom jobmaster import JobMaster\nfrom .. import db_engine\nfrom '\u003cany module with tasks\u003e' import *\n\njobmaster = JobMaster(db_engine=db_engine, _validate_dependencies=True)\n\n# Run jobs from the queue until the queue is empty\nif __name__ == '__main__':\n    jobs = jobmaster.run()\n```\nThis could be run in a loop, or in a cron job, or in a web server, etc..\n\nTasks can depend on other tasks, and JobMaster will automatically run them in the correct order.\n\n## Simple example\n\nIn one file you might have:\n```python\n# module nice_tasks.py\n\nfrom jobmaster import task, Dependency, same\n\n@task\ndef foo(file_path: str, number: int):\n    with open(file_path, 'w') as f:\n        f.write(str(number))\n    \n\n@task(dependencies=Dependency(foo, 6, file_path=same))\ndef bar(file_path: str, number: int, letter: ['A', 'B', 'C'] = 'A'):\n    with open(file_path, 'r') as f:\n        _n = int(f.read())\n        \n    with open(file_path, 'w') as f:\n        f.write(f\"{number + _n}{letter}\")\n\n@task(\n    process_limit=3,\n    dependencies=[\n        Dependency(foo, 2, number=10, file_path=same), \n        Dependency(bar, 2, file_path=same)\n    ]\n)\ndef baz(file_path: str):\n    with open(file_path, 'r') as f:\n        s = f.read()\n```\n\nAnd in another file you might have:\n```python\n# module main.py\n\nimport sqlalchemy\nfrom jobmaster import JobMaster\n\n# Create a SQLAlchemy engine for your PostgreSQL database\nmy_database_engine = sqlalchemy.create_engine(\"postgresql+pg8000://\", ...)\n\n# Create a JobMaster instance\njobmaster = JobMaster(db_engine=my_database_engine)\n```\n\n## Setup\n\nThe first time you run your code, you'll need to create the necessary tables in your database. You can do this by using the `JobMaster.deploy()` method:\n```python\n# module deploy_jobmaster.py\n\nfrom .main import jobmaster\n\n# Deploy the necessary tables\njobmaster.deploy()\n```\n\n```bash\npython3 deploy_jobmaster.py\n```\n\nThis will create a schema in your database called `jobmaster` and create the necessary tables and functions for JobMaster to work.\nIf you already have a schema called `jobmaster` in your database, you can specify a different schema name when creating the `JobMaster` instance:\n```python\njobmaster = JobMaster(db_engine=my_database_engine, schema=\"job_mistress\")\n```\n\nIf you change any of the task definitions, you must update the database by running `jobmaster.deploy()` again. Running `jobmaster.deploy(_reset=True)` will drop the schema and all tables then recreate them from scratch, losing any jobs you had in your queue.\n\n## Tasks\n\nWe have already introduced the `@task` decorator. \nThis can be used without arguments as in\n```python\n# module my_tasks.py\n\nfrom jobmaster import task\n\n@task\ndef foo(a: int, b: int):\n    # do something\n```\nThis will register the function `foo` as a task with type key `'my_tasks'` and task key `'foo'`.\nJobMaster will register the parameters `a` and `b`, and their types (both `int`) are infered from the function signature, it is therefore important to use python type hints.\n\n#### Optional parameters\n\nIf a parameter is optional, you can specify a default value for it:\n```python\n@task\ndef foo(a: int = 1, b: int = 2):\n    # do something\n```\n\n#### \"select-from\" parameters\n\nIf a parameter has a number of possible options, this should be specified in the type hint:\n```python\n@task\ndef foo(a: int, b: [1, 2, 3]):\n    # do something\n```\nthe argument values `1`, `2`, and `3` are the only valid values for `b`.\n\n#### \"write-all\" parameters\n\nFor such parameters, it may be desirable to insert a job to the queue which performs the task for all possible values of the parameter.\nThis can be achieved by specifying the relevant parameters in the `write_all` argument of the `@task` decorator:\n```python\n@task(write_all=['b'])\ndef foo(a: int, b: [1, 2, 3]):\n    # do something\n```\nthen calling\n```postgresql\ncall jobmaster.insert_job('my_tasks', 'foo', 10, '{\"a\": 1, \"b\": \"ALL\"}'::json)\n```\n(`'ALL'` in upper-case). When this job is executed, instead of actually executing the task `foo`, JobMaster will insert new a job for each possible value of `b` into the queue.\n\n#### Custom type keys\n\nThe type key is the name of the module by default, but you can specify a different type key by passing it as an argument to the decorator:\n```python\n@task(type_key='cool_tasks', write_all=['b'])\ndef foo(a: int, b: [1, 2, 3]):\n    # do something\n```\n\n#### Dependencies\n\nTasks can depend on other tasks. This is specified by passing `Dependency` object (or a list of `Dependency` objects) to the `dependencies` argument of the `@task` decorator:\n```python\nfrom jobmaster import task, Dependency, same\n\n@task(type_key='cool_tasks', write_all=['b'])\ndef foo(a: int, b: [1, 2, 3]):\n    # do something\n\n@task(\n    type_key='cool_tasks', \n    write_all=['b'],\n    dependencies=Dependency(foo, 2, a=1, b=same)\n)\ndef bar(a: int, b: [1, 2, 3], c: str):\n    # do something\n```\nThe `Dependency` object takes the task function, a time (in hours), and the arguments to pass to the task.\nIn this example, the task `bar` depends on the task `foo` with `a=1` and `b` the same as the `b` of the job for `bar`.\nWhen a job with task-type `\"bar\"` is popped from the queue, JobMaster will first check if there is a job for task-type `\"foo\"` with `a=1` and `b` the same as the `b` of the job for `bar`, which has been **completed in the past 2 hours**.\nIf there isn't, it will insert a job for task-type `\"foo\"` with `a=1` and `b` the same as the `b` of the job for `bar` into the queue, with a higher priority than the job for `bar`, then re-insert the job for `bar` into the queue.\n\n#### Process units\n\nJobMaster can limit jobs using process units. \n\nWhen a JobMaster object is initialised, the number of process units available on the system is passed as an argument:\n```python\njobmaster = JobMaster(db_engine=my_database_engine, system_process_units=1_000)\n```\nIf this argument is not specified, JobMaster will check the environment variable `JOBMASTER_SYSTEM_PROCESS_UNITS` for the number of process units available. If this is not set, or is not formatted like an integer, it will default to 10,000.\n\nYou can then specify the number of process units a task requires by passing an integer to the `process_units` argument of the `@task` decorator. If this argument is not specified, the task will require 1 process unit by default.\n\nYou can potentially think of process units as Megabytes of RAM. So if you want to restrict JobMaster to using 1GB, set `system_process_units=1_000` and set `process_units` for each task according to how much RAM you expect it to use in MB.\n\nWhen JobMaster attempts to pop a job from the queue, it checks the queue for all \"running\" jobs on the same system and sums their process units. This is subtracted from the system process units to get the available process units.\nThen, when checking the queue for \"waiting\" jobs to pop, there is a `WHERE process_units \u003c= available_process_units` clause.\n\nExample:\n```python\nfrom jobmaster import task, Dependency, same\n\n@task(process_units=10, write_all=['b'])\ndef foo(a: int, b: [1, 2, 3]):\n    # do something\n\n@task(\n    type_key='cool_tasks', \n    write_all=['b'],\n    dependencies=Dependency(foo, 2, a=1, b=same),\n    process_units=20\n)\ndef bar(a: int, b: [1, 2, 3], c: str):\n    # do something\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdrprettyman%2Fjobmaster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdrprettyman%2Fjobmaster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdrprettyman%2Fjobmaster/lists"}