{"id":15628985,"url":"https://github.com/pwwang/xqute","last_synced_at":"2026-04-17T00:05:54.142Z","repository":{"id":62590151,"uuid":"305843260","full_name":"pwwang/xqute","owner":"pwwang","description":"A job management system for python","archived":false,"fork":false,"pushed_at":"2025-04-18T15:25:24.000Z","size":1505,"stargazers_count":9,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-29T09:56:20.841Z","etag":null,"topics":["job-queue","job-scheduler","queue-tasks","queue-workers","task-management","task-manager","task-queue","task-runner","task-scheduler"],"latest_commit_sha":null,"homepage":"https://pwwang.github.io/xqute/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pwwang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-10-20T21:59:56.000Z","updated_at":"2025-04-18T04:36:49.000Z","dependencies_parsed_at":"2024-03-13T22:41:27.700Z","dependency_job_id":"0aedf2b1-7398-4d4d-9fb8-8254b2549be7","html_url":"https://github.com/pwwang/xqute","commit_stats":{"total_commits":41,"total_committers":1,"mean_commits":41.0,"dds":0.0,"last_synced_commit":"44ba7e95d5208e816c63aa37088e41f6e0670db6"},"previous_names":[],"tags_count":42,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pwwang%2Fxqute","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pwwang%2Fxqute/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pwwang%2Fxqute/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pwwang%2Fxqute/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pwwang","download_url":"https://codeload.github.com/pwwang/xqute/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251480069,"owners_count":21596016,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["job-queue","job-scheduler","queue-tasks","queue-workers","task-management","task-manager","task-queue","task-runner","task-scheduler"],"created_at":"2024-10-03T10:25:12.440Z","updated_at":"2026-04-17T00:05:54.119Z","avatar_url":"https://github.com/pwwang.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# xqute\n\nA job management system for Python, designed to simplify job scheduling and execution with support for multiple schedulers and plugins.\n\n## Features\n\n- Written in async for high performance\n- Plugin system for extensibility\n- Scheduler adaptor for various backends\n- Job retrying and pipeline halting on failure\n- Support for cloud-based working directories\n- Built-in support for Google Batch Jobs, Slurm, SGE, SSH, and container schedulers\n\n## Installation\n\n```shell\npip install xqute\n```\n\n## A Toy Example\n\n```python\nimport asyncio\nfrom xqute import Xqute\n\nasync def main():\n    # Initialize Xqute with 3 jobs allowed to run concurrently\n    xqute = Xqute(forks=3)\n    for _ in range(10):\n        await xqute.feed(['sleep', '1'])\n    await xqute.run_until_complete()\n\nif __name__ == '__main__':\n    asyncio.run(main())\n```\n\n### Daemon Mode (Keep Feeding)\n\nYou can also run Xqute in daemon mode, where jobs can be added continuously after starting:\n\n```python\nimport asyncio\nfrom xqute import Xqute\n\nasync def main():\n    xqute = Xqute(forks=3)\n\n    # Add initial job\n    await xqute.feed(['echo', 'Job 1'])\n\n    # Start in keep_feeding mode (returns immediately)\n    await xqute.run_until_complete(keep_feeding=True)\n\n    # Continue adding jobs dynamically\n    for i in range(2, 11):\n        await xqute.feed(['sleep', '1'])\n        await asyncio.sleep(0.1)  # Jobs can be added over time\n\n    # Signal completion and wait for all jobs to finish\n    await xqute.stop_feeding()\n\nif __name__ == '__main__':\n    asyncio.run(main())\n```\n\n**Tip:** Use `xqute.is_feeding()` to check if you need to call `stop_feeding()`.\n\n![xqute](./xqute.png)\n\n## API Documentation\n\nFull API documentation is available at: \u003chttps://pwwang.github.io/xqute/\u003e\n\n## Usage\n\n### Xqute Object\n\nAn `Xqute` object is initialized as follows:\n\n```python\nxqute = Xqute(...)\n```\n\nAvailable arguments are:\n\n- `scheduler`: The scheduler class or name (default: `local`)\n- `plugins`: Plugins to enable/disable for this session\n- `workdir`: Directory for job metadata (default: `./.xqute/`)\n- `forks`: Number of jobs allowed to run concurrently\n- `error_strategy`: Strategy for handling errors (e.g., `halt`, `retry`)\n- `num_retries`: Maximum number of retries when `error_strategy` is set to `retry`\n- `submission_batch`: Number of jobs to submit in a batch\n- `scheduler_opts`: Additional keyword arguments for the scheduler\n- `jobname_prefix`: Prefix for job names\n- `recheck_interval`: Interval to recheck job status. The actual interval will be `\u003crecheck_interval\u003e * \u003cxqute.defaults.SLEEP_INTERVAL_POLLING_JOBS\u003e`\n\n**Note:** The producer must be initialized within an event loop.\n\nTo add a job to the queue:\n\n```python\nawait xqute.feed(['echo', 'Hello, World!'])\n```\n\nTo run until all jobs complete:\n\n```python\n# Traditional mode - wait for all jobs to complete\nawait xqute.run_until_complete()\n\n# Or daemon mode - add jobs continuously\nawait xqute.run_until_complete(keep_feeding=True)\n# ... add more jobs ...\nawait xqute.stop_feeding()  # Signal completion and wait\n```\n\n### Using SGE Scheduler\n\n```python\nxqute = Xqute(\n    scheduler='sge',\n    forks=100,\n    scheduler_opts={\n        'qsub': '/path/to/qsub',\n        'qdel': '/path/to/qdel',\n        'qstat': '/path/to/qstat',\n        'q': '1-day',  # or qsub_q='1-day'\n    }\n)\n```\n\nKeyword arguments starting with `sge_` are interpreted as `qsub` options. For example:\n\n```python\n'l': ['h_vmem=2G', 'gpu=1']\n```\nwill be expanded in the job script as:\n\n```shell\n#$ -l h_vmem=2G\n#$ -l gpu=1\n```\n\n### Using Slurm Scheduler\n\n```python\nxqute = Xqute(\n    scheduler='slurm',\n    forks=100,\n    scheduler_opts={\n        'sbatch': '/path/to/sbatch',\n        'scancel': '/path/to/scancel',\n        'squeue': '/path/to/squeue',\n        'partition': '1-day',\n        'time': '01:00:00',\n    }\n)\n```\n\n### Using SSH Scheduler\n\n```python\nxqute = Xqute(\n    scheduler='ssh',\n    forks=100,\n    scheduler_opts={\n        'ssh': '/path/to/ssh',\n        'servers': {\n            'server1': {\n                'user': 'username',\n                'port': 22,\n                'keyfile': '/path/to/keyfile',\n                'ctrl_persist': 600,\n                'ctrl_dir': '/tmp',\n            }\n        }\n    }\n)\n```\n\n**Note:** SSH servers must share the same filesystem and use keyfile authentication.\n\n### Using Google Batch Jobs Scheduler\n\n```python\nxqute = Xqute(\n    scheduler='gbatch',\n    forks=100,\n    scheduler_opts={\n        'project': 'your-gcp-project-id',\n        'location': 'us-central1',\n        'gcloud': '/path/to/gcloud',\n        'taskGroups': [ ... ],\n    }\n)\n```\n\n### Using Container Scheduler\n\n```python\nxqute = Xqute(\n    scheduler='container',\n    forks=100,\n    scheduler_opts={\n        'image': 'docker://bash:latest',\n        'entrypoint': '/usr/local/bin/bash',\n        'bin': 'docker',\n        'volumes': '/host/path:/container/path',\n        'envs': {'MY_ENV_VAR': 'value'},\n        'remove': True,\n        'bin_args': ['--hostname', 'xqute-container'],\n    }\n)\n```\n\n### Plugins\n\nTo create a plugin for `xqute`, implement the following hooks:\n\n- `def on_init(scheduler)`: Called after the scheduler is initialized\n- `def on_shutdown(scheduler, sig)`: Called when the scheduler shuts down\n- `async def on_job_init(scheduler, job)`: Called when a job is initialized\n- `async def on_job_queued(scheduler, job)`: Called when a job is queued\n- `async def on_job_submitted(scheduler, job)`: Called when a job is submitted\n- `async def on_job_started(scheduler, job)`: Called when a job starts running\n- `async def on_job_polling(scheduler, job, counter)`: Called during job status polling\n- `async def on_job_killing(scheduler, job)`: Called when a job is being killed\n- `async def on_job_killed(scheduler, job)`: Called when a job is killed\n- `async def on_job_failed(scheduler, job)`: Called when a job fails\n- `async def on_job_succeeded(scheduler, job)`: Called when a job succeeds\n- `def on_jobcmd_init(scheduler, job) -\u003e str`: Called during job command initialization\n- `def on_jobcmd_prep(scheduler, job) -\u003e str`: Called before the job command runs\n- `def on_jobcmd_end(scheduler, job) -\u003e str`: Called after the job command completes\n\nTo implement a hook, use the `simplug` plugin manager:\n\n```python\nfrom xqute import simplug as pm\n\n@pm.impl\ndef on_init(scheduler):\n    ...\n```\n\n### Implementing a Scheduler\n\nTo create a custom scheduler, subclass the `Scheduler` abstract class and implement the following methods:\n\n```python\nfrom xqute import Scheduler\n\nclass MyScheduler(Scheduler):\n    name = 'mysched'\n\n    async def submit_job(self, job):\n        \"\"\"Submit a job and return its unique ID.\"\"\"\n\n    async def kill_job(self, job):\n        \"\"\"Kill a job.\"\"\"\n\n    async def job_is_running(self, job):\n        \"\"\"Check if a job is running.\"\"\"\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpwwang%2Fxqute","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpwwang%2Fxqute","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpwwang%2Fxqute/lists"}