{"id":24651785,"url":"https://github.com/jizongfox/gpuqueues","last_synced_at":"2025-10-07T11:31:23.645Z","repository":{"id":57435648,"uuid":"218174878","full_name":"jizongFox/GPUQueues","owner":"jizongFox","description":"A very simple GPU job scheduler - To run multiple jobs with assigned (limited) GPU resources in a dynamic way","archived":false,"fork":false,"pushed_at":"2024-03-31T21:41:05.000Z","size":25,"stargazers_count":12,"open_issues_count":2,"forks_count":4,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-07T04:13:28.020Z","etag":null,"topics":["gpu","gpu-scheduler","machine-learning","python","scheduler"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jizongFox.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2019-10-29T01:01:35.000Z","updated_at":"2024-10-10T15:28:52.000Z","dependencies_parsed_at":"2024-03-30T08:26:00.852Z","dependency_job_id":"ad0828d6-4771-495e-aa4f-7d925f9ac152","html_url":"https://github.com/jizongFox/GPUQueues","commit_stats":{"total_commits":8,"total_committers":1,"mean_commits":8.0,"dds":0.0,"last_synced_commit":"f7ecacfe42d1f8e2dc92fd4ed9c0442225ed329d"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jizongFox%2FGPUQueues","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jizongFox%2FGPUQueues/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jizongFox%2FGPUQueues/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jizongFox%2FGPUQueues/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jizongFox","download_url":"https://codeload.github.com/jizongFox/GPUQueues/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235621555,"owners_count":19019522,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gpu","gpu-scheduler","machine-learning","python","scheduler"],"created_at":"2025-01-25T19:13:05.300Z","updated_at":"2025-10-07T11:31:23.315Z","avatar_url":"https://github.com/jizongFox.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# `GPUQueue` A very simple GPU tool - To run multiple jobs with assigned (limited) GPU resources\n\nIt provides very simple and basic function of dynamically utilize given GPUs with a large job array. It can be used to automatically identify the GPU that has been released by a newly-ended program.\n\n### Examples\n\n---\n`python` interface\n``` python\nfrom gpu_queue import JobSubmitter\n\njob_array = [\n    \"python -c 'import os, time;print(\\\"GPU num utilized\\\",os.environ[\\\"CUDA_VISIBLE_DEVICES\\\"]);time.sleep(3)'\",\n    \"python -c 'import os, time;print(\\\"GPU num utilized\\\",os.environ[\\\"CUDA_VISIBLE_DEVICES\\\"]);time.sleep(2)'\",\n    \"python -c 'import os, time;print(\\\"GPU num utilized\\\",os.environ[\\\"CUDA_VISIBLE_DEVICES\\\"]);time.sleep(0.5)'\",\n    \"python -c 'import os, time;print(\\\"GPU num utilized\\\",os.environ[\\\"CUDA_VISIBLE_DEVICES\\\"]);time.sleep(0.5)'\",\n    \"python -c 'import os, time;print(\\\"GPU num utilized\\\",os.environ[\\\"CUDA_VISIBLE_DEVICES\\\"]);time.sleep(3)'\",\n    \"python -c 'import os, time;print(\\\"GPU num utilized\\\",os.environ[\\\"CUDA_VISIBLE_DEVICES\\\"]);time.sleep(1)'\",\n]\n\nJ = JobSubmitter(job_array, [0, 1, 2])\nJ.submit_jobs()\n```\nOutput:\n```\n6 jobs has been saved\nGPU num utilized 0\nGPU num utilized 2\nGPU num utilized 1\nGPU num utilized 2\nGPU num utilized 2\nGPU num utilized 1\nall jobs has been run\nsucessful jobs: 6\n\nfailed jobs: 0\n```\n`gpuqueue` can be directly used in the bash   \n`Bash` interface\n```bash\n#!/usr/bin/env bash\n\n# example of typical machine learning hyper-parameter tuning \n# mean teacher for semi supervised learning\n\nsave_dir=cifar10/labeled_sample_4000/augment_img\nEMA_decay=0.999\n\ndeclare -a StringArray=(\n\"python classify_main.py Trainer.name=MeanTeacherTrainer Config=config/cifar_mt_config.yaml Trainer.save_dir=${save_dir}/meanteacherbaseline  RegScheduler.max_value=0  Trainer.EMA_decay=${EMA_decay}  \"\n\"python classify_main.py Trainer.name=MeanTeacherTrainer Config=config/cifar_mt_config.yaml Trainer.save_dir=${save_dir}/meanteacher_0.1      RegScheduler.max_value=0.1  Trainer.EMA_decay=${EMA_decay} \"\n\"python classify_main.py Trainer.name=MeanTeacherTrainer Config=config/cifar_mt_config.yaml Trainer.save_dir=${save_dir}/meanteacher_1        RegScheduler.max_value=1  Trainer.EMA_decay=${EMA_decay}  \"\n\"python classify_main.py Trainer.name=MeanTeacherTrainer Config=config/cifar_mt_config.yaml Trainer.save_dir=${save_dir}/meanteacher_10       RegScheduler.max_value=10  Trainer.EMA_decay=${EMA_decay} \"\n\"python classify_main.py Trainer.name=MeanTeacherTrainer Config=config/cifar_mt_config.yaml Trainer.save_dir=${save_dir}/meanteacher_20       RegScheduler.max_value=20  Trainer.EMA_decay=${EMA_decay} \"\n\"python classify_main.py Trainer.name=MeanTeacherTrainer Config=config/cifar_mt_config.yaml Trainer.save_dir=${save_dir}/meanteacher_50       RegScheduler.max_value=50  Trainer.EMA_decay=${EMA_decay} \"\n)\n# just using 0 and 1 gpus for those jobs\ngpuqueue \"${StringArray[@]}\" --available_gpus 0 1\n\n# you may want to run 2 jobs on each gpu to fully use the memory? simple\ngpuqueue \"${StringArray[@]}\" --available_gpus 0 1 0 1\n\n```\n---\n### Log\nlogs are stored in automatically generated `log` folder.\n\n\n---\n### install \n```bash\npip install -U gpuqueue\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjizongfox%2Fgpuqueues","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjizongfox%2Fgpuqueues","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjizongfox%2Fgpuqueues/lists"}