https://github.com/jizongfox/gpuqueues
A very simple GPU job scheduler - To run multiple jobs with assigned (limited) GPU resources in a dynamic way
https://github.com/jizongfox/gpuqueues
gpu gpu-scheduler machine-learning python scheduler
Last synced: 9 months ago
JSON representation
A very simple GPU job scheduler - To run multiple jobs with assigned (limited) GPU resources in a dynamic way
- Host: GitHub
- URL: https://github.com/jizongfox/gpuqueues
- Owner: jizongFox
- Created: 2019-10-29T01:01:35.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2024-03-31T21:41:05.000Z (over 2 years ago)
- Last Synced: 2025-01-07T04:13:28.020Z (over 1 year ago)
- Topics: gpu, gpu-scheduler, machine-learning, python, scheduler
- Language: Python
- Homepage:
- Size: 24.4 KB
- Stars: 12
- Watchers: 1
- Forks: 4
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# `GPUQueue` A very simple GPU tool - To run multiple jobs with assigned (limited) GPU resources
It provides very simple and basic function of dynamically utilize given GPUs with a large job array. It can be used to automatically identify the GPU that has been released by a newly-ended program.
### Examples
---
`python` interface
``` python
from gpu_queue import JobSubmitter
job_array = [
"python -c 'import os, time;print(\"GPU num utilized\",os.environ[\"CUDA_VISIBLE_DEVICES\"]);time.sleep(3)'",
"python -c 'import os, time;print(\"GPU num utilized\",os.environ[\"CUDA_VISIBLE_DEVICES\"]);time.sleep(2)'",
"python -c 'import os, time;print(\"GPU num utilized\",os.environ[\"CUDA_VISIBLE_DEVICES\"]);time.sleep(0.5)'",
"python -c 'import os, time;print(\"GPU num utilized\",os.environ[\"CUDA_VISIBLE_DEVICES\"]);time.sleep(0.5)'",
"python -c 'import os, time;print(\"GPU num utilized\",os.environ[\"CUDA_VISIBLE_DEVICES\"]);time.sleep(3)'",
"python -c 'import os, time;print(\"GPU num utilized\",os.environ[\"CUDA_VISIBLE_DEVICES\"]);time.sleep(1)'",
]
J = JobSubmitter(job_array, [0, 1, 2])
J.submit_jobs()
```
Output:
```
6 jobs has been saved
GPU num utilized 0
GPU num utilized 2
GPU num utilized 1
GPU num utilized 2
GPU num utilized 2
GPU num utilized 1
all jobs has been run
sucessful jobs: 6
failed jobs: 0
```
`gpuqueue` can be directly used in the bash
`Bash` interface
```bash
#!/usr/bin/env bash
# example of typical machine learning hyper-parameter tuning
# mean teacher for semi supervised learning
save_dir=cifar10/labeled_sample_4000/augment_img
EMA_decay=0.999
declare -a StringArray=(
"python classify_main.py Trainer.name=MeanTeacherTrainer Config=config/cifar_mt_config.yaml Trainer.save_dir=${save_dir}/meanteacherbaseline RegScheduler.max_value=0 Trainer.EMA_decay=${EMA_decay} "
"python classify_main.py Trainer.name=MeanTeacherTrainer Config=config/cifar_mt_config.yaml Trainer.save_dir=${save_dir}/meanteacher_0.1 RegScheduler.max_value=0.1 Trainer.EMA_decay=${EMA_decay} "
"python classify_main.py Trainer.name=MeanTeacherTrainer Config=config/cifar_mt_config.yaml Trainer.save_dir=${save_dir}/meanteacher_1 RegScheduler.max_value=1 Trainer.EMA_decay=${EMA_decay} "
"python classify_main.py Trainer.name=MeanTeacherTrainer Config=config/cifar_mt_config.yaml Trainer.save_dir=${save_dir}/meanteacher_10 RegScheduler.max_value=10 Trainer.EMA_decay=${EMA_decay} "
"python classify_main.py Trainer.name=MeanTeacherTrainer Config=config/cifar_mt_config.yaml Trainer.save_dir=${save_dir}/meanteacher_20 RegScheduler.max_value=20 Trainer.EMA_decay=${EMA_decay} "
"python classify_main.py Trainer.name=MeanTeacherTrainer Config=config/cifar_mt_config.yaml Trainer.save_dir=${save_dir}/meanteacher_50 RegScheduler.max_value=50 Trainer.EMA_decay=${EMA_decay} "
)
# just using 0 and 1 gpus for those jobs
gpuqueue "${StringArray[@]}" --available_gpus 0 1
# you may want to run 2 jobs on each gpu to fully use the memory? simple
gpuqueue "${StringArray[@]}" --available_gpus 0 1 0 1
```
---
### Log
logs are stored in automatically generated `log` folder.
---
### install
```bash
pip install -U gpuqueue
```