https://github.com/luocfprime/labtasker-plugin-script-generate
This is a plugin for Labtasker that helps you to automatically split your bash script into submit script and job script.
https://github.com/luocfprime/labtasker-plugin-script-generate
Last synced: 7 months ago
JSON representation
This is a plugin for Labtasker that helps you to automatically split your bash script into submit script and job script.
- Host: GitHub
- URL: https://github.com/luocfprime/labtasker-plugin-script-generate
- Owner: luocfprime
- Created: 2025-03-26T12:08:20.000Z (10 months ago)
- Default Branch: main
- Last Pushed: 2025-04-09T09:51:21.000Z (9 months ago)
- Last Synced: 2025-06-12T18:36:50.889Z (7 months ago)
- Language: Python
- Homepage:
- Size: 23.4 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# labtasker-plugin-script-generate
This is a plugin for [Labtasker](https://github.com/luocfprime/labtasker) that
helps you to automatically split your bash script into submit script and job script.
## Install
```bash
pip install labtasker-plugin-script-generate
```
## Example
You can find the code in the [demo](./demo) directory.
Imagine you have a bash wrapper script for your machine learning experiment. To divide the script into a submit script
and a job script, simply add a few special comments (e.g., `#@submit`, `#@task`, `#@end`).
```diff
#!/bin/bash
export CUDA_HOME=/usr/local/cuda-12.1
+#@submit
for dataset in imagenet cifar10 mnist; do
for model in resnet50 vit transformer; do
LOG_DIR=/path/to/logs/$dataset/$model
+ #@task
python train.py --dataset $dataset \
--model $model \
--cuda-home $CUDA_HOME \
--log-dir $LOG_DIR
+ #@end
done
done
+#@end
echo "done"
```
Run:
```bash
labtasker generate original.sh
```
You will see `original_submit.sh` and `original_run.sh` generated at the same directory, with
the following content:
### `original_submit.sh`:
```bash
#!/bin/bash
# This script is generated by Labtasker from original.sh
export CUDA_HOME=/usr/local/cuda-12.1
for dataset in imagenet cifar10 mnist; do
for model in resnet50 vit transformer; do
LOG_DIR=/path/to/logs/$dataset/$model
labtasker task submit -- --CUDA_HOME "$CUDA_HOME" --LOG_DIR "$LOG_DIR" --dataset "$dataset" --model "$model"
done
done
echo "done"
```
### `original_run.sh`:
```bash
#!/bin/bash
# This script is generated by Labtasker from original.sh
export CUDA_HOME=/usr/local/cuda-12.1
LABTASKER_TASK_SCRIPT=$(mktemp)
cat <<'LABTASKER_LOOP_EOF' > "$LABTASKER_TASK_SCRIPT"
CUDA_HOME=%(CUDA_HOME)
LOG_DIR=%(LOG_DIR)
dataset=%(dataset)
model=%(model)
python train.py --dataset $dataset \
--model $model \
--cuda-home $CUDA_HOME \
--log-dir $LOG_DIR
LABTASKER_LOOP_EOF
labtasker loop --executable /bin/bash --script-path $LABTASKER_TASK_SCRIPT
echo "done"
```