{"id":20541347,"url":"https://github.com/link89/oh-my-batch","last_synced_at":"2025-04-14T08:43:40.704Z","repository":{"id":257885216,"uuid":"873056221","full_name":"link89/oh-my-batch","owner":"link89","description":"A toolkit to manipulate batch tasks with command line. Designed for scientific computing community.","archived":false,"fork":false,"pushed_at":"2025-04-10T09:26:16.000Z","size":724,"stargazers_count":16,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-10T10:47:34.626Z","etag":null,"topics":["bash","batch","command-line-tool","hpc","job-scheduler","pipeline","python3","scientific-computing","shell","slurm","workflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/link89.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-15T14:25:09.000Z","updated_at":"2025-04-10T09:26:19.000Z","dependencies_parsed_at":"2024-10-16T22:11:30.012Z","dependency_job_id":"378e02c1-2aa2-487f-be11-15af4a1840a9","html_url":"https://github.com/link89/oh-my-batch","commit_stats":null,"previous_names":["link89/oh-my-batch"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/link89%2Foh-my-batch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/link89%2Foh-my-batch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/link89%2Foh-my-batch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/link89%2Foh-my-batch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/link89","download_url":"https://codeload.github.com/link89/oh-my-batch/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248850140,"owners_count":21171704,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bash","batch","command-line-tool","hpc","job-scheduler","pipeline","python3","scientific-computing","shell","slurm","workflow"],"created_at":"2024-11-16T01:22:14.393Z","updated_at":"2025-04-14T08:43:40.698Z","avatar_url":"https://github.com/link89.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# oh-my-batch\n\n[![PyPI version](https://badge.fury.io/py/oh-my-batch.svg)](https://badge.fury.io/py/oh-my-batch)\n[![PyPI - Downloads](https://img.shields.io/pypi/dm/oh-my-batch)](https://pypi.org/project/oh-my-batch/)\n[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/oh-my-batch)](https://pypi.org/project/oh-my-batch/)\n\nA toolkit to manipulate batch tasks with command line. Designed for scientific computing community.\n\n## Features\n* `omb combo`: generate folders/files from different combinations of parameters\n* `omb batch`: generate batch scripts from multiple working directories\n* `omb job`: track the state of job in job scheduler\n* `omb misc`: miscellaneous commands\n\n## Install\n```bash\npip install oh-my-batch\n```\n\n## Examples\n* [TESLA workflow](./examples/tesla/): A customizable active learning workflow for training machine learning potentials.\n* [TESLA PIMD workflow](./examples/tesla-pimd/): A customizable active learning workflow for training machine learning potentials with path integral molecular dynamics.\n* [LAMMPS benchmark](./examples/lammps-benchmark/): Find out the best MPI and OpenMP setup for LAMMPS through benchmarking.\n\n\n## Use cases\n\n### Generate files from different combinations of parameters\n\nIt's common to generate files with different combinations of parameters in scientific computing.\nFor example, you have 3 LAMMPS data files in `tmp` directory: `tmp/1.data`, `tmp/2.data`, `tmp/3.data`.\nAnd you want to generate a series of input files with different parameters,\nfor example, different temperatures 300K, 400K, 500K, against each data file.\n\nIn this case, you can use `omb combo` command to generate a series of input files for you.\n\n```bash\n# prepare fake data files\nmkdir -p tmp/\ntouch tmp/1.data tmp/2.data tmp/3.data\n\n# prepare a lammps input file template\ncat \u003e tmp/in.lmp.tmp \u003c\u003cEOF\nread_data @DATA_FILE\nvelocity all create @TEMP @RANDOM\nrun 1000\nEOF\n\n# prepare a run script template\ncat \u003e tmp/run.sh.tmp \u003c\u003cEOF\ncat in.lmp  # simulate running lammps\nEOF\n\n# generate input files\nomb combo \\\n    add_files DATA_FILE tmp/*.data - \\\n    add_var TEMP 300 400 500 - \\\n    add_randint RANDOM -n 3 -a 1 -b 1000 - \\\n    set_broadcast RANDOM - \\\n    make_files tmp/tasks/{i}-T-{TEMP}/in.lmp --template tmp/in.lmp.tmp - \\\n    make_files tmp/tasks/{i}-T-{TEMP}/run.sh --template tmp/run.sh.tmp --mode 755 - \\\n    done\n```\n\nThe above script will generate 9 folders in `tmp/tasks` directory\nwith names from `0-T-300`, `1-T-400`, `2-T-500`, `3-T-300` to `8-T-500`.\nEach folder will contain a `in.lmp` file and a `run.sh` file.\n\nThe 9 folders are the combinations of 3 data files and 3 temperatures,\nand each input file will have a independent random number between 1 and 1000 as `RANDOM`.\n\nYou can run the about script by `./examples/omb-combo.sh`,\nand you can also run `omb combo --help` to see the detailed usage of `combo` command.\n\n### Generate batch scripts from multiple working directories\nIt's common to submit a lot of jobs to a job scheduler. `omb batch` is designed to help you generate batch scripts from multiple working directories and package them into several batch scripts.\n\nLet's continue the above example, now you have 9 folders in `tmp/tasks` directory.\nYou want to package them into 2 batch scripts to submit to a job scheduler.\n\nYou can use `omb batch` to generate batch scripts for you like this:\n\n```bash\ncat \u003e tmp/lammps_header.sh \u003c\u003cEOF\n#!/bin/bash\n#SBATCH -J lmp\n#SBATCH -n 1\n#SBATCH -t 1:00:00\nEOF\n\nomb batch \\\n    add_work_dirs tmp/tasks/* - \\\n    add_header_files tmp/lammps_header.sh - \\\n    add_cmds \"./run.sh\" - \\\n    make tmp/lmp-{i}.slurm --concurrency 2\n```\n\nYou will find batch scripts `tmp/lmp-0.slurm` and `tmp/lmp-1.slurm` in `tmp` directory.\n\nYou can run the above script by `./examples/omb-batch.sh`,\n\n### Track the state of job in job schedular\n\nLet's continue the above example, now you have submitted the batch scripts to the job scheduler.\nIn this case, you can use `omb job` to track the state of the jobs.\n\n```bash\nomb job slurm submit tmp/*.slurm --max_tries 3 --wait --recovery lammps-jobs.json\n```\n\nThe above command will submit the batch scripts to the job scheduler,\nand wait for the jobs to finish. If the job fails, it will retry for at most 3 times.\n\nThe `--recovery` option will save the job information to `lammps-jobs.json` file.\nIf `omb job` is interrupted, you can rerun the exact same command to recover the job status,\nso that you don't need to resubmit the jobs that are still running or completed.\n\n\n## Shell tips\n`oh-my-batch` is intended to help you implement computational workflows with shell scripts.\nTo make the best use of `oh-my-batch`, you need to know some shell tips.\n\n* [Retry commands until success in shell script](https://stackoverflow.com/a/79191004/3099733)\n* [Run multiple line shell script with ssh](https://stackoverflow.com/a/32082912/3099733)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flink89%2Foh-my-batch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flink89%2Foh-my-batch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flink89%2Foh-my-batch/lists"}