https://github.com/dunnkers/slurm-to-wandb
Monitor SLURM jobs using Weights and Biases (wandb) 📊
https://github.com/dunnkers/slurm-to-wandb
hpc monitor-slurm-jobs monitoring slurm wandb weights-and-biases
Last synced: 3 months ago
JSON representation
Monitor SLURM jobs using Weights and Biases (wandb) 📊
- Host: GitHub
- URL: https://github.com/dunnkers/slurm-to-wandb
- Owner: dunnkers
- License: mit
- Created: 2021-05-30T13:33:11.000Z (over 4 years ago)
- Default Branch: master
- Last Pushed: 2021-06-17T14:20:48.000Z (over 4 years ago)
- Last Synced: 2025-06-14T16:53:16.369Z (4 months ago)
- Topics: hpc, monitor-slurm-jobs, monitoring, slurm, wandb, weights-and-biases
- Language: Python
- Homepage:
- Size: 32.2 KB
- Stars: 3
- Watchers: 3
- Forks: 2
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# slurm-to-wandb
[](https://github.com/dunnkers/slurm-to-wandb/actions/workflows/python-app.yml) [](https://pypi.org/project/slurm-to-wandb/)Monitor SLURM jobs using Weights and Biases (wandb) 📊
## Usage
On your cluster, login to wandb using the [cli](https://github.com/wandb/client), then run:```shell
pip install slurm-to-wandb
slurm_to_wandb
```... to upload all `sacct` information to wandb. Currently, uploads to a project called "peregrine" - support for configuring this is planned #4. `` can be multiple space-separated job ids. In the case multiple job id's match, all matched job ids are uploaded (useful, for example, when using job arrays).
To construct a DataFrame with the information yourself, use the function `slurm_to_wandb.sacct_as_df(*job_ids, **sacct_args)`. `sacct_args` can be any additional args to pass to `sacct`. Run it on the cluster:
```python
from slurm_to_wandb import sacct_as_dfdf = sacct_as_df("job_id_123", "another_job_id", starttime="2021-05-20")
df
```You can now upload the information to wandb however you like 🙌🏻
## About
By [Jeroen Overschie](https://dunnkers.com/).