{"id":19644826,"url":"https://github.com/uscbiostats/slurmr","last_synced_at":"2025-04-28T13:31:49.771Z","repository":{"id":37587672,"uuid":"145923875","full_name":"USCbiostats/slurmR","owner":"USCbiostats","description":"slurmR: A Lightweight Wrapper for Slurm","archived":false,"fork":false,"pushed_at":"2024-06-17T12:30:05.000Z","size":2083,"stargazers_count":60,"open_issues_count":19,"forks_count":11,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-22T08:43:27.752Z","etag":null,"topics":["bioinformatics","hpc","rpackage","rstats","slurm"],"latest_commit_sha":null,"homepage":"https://uscbiostats.github.io/slurmR/","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/USCbiostats.png","metadata":{"files":{"readme":"README.Rmd","changelog":"NEWS.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-08-24T00:58:12.000Z","updated_at":"2025-02-24T09:56:34.000Z","dependencies_parsed_at":"2024-01-12T04:45:05.019Z","dependency_job_id":"b1c4904f-07c9-46e5-bdd6-02ef05b5cd59","html_url":"https://github.com/USCbiostats/slurmR","commit_stats":{"total_commits":275,"total_committers":3,"mean_commits":91.66666666666667,"dds":"0.036363636363636376","last_synced_commit":"df891c49cb81abf876cdb597f89e3786a54c37cb"},"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/USCbiostats%2FslurmR","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/USCbiostats%2FslurmR/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/USCbiostats%2FslurmR/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/USCbiostats%2FslurmR/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/USCbiostats","download_url":"https://codeload.github.com/USCbiostats/slurmR/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251319757,"owners_count":21570451,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bioinformatics","hpc","rpackage","rstats","slurm"],"created_at":"2024-11-11T14:30:10.856Z","updated_at":"2025-04-28T13:31:44.730Z","avatar_url":"https://github.com/USCbiostats.png","language":"R","funding_links":[],"categories":[],"sub_categories":[],"readme":"---\noutput:\n  github_document:\n    md_extensions: -auto_identifiers+header_attributes\n    html_preview: false\n---\n\n[![DOI](http://joss.theoj.org/papers/10.21105/joss.01493/status.svg)](https://doi.org/10.21105/joss.01493)\n[![R CI release](https://github.com/USCbiostats/slurmR/actions/workflows/ci.yml/badge.svg)](https://github.com/USCbiostats/slurmR/actions/workflows/ci.yml)\n[![codecov](https://codecov.io/gh/USCbiostats/slurmR/branch/master/graph/badge.svg)](https://app.codecov.io/gh/USCbiostats/slurmR)\n[![CRAN status](https://www.r-pkg.org/badges/version/slurmR)](https://CRAN.R-project.org/package=slurmR)\n[![CRAN downloads](http://cranlogs.r-pkg.org/badges/grand-total/slurmR)](https://cran.r-project.org/package=slurmR)\n[![status](https://tinyverse.netlify.com/badge/slurmR)](https://CRAN.R-project.org/package=slurmR)\n[![Integrative Methods of Analysis for Genetic Epidemiology](https://raw.githubusercontent.com/USCbiostats/badges/master/tommy-image-badge.svg)](https://p01.uscbiostatistics.org/)\n\n\u003c!-- README.md is generated from README.Rmd. Please edit that file --\u003e\n\n```{r setup, include = FALSE}\noptions(width = 80)\nknitr::opts_chunk$set(\n  collapse = TRUE,\n  comment = \"# \", \n  fig.path = \"man/figures/README-\",\n  out.width = \"100%\"\n)\n```\n\n\n# slurmR: A Lightweight Wrapper for Slurm \u003cimg src=\"man/figures/logo.png\" height=\"180px\" align=\"right\"/\u003e\n\nSlurm Workload Manager is a popular HPC cluster job scheduler found in many of the top 500 supercomputers. The `slurmR` R package provides an R wrapper to it that matches the parallel package's syntax, this is, just like `parallel` provides the `parLapply`, `clusterMap`, `parSapply`, etc., `slurmR` provides `Slurm_lapply`, `Slurm_Map`, `Slurm_sapply`, etc.\n\nWhile there are other alternatives such as `future.batchtools`, `batchtools`, `clustermq`, and `rslurm`, this R package has the following goals:\n\n1.  It is dependency-free, which means that it works out-of-the-box\n\n2.  Emphasizes been similar to the workflow in the R package `parallel`\n\n3.  It provides a general framework for creating personalized own wrappers without using template files.\n\n4.  Is specialized on Slurm, meaning more flexibility (no need to modify template files) and debugging tools (e.g., job resubmission).\n\n5.  Provide a backend for the\n    [parallel](https://CRAN.R-project.org/view=HighPerformanceComputing)\n    package, providing an out-of-the-box method for creating Socket cluster objects for multi-node operations. (See the examples below on how to use it with other R packages)\n\nCheckout the [VS section](#vs) section for comparing `slurmR` with other R packages.\nWondering who is using Slurm? Check out the [list at the end of this document](#who-uses-slurm).\n\n# Installation\n\nFrom your HPC command line, you can install the development version from [GitHub](https://github.com/) with:\n\n```bash\n$ git clone https://github.com/USCbiostats/slurmR.git\n$ R CMD INSTALL slurmR/ \n```\n\nThe second line assumes you have R available in your system (usually loaded via\n`module R` or some other command). Or using the `devtools` from within R:\n\n``` r\n# install.packages(\"devtools\")\ndevtools::install_github(\"USCbiostats/slurmR\")\n```\n\n# Citation\n\n```{r cite, echo=FALSE, comment=\"\"}\ncitation(\"slurmR\")\n```\n\n# Running slurmR with Docker\n\nFor testing purposes, slurmR is available in [Dockerhub](https://hub.docker.com/repository/docker/uscbiostats/slurmr).\nThe `rcmdcheck` and `interactive` images are built on top of\n[`xenonmiddleware/slurm`](https://hub.docker.com/r/xenonmiddleware/slurm).\n\nOnce you download the files contained in the [`slurmR` repository](https://github.com/USCbiostats/slurmR),\nyou can go to the `docker` folder and use the `Makefile` included\nthere to start a Unix session with slurmR and Slurm included.\n\nTo test `slurmR` using docker, check the README.md file located at\nhttps://github.com/USCbiostats/slurmR/tree/master/docker.\n\n\n# Examples\n\n## Example 1: Computing means (and looking under the hood)\n\n```{r simple-example}\nlibrary(slurmR)\n\n# Suppose that we have 100 vectors of length 50 ~ Unif(0,1)\nset.seed(881)\nx \u003c- replicate(100, runif(50), simplify = FALSE)\n```\n\nWe can use the function `Slurm_lapply` to distribute computations\n\n```{r example1}\nans \u003c- Slurm_lapply(x, mean, plan = \"none\")\nSlurm_clean(ans) # Cleaning after you\n```\n\nNotice the `plan = \"none\"` option; this tells `Slurm_lapply` to only create the job object but do nothing with it, i.e., skip submission. To get more info, we can set the verbose mode on\n\n```{r example1-with-verb}\nopts_slurmR$verbose_on()\nans \u003c- Slurm_lapply(x, mean, plan = \"none\")\nSlurm_clean(ans) # Cleaning after you\n```\n\n## Example 2: Job resubmission \n\nThe following example was extracted from the package's manual.\n\n```r\n# Submitting a simple job\njob \u003c- Slurm_EvalQ(slurmR::WhoAmI(), njobs = 20, plan = \"submit\")\n\n# Checking the status of the job (we can simply print)\njob\nstatus(job) # or use the state function\nsacct(job) # or get more info with the sactt wrapper.\n\n# Suppose some of the jobs are taking too long to complete (say 1, 2, and 15 through 20)\n# we can stop it and resubmit the job as follows:\nscancel(job)\n\n# Resubmitting only \nsbatch(job, array = \"1,2,15-20\") # A new jobid will be assigned\n\n# Once its done, we can collect all the results at once\nres \u003c- Slurm_collect(job)\n\n# And clean up if we don't need to use it again\nSlurm_clean(res)\n```\n\nTake a look at the vignette [here](vignettes/getting-started.Rmd).\n\n## Example 3: Using slurmR and future/doParallel/boot/... \n\nThe function `makeSlurmCluster` creates a PSOCK cluster within a Slurm HPC network,\nmeaning that users can go beyond a single node cluster object and take advantage\nof Slurm to create a multi-node cluster object. This feature allows using\n`slurmR` with other R packages that support working with `SOCKcluster` class objects. Here are some examples\n\nWith the [`future`](https://cran.r-project.org/package=future) package\n\n```r\nlibrary(future)\nlibrary(slurmR)\n\ncl \u003c- makeSlurmCluster(50)\n\n# It only takes using a cluster plan!\nplan(cluster, cl)\n\n...your fancy futuristic code...\n\n# Slurm Clusters are stopped in the same way any cluster object is\nstopCluster(cl)\n```\n\nWith the [`doParallel`](https://cran.r-project.org/package=doParallel) package\n\n```r\nlibrary(doParallel)\nlibrary(slurmR)\n\ncl \u003c- makeSlurmCluster(50)\n\nregisterDoParallel(cl)\nm \u003c- matrix(rnorm(9), 3, 3)\nforeach(i=1:nrow(m), .combine=rbind) \n\nstopCluster(cl)\n```\n\n## Example 4: Using slurmR directly from the command line\n\nThe `slurmR` package has a couple of convenient functions designed for the user\nto save time. First, the function `sourceSlurm()` allows skipping the explicit\ncreating of a bash script file to be used together with `sbatch` by putting all\nthe required config files on the first lines of an R scripts, for example:\n\n```{r, results='asis', echo=FALSE}\ncat(\"```\\n\")\ncat(readLines(system.file(\"example.R\", package=\"slurmR\")), sep=\"\\n\")\ncat(\"```\\n\")\n```\n\nIs an R script that on the first line coincides with that of a bash script for\nSlurm: `#!/bin/bash`. The following lines start with `#SBATCH` explicitly\nspecifying options for `sbatch`, and the reminder lines are just R code.\n\nThe previous R script is included in the package (type `system.file(\"example.R\", package=\"slurmR\")`).\n\nImagine that that R script is named `example.R`, then you use the `sourceSlurm`\nfunction to submit it to Slurm as follows:\n\n```r\nslurmR::sourceSlurm(\"example.R\")\n```\n\nThis will create the corresponding bash file required to be used with `sbatch`,\nand submit it to Slurm.\n\nAnother nice tool is the `slurmr_cmd()`. This function will create a simple bash-script\nthat we can use as a command-line tool to submit this type of R-scripts.\nMoreover, this command will can add the command to your session's\n[**alias**](https://en.wikipedia.org/wiki/Alias_(command)) as follows:\n\n```r\nlibrary(slurmR)\nslurmr_cmd(\"~\", add_alias = TRUE)\n```\n\nOnce that's done, you can submit R scripts with \"Slurm-like headers\" (as\nshown previously) as follows:\n\n```bash\n$ slurmr example.R\n```\n\n## Example 5: Using the preamble\n\nSince version 0.4-3, `slurmR` includes the option `preamble`. This provides a way\nfor the user to specify commands/modules that need to be executed before running\nthe Rscript. Here is an example using `module load`:\n\n```{r preamble, warning=FALSE, eval = TRUE}\n# Turning the verbose mode off\nopts_slurmR$verbose_off()\n\n# Setting the preamble can be done globally\nopts_slurmR$set_preamble(\"module load gcc/6.0\")\n\n# Or on the fly\nans \u003c- Slurm_lapply(1:10, mean, plan = \"none\", preamble = \"module load pandoc\")\n\n# Printing out the bashfile\ncat(readLines(ans$bashfile), sep = \"\\n\")\n\nSlurm_clean(ans) # Cleaning after you\n```\n\n## VS \n\nThere are several ways to enhance R for HPC. Depending on what are your goals/restrictions/preferences, you can use any of the following from this **manually curated** list:\n\n\n```{r vs-table, echo = FALSE}\ndat \u003c- read.csv(\"comparing-projects.csv\", check.names = FALSE)\ndat$Dependencies \u003c- sprintf(\"[![status](https://tinyverse.netlify.com/badge/%s)](https://CRAN.R-project.org/package=%1$s)\", dat$Package)\ndat$Activity \u003c- sprintf(\"[![Activity](https://img.shields.io/github/last-commit/%s)](https://github.com/%1$s)\", dat$github)\n\ndat$Package \u003c- sprintf(\"[**%s**](https://cran.r-project.org/package=%1$s)\", dat$Package)\n\n# Packages that only work with Slurm\nonly_w_slurm \u003c- dat$Package[dat$`System [blank]` == \"specific\"]\nonly_w_slurm \u003c- paste(only_w_slurm, collapse = \", \")\n\ndat$github \u003c- NULL\ndat$`System [blank]` \u003c- NULL\ndat$`Focus on [blank]` \u003c- NULL\nknitr::kable(dat)\n```\n\n(1) After errors, a part or the entire job can be resubmitted.\n(2) Functionality similar to the apply family in base R, e.g., lapply, sapply, mapply or similar.\n(3) Creating a cluster object using either MPI or Socket connection.\n\n\nThe packages `r only_w_slurm` work only on Slurm. The [**drake**](https://cran.r-project.org/package=drake) package is focused on workflows.\n\n\n## Contributing\n\nWe welcome contributions to `slurmR`. Whether it is reporting a bug, starting a discussion by asking a question, or proposing/requesting a new feature, please go by creating a new issue [here](https://github.com/USCbiostats/slurmR/issues) so that we can talk about it.\n\nPlease note that this project is released with a Contributor Code of Conduct (see\nthe CODE_OF_CONDUCT.md file included in this project). By participating in this\nproject, you agree to abide by its terms.\n\n## Who uses Slurm\n\nHere is a manually curated list of institutions using Slurm:\n\n|Institution | Country | Link |\n|------------|---------|------|\n| University of Utah's CHPC | US | [link](https://chpc.utah.edu/documentation/software/slurm.php) |\n| USC Center for Advance Research Computing | US | [link](https://carc.usc.edu/) |\n| Princeton Research Computing | US | [link](https://researchcomputing.princeton.edu/support/knowledge-base/slurm) |\n| Harvard FAS | US | [link](https://docs.rc.fas.harvard.edu/kb/quickstart-guide/)|\n| Harvard HMS research computing | US | [link](https://rc.hms.harvard.edu/) |\n| UCSan Diego WM Keck Lab for Integrated Biology | US | [link](https://keck2.ucsd.edu/dokuwiki/doku.php/wiki:slurm) |\n| Stanford Sherlock | US | [link](https://www.sherlock.stanford.edu/docs/overview/introduction/) |\n| Stanford SCG Informatics Cluster | US | [link](https://login.scg.stanford.edu/tutorials/job_scripts/) |\n| UC Berkeley Open Computing Facility | US | [link](https://www.ocf.berkeley.edu/docs/services/hpc/) |\n| University of Utah CHPC | US | [link](https://www.chpc.utah.edu/documentation/software/slurm.php) |\n| The University of Kansas Center for Research Computing | US | [link](https://docs.crc.ku.edu/how-to/submitting-jobs/) |\n| University of Cambridge | UK | [link](https://www.ch.cam.ac.uk/computing/slurm-usage) |\n| Indiana University | US | [link](https://kb.iu.edu/d/awrz) |\n| Caltech HPC Center | US | [link](https://www.hpc.caltech.edu/documentation/slurm-commands) |\n| Institute for Advanced Study | US | [link](https://www.ias.edu/sns/computing/slurm) |\n| UTSouthwestern Medical Center BioHPC | US | [link](https://portal.biohpc.swmed.edu/content/guides/slurm/) |\n| Vanderbilt University ACCRE | US | [link](https://www.vanderbilt.edu/accre/documentation/slurm/) |\n| University of Virginia Research Computing | US | [link](https://www.rc.virginia.edu/userinfo/rivanna/slurm/) |\n| Center for Advanced Computing | CA | [link](https://info.cac.queensu.ca) |\n| SciNet | CA | [link](https://docs.scinet.utoronto.ca/index.php/Slurm) |\n| NLHPC | CL | [link](https://wiki.nlhpc.cl/P%C3%A1gina_principal) |\n| Kultrun | CL | [link](http://www.astro.udec.cl/kultrun/) |\n| Matbio | CL | [link](http://www.matbio.cl/cluster/) |\n| TIG MIT | US | [link](https://tig.csail.mit.edu/shared-computing/slurm/) |\n| MIT Supercloud | US | supercloud.mit.edu/ |\n| Oxford's ARC | UK | [link](https://arc-user-guide.readthedocs.io/en/latest/slurm-reference.html) |\n\n## Funding\n\nWith project is supported by the National Cancer Institute, Grant #1P01CA196596.\n\nComputation for the work described in this paper was supported by the University of Southern California's Center for High-Performance Computing (hpcc.usc.edu).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuscbiostats%2Fslurmr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fuscbiostats%2Fslurmr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuscbiostats%2Fslurmr/lists"}