{"id":22978252,"url":"https://github.com/futureverse/future.batchtools","last_synced_at":"2025-12-12T00:50:37.578Z","repository":{"id":48243393,"uuid":"56456829","full_name":"futureverse/future.batchtools","owner":"futureverse","description":":rocket: R package future.batchtools: A Future API for Parallel and Distributed Processing using batchtools","archived":false,"fork":false,"pushed_at":"2025-03-30T16:32:41.000Z","size":2086,"stargazers_count":84,"open_issues_count":25,"forks_count":9,"subscribers_count":3,"default_branch":"develop","last_synced_at":"2025-04-02T06:11:13.806Z","etag":null,"topics":["distributed-computing","hpc","job-scheduler","package","parallel","pbs","r","sge","slurm","torque"],"latest_commit_sha":null,"homepage":"https://future.batchtools.futureverse.org","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/futureverse.png","metadata":{"files":{"readme":"README.md","changelog":"NEWS.md","contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-04-17T20:27:22.000Z","updated_at":"2025-03-24T19:21:06.000Z","dependencies_parsed_at":"2024-01-12T15:49:26.715Z","dependency_job_id":"5a19b096-6f41-4edf-889f-27fc5c89b890","html_url":"https://github.com/futureverse/future.batchtools","commit_stats":{"total_commits":438,"total_committers":4,"mean_commits":109.5,"dds":0.07990867579908678,"last_synced_commit":"fa34d531e0a0520e789b924c9188e46ca03dc3ac"},"previous_names":["futureverse/future.batchtools","henrikbengtsson/future.batchtools"],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/futureverse%2Ffuture.batchtools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/futureverse%2Ffuture.batchtools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/futureverse%2Ffuture.batchtools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/futureverse%2Ffuture.batchtools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/futureverse","download_url":"https://codeload.github.com/futureverse/future.batchtools/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247994121,"owners_count":21030050,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-computing","hpc","job-scheduler","package","parallel","pbs","r","sge","slurm","torque"],"created_at":"2024-12-15T01:06:09.087Z","updated_at":"2025-12-12T00:50:37.571Z","avatar_url":"https://github.com/futureverse.png","language":"R","funding_links":[],"categories":["R"],"sub_categories":[],"readme":"\n\n\u003cdiv id=\"badges\"\u003e\u003c!-- pkgdown markup --\u003e\n\u003ca href=\"https://CRAN.R-project.org/web/checks/check_results_future.batchtools.html\"\u003e\u003cimg border=\"0\" src=\"https://www.r-pkg.org/badges/version/future.batchtools\" alt=\"CRAN check status\"/\u003e\u003c/a\u003e \u003ca href=\"https://github.com/futureverse/future.batchtools/actions?query=workflow%3AR-CMD-check\"\u003e\u003cimg border=\"0\" src=\"https://github.com/futureverse/future.batchtools/actions/workflows/R-CMD-check.yaml/badge.svg?branch=develop\" alt=\"R CMD check status\"/\u003e\u003c/a\u003e  \u003ca href=\"https://github.com/futureverse/future.batchtools/actions?query=workflow%3Afuture_tests\"\u003e\u003cimg border=\"0\" src=\"https://github.com/futureverse/future.batchtools/actions/workflows/future_tests.yaml/badge.svg?branch=develop\" alt=\"future.tests checks status\"/\u003e\u003c/a\u003e   \u003ca href=\"https://app.codecov.io/gh/futureverse/future.batchtools\"\u003e\u003cimg border=\"0\" src=\"https://codecov.io/gh/futureverse/future.batchtools/branch/develop/graph/badge.svg\" alt=\"Coverage Status\"/\u003e\u003c/a\u003e \n\u003c/div\u003e\n\n# future.batchtools: A Future API for Parallel and Distributed Processing using 'batchtools' \n\n## TL;DR\n\nHere is an example on how evaluate R expression on a Slurm\nhigh-performance compute (HPC) cluster.\n\n```r\nlibrary(future)\n\n# Limit runtime to 10 minutes and memory to 400 MiB per future,\n# request a parallel environment with four slots on a single host.\n# On this system, R is available via environment module 'r'. By\n# specifying 'r/4.5.1', 'module load r/4.5.1' will be added to\n# the submitted job script.\nplan(future.batchtools::batchtools_slurm, resources = list(\n  time = \"00:10:00\", mem = \"400M\", nodes = 1, ntasks = 4,\n  modules = c(\"r/4.5.1\")\n))\n\n# Give it a spin\nf \u003c- future({\n  data.frame(\n    hostname = Sys.info()[[\"nodename\"]],\n          os = Sys.info()[[\"sysname\"]],\n       cores = unname(parallelly::availableCores()),\n     modules = Sys.getenv(\"LOADEDMODULES\")\n  )\n})\ninfo \u003c- value(f)\nprint(info)\n#\u003e   hostname    os cores  modules\n#\u003e 1      n12 Linux     4  r/4.5.1\n```\n\n## Introduction\n\nThe **[future]** package provides a generic API for using futures in\nR.  A future is a simple yet powerful mechanism to evaluate an R\nexpression and retrieve its value at some point in time.  Futures can\nbe resolved in many different ways depending on which strategy is\nused.  There are various types of synchronous and asynchronous futures\nto choose from in the **[future]** package.\n\nThis package, **[future.batchtools]**, provides a type of futures that\nutilizes the **[batchtools]** package.  This means that _any_ type of\nbackend that the **batchtools** package supports can be used as a\nfuture.  More specifically, **future.batchtools** will allow you or\nusers of your package to leverage the compute power of\nhigh-performance computing (HPC) clusters via a simple switch in\nsettings - without having to change any code at all.\n\nFor instance, if **batchtools** is properly configured, the below two\nexpressions for two futures will be processed on two different compute\nnodes:\n\n```r\nlibrary(future)\nplan(future.batchtools::batchtools_slurm)\n\nf_x \u003c- future({ Sys.sleep(5); 3.14 })\nf_y \u003c- future({ Sys.sleep(5); 2.71 })\nx \u003c- value(f_x)\ny \u003c- value(f_y)\nx + y\n#\u003e [1] 5.85\n```\n\nThis is just a toy example to illustrate what futures look like and\nhow to work with them.\n\nFor an introduction as well as full details on how to use futures,\nplease see \u003chttps://www.futureverse.org\u003e or consult the package\nvignettes of the **[future]** package.\n\n\n## Demos\n\nThe **[future]** package provides a demo using futures for calculating\na set of Mandelbrot planes.  The demo does not assume anything about\nwhat type of futures are used.  _The user has full control of how\nfutures are evaluated_.  For instance, to use local batchtools\nfutures, run the demo as:\n\n```r\nlibrary(future)\nplan(future.batchtools::batchtools_local)\ndemo(\"mandelbrot\", package = \"future\", ask = FALSE)\n```\n\n\n## Available batchtools backend\n\nThe **future.batchtools** package implements a generic future wrapper\nfor all batchtools backends.  Below are the most common types of\nbatchtools backends. For other types of parallel and distributed\nbackends, please see \u003chttps://www.futureverse.org/backends.html\u003e.\n\n\n| Backend                  | Description                                                              | Alternative in future package\n|:-------------------------|:-------------------------------------------------------------------------|:------------------------------------\n| `batchtools_lsf`         | Futures are evaluated via a [Load Sharing Facility (LSF)] job scheduler  | N/A\n| `batchtools_openlava`    | Futures are evaluated via an [OpenLava] job scheduler                    | N/A\n| `batchtools_sge`         | Futures are evaluated via a [Sun/Son of/Oracle/Univa/Altair Grid Engine (SGE)] job scheduler | N/A\n| `batchtools_slurm`       | Futures are evaluated via a [Slurm] job scheduler                        | N/A\n| `batchtools_torque`      | Futures are evaluated via a [TORQUE] / PBS job scheduler                 | N/A\n| `batchtools_custom`      | Futures are evaluated via a custom batchtools configuration R script or via a set of cluster functions  | N/A\n| `batchtools_multicore`   | parallel evaluation by forking the current R process                     | `plan(multicore)`\n| `batchtools_local`       | sequential evaluation in a separate R process (on current machine)       | `plan(cluster, workers = I(1))`\n\n\n\n\n[batchtools]: https://cran.r-project.org/package=batchtools\n[future]: https://cran.r-project.org/package=future\n[future.batchtools]: https://cran.r-project.org/package=future.batchtools\n[TORQUE]: https://en.wikipedia.org/wiki/TORQUE\n[Slurm]: https://en.wikipedia.org/wiki/Slurm_Workload_Manager\n[Sun/Son of/Oracle/Univa/Altair Grid Engine (SGE)]: https://en.wikipedia.org/wiki/Oracle_Grid_Engine\n[Load Sharing Facility (LSF)]: https://en.wikipedia.org/wiki/Platform_LSF\n[OpenLava]: https://en.wikipedia.org/wiki/OpenLava\n\n## Installation\nR package future.batchtools is available on [CRAN](https://cran.r-project.org/package=future.batchtools) and can be installed in R as:\n```r\ninstall.packages(\"future.batchtools\")\n```\n\n\n### Pre-release version\n\nTo install the pre-release version that is available in Git branch `develop` on GitHub, use:\n```r\nremotes::install_github(\"futureverse/future.batchtools\", ref=\"develop\")\n```\nThis will install the package from source.  \n\n\u003c!-- pkgdown-drop-below --\u003e\n\n\n## Contributing\n\nTo contribute to this package, please see [CONTRIBUTING.md](CONTRIBUTING.md).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffutureverse%2Ffuture.batchtools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffutureverse%2Ffuture.batchtools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffutureverse%2Ffuture.batchtools/lists"}