https://github.com/wlandau/workers

Persistent and transient parallel workers for R
https://github.com/wlandau/workers

Last synced: about 2 months ago
JSON representation

Persistent and transient parallel workers for R

• Host: GitHub
• URL: https://github.com/wlandau/workers
• Owner: wlandau
• Created: 2018-02-28T14:23:02.000Z (almost 7 years ago)
• Default Branch: main
• Last Pushed: 2020-10-27T16:14:51.000Z (about 4 years ago)
• Last Synced: 2024-08-13T07:14:20.303Z (4 months ago)
• Language: R
• Homepage:
• Size: 97.7 KB
• Stars: 7
• Watchers: 4
• Forks: 0
• Open Issues: 4
• Metadata Files:
  • Readme: README.Rmd

Awesome Lists containing this project

• jimsghstars - wlandau/workers - Persistent and transient parallel workers for R (R)

README

        
---

output:

  github_document:

    html_preview: false

---

```{r, echo = FALSE}

knitr::opts_chunk$set(

  collapse = TRUE,

  comment = "#>",

  fig.path = "inst/img/README-"

)

options(tibble.print_min = 5, tibble.print_max = 5)

```

![stability-wip](https://img.shields.io/badge/stability-work_in_progress-lightgrey.svg)

[![CRAN](http://www.r-pkg.org/badges/version/workers)](http://cran.r-project.org/package=workers)

[![downloads](http://cranlogs.r-pkg.org/badges/workers)](http://cran.rstudio.com/package=workers)

[![Travis build status](https://travis-ci.org/wlandau/workers.svg?branch=master)](https://travis-ci.org/wlandau/workers)

[![AppVeyor Build Status](https://ci.appveyor.com/api/projects/status/github/wlandau/workers?branch=master&svg=true)](https://ci.appveyor.com/project/wlandau/workers)

[![Codecov](https://codecov.io/github/wlandau/workers/coverage.svg?branch=master)](https://codecov.io/github/wlandau/workers?branch=master)

# Purpose

The `workers` package is a platform-agnostic R-focused parallel job scheduler. For computationally-demanding workflows, schedulers are important. Some tasks need to complete before others start (for example, the data munging steps that precede analysis) and `workers` takes advantages of parallel computing opportunities while saving you the trouble of figuring out what needs to run when.

# Installation

```{r install, eval = FALSE}

devtools::install_github("wlandau/workers")

```

# Usage

Represent your workflow as a dependency graph with functions as attributes. Each function is a step in the pipeline.

```{r use}

success <- function() {

  future::future(list(success = TRUE))

}

code <- list(

  a = function() {

    x <<- 2

    success()

  },

  b = function() {

    y <<- x + 1

    success()

  },

  c = function() {

    z <<- x * 2

    success()

  },

  d = function() {

    w <<- 3 * y + z

    success()

  }

)

vertices <- tibble::tibble(name = letters[1:4], code)

edges <- tibble::tibble(

  from = c("a", "a", "b", "c"),

  to = c("b", "c", "d", "d")

)

graph <- igraph::graph_from_data_frame(edges, vertices = vertices)

plot(graph)

```

Then, run your workflow with `schedule(graph)`.

```{r schedule}

library(workers)

schedule(graph)

```