Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/shikokuchuo/mirai
mirai - Minimalist Async Evaluation Framework for R
https://github.com/shikokuchuo/mirai
asynchronous-tasks concurrency cran distributed-computing high-performance-computing parallel-programming promises r r-package rstats
Last synced: 7 days ago
JSON representation
mirai - Minimalist Async Evaluation Framework for R
- Host: GitHub
- URL: https://github.com/shikokuchuo/mirai
- Owner: shikokuchuo
- License: gpl-3.0
- Created: 2022-02-14T22:11:55.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2024-05-22T22:46:22.000Z (8 months ago)
- Last Synced: 2024-05-23T02:27:51.711Z (8 months ago)
- Topics: asynchronous-tasks, concurrency, cran, distributed-computing, high-performance-computing, parallel-programming, promises, r, r-package, rstats
- Language: R
- Homepage: https://shikokuchuo.net/mirai/
- Size: 6.33 MB
- Stars: 135
- Watchers: 6
- Forks: 5
- Open Issues: 1
-
Metadata Files:
- Readme: README.Rmd
- License: LICENSE.md
- Code of conduct: .github/CODE_OF_CONDUCT.md
Awesome Lists containing this project
- jimsghstars - shikokuchuo/mirai - mirai - Minimalist Async Evaluation Framework for R (R)
README
---
output: github_document
---
```{r, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "man/figures/README-",
out.width = "100%"
)
```
# mirai 
[](https://CRAN.R-project.org/package=mirai)
[](https://community.r-multiverse.org/mirai)
[](https://shikokuchuo.r-universe.dev/mirai)
[](https://github.com/shikokuchuo/mirai/actions)
[](https://app.codecov.io/gh/shikokuchuo/mirai)
[](https://zenodo.org/badge/latestdoi/459341940)
### ミライ
みらい 未来
Minimalist Async Evaluation Framework for R
Designed for simplicity, a 'mirai' evaluates an R expression asynchronously in a parallel process, locally or distributed over the network, with the result automatically available upon completion.
Modern networking and concurrency built on [nanonext](https://github.com/shikokuchuo/nanonext/) and [NNG](https://nng.nanomsg.org/) (Nanomsg Next Gen) ensure reliable and efficient scheduling, over fast inter-process communications or TCP/IP secured by TLS.
Advantages include being inherently queued thus handling many more tasks than available processes, no storage on the file system, support for otherwise non-exportable reference objects, an event-driven promises implementation, and built-in asynchronous parallel map.
### Quick Start
Use `mirai()` to evaluate an expression asynchronously in a separate, clean R process.
The following mimics an expensive calculation that eventually returns a vector of random values.
```{r exec}
library(mirai)
m <- mirai({Sys.sleep(n); rnorm(n, mean)}, n = 5L, mean = 7)
```
The mirai expression is evaluated in another process and hence must be self-contained, not referring to variables that do not already exist there. Above, the variables `n` and `mean` are passed as part of the `mirai()` call.
A 'mirai' object is returned immediately - creating a mirai never blocks the session.
Whilst the async operation is ongoing, attempting to access a mirai's data yields an 'unresolved' logical NA.
```{r do}
m
m$data
```
To check whether a mirai remains unresolved (yet to complete):
```{r unres}
unresolved(m)
```
To wait for and collect the return value, use the mirai's `[]` method:
```{r call}
m[]
```
As a mirai represents an async operation, it is never necessary to wait for it. Other code can continue to run. Once it completes, the return value automatically becomes available at `$data`.
```{r resolv}
while (unresolved(m)) {
# do work here that does not depend on 'm'
}
m
m$data
```
#### Daemons
Daemons are persistent background processes for receiving mirai requests, and are created as easily as:
```{r daemons}
daemons(4)
```
Daemons may also be deployed [remotely](https://shikokuchuo.net/mirai/articles/mirai.html#distributed-computing-remote-daemons) for distributed computing and [launchers](https://shikokuchuo.net/mirai/articles/mirai.html#distributed-computing-launching-daemons) can start daemons across the network via (tunnelled) SSH or a cluster resource manager.
[Secure TLS connections](https://shikokuchuo.net/mirai/articles/mirai.html#distributed-computing-tls-secure-connections) can be used for remote daemon connections, with zero configuration required.
#### Async Parallel Map
`mirai_map()` maps a function over a list or vector, with each element processed in a separate parallel process. It also performs multiple map over the rows of a dataframe or matrix.
```{r map}
df <- data.frame(
fruit = c("melon", "grapes", "coconut"),
price = c(3L, 5L, 2L)
)
m <- mirai_map(df, \(...) sprintf("%s: $%d", ...))
```
A 'mirai_map' object is returned immediately. Other code can continue to run at this point. Its value may be retrieved at any time using its `[]` method to return a list, just like `purrr::map()` or `base::lapply()`. The `[]` method also provides options for flatmap, early stopping and/or progress indicators.
```{r mapvalue}
m
m[.flat]
```
All errors are returned as 'errorValues', facilitating recovery from partial failure. There are further [advantages](https://shikokuchuo.net/mirai/articles/mirai.html#asynchronous-parallel-map) over alternative map implementations.
### Design Concepts
`mirai` is designed from the ground up to provide a production-grade experience.
- Fast
+ 1,000x more responsive compared to common alternatives [[1]](https://github.com/shikokuchuo/mirai/pull/142#issuecomment-2457589563)
+ Built for low-latency applications such as real time inference or Shiny apps
- Reliable
+ Consistent behaviour with no reliance on global options or variables
+ Each mirai call is evaluated explicitly for transparent and predictable results
- Scalable
+ Launch millions of tasks simultaneously over thousands of connections
+ Proven track record handling heavy-duty workloads in the life sciences industry
[
](https://youtu.be/GhX0PcEm3CY?t=1740)
[
](https://youtu.be/cyF2dzloVLo?t=5127)
> *mirai パッケージを試してみたところ、かなり速くて驚きました*
### Integrations
The following core integrations are documented, with usage examples in the linked vignettes:
[
](https://shikokuchuo.net/mirai/articles/parallel.html) Provides an alternative communications backend for R, implementing a new parallel cluster type, a feature request by R-Core at R Project Sprint 2023. 'miraiCluster' may also be used with `foreach` via `doParallel`.
[
](https://shikokuchuo.net/mirai/articles/promises.html) Implements the next generation of completely event-driven, non-polling promises. 'mirai' may be used interchageably with 'promises', including with the promise pipe `%...>%`.
[
](https://shikokuchuo.net/mirai/articles/shiny.html) Asynchronous parallel / distributed backend, supporting the next level of responsiveness and scalability within Shiny, with native support for ExtendedTask.
[
](https://shikokuchuo.net/mirai/articles/plumber.html) Asynchronous parallel / distributed backend for scaling Plumber applications in production.
[
](https://shikokuchuo.net/mirai/articles/databases.html) Allows queries using the Apache Arrow format to be handled seamlessly over ADBC database connections hosted in background processes.
[
](https://shikokuchuo.net/mirai/articles/torch.html) Allows Torch tensors and complex objects such as models and optimizers to be used seamlessly across parallel processes.
### Powering Crew and Targets High Performance Computing
[
](https://docs.ropensci.org/targets/) Targets, a Make-like pipeline tool for statistics and data science, has integrated and adopted `crew` as its default high-performance computing backend.
[
](https://wlandau.github.io/crew/) Crew is a distributed worker-launcher extending `mirai` to different distributed computing platforms, from traditional clusters to cloud services.
[
](https://wlandau.github.io/crew.cluster/) `crew.cluster` enables mirai-based workflows on traditional high-performance computing clusters using LFS, PBS/TORQUE, SGE and Slurm.
[
](https://wlandau.github.io/crew.aws.batch/) `crew.aws.batch` extends `mirai` to cloud computing using AWS Batch.
### Thanks
We would like to thank in particular:
[Will Landau](https://github.com/wlandau/) for being instrumental in shaping development of the package, from initiating the original request for persistent daemons, through to orchestrating robustness testing for the high performance computing requirements of `crew` and `targets`.
[Joe Cheng](https://github.com/jcheng5/) for integrating the `promises` method to work seamlessly within Shiny, and prototyping event-driven promises.
[Luke Tierney](https://github.com/ltierney/) of R Core, for discussion on L'Ecuyer-CMRG streams to ensure statistical independence in parallel processing, and making it possible for `mirai` to be the first 'alternative communications backend for R'.
[Henrik Bengtsson](https://github.com/HenrikBengtsson/) for valuable insights leading to the interface accepting broader usage patterns.
[Daniel Falbel](https://github.com/dfalbel/) for discussion around an efficient solution to serialization and transmission of `torch` tensors.
[Kirill Müller](https://github.com/krlmlr/) for discussion on using 'daemons' to host Arrow database connections.
[
](https://r-consortium.org/) for funding work on the TLS implementation in `nanonext`, used to provide secure connections in `mirai`.
### Installation
Install the latest release from CRAN:
```{r cran, eval=FALSE}
install.packages("mirai")
```
The current development version is available from R-universe:
```{r universe, eval=FALSE}
install.packages("mirai", repos = "https://shikokuchuo.r-universe.dev")
```
### Links & References
◈ mirai R package:
◈ nanonext R package:
mirai is listed in CRAN High Performance Computing Task View:
--
Please note that this project is released with a [Contributor Code of Conduct](https://shikokuchuo.net/mirai/CODE_OF_CONDUCT.html). By participating in this project you agree to abide by its terms.