https://github.com/henrikbengtsson/future.mapreduce
[EXPERIMENTAL] R package: future.mapreduce - Utility Functions for Future Map-Reduce API Packages
https://github.com/henrikbengtsson/future.mapreduce
futures map-reduce package r
Last synced: 5 months ago
JSON representation
[EXPERIMENTAL] R package: future.mapreduce - Utility Functions for Future Map-Reduce API Packages
- Host: GitHub
- URL: https://github.com/henrikbengtsson/future.mapreduce
- Owner: HenrikBengtsson
- Created: 2020-06-22T00:53:16.000Z (almost 5 years ago)
- Default Branch: develop
- Last Pushed: 2024-04-19T01:13:33.000Z (about 1 year ago)
- Last Synced: 2024-05-09T10:35:51.949Z (about 1 year ago)
- Topics: futures, map-reduce, package, r
- Language: R
- Homepage: https://future.mapreduce.futureverse.org/
- Size: 279 KB
- Stars: 13
- Watchers: 4
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
Awesome Lists containing this project
README
# future.mapreduce: Utility Functions for Future Map-Reduce API Packages
The **future.mapreduce** package provides utility functions for other packages implementing map-reduce APIs on top of the **[future]** framework. Specifically, it will provide general functions for "load balancing", that is, methods for partitioning the elements to iterate over into chunks so that each chunk is processed by a single futures. Load balancing helps lower the overhead in parallel processing that comes from communicating with and orchestrating parallel workers. It will provide methods for common tasks such as globals handling and critical tasks such as parallel RNG in map-reduce contexts.
_WARNING: This package is currently under development. Please stay tuned._
This will benefit existing map-reduce packages **[future.apply]**, **[furrr]**, and **[doFuture]**, but also other similar efforts. This will further simply the implementing of these existing solutions as well as other future-based map-reduce APIs that might be on the horizon.
## Roadmap
* Migrate core functions from **future.apply** and **doFuture** to **future.mapreduce**
* Only export a minimal API and keep most functions internal for now
* Update **future.apply** and **doFuture** to import (also internal functions) from **future.mapreduce**
* Submit **future.mapreduce** to CRAN with a warning that the API is under development and should not be used by other packages than **future.apply** and **doFuture** until further notice
* Submit updated versions of **future.apply** and **doFuture**
* Work with **furrr** to make use of **future.mapreduce**
* When the internal API has stabilized, export it and submit to CRAN
[future]: https://cran.r-project.org/package=future
[future.apply]: https://cran.r-project.org/package=future.apply
[furrr]: https://cran.r-project.org/package=furrr
[doFuture]: https://cran.r-project.org/package=doFuture
## Installation
R package future.mapreduce is only available via [GitHub](https://github.com/HenrikBengtsson/future.mapreduce) and can be installed in R as:
```r
remotes::install_github("HenrikBengtsson/future.mapreduce", ref="master")
```
### Pre-release version
To install the pre-release version that is available in Git branch `develop` on GitHub, use:
```r
remotes::install_github("HenrikBengtsson/future.mapreduce", ref="develop")
```
This will install the package from source.
## Contributing
To contribute to this package, please see [CONTRIBUTING.md](CONTRIBUTING.md).