{"id":18520192,"url":"https://github.com/mlr-org/rush","last_synced_at":"2025-04-09T09:32:42.149Z","repository":{"id":189699907,"uuid":"676961990","full_name":"mlr-org/rush","owner":"mlr-org","description":"Parallel and distributed computing in R.","archived":false,"fork":false,"pushed_at":"2024-04-28T16:13:00.000Z","size":9180,"stargazers_count":4,"open_issues_count":8,"forks_count":0,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-05-01T09:37:23.024Z","etag":null,"topics":["mlr3","parallel-computing","r"],"latest_commit_sha":null,"homepage":"https://rush.mlr-org.com/","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mlr-org.png","metadata":{"files":{"readme":"README.Rmd","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"mlr-org"}},"created_at":"2023-08-10T12:17:33.000Z","updated_at":"2024-05-21T11:39:28.705Z","dependencies_parsed_at":"2023-10-12T02:57:30.165Z","dependency_job_id":"dc812781-831e-46f8-89ff-cb29e0de2edd","html_url":"https://github.com/mlr-org/rush","commit_stats":{"total_commits":71,"total_committers":2,"mean_commits":35.5,"dds":"0.18309859154929575","last_synced_commit":"1451787f0229d1a5fd9ac3bef52037081378f60e"},"previous_names":["mlr-org/rush"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlr-org%2Frush","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlr-org%2Frush/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlr-org%2Frush/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlr-org%2Frush/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mlr-org","download_url":"https://codeload.github.com/mlr-org/rush/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247994108,"owners_count":21030049,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mlr3","parallel-computing","r"],"created_at":"2024-11-06T17:18:57.277Z","updated_at":"2025-04-09T09:32:41.770Z","avatar_url":"https://github.com/mlr-org.png","language":"R","funding_links":["https://github.com/sponsors/mlr-org"],"categories":[],"sub_categories":[],"readme":"---\noutput: github_document\n---\n\n\u003c!-- README.md is generated from README.Rmd. Please edit that file --\u003e\n\n# rush\n\n*rush* is a package for parallel and distributed computing in R.\nIt evaluates an R expression asynchronously on a cluster of workers and provides a shared storage between the workers.\nThe shared storage is a [Redis](https://redis.io) data base.\nRush offers a centralized and decentralized network architecture.\nThe centralized network has a single controller (`Rush`) and multiple workers (`RushWorker`).\nTasks are created centrally and distributed to workers by the controller.\nThe decentralized network has no controller.\nThe workers sample tasks and communicate the results asynchronously with other workers.\n\n# Features\n\n* Parallelize arbitrary R expressions.\n* Centralized and decentralized network architecture.\n* Small overhead of a few milliseconds per task.\n* Easy start of local workers with `processx`\n* Start workers on any platform with a batch script.\n* Designed to work with [`data.table`](https://CRAN.R-project.org/package=data.table).\n* Results are cached in the R session to minimize read and write operations.\n* Detect and recover from worker failures.\n* Start heartbeats to monitor workers on remote machines.\n* Snapshot the in-memory data base to disk.\n* Store [`lgr`](https://CRAN.R-project.org/package=lgr) messages of the workers in the Redis data base.\n* Light on dependencies.\n\n## Install\n\nInstall the development version from GitHub.\n\n```{r eval = FALSE}\nremotes::install_github(\"mlr-org/rush\")\n```\n\nAnd install [Redis](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/).\n\n## Centralized Rush Network\n\n![](man/figures/README-flow.png)\n\n*Centralized network with a single controller and three workers.*\n\n```{r, include=FALSE}\nconfig = redux::redis_config()\nr = redux::hiredis(config)\nr$FLUSHDB()\n```\n\nThe example below shows the evaluation of a simple function in a centralized network.\nThe `network_id` identifies the instance and workers in the network.\nThe `config` is a list of parameters for the connection to Redis.\n\n```{r}\nlibrary(rush)\n\nconfig = redux::redis_config()\nrush = Rush$new(network_id = \"test\", config)\n\nrush\n```\n\nNext, we define a function that we want to evaluate on the workers.\n\n```{r}\nfun = function(x1, x2, ...) {\n  list(y = x1 + x2)\n}\n```\n\nWe start two workers.\n\n```{r}\nrush$start_local_workers(fun = fun, n_workers = 2)\n```\n\nNow we can push tasks to the workers.\n\n```{r}\nxss = list(list(x1 = 3, x2 = 5), list(x1 = 4, x2 = 6))\nkeys = rush$push_tasks(xss)\nrush$wait_for_tasks(keys)\n```\n\nAnd retrieve the results.\n\n```{r}\nrush$fetch_finished_tasks()\n```\n\n## Decentralized Rush Network\n\n![](man/figures/README-flow-2.png)\n\n*Decentralized network with four workers.*\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmlr-org%2Frush","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmlr-org%2Frush","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmlr-org%2Frush/lists"}