{"id":16572101,"url":"https://github.com/eddelbuettel/prrd","last_synced_at":"2025-03-16T20:31:21.502Z","repository":{"id":46621594,"uuid":"115268474","full_name":"eddelbuettel/prrd","owner":"eddelbuettel","description":"Parallel Running of Reverse Depends","archived":false,"fork":false,"pushed_at":"2025-02-15T02:16:25.000Z","size":1181,"stargazers_count":12,"open_issues_count":4,"forks_count":4,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-02-27T13:21:50.733Z","etag":null,"topics":["cran","hacktoberfest","r","r-package","reverse-dependencies"],"latest_commit_sha":null,"homepage":"","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eddelbuettel.png","metadata":{"files":{"readme":"README.md","changelog":"ChangeLog","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-12-24T14:34:49.000Z","updated_at":"2025-02-15T02:16:29.000Z","dependencies_parsed_at":"2023-12-23T20:45:18.780Z","dependency_job_id":"db82f32f-c80c-415e-bb77-ac07e6cafa55","html_url":"https://github.com/eddelbuettel/prrd","commit_stats":{"total_commits":120,"total_committers":4,"mean_commits":30.0,"dds":"0.025000000000000022","last_synced_commit":"060b0425535df5e0bdfcccce649d3353875ef73a"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eddelbuettel%2Fprrd","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eddelbuettel%2Fprrd/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eddelbuettel%2Fprrd/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eddelbuettel%2Fprrd/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eddelbuettel","download_url":"https://codeload.github.com/eddelbuettel/prrd/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243830912,"owners_count":20354848,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cran","hacktoberfest","r","r-package","reverse-dependencies"],"created_at":"2024-10-11T21:26:22.513Z","updated_at":"2025-03-16T20:31:20.735Z","avatar_url":"https://github.com/eddelbuettel.png","language":"R","funding_links":[],"categories":[],"sub_categories":[],"readme":"## prrd: Parallel Running [of] Reverse Depends\n \n[![CI](https://github.com/eddelbuettel/prrd/workflows/ci/badge.svg)](https://github.com/eddelbuettel/prrd/actions?query=workflow%3Aci)\n[![License](https://eddelbuettel.github.io/badges/GPL2+.svg)](https://www.gnu.org/licenses/gpl-2.0.html)\n[![CRAN](https://www.r-pkg.org/badges/version/prrd)](https://cran.r-project.org/package=prrd)\n[![r-universe](https://eddelbuettel.r-universe.dev/badges/prrd)](https://eddelbuettel.r-universe.dev/prrd)\n[![Downloads](https://cranlogs.r-pkg.org/badges/prrd?color=brightgreen)](https://www.r-pkg.org:443/pkg/prrd)\n[![Last Commit](https://img.shields.io/github/last-commit/eddelbuettel/prrd)](https://github.com/eddelbuettel/prrd)\n\n### Motivation\n\n[R](https://www.r-project.org) packages available via the [CRAN](https://cran.r-project.org) mirror\nnetwork system are of consistently high quality and tend to \"Just Work\".  One of the many reasons\nfor this is a good culture of \"do not break other packages\" which is controlled for / enacted by the\nCRAN maintainers. Package maintainers are expected to do their part---by checking their packages!\n\nTo take one example, [Rcpp](http://dirk.eddelbuettel.com/code/rcpp.html) is package with a pretty\nlarge tail of dependencies: as of this writing in late 2017, about 1270 other packages use it.  So\n1270 other packages need to be tested.  This takes time, especially when running serially.  But it\nis easy to parallelise.\n\n### How\n\nPreviously, a few ad-hoc scripts (available\n[here](https://github.com/RcppCore/rcpp-logs/tree/master/scripts)) were used for a number of\npackages.  The scripts were one-offs and did their job. But with the idea of running jobs in\nparallel, the [liteq](https://cran.r-project.org/package=liteq) package by\n[Gabor Csardi](https://github.com/gaborcsardi) fit the requirements nicely.\n\n#### Enqueuing\n\nThe first operation is to _enqueue_ jobs. In the simplest form we do (assuming the included script\nis in the `PATH`)\n\n```sh\n$ enqueueJobs -q queueDirectory Rcpp\n```\n\nThe same operation can also be done from R itself, see `help(enqueueJobs)`.  A package name has to\nbe supplied; a directory name (for the queue directory) is optional.  This function uses two base R \nfunctions to get all available packages and then determine the (non-recursive, first-level) reverse\ndependencies of the given package. These are then added to the queue as \"jobs\".\n\n#### Dequeuing\n\nThis is the second operation, and it can be done in parallel.  In other words, in several shells \ndo:\n\n```sh\n$ dequeueJobs -q queueDirectory Rcpp\n```\n\nwhich will find the (current) queue file in the specified directory for the given package---here\n`Rcpp`.  Again, this can also be done from an R prompt if prefered, see `help(dequeueJobs)`.\n\nEach worker, when idle, goes to the queue and requests a job, which it then labors over by testing\nthe thus-given reverse depedency.  Once done, the worker is idle and returns to the queue and the\ncycle repeats. \n\nAs there is absolutely no interdepedence between the tests, this parallelises easily up to the\nresource level of the machine.\n\n### Performance \n\nTo illustrate, \"wall time\" for a reverse-dependecy check of\n[Rcpp](http://dirk.eddelbuettel.com/code/rcpp.html) decreased from 14.91 hours to 3.75 hours (or\nalmost four-fold) using six workers. An earlier run of\n[RcppArmadillo](http://dirk.eddelbuettel.com/code/rcpp.armadillo.html) decreased from 5.87 hours to\n1.92 hours (or just over three-fold) using four workers, and to 1.29 hours (or by 4.5) using six\nworkers (and a fresh `ccache`, see\n[here](https://dirk.eddelbuettel.com/blog/2017/11/27/) for its\nimpact).  In all cases the machine which was used was generally not idle.\n\nThe following screenshot shows a run for\n[RcppArmadillo](http://dirk.eddelbuettel.com/code/rcpp.armadillo.html) with six workers. It shows\nthe successes in green, skipped jobs in blue (from packages which sometimes would result in runaway tests), and no failures (which would be shown in red).\n\n![](https://github.com/eddelbuettel/prrd/raw/master/local/screenshot_prrd_rcpparmadillo.png)\n\nThe split screen and additional tables are thanks to the wonderful\n[byobu](https://byobu.org) wrapper around [tmux](https://github.com/tmux/tmux).\n\n### Configuration\n\nThe scripts use an internal YAML file access via the\n[config](https://cran.r-project.org/package=config) package by JJ. The following locations are\nsearched: `.prrd.yaml`, `~/.R/prrd.yaml`, `~/.prrd.yaml`, and `/etc/R/prrd.yaml`.  For my initial\ntests I used these values:\n\n```\n## config for prrd package\ndefault:\n  setup: \"~/git/prrd/local/setup.R\"\n  workdir: \"/tmp/prrd\"\n  libdir: \"/tmp/prrd/lib\"\n  debug: \"false\"\n  verbose: \"false\"\n```\n\nThe `workdir` and `libdir` variables specify where tests are run and which additonal library\ndirectory is used.  A more interesting variable is `setup`, which points to a helper script which\ngets sourced.  This permits setting of the CRAN repo address and any additonal environment\nvariables needed for tests. My current script is\n[in the repository](https://github.com/eddelbuettel/prrd/blob/master/local/setup.R).\n\n\n### Status\n\nWhile the package is new, it has already been used for a few complete reverse depends tests runs.\n\n### Installation\n\nThe package is now on [CRAN](https://cran.r-project.org/) and can be installed via a standard\n```\ninstall.packages(\"prrd\")\n```\n\n### Authors\n\nDirk Eddelbuettel\n\n### License\n\nGPL (\u003e= 2)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feddelbuettel%2Fprrd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feddelbuettel%2Fprrd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feddelbuettel%2Fprrd/lists"}