{"id":14067618,"url":"https://github.com/lorenzwalthert/touchstone","last_synced_at":"2025-07-30T16:10:26.628Z","repository":{"id":41867232,"uuid":"311152527","full_name":"lorenzwalthert/touchstone","owner":"lorenzwalthert","description":"Smart benchmarking of pull requests with statistical confidence","archived":false,"fork":false,"pushed_at":"2025-04-21T19:09:01.000Z","size":1490,"stargazers_count":56,"open_issues_count":16,"forks_count":8,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-24T06:02:19.264Z","etag":null,"topics":["benchmark","ci-cd","r"],"latest_commit_sha":null,"homepage":"https://lorenzwalthert.github.io/touchstone","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lorenzwalthert.png","metadata":{"files":{"readme":"README.Rmd","changelog":"NEWS.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-11-08T20:51:56.000Z","updated_at":"2025-04-09T20:48:02.000Z","dependencies_parsed_at":"2024-02-17T05:19:48.009Z","dependency_job_id":"9aed9fba-3d1e-4ff6-924c-f184db5c6e83","html_url":"https://github.com/lorenzwalthert/touchstone","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lorenzwalthert%2Ftouchstone","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lorenzwalthert%2Ftouchstone/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lorenzwalthert%2Ftouchstone/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lorenzwalthert%2Ftouchstone/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lorenzwalthert","download_url":"https://codeload.github.com/lorenzwalthert/touchstone/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252996857,"owners_count":21837721,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","ci-cd","r"],"created_at":"2024-08-13T07:05:41.547Z","updated_at":"2025-05-08T04:19:30.832Z","avatar_url":"https://github.com/lorenzwalthert.png","language":"R","funding_links":[],"categories":["R"],"sub_categories":[],"readme":"---\noutput: github_document\n---\n\n\u003c!-- badges: start --\u003e [![Lifecycle:\nexperimental](https://img.shields.io/badge/lifecycle-experimental-orange.svg)](https://www.tidyverse.org/lifecycle/#experimental)\n[![R build\nstatus](https://github.com/lorenzwalthert/touchstone/workflows/R-CMD-check/badge.svg)](https://github.com/lorenzwalthert/touchstone/actions)\n\u003c!-- badges: end --\u003e\n\n```{r, include = FALSE}\nknitr::opts_chunk$set(\n  collapse = TRUE,\n  comment = \"#\u003e\",\n  fig.path = \"man/figures/README-\",\n  out.width = \"100%\",\n  eval = FALSE\n)\n```\n\n# touchstone\n\n{touchstone} is a developer tool for continuous benchmarking with a focus\non reliable relative measurement, uncertainty reporting and user convenience.\nThe results are directly reported as a comment in GitHub Pull Requests.\n\n![](man/figures/screenshot-pr-comment.png)\n\n## Installation\n\nYou can install the package from CRAN:\n\n``` r\ninstall.packages(\"touchstone\")\n```\n\nAnd the development version from [GitHub](https://github.com/lorenzwalthert/touchstone){target=\"_blank\"} with:\n\n``` r\n# install.packages(\"devtools\")\ndevtools::install_github(\"lorenzwalthert/touchstone\")\n```\n\n## Getting Started\nYou can start using {touchstone} in your package repository with:\n```{r, eval = FALSE}\ntouchstone::use_touchstone()\n```\nFor a detailed explanation on how to configure and use {touchstone} see the \n[\"Using touchstone\"](https://lorenzwalthert.github.io/touchstone/articles/touchstone.html) vignette.\n\n## Motivation\n\nThe motivation for touchstone is to provide accurate benchmarking results for\npackage developers. The following insights were the motivation:\n\n- **Compute power in GitHub Action VMs varies too much for reliable isolated\n  benchmarks**: Experience with styler showed that a variation [around\n  30%](https://github.com/r-lib/styler/pull/679) for identical benchmarking code\n  is possible. The solution to this is to benchmark the two branches in one\n  CI/CD run and look at *relative difference* between branches. This matters in\n  particular when running one iteration of a benchmark takes long (\u003e\u003e a few\n  seconds) and speed implications are not huge.\n\n- **Timelines of absolute changes are mostly noise:** Maintaining a timeline of\n  absolute benchmark times is of limited use because of the first bullet, at\n  least when benchmark results don't vary significantly (\u003e 50%).\n\n- **Dependencies should be identical across versions:** R and package versions\n  of dependencies must be fixed via [RSPM](http://packagemanager.rstudio.com) to\n  allow as much continuation as possible anyways. Changing the timestamp of RSPM\n  can happen in PRs that are only dedicated to dependency updates.\n\n- **Pull requests are a natural unit for measuring speed:** Linking benchmarking\n  to pull requests make sense because you can easily benchmark any revision\n  against any other. Just open a pull request from one branch on another. You\n  can also easily keep track of how performance of your development version\n  evolves by opening a PR against a branch with the released version. No need to\n  ever merge these. Once you release a new version of you package, you can\n  change the target branch of the pull request to start anew. The pull request\n  comments will preserve the information.\n\n## Conceptual \n\nFor your PR branch and the target branch, {touchstone} will:\n\n* build the two versions of the package in isolated libraries.\n\n* run the code you want to benchmark, many times, in [random\n  order](https://lorenzwalthert.github.io/touchstone/articles/inference.html#sampling).\n  This ensures the accurate measurement of relative differences between the\n  branches.\n\nOnce done with the measurements, it will\n\n* comment the results of the benchmarking on the PR.\n\n* create visualizations as Github Action artifacts that show how the\n  distribution of the timings for both branches.\n\n## Status\n\nThis package is experimental. You can see an example usage in\n[{styler}](https://github.com/r-lib/styler/pull/799).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Florenzwalthert%2Ftouchstone","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Florenzwalthert%2Ftouchstone","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Florenzwalthert%2Ftouchstone/lists"}