{"id":32199557,"url":"https://github.com/ropensci/butterfly","last_synced_at":"2026-02-20T16:01:58.787Z","repository":{"id":257846298,"uuid":"870725403","full_name":"ropensci/butterfly","owner":"ropensci","description":"Verification of continually updating timeseries data where we expect new values, but want to ensure previous data remains unchanged. Maintained by @thomaszwagerman","archived":false,"fork":false,"pushed_at":"2025-04-04T08:09:16.000Z","size":2762,"stargazers_count":10,"open_issues_count":4,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-12-09T04:30:35.579Z","etag":null,"topics":["data-versioning","qaqc","r","r-package","rstats","timeseries","verification"],"latest_commit_sha":null,"homepage":"https://docs.ropensci.org/butterfly","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ropensci.png","metadata":{"files":{"readme":"README.Rmd","changelog":"NEWS.md","contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":"codemeta.json","zenodo":null}},"created_at":"2024-10-10T14:55:30.000Z","updated_at":"2025-11-20T19:36:25.000Z","dependencies_parsed_at":"2025-01-10T16:45:15.982Z","dependency_job_id":"1cd48100-ad85-4d4f-ae71-63ead00eea25","html_url":"https://github.com/ropensci/butterfly","commit_stats":null,"previous_names":["thomaszwagerman/butterfly","antarctica/butterfly","ropensci/butterfly"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/ropensci/butterfly","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ropensci%2Fbutterfly","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ropensci%2Fbutterfly/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ropensci%2Fbutterfly/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ropensci%2Fbutterfly/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ropensci","download_url":"https://codeload.github.com/ropensci/butterfly/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ropensci%2Fbutterfly/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29656589,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-20T09:27:29.698Z","status":"ssl_error","status_checked_at":"2026-02-20T09:26:12.373Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-versioning","qaqc","r","r-package","rstats","timeseries","verification"],"created_at":"2025-10-22T03:19:20.316Z","updated_at":"2026-02-20T16:01:58.782Z","avatar_url":"https://github.com/ropensci.png","language":"R","funding_links":[],"categories":[],"sub_categories":[],"readme":"---\noutput: github_document\n---\n\n\u003c!-- README.md is generated from README.Rmd. Please edit that file --\u003e\n\n```{r, include = FALSE}\nknitr::opts_chunk$set(\n  collapse = TRUE,\n  comment = \"#\u003e\",\n  fig.path = \"man/figures/README-\",\n  out.width = \"100%\"\n)\n```\n\n# butterfly \u003ca href=\"https://github.com/ropensci/butterfly\"\u003e\u003cimg src=\"man/figures/logo.png\" align=\"right\" height=\"139\" alt=\"butterfly website\" /\u003e\u003c/a\u003e\n\n\u003c!-- badges: start --\u003e\n[![R-CMD-check](https://github.com/ropensci/butterfly/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/ropensci/butterfly/actions/workflows/R-CMD-check.yaml)\n[![Codecov test coverage](https://codecov.io/gh/thomaszwagerman/butterfly/branch/main/graph/badge.svg)](https://app.codecov.io/gh/ropensci/butterfly?branch=main)\n[![Lifecycle: stable](https://img.shields.io/badge/lifecycle-stable-brightgreen.svg)](https://lifecycle.r-lib.org/articles/stages.html#stable)\n[![Project Status: Active – The project has reached a stable, usable state and is being actively developed.](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active)\n[![pkgcheck](https://github.com/ropensci/butterfly/workflows/pkgcheck/badge.svg)](https://github.com/ropensci/butterfly/actions?query=workflow%3Apkgcheck)\n[![Status at rOpenSci Software Peer Review](https://badges.ropensci.org/676_status.svg)](https://github.com/ropensci/software-review/issues/676)\n\u003c!-- badges: end --\u003e\n\nThe goal of butterfly is to aid in the verification of continually updating timeseries data, where we expect new values over time, but want to ensure previous data remains unchanged, and timesteps remain continuous. \n\n```{r butterfly_diagram, echo=FALSE, out.width=\"100%\", fig.cap=\"An illustration of continually updating timeseries data where a previous value unexpectedly changes.\"}\nknitr::include_graphics(\"man/figures/README-butterfly_diagram.png\")\n```\n\n\nData previously recorded could change for a number of reasons, such as discovery of an error in model code, a change in methodology or instrument recalibration. Monitoring data sources for these changes is not always possible.\n\nUnnoticed changes in previous data could have unintended consequences, such as invalidating a published dataset's Digital Object Identfier (DOI), or altering future predictions if used as input in forecasting models.\n\nOther unnoticed changes could include a jump in time or measurement frequency, due to instrument failure or software updates.\n\n```{r timeseries_diagram, echo=FALSE, out.width=\"100%\", fig.cap=\"An illustration of timeseries data not being continuous in the way it is expected to be.\"}\nknitr::include_graphics(\"man/figures/README-timeseries_dark.png\")\n```\n\nThis package provides functionality that can be used as part of a data pipeline, to check and flag changes to previous data to prevent changes going unnoticed.\n\n## Installation\n\nYou can install butterfly with:\n\n``` r\ninstall.packages(\"butterfly\", repos = \"https://ropensci.r-universe.dev\")\n```\n\n## Overview\nThe butterfly package contains the following functions:\n\n  * `butterfly::loupe()` - examines in detail whether previous values have changed, and returns TRUE/FALSE for no change/change.\n  * `butterfly::catch()` - returns rows which contain previously changed values in a dataframe.\n  * `butterfly::release()` - drops rows which contain previously changed values, and returns a dataframe containing new and unchanged rows.\n  * `butterfly::create_object_list()` - returns a list of objects required by all of `loupe()`, `catch()` and `release()`. Contains underlying functionality.\n  * `butterfly::timeline()` - check if a timeseries is continuous between timesteps.\n  * `butterfly::timeline_group()` - group distinct, but continuous sequences of a timeseries.\n\nThere are also dummy datasets, which a fictional and purely to demonstrate butterfly functionality:\n\n  * `butterflycount` - a list of monthly dataframes, which contain fictional butterfly counts for a given date.\n  * `forestprecipitation` - a list of monthly dataframes, which contain fictional daily precipitation measurements for a given date.\n  * `butterflymess` - a messy version of `butterflycount`, provided for testing purposes.\n\n## Examples\n\nThis is a basic example which shows you how to use butterfly:\n\n```{r simple_example}\nlibrary(butterfly)\n\n# Imagine a continually updated dataset that starts in January and is updated once a month\nbutterflycount$january\n\n# In February an additional row appears, all previous data remains the same\nbutterflycount$february\n\n# In March an additional row appears again\n# ...but a previous value has unexpectedly changed\nbutterflycount$march\n```\n\nWe can use `butterfly::loupe()` to examine in detail whether previous values have changed.\n\n```{r butterfly_example}\nbutterfly::loupe(\n  butterflycount$february,\n  butterflycount$january,\n  datetime_variable = \"time\"\n)\n\nbutterfly::loupe(\n  butterflycount$march,\n  butterflycount$february,\n  datetime_variable = \"time\"\n)\n```\n\n`butterfly::loupe()` uses `dplyr::semi_join()` to match the new and old objects using a common unique identifier, which in a timeseries will be the timestep. `waldo::compare()` is then used to compare these and provide a detailed report of the differences.\n\n`butterfly` follows the `waldo` philosophy of erring on the side of providing too much information, rather than too little. It will give a detailed feedback message on the status between two objects.\n\n### Using butterfly for data wrangling\nYou might want to return changed rows as a dataframe, or drop them altogether. For this `butterfly::catch()` and `butterfly::release()` are provided.\n\nHere, `butterfly::catch()` only returns rows which have **changed** from the previous version. It will not return new rows.\n\n```{r butterfly_catch}\ndf_caught \u003c- butterfly::catch(\n  butterflycount$march,\n  butterflycount$february,\n  datetime_variable = \"time\"\n)\n\ndf_caught\n```\n\nConversely, `butterfly::release()` drops all rows which had changed from the previous version. Note it retains new rows, as these were expected.\n\n```{r butterfly_release}\ndf_released \u003c- butterfly::release(\n  butterflycount$march,\n  butterflycount$february,\n  datetime_variable = \"time\"\n)\n\ndf_released\n```\n\n### Checking for continuity: `timeline()`\n\nTo check if a timeseries is continuous, `timeline()` and `timeline_group()` are\nprovided.\n\n```{r rain_gauge_data}\n# A rain gauge which measures precipitation every day\nbutterfly::forestprecipitation$january\n\n# In February there is a power failure in the instrument\nbutterfly::forestprecipitation$february\n```\n\nTo check if a timeseries is continuous:\n\n```{r check_continuity}\nbutterfly::timeline(\n   forestprecipitation$january,\n   datetime_variable = \"time\",\n   expected_lag = 1\n )\n```\n\nIn February our imaginary rain gauge's onboard computer had a failure.\n\nThe timestamp was reset to `1970-01-01`:\n \n```{r not_continuous}\nforestprecipitation$february\n\nbutterfly::timeline(\n  forestprecipitation$february,\n   datetime_variable = \"time\",\n   expected_lag = 1\n )\n```\n\nIf we wanted to group chunks of our timeseries that are distinct, or broken up\nin some way, but still continuous, we can use `timeline_group()`:\n\n```{r timeline_group}\nbutterfly::timeline_group(\n  forestprecipitation$february,\n   datetime_variable = \"time\",\n   expected_lag = 1\n )\n```\n\n## Relevant packages and functions\nThe butterfly package was created for a specific use case of handling continuously updating/overwritten timeseries data, where previous values may change without notice. \n\nThere are other R packages and functions which handle object comparison, which may suit your specific needs better. Below we describe their overlap and differences to `butterfly`:\n\n* [waldo](https://github.com/r-lib/waldo) - `butterfly` uses `waldo::compare()` in every function to provide a report on difference. There is therefore significant overlap, however `butterfly` builds on `waldo` by providing the functionality of comparing objects where we expect some changes, with previous versions but not others. `butterfly` also provides extra user feedback to provide clarity on what it is and isn't comparing, due to the nature of comparing only \"matched\" rows.\n* [diffdf](https://github.com/gowerc/diffdf) - similar to `waldo`, but specifically for data frames, `diffdf` provides the ability to compare data frames directly. We could have used `diffdf::diffdf()` in our case, but we prefer `waldo`'s more explicit and clear user feedback. That said, there is significant overlap in functionality: `butterfly::loupe()` and `diffdf::diffdf_has_issues()` both provide a TRUE/FALSE difference check, while `diffdf::diffdf_issue_rows()` and `butterfly::catch()` both return the rows where changes have occurred. However, it lacks the flexibility of `butterfly` to compare object where we expect some changes, but not others.\n* [assertr](https://github.com/tonyfischetti/assertr) - `assertr` provides assertion functionality that can be used as part of a pipeline, and test assertions on a particular dataset, but it does not offer tools for comparison. We do highly recommend using `assertr` for checks, prior to using `butterfly`, as any data quality issues will be caught first.\n* [daquiri](https://github.com/ropensci/daiquiri/) - `daquiri` provides tools to check data quality and visually inspect timeseries data. It is also quality assurance package for timeseries, but has a very different purpose to `butterfly`.\n\nOther functions include `all.equal()` (base R) or [dplyr](https://github.com/tidyverse/dplyr)'s `setdiff()`.\n\n## `butterfly` in production\n\nRead more about how `butterfly` is [used in an operational data pipeline](https://docs.ropensci.org/butterfly/articles/butterfly_in_pipeline.html) to verify a continually updated **and** published dataset.\n\n## Contributing\n\nFor full guidance on contributions, please refer to `.github/CONTRIBUTING.md`.\n\n### Without write access\nCorrections, suggestions and general improvements are welcome as issues.\n\nYou can also suggest changes by forking this repository, and opening a pull request. Please target your pull requests to the main branch.\n\n### With write access\nYou can push directly to main for small fixes. Please use PRs to main for discussing larger updates.\n\n## Code of Conduct\n\nPlease note that this package is released with a [Contributor\nCode of Conduct](https://ropensci.org/code-of-conduct/). \n\nBy contributing to this project, you agree to abide by its terms.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fropensci%2Fbutterfly","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fropensci%2Fbutterfly","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fropensci%2Fbutterfly/lists"}