{"id":28549790,"url":"https://github.com/tidyverse/duckplyr","last_synced_at":"2025-07-05T03:31:40.845Z","repository":{"id":66206536,"uuid":"571945396","full_name":"tidyverse/duckplyr","owner":"tidyverse","description":"A drop-in replacement for dplyr, powered by DuckDB for speed.","archived":false,"fork":false,"pushed_at":"2025-05-10T02:14:09.000Z","size":16354,"stargazers_count":333,"open_issues_count":70,"forks_count":20,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-07-02T13:55:17.019Z","etag":null,"topics":["analytics","dataframe","dplyr","duckdb","performance","r"],"latest_commit_sha":null,"homepage":"https://duckplyr.tidyverse.org/","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tidyverse.png","metadata":{"files":{"readme":"README.Rmd","changelog":"NEWS.md","contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-11-29T08:20:48.000Z","updated_at":"2025-07-02T12:28:39.000Z","dependencies_parsed_at":"2023-10-05T03:43:17.011Z","dependency_job_id":"e72ae1d3-5659-450a-a659-ff74bb75f053","html_url":"https://github.com/tidyverse/duckplyr","commit_stats":null,"previous_names":["tidyverse/duckplyr","duckdblabs/duckplyr"],"tags_count":119,"template":false,"template_full_name":null,"purl":"pkg:github/tidyverse/duckplyr","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tidyverse%2Fduckplyr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tidyverse%2Fduckplyr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tidyverse%2Fduckplyr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tidyverse%2Fduckplyr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tidyverse","download_url":"https://codeload.github.com/tidyverse/duckplyr/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tidyverse%2Fduckplyr/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263154352,"owners_count":23422011,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","dataframe","dplyr","duckdb","performance","r"],"created_at":"2025-06-10T02:30:43.698Z","updated_at":"2025-07-05T03:31:40.840Z","avatar_url":"https://github.com/tidyverse.png","language":"R","funding_links":[],"categories":[],"sub_categories":[],"readme":"---\noutput:\n  github_document:\n    html_preview: false\n---\n\n\u003c!-- README.md and index.md are generated from README.Rmd. Please edit that file. --\u003e\n\n```{r setup, include = FALSE}\nknitr::opts_chunk$set(\n  collapse = TRUE,\n  comment = \"#\u003e\",\n  fig.path = \"man/figures/dropin-\",\n  out.width = \"100%\"\n)\n\nSys.setenv(DUCKPLYR_META_ENABLE = FALSE)\n\nset.seed(20230702)\n\nclean_output \u003c- function(x, options) {\n  x \u003c- gsub(\"0x[0-9a-f]+\", \"0xdeadbeef\", x)\n  x \u003c- gsub(\"dataframe_[0-9]*_[0-9]*\", \"      dataframe_42_42      \", x)\n  x \u003c- gsub(\"[0-9]*\\\\.___row_number ASC\", \"42.___row_number ASC\", x)\n\n  index \u003c- x\n  index \u003c- gsub(\"─\", \"-\", index)\n  index \u003c- strsplit(paste(index, collapse = \"\\n\"), \"\\n---\\n\")[[1]][[2]]\n  writeLines(index, \"index.md\")\n\n  x \u003c- gsub('(`vignette[(]\"([^\"]+)\"[)]`)', \"[\\\\1](https://duckplyr.tidyverse.org/articles/\\\\2.html)\", x)\n  x \u003c- fansi::strip_sgr(x)\n  x\n}\n\noptions(\n  cli.num_colors = 256,\n  cli.width = 71,\n  width = 71,\n  pillar.bold = TRUE,\n  pillar.max_title_chars = 5,\n  pillar.min_title_chars = 5,\n  pillar.max_footer_lines = 12,\n  conflicts.policy = list(warn = FALSE)\n)\n\nlocal({\n  hook_source \u003c- knitr::knit_hooks$get(\"document\")\n  knitr::knit_hooks$set(document = clean_output)\n})\n```\n\n# duckplyr \u003ca href=\"https://duckplyr.tidyverse.org\"\u003e\u003cimg src=\"man/figures/logo.png\" align=\"right\" height=\"138\" /\u003e\u003c/a\u003e\n\n\u003c!-- badges: start --\u003e\n[![Lifecycle: stable](https://img.shields.io/badge/lifecycle-stable-brightgreen.svg)](https://lifecycle.r-lib.org/articles/stages.html#stable)\n[![R-CMD-check](https://github.com/tidyverse/duckplyr/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/tidyverse/duckplyr/actions/workflows/R-CMD-check.yaml)\n[![Codecov test coverage](https://codecov.io/gh/tidyverse/duckplyr/graph/badge.svg)](https://app.codecov.io/gh/tidyverse/duckplyr)\n\u003c!-- badges: end --\u003e\n\n\u003e A **drop-in replacement** for dplyr, powered by DuckDB for **speed**.\n\n[dplyr](https://dplyr.tidyverse.org/) is the grammar of data manipulation in the tidyverse.\nThe duckplyr package will run all of your existing dplyr code with identical results, using [DuckDB](https://duckdb.org/) where possible to compute the results faster.\nIn addition, you can analyze larger-than-memory datasets straight from files on your disk or from the web.\n\nIf you are new to dplyr, the best place to start is the [data transformation chapter](https://r4ds.hadley.nz/data-transform) in _R for Data Science_.\n\n\n## Installation\n\nInstall duckplyr from CRAN with:\n\n``` r\ninstall.packages(\"duckplyr\")\n```\n\nYou can also install the development version of duckplyr from [R-universe](https://tidyverse.r-universe.dev/builds):\n\n``` r\ninstall.packages(\"duckplyr\", repos = c(\"https://tidyverse.r-universe.dev\", \"https://cloud.r-project.org\"))\n```\n\nOr from [GitHub](https://github.com/) with:\n\n``` r\n# install.packages(\"pak\")\npak::pak(\"tidyverse/duckplyr\")\n```\n\n## Drop-in replacement for dplyr\n\nCalling `library(duckplyr)` overwrites dplyr methods, enabling duckplyr for the entire session.\n\n```{r dropin-attach}\nlibrary(conflicted)\nlibrary(duckplyr)\n```\n\n```{r load-all, include = FALSE}\n# Done after library(duckplyr) to leave the original output\npkgload::load_all()\n```\n\n```{r dropin-simulate-library, echo = FALSE}\nSys.setenv(DUCKPLYR_FALLBACK_COLLECT = 0)\n```\n\n```{r dropin-dplyr}\nconflict_prefer(\"filter\", \"dplyr\")\n```\n\nThe following code aggregates the inflight delay by year and month for the first half of the year.\nWe use a variant of the `nycflights13::flights` dataset, where the timezone has been set to UTC to work around a current limitation of duckplyr, see `vignette(\"limits\")`.\n\n```{r dropin-pipeline}\nflights_df()\n\nout \u003c-\n  flights_df() |\u003e\n  filter(!is.na(arr_delay), !is.na(dep_delay)) |\u003e\n  mutate(inflight_delay = arr_delay - dep_delay) |\u003e\n  summarize(\n    .by = c(year, month),\n    mean_inflight_delay = mean(inflight_delay),\n    median_inflight_delay = median(inflight_delay),\n  ) |\u003e\n  filter(month \u003c= 6)\n```\n\nThe result is a plain tibble:\n\n```{r dropin-class}\nclass(out)\n```\n\nNothing has been computed yet.\nQuerying the number of rows, or a column, starts the computation:\n\n```{r dropin-touch, cache = TRUE}\nout$month\n```\n\nNote that, unlike dplyr, the results are not ordered, see `?config` for details.\nHowever, once materialized, the results are stable:\n\n```{r dropin-stable, cache = TRUE}\nout\n```\n\nIf a computation is not supported by DuckDB, duckplyr will automatically fall back to dplyr.\n\n```{r dropin-fallback, cache = TRUE}\nflights_df() |\u003e\n  summarize(\n    .by = origin,\n    dest = paste(sort(unique(dest)), collapse = \" \")\n  )\n```\n\nRestart R, or call `duckplyr::methods_restore()` to revert to the default dplyr implementation.\n\n```{r dropin-restore}\nduckplyr::methods_restore()\n```\n\n## Analyzing larger-than-memory data\n\nAn extended variant of the `nycflights13::flights` dataset is also available for download as Parquet files.\n\n```{r remote-url}\nyear \u003c- 2022:2024\nbase_url \u003c- \"https://blobs.duckdb.org/flight-data-partitioned/\"\nfiles \u003c- paste0(\"Year=\", year, \"/data_0.parquet\")\nurls \u003c- paste0(base_url, files)\ntibble(urls)\n```\n\nUsing the [httpfs DuckDB extension](https://duckdb.org/docs/extensions/httpfs/overview.html), we can query these files directly from R, without even downloading them first.\n\n```{r remote-https}\ndb_exec(\"INSTALL httpfs\")\ndb_exec(\"LOAD httpfs\")\n\nflights \u003c- read_parquet_duckdb(urls)\n```\n\nLike with local data frames, queries on the remote data are executed lazily.\nUnlike with local data frames, the default is to disallow automatic materialization if the result is too large in order to protect memory: the results are not materialized until explicitly requested, with a `collect()` call for instance.\n\n```{r remote-thrifty, error = TRUE}\nnrow(flights)\n```\n\nFor printing, only the first few rows of the result are fetched.\n\n```{r remote-print, cache = TRUE}\nflights\n```\n\n```{r remote-count, cache = TRUE}\nflights |\u003e\n  count(Year)\n```\n\nComplex queries can be executed on the remote data.\nNote how only the relevant columns are fetched and the 2024 data isn't even touched, as it's not needed for the result.\n\n```{r remote-exec, cache = TRUE}\nout \u003c-\n  flights |\u003e\n  mutate(InFlightDelay = ArrDelay - DepDelay) |\u003e\n  summarize(\n    .by = c(Year, Month),\n    MeanInFlightDelay = mean(InFlightDelay, na.rm = TRUE),\n    MedianInFlightDelay = median(InFlightDelay, na.rm = TRUE),\n  ) |\u003e\n  filter(Year \u003c 2024)\n\nout |\u003e\n  explain()\n\nout |\u003e\n  print() |\u003e\n  system.time()\n```\n\nOver 10M rows analyzed in about 10 seconds over the internet, that's not bad.\nOf course, working with Parquet, CSV, or JSON files downloaded locally is possible as well.\n\nFor full compatibility, `na.rm = FALSE` by default in the aggregation functions:\n\n```{r remote-na-rm, cache = TRUE}\nflights |\u003e\n  summarize(mean(ArrDelay - DepDelay))\n```\n\n\n## Further reading\n\n- `vignette(\"large\")`: Tools for working with large data\n\n- `vignette(\"prudence\")`: How duckplyr can help protect memory when working with large data\n\n- `vignette(\"fallback\")`: How the fallback to dplyr works internally\n\n- `vignette(\"limits\")`: Translation of dplyr employed by duckplyr, and current limitations\n\n- `vignette(\"duckdb\")`: Using the full power of DuckDB\n\n- `vignette(\"developers\")`: Using duckplyr for individual data frames and in other packages\n\n- `vignette(\"telemetry\")`: Telemetry in duckplyr\n\n\n## Getting help\n\nIf you encounter a clear bug, please file an issue with a minimal reproducible example on [GitHub](https://github.com/tidyverse/duckplyr/issues). For questions and other discussion, please use [forum.posit.co](https://forum.posit.co/).\n\n\n## Code of conduct\n\nPlease note that this project is released with a [Contributor Code of Conduct](https://duckplyr.tidyverse.org/CODE_OF_CONDUCT).\nBy participating in this project you agree to abide by its terms.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftidyverse%2Fduckplyr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftidyverse%2Fduckplyr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftidyverse%2Fduckplyr/lists"}