{"id":26822673,"url":"https://github.com/dylanpieper/redquack","last_synced_at":"2025-04-26T10:46:41.478Z","repository":{"id":282922483,"uuid":"949154933","full_name":"dylanpieper/redquack","owner":"dylanpieper","description":"REDCap and DuckDB Flock Together","archived":false,"fork":false,"pushed_at":"2025-04-25T13:23:58.000Z","size":9990,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-25T22:38:41.735Z","etag":null,"topics":["duckdb","r","redcap"],"latest_commit_sha":null,"homepage":"https://dylanpieper.github.io/redquack/","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dylanpieper.png","metadata":{"files":{"readme":"README.md","changelog":"NEWS.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-03-15T19:58:05.000Z","updated_at":"2025-04-25T13:22:23.000Z","dependencies_parsed_at":"2025-04-25T13:57:44.376Z","dependency_job_id":null,"html_url":"https://github.com/dylanpieper/redquack","commit_stats":null,"previous_names":["dylanpieper/redquack"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dylanpieper%2Fredquack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dylanpieper%2Fredquack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dylanpieper%2Fredquack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dylanpieper%2Fredquack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dylanpieper","download_url":"https://codeload.github.com/dylanpieper/redquack/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250926213,"owners_count":21508917,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["duckdb","r","redcap"],"created_at":"2025-03-30T08:20:11.698Z","updated_at":"2025-04-26T10:46:41.429Z","avatar_url":"https://github.com/dylanpieper.png","language":"R","funding_links":[],"categories":[],"sub_categories":[],"readme":"# redquack \u003cimg src=\"man/figures/redquack-hex.png\" align=\"right\" width=\"140\"/\u003e\n\n[![CRAN status](https://www.r-pkg.org/badges/version/redquack)](https://cran.r-pkg.org/package=redquack) [![R-CMD-check](https://github.com/dylanpieper/redquack/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/dylanpieper/redquack/actions/workflows/R-CMD-check.yaml)\n\nTransfer [REDCap](https://www.project-redcap.org/) data to [DuckDB](https://duckdb.org/) with minimal memory overhead, designed for large datasets that exceed available RAM.\n\n## Motivation\n\nR objects live entirely in RAM, causing three problems if not using a specialized framework:\n\n1.  You must load full datasets even if you only need a subset\n2.  Unused objects still consume memory\n3.  Large datasets can easily exceed available RAM\n\nredquack's solution to this problem is to:\n\n1.  Request all of the REDCap record IDs to sequence in chunks\n2.  Process each chunk of the REDCap data in one R object at a time\n3.  Remove each object from memory after it has been transferred to DuckDB\n\n## Features\n\n-   Chunked transfers for memory efficiency\n-   Auto-resume from interruptions\n-   Optimal data type conversion\n-   Timestamped operation logs\n-   Configurable API request retries\n-   Real-time progress indicators\n-   Completion notifications (🔊 🦆)\n\n## Installation\n\nFrom CRAN:\n\n``` r\ninstall.packages(\"redquack\")\n```\n\nDevelopment version:\n\n``` r\n# install.packages(\"pak\")\npak::pak(\"dylanpieper/redquack\")\n```\n\n## Basic Usage\n\nData from REDCap is transferred to DuckDB in configurable chunks of record IDs:\n\n``` r\nlibrary(redquack)\n\ncon \u003c- redcap_to_duckdb(\n  redcap_uri = \"https://redcap.example.org/api/\",\n  token = \"YOUR_API_TOKEN\",\n  record_id_name = \"record_id\",\n  chunk_size = 1000  \n  # Increase chunk size for memory-efficient systems (faster)\n  # Decrease chunk size for memory-constrained systems (slower)\n)\n```\n\nBy default, the function returns the DuckDB connection from the output file `redcap.duckdb`.\n\n### Data Manipulation\n\nQuery and collect the data with `dplyr`:\n\n``` r\nlibrary(dplyr)\n\ndemographics \u003c- tbl(con, \"data\") |\u003e\n  filter(demographics_complete == 2) |\u003e\n  select(record_id, age, race, gender) |\u003e\n  collect()\n```\n\nCreate a Parquet file directly from DuckDB (efficient for sharing data):\n\n``` r\nDBI::dbExecute(con, \"COPY (SELECT * FROM data) TO 'redcap.parquet' (FORMAT PARQUET)\")\n```\n\nRemember to close the connection when finished:\n\n``` r\nDBI::dbDisconnect(con)\n```\n\n### Workflow Mode\n\nFor scripted workflows or automated processes where you don't need to return the connection, you can use the function in workflow mode:\n\n```r\nsuccess \u003c- redcap_to_duckdb(\n  redcap_uri = \"https://redcap.example.org/api/\",\n  token = \"YOUR_API_TOKEN\",\n  record_id_name = \"record_id\",\n  return_duckdb = FALSE\n)\n\nif (success) {\n  message(\"Data transfer completed successfully!\")\n} else {\n  stop(\"Data transfer failed or is incomplete!\")\n}\n```\n\nWhen `return_duckdb = FALSE`, the function returns a logical value:\n\n-   `TRUE` for a complete successful transfer\n-   `FALSE` for a failed or partially completed transfer\n\nWorkflow mode automatically tries to resume incomplete transfers up to `max_retries` times.\n\n## Database Structure\n\nThe DuckDB database created by `redcap_to_duckdb()` contains two tables:\n\n1.  `data`: Contains all exported REDCap records with optimized column types\n\n    ``` r\n    DBI::dbGetQuery(con, \"SELECT * FROM data LIMIT 10\")\n    ```\n\n2.  `log`: Contains timestamped logs of the transfer process for troubleshooting\n\n    ``` r\n    DBI::dbGetQuery(con, \"SELECT timestamp, type, message FROM log ORDER BY timestamp\")\n    ```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdylanpieper%2Fredquack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdylanpieper%2Fredquack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdylanpieper%2Fredquack/lists"}