{"id":13857925,"url":"https://github.com/ropensci/arkdb","last_synced_at":"2025-04-09T10:07:44.769Z","repository":{"id":32896831,"uuid":"136522042","full_name":"ropensci/arkdb","owner":"ropensci","description":"Archive and unarchive databases as flat text files","archived":false,"fork":false,"pushed_at":"2024-01-15T23:08:30.000Z","size":1123,"stargazers_count":79,"open_issues_count":6,"forks_count":7,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-04-02T02:32:59.643Z","etag":null,"topics":["archiving","database","dbi","peer-reviewed","r","r-package","rstats"],"latest_commit_sha":null,"homepage":"https://docs.ropensci.org/arkdb","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ropensci.png","metadata":{"files":{"readme":"README.Rmd","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2018-06-07T19:29:36.000Z","updated_at":"2024-12-10T22:19:27.000Z","dependencies_parsed_at":"2024-01-18T08:08:29.135Z","dependency_job_id":"d5c42a81-ddf2-4e85-8d81-1ffced4801af","html_url":"https://github.com/ropensci/arkdb","commit_stats":{"total_commits":218,"total_committers":9,"mean_commits":24.22222222222222,"dds":"0.24770642201834858","last_synced_commit":"0a836d03eac94c3d7d9f697d190546df2b3bd3d0"},"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ropensci%2Farkdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ropensci%2Farkdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ropensci%2Farkdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ropensci%2Farkdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ropensci","download_url":"https://codeload.github.com/ropensci/arkdb/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248018060,"owners_count":21034048,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["archiving","database","dbi","peer-reviewed","r","r-package","rstats"],"created_at":"2024-08-05T03:01:50.935Z","updated_at":"2025-04-09T10:07:44.740Z","avatar_url":"https://github.com/ropensci.png","language":"R","funding_links":[],"categories":["R"],"sub_categories":[],"readme":"---\noutput: github_document\n---\n\n# arkdb \u003cimg src=\"man/figures/logo.svg\" align=\"right\" alt=\"\" width=\"120\" /\u003e\n\n[![R build status](https://github.com/ropensci/arkdb/workflows/R-CMD-check/badge.svg)](https://github.com/ropensci/arkdb/actions)\n[![Travis build status](https://app.travis-ci.com/ropensci/arkdb.svg?branch=master)](https://app.travis-ci.com/ropensci/arkdb)\n[![Coverage status](https://codecov.io/gh/ropensci/arkdb/branch/master/graph/badge.svg)](https://app.codecov.io/github/ropensci/arkdb?branch=master)\n[![CRAN_Status_Badge](http://www.r-pkg.org/badges/version/arkdb)](https://cran.r-project.org/package=arkdb)\n[![](https://badges.ropensci.org/224_status.svg)](https://github.com/ropensci/software-review/issues/224)\n[![lifecycle](https://img.shields.io/badge/lifecycle-stable-brightgreen.svg)](https://lifecycle.r-lib.org/articles/stages.html)\n [![CRAN RStudio mirror downloads](http://cranlogs.r-pkg.org/badges/grand-total/arkdb)](https://CRAN.R-project.org/package=arkdb) \n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.1343943.svg)](https://doi.org/10.5281/zenodo.1343943)\n  \u003c!-- badges: end --\u003e\n  \n\u003c!-- README.md is generated from README.Rmd. Please edit that file --\u003e\n\n```{r, echo = FALSE}\nknitr::opts_chunk$set(\n  collapse = TRUE,\n  comment = \"#\u003e\",\n  fig.path = \"README-\"\n)\n```\n\nThe goal of `arkdb` is to provide a convenient way to move data from large compressed text files (tsv, csv, etc) into any DBI-compliant database connection (e.g. MYSQL, Postgres, SQLite; see [DBI](https://solutions.rstudio.com/db/r-packages/DBI/)), and move tables out of such databases into text files. The key feature of `arkdb` is that files are moved between databases and text files in chunks of a fixed size, allowing the package functions to work with tables that would be much too large to read into memory all at once. There is also functionality for filtering and applying transformation to data as it is extracted from the database.  \n\nThe `arkdb` package is easily extended to use custom read and write methods allowing you to dictate your own output formats. See `R/streamable_table.R` for examples that include using: \n\n- Base c/tsv\n- Apache arrow's parquet\n- The `readr` package for c/tsv\n\n## Links\n\n- A more detailed introduction to package design and use can be found in the package [Vignette](https://docs.ropensci.org/arkdb/articles/arkdb.html)\n- [Online versions of package documentation](https://docs.ropensci.org/arkdb/)\n\n## Installation\n\nYou can install arkdb from GitHub with:\n\n```{r gh-installation, eval = FALSE}\n# install.packages(\"devtools\")\ndevtools::install_github(\"cboettig/arkdb\")\n```\n\n\n# Basic use\n\n```{r message = FALSE}\nlibrary(arkdb)\n\n# additional libraries just for this demo\nlibrary(dbplyr)\nlibrary(dplyr)\nlibrary(fs)\n```\n\n## Creating an archive of a database\n\nConsider the `nycflights` database in SQLite:\n\n```{r example}\ntmp \u003c- tempdir() # Or can be your working directory, \".\"\ndb \u003c- dbplyr::nycflights13_sqlite(tmp)\n```\n\nCreate an archive of the database: \n\n```{r}\ndir \u003c- fs::dir_create(fs::path(tmp, \"nycflights\"))\nark(db, dir, lines = 50000)\n```\n\n## Unarchive\n\nImport a list of compressed tabular files (i.e. `*.csv.bz2`) into a local SQLite database:\n\n\n```{r}\nfiles \u003c- fs::dir_ls(dir)\nnew_db \u003c- DBI::dbConnect(RSQLite::SQLite(), fs::path(tmp, \"local.sqlite\"))\n\nunark(files, new_db, lines = 50000)\n```\n\n\n\n```{r include=FALSE}\ndisconnect \u003c- function(db){\n  ## Cleanup \n  if(inherits(db, \"DBIConnection\")){\n    DBI::dbDisconnect(db)\n  } else {\n    DBI::dbDisconnect(db$con)\n  }\n}\n\nDBI::dbDisconnect(db)\nDBI::dbDisconnect(new_db)\n\ncodemeta::write_codemeta()\n```\n\n## Using filters\n\nThis package can also be used to generate slices of data that are required for analytical or operational purposes. In the example below we archive to disk only the flight data that occurred in the month of December. It is recommended to use filters on a single table at a time. \n\n```{r, eval=FALSE}\nark(db, dir, lines = 50000, tables = \"flights\", filter_statement = \"WHERE month = 12\")\n```\n\n## Using callbacks\n\nIt is possible to use a callback to perform just-in-time data transformations before ark writes your data object to disk in your preferred format. In the example below, we write a simple transformation to convert the flights data `arr_delay` field, from minutes, to hours. It is recommended to use callbacks on a single table at a time. A callback function can be anything you can imagine so long as it returns a data.frame that can be written to disk.\n\n```{r, eval=FALSE}\nmins_to_hours \u003c- function(data) {\n  data$arr_delay \u003c- data$arr_delay/60\n  data\n}\n\nark(db, dir, lines = 50000, tables = \"flights\", callback = mins_to_hours)\n```\n\n## ark() in parallel\n\nThere are two strategies for using `ark` in parallel. One is to loop over the tables, re-using the ark function per table in parallel. The other, introduced in 0.0.15, is to use the \"window-parallel\" method which loops over chunks of your table. This is particularly useful if your tables are very large and can speed up the process significantly. \n\nNote: `window-parallel` currently only works in conjunction with `streamable_parquet`\n\n```{r, eval = FALSE}\n# Strategy 1: Parallel over tables\nlibrary(arkdb)\nlibrary(future.apply)\n\nplan(multisession)\n\n# Any streamable_table method is acceptable\nfuture_lapply(vector_of_tables, function(x) ark(db, dir, lines, tables = x))\n\n# Strategy 2: Parallel over chunks of a table\nlibrary(arkdb)\nlibrary(future.apply)\n\nplan(multisession)\n\nark(\n  db, \n  dir, \n  streamable_table = streamable_parquet(), # required for window-parallel\n  lines = 50000, \n  tables = \"flights\", \n  method = \"window-parallel\"\n)\n\n# Strategy 3: Parallel over tables and chunks of tables\nlibrary(arkdb)\nlibrary(future.apply)\n# 16 core machine for example\nplan(list(tweak(multisession, n = 4), tweak(multisession, n = 4)))\n\n# 4 tables at a time, 4 threads per table\nfuture_lapply(vector_of_tables, function(x) { \n  ark(\n    db, \n    dir, \n    streamable_table = streamable_parquet(), # required for window-parallel\n    lines = 50000, \n    tables = x, \n    method = \"window-parallel\")\n  }\n)\n\n```\n\n## ETLs with arkdb\n\nThe `arkdb` package can also be used to create a number of ETL pipelines involving text archives or databases given its ability to filter, and use callbacks. In the example below, we leverage `duckdb` to read a fictional folder of files by US state, filter by `var_filtered`, apply a callback transformation `transform_fun` to `var_transformed` save as parquet, and then load a folder of parquet files for analysis with Apache Arrow. \n\n```{r, eval = FALSE}\nlibrary(arrow)\nlibrary(duckdb)\n\ndb \u003c- dbConnect(duckdb::duckdb())\n\ntransform_fun \u003c- function(data) {\n  data$var_transformed \u003c- sqrt(data$var_transformed)\n  data\n}\n\nfor(state in c(\"DC\", state.abb)) {\n  path \u003c- paste0(\"path/to/archives/\", state, \".gz\")\n  \n  ark(\n    db,\n    dir = paste0(\"output/\", state),\n    streamable_table = streamable_parquet(), # parquet files of nline rows\n    lines = 100000,\n    # See: https://duckdb.org/docs/data/csv\n    tables = sprintf(\"read_csv_auto('%s')\", path), \n    compress = \"none\", # Compression meaningless for parquet as it's already compressed\n    overwrite = T, \n    filenames = state, # Overload tablename\n    filter_statement = \"WHERE var_filtered = 1\",\n    callback = transform_fun\n  )\n}\n\n# The result is trivial to read in with arrow \nds \u003c- open_dataset(\"output\", partitioning = \"state\")\n```\n\n-----\n\nPlease note that this project is released with a [Contributor Code of Conduct](https://ropensci.org/code-of-conduct/).\nBy participating in this project you agree to abide by its terms.\n\n\n[![ropensci_footer](https://ropensci.org/public_images/ropensci_footer.png)](https://ropensci.org)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fropensci%2Farkdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fropensci%2Farkdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fropensci%2Farkdb/lists"}