{"id":13857978,"url":"https://github.com/ankane/dbx","last_synced_at":"2025-11-17T14:12:09.048Z","repository":{"id":56935332,"uuid":"136569691","full_name":"ankane/dbx","owner":"ankane","description":"A fast, easy-to-use database library for R","archived":false,"fork":false,"pushed_at":"2025-10-15T08:10:54.000Z","size":404,"stargazers_count":189,"open_issues_count":0,"forks_count":15,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-10-29T00:52:54.840Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ankane.png","metadata":{"files":{"readme":"README.md","changelog":"NEWS.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2018-06-08T05:18:01.000Z","updated_at":"2025-10-19T20:29:56.000Z","dependencies_parsed_at":"2025-10-15T19:37:32.411Z","dependency_job_id":"25cc1b25-42b7-4efe-8a09-5f4ff6579657","html_url":"https://github.com/ankane/dbx","commit_stats":{"total_commits":472,"total_committers":4,"mean_commits":118.0,"dds":"0.19703389830508478","last_synced_commit":"e7e4a5094719518f339b49c9aeeb6050e74ae439"},"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"purl":"pkg:github/ankane/dbx","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fdbx","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fdbx/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fdbx/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fdbx/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ankane","download_url":"https://codeload.github.com/ankane/dbx/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fdbx/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":284894178,"owners_count":27080659,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-17T02:00:06.431Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-05T03:01:52.989Z","updated_at":"2025-11-17T14:12:09.032Z","avatar_url":"https://github.com/ankane.png","language":"R","funding_links":[],"categories":["R"],"sub_categories":[],"readme":"# dbx\n\n:fire: A fast, easy-to-use database library for R\n\n- Intuitive functions\n- High performance batch operations\n- Safe inserts, updates, and deletes without writing SQL\n- Upserts!!\n- Great date and time support\n- Works well with auto-incrementing primary keys\n- Built on top of [DBI](https://cran.r-project.org/package=DBI)\n\nDesigned for both research and production environments\n\nSupports Postgres, MySQL, MariaDB, SQLite, SQL Server, and more\n\n[![Build Status](https://github.com/ankane/dbx/actions/workflows/build.yml/badge.svg)](https://github.com/ankane/dbx/actions) [![CRAN status](https://www.r-pkg.org/badges/version/dbx)](https://cran.r-project.org/package=dbx)\n\n## Installation\n\nInstall dbx\n\n```r\ninstall.packages(\"dbx\")\n```\n\nAnd follow the instructions for your database\n\n- [Postgres](#postgres)\n- [MySQL \u0026 MariaDB](#mysql--mariadb)\n- [SQLite](#sqlite)\n- [SQL Server](#sql-server)\n- [Redshift](#redshift)\n- [DuckDB](#duckdb)\n- [Others](#others)\n\nTo install with [Jetpack](https://github.com/ankane/jetpack), use:\n\n```r\njetpack::add(\"dbx\")\n```\n\n### Postgres\n\nInstall the R package\n\n```r\ninstall.packages(\"RPostgres\")\n```\n\nAnd use:\n\n```r\nlibrary(dbx)\n\ndb \u003c- dbxConnect(adapter=\"postgres\", dbname=\"mydb\")\n```\n\nYou can also pass `user`, `password`, `host`, `port`, and `url`.\n\n\u003e Works with RPostgreSQL as well\n\n### MySQL \u0026 MariaDB\n\nInstall the R package\n\n```r\ninstall.packages(\"RMySQL\")\n```\n\nAnd use:\n\n```r\nlibrary(dbx)\n\ndb \u003c- dbxConnect(adapter=\"mysql\", dbname=\"mydb\")\n```\n\nYou can also pass `user`, `password`, `host`, `port`, and `url`.\n\n\u003e Works with RMariaDB as well\n\n### SQLite\n\nInstall the R package\n\n```r\ninstall.packages(\"RSQLite\")\n```\n\nAnd use:\n\n```r\nlibrary(dbx)\n\ndb \u003c- dbxConnect(adapter=\"sqlite\", dbname=\":memory:\")\n```\n\n### SQL Server\n\nInstall the R package\n\n```r\ninstall.packages(\"odbc\")\n```\n\nAnd use:\n\n```r\nlibrary(dbx)\n\ndb \u003c- dbxConnect(adapter=odbc::odbc(), database=\"mydb\")\n```\n\nYou can also pass `uid`, `pwd`, `server`, and `port`.\n\n### Redshift\n\nFor Redshift, follow the [Postgres instructions](#postgres).\n\n### DuckDB\n\nInstall the R package\n\n```r\ninstall.packages(\"duckdb\")\n```\n\nAnd use:\n\n```r\nlibrary(dbx)\n\ndb \u003c- dbxConnect(adapter=duckdb::duckdb(), dbdir=\":memory:\")\n```\n\n### Others\n\nInstall the appropriate R package and use:\n\n```r\ndb \u003c- dbxConnect(adapter=odbc::odbc(), database=\"mydb\")\n```\n\n## Operations\n\n### Select\n\nCreate a data frame of records from a SQL query\n\n```r\nrecords \u003c- dbxSelect(db, \"SELECT * FROM forecasts\")\n```\n\nPass parameters\n\n```r\ndbxSelect(db, \"SELECT * FROM forecasts WHERE period = ? AND temperature \u003e ?\", params=list(\"hour\", 27))\n```\n\nParameters can also be vectors\n\n```r\ndbxSelect(db, \"SELECT * FROM forecasts WHERE id IN (?)\", params=list(1:3))\n```\n\n### Insert\n\nInsert records\n\n```r\ntable \u003c- \"forecasts\"\nrecords \u003c- data.frame(temperature=c(32, 25))\ndbxInsert(db, table, records)\n```\n\nIf you use auto-incrementing ids, you can get the ids of newly inserted rows by passing the column name:\n\n```r\ndbxInsert(db, table, records, returning=c(\"id\"))\n```\n\n\u003e `returning` is not available for MySQL or Redshift\n\n### Update\n\nUpdate records\n\n```r\nrecords \u003c- data.frame(id=c(1, 2), temperature=c(16, 13))\ndbxUpdate(db, table, records, where_cols=c(\"id\"))\n```\n\nUse `where_cols` to specify the columns used for lookup. Other columns are written to the table.\n\n\u003e Updates are batched when possible, but often need to be run as multiple queries. We recommend upsert when possible for better performance, as it can always be run as a single query. Turn on logging to see the difference.\n\n### Upsert\n\n*Atomically* insert records if they don’t exist, otherwise update them\n\n```r\nrecords \u003c- data.frame(id=c(2, 3), temperature=c(20, 25))\ndbxUpsert(db, table, records, where_cols=c(\"id\"))\n```\n\nUse `where_cols` to specify the columns used for lookup. There must be a unique index on them, or an error will be thrown.\n\nTo skip existing rows instead of updating them, use:\n\n```r\ndbxUpsert(db, table, records, where_cols=c(\"id\"), skip_existing=TRUE)\n```\n\nIf you use auto-incrementing ids, you can get the ids of newly upserted rows by passing the column name:\n\n```r\ndbxUpsert(db, table, records, where_cols=c(\"id\"), returning=c(\"id\"))\n```\n\n\u003e Upsert is not available for Redshift, and `returning` is not available for MySQL\n\n### Delete\n\nDelete specific records\n\n```r\nbad_records \u003c- data.frame(id=c(1, 2))\ndbxDelete(db, table, where=bad_records)\n```\n\nDelete all records (uses `TRUNCATE` when possible for performance)\n\n```r\ndbxDelete(db, table)\n```\n\n### Execute\n\nExecute a statement\n\n```r\ndbxExecute(db, \"UPDATE forecasts SET temperature = temperature + 1\")\n```\n\nPass parameters\n\n```r\ndbxExecute(db, \"UPDATE forecasts SET temperature = ? WHERE id IN (?)\", params=list(27, 1:3))\n```\n\n## Logging\n\nLog all SQL queries with:\n\n```r\noptions(dbx_logging=TRUE)\n```\n\nCustomize logging by passing a function\n\n```r\nlogQuery \u003c- function(sql) {\n  # your logging code\n}\n\noptions(dbx_logging=logQuery)\n```\n\n## Database Credentials\n\nEnvironment variables are a convenient way to store database credentials. This keeps them outside your source control. It’s also how platforms like [Heroku](https://www.heroku.com) store them.\n\nCreate an `.Renviron` file in your home directory with:\n\n```\nDATABASE_URL=postgres://user:pass@host/dbname\n```\n\nInstall [urltools](https://cran.r-project.org/package=urltools):\n\n```r\ninstall.packages(\"urltools\")\n```\n\nAnd use:\n\n```r\ndb \u003c- dbxConnect()\n```\n\nIf you have multiple databases, use a different variable name, and:\n\n```r\ndb \u003c- dbxConnect(url=Sys.getenv(\"OTHER_DATABASE_URL\"))\n```\n\nYou can also use a package like [keyring](https://cran.r-project.org/package=keyring).\n\n## Batching\n\nBy default, operations are performed in a single statement or transaction. This is better for performance and prevents partial writes on failures. However, when working with large data frames on production systems, it can be better to break writes into batches. Use the `batch_size` option to do this.\n\n```r\ndbxInsert(db, table, records, batch_size=1000)\ndbxUpdate(db, table, records, where_cols, batch_size=1000)\ndbxUpsert(db, table, records, where_cols, batch_size=1000)\ndbxDelete(db, table, records, where, batch_size=1000)\n```\n\n## Query Comments\n\nAdd comments to queries to make it easier to see where time-consuming queries are coming from.\n\n```r\noptions(dbx_comment=TRUE)\n```\n\nThe comment will be appended to queries, like:\n\n```sql\nSELECT * FROM users /*script:forecast.R*/\n```\n\nSet a custom comment with:\n\n```r\noptions(dbx_comment=\"hi\")\n```\n\n## Transactions\n\nTo perform multiple operations in a single transaction, use:\n\n```r\nDBI::dbWithTransaction(db, {\n  dbxInsert(db, ...)\n  dbxDelete(db, ...)\n})\n```\n\nFor updates inside a transaction, use:\n\n```r\ndbxUpdate(db, transaction=FALSE)\n```\n\n## Schemas\n\nTo specify a schema, use:\n\n```r\ntable \u003c- DBI::Id(schema=\"schema\", table=\"table\")\n```\n\n## Data Type Notes\n\n### Dates \u0026 Times\n\nDates are returned as `Date` objects and times as `POSIXct` objects. Times are stored in the database in UTC and converted to your local time zone when retrieved.\n\nTimes without dates are returned as `character` vectors since R has no built-in support for this type. If you use [hms](https://cran.r-project.org/package=hms), you can convert columns with:\n\n```r\nrecords$column \u003c- hms::as_hms(records$column)\n```\n\nSQLite does not have support for `TIME` columns, so we recommend storing as `VARCHAR`.\n\n### JSON\n\nJSON and JSONB columns are returned as `character` vectors. You can use [jsonlite](https://cran.r-project.org/package=jsonlite) to parse them with:\n\n```r\nrecords$column \u003c- lapply(records$column, jsonlite::fromJSON)\n```\n\nSQLite does not have support for `JSON` columns, so we recommend storing as `TEXT`.\n\n### Binary Data\n\nBLOB and BYTEA columns are returned as `raw` vectors.\n\n## Data Type Limitations\n\n### Dates \u0026 Times\n\nRSQLite does not currently provide enough info to automatically typecast dates and times. You can manually typecast date columns with:\n\n```r\nrecords$column \u003c- as.Date(records$column)\n```\n\nAnd time columns with:\n\n```r\nrecords$column \u003c- as.POSIXct(records$column, tz=\"Etc/UTC\")\nattr(records$column, \"tzone\") \u003c- Sys.timezone()\n```\n\n### Booleans\n\nRMariaDB and RSQLite do not currently provide enough info to automatically typecast booleans. You can manually typecast with:\n\n```r\nrecords$column \u003c- records$column != 0\n```\n\n### JSON\n\nRMariaDB does [not currently support JSON](https://github.com/r-dbi/DBI/issues/203).\n\n### Binary Data\n\nRMySQL can write BLOB columns, but [can’t retrieve them directly](https://github.com/r-dbi/RMySQL/issues/123). To workaround this, use:\n\n```r\nrecords \u003c- dbxSelect(db, \"SELECT HEX(column) AS column FROM table\")\n\nhexToRaw \u003c- function(x) {\n  y \u003c- strsplit(x, \"\")[[1]]\n  z \u003c- paste0(y[c(TRUE, FALSE)], y[c(FALSE, TRUE)])\n  as.raw(as.hexmode(z))\n}\n\nrecords$column \u003c- lapply(records$column, hexToRaw)\n```\n\n### Bigint\n\nBIGINT columns are returned as `numeric` vectors. The `numeric` type in R loses precision above 2\u003csup\u003e53\u003c/sup\u003e. Some libraries (RPostgres, RMariaDB, RSQLite, ODBC) support returning `bit64::integer64` vectors instead.\n\n```r\ndbxConnect(bigint=\"integer64\")\n```\n\n## Connection Pooling\n\nInstall the [pool](https://cran.r-project.org/package=pool) package\n\n```r\ninstall.packages(\"pool\")\n```\n\nCreate a pool\n\n```r\nlibrary(pool)\n\nfactory \u003c- function() {\n  dbxConnect(adapter=\"postgres\", ...)\n}\n\npool \u003c- poolCreate(factory, maxSize=5)\n```\n\nRun queries\n\n```ruby\nconn \u003c- poolCheckout(pool)\n\ntryCatch({\n  dbxSelect(conn, \"SELECT * FROM forecasts\")\n}, finally={\n  poolReturn(conn)\n})\n```\n\nIn the future, dbx commands may work directly with pools.\n\n## Security\n\nWhen connecting to a database over a network you don’t fully trust, make sure your [connection is secure](https://ankane.org/postgres-sslmode-explained).\n\nWith Postgres, use:\n\n```r\ndb \u003c- dbxConnect(adapter=\"postgres\", sslmode=\"verify-full\", sslrootcert=\"ca.pem\")\n```\n\nWith RMariaDB, use:\n\n```r\ndb \u003c- dbxConnect(adapter=\"mysql\", ssl.ca=\"ca.pem\")\n```\n\nPlease [let us know](https://github.com/ankane/dbx/issues/new) if you have a way that works with RMySQL.\n\n## Variables\n\nSet session variables with:\n\n```r\ndb \u003c- dbxConnect(variables=list(search_path=\"archive\"))\n```\n\n## Timeouts\n\nSet a statement timeout with:\n\n```r\n# Postgres\ndb \u003c- dbxConnect(variables=list(statement_timeout=1000)) # ms\n\n# MySQL\ndb \u003c- dbxConnect(variables=list(max_execution_time=1000)) # ms\n\n# MariaDB\ndb \u003c- dbxConnect(variables=list(max_statement_time=1)) # sec\n```\n\nSet a connect timeout with:\n\n```r\n# Postgres\ndb \u003c- dbxConnect(connect_timeout=3) # sec\n\n# RMariaDB and SQL Server\ndb \u003c- dbxConnect(timeout=3) # sec\n```\n\n## Compatibility\n\nAll connections are simply [DBI](https://cran.r-project.org/package=DBI) connections, so you can use them anywhere you use DBI.\n\n```r\ndbCreateTable(db, ...)\n```\n\nInstall [dbplyr](https://cran.r-project.org/package=dbplyr) to use data with [dplyr](https://cran.r-project.org/package=dplyr).\n\n```r\nforecasts \u003c- tbl(db, \"forecasts\")\n```\n\n## Reference\n\nTo close a connection, use:\n\n```r\ndbxDisconnect(db)\n```\n\n## History\n\nView the [changelog](https://github.com/ankane/dbx/blob/master/NEWS.md)\n\n## Contributing\n\nEveryone is encouraged to help improve this project. Here are a few ways you can help:\n\n- [Report bugs](https://github.com/ankane/dbx/issues)\n- Fix bugs and [submit pull requests](https://github.com/ankane/dbx/pulls)\n- Write, clarify, or fix documentation\n- Suggest or add new features\n\nTo get started with development:\n\n```sh\ngit clone https://github.com/ankane/dbx.git\ncd dbx\n\n# create Postgres database\ncreatedb dbx_test\n\n# create MySQL database\nmysqladmin create dbx_test\n```\n\nIn R, do:\n\n```r\ninstall.packages(\"devtools\")\ndevtools::install_deps(dependencies=TRUE)\ndevtools::test()\n```\n\nTo test a single file, use:\n\n```r\ndevtools::install() # to use latest updates\ndevtools::test_active_file(\"tests/testthat/test-postgres.R\")\n```\n\nTo test the ODBC adapter, use:\n\n```sh\nbrew install mariadb-connector-odbc psqlodbc\n# or\nsudo apt-get install odbc-mariadb odbc-postgresql\n```\n\nTo test SQL Server, use:\n\n```sh\nbrew install freetds\n# or\nsudo apt-get install tdsodbc\n\ndocker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=YourStrong!Passw0rd' -p 1433:1433 -d mcr.microsoft.com/mssql/server:2022-latest\ndocker exec -it \u003ccontainer-id\u003e /opt/mssql-tools18/bin/sqlcmd -S localhost -U SA -P YourStrong\\!Passw0rd -C -Q \"CREATE DATABASE dbx_test\"\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fankane%2Fdbx","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fankane%2Fdbx","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fankane%2Fdbx/lists"}