{"id":29992927,"url":"https://github.com/extendr/arrow-extendr","last_synced_at":"2025-08-05T00:22:08.153Z","repository":{"id":208673081,"uuid":"722201437","full_name":"extendr/arrow-extendr","owner":"extendr","description":"Integration between arrow-rs and extendr ","archived":false,"fork":false,"pushed_at":"2025-05-28T16:50:16.000Z","size":79,"stargazers_count":23,"open_issues_count":1,"forks_count":1,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-07-30T01:48:18.446Z","etag":null,"topics":["apache-arrow","ffi","rstats","rust"],"latest_commit_sha":null,"homepage":"https://docs.rs/arrow-extendr","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/extendr.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-11-22T16:45:22.000Z","updated_at":"2025-07-27T21:12:18.000Z","dependencies_parsed_at":"2024-01-24T16:29:29.934Z","dependency_job_id":"8899edb7-21cd-40ab-b415-b15ecfc52d44","html_url":"https://github.com/extendr/arrow-extendr","commit_stats":{"total_commits":52,"total_committers":2,"mean_commits":26.0,"dds":"0.019230769230769273","last_synced_commit":"1ff628cd5e9c208c1aff99bc8aa92a3b7b9303dc"},"previous_names":["josiahparry/arrow-extendr","extendr/arrow-extendr"],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/extendr/arrow-extendr","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/extendr%2Farrow-extendr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/extendr%2Farrow-extendr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/extendr%2Farrow-extendr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/extendr%2Farrow-extendr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/extendr","download_url":"https://codeload.github.com/extendr/arrow-extendr/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/extendr%2Farrow-extendr/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268808613,"owners_count":24310568,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-04T02:00:09.867Z","response_time":79,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-arrow","ffi","rstats","rust"],"created_at":"2025-08-05T00:22:02.815Z","updated_at":"2025-08-05T00:22:08.131Z","avatar_url":"https://github.com/extendr.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# arrow_extendr\n\narrow-extendr is a crate that facilitates the transfer of [Apache Arrow](https://arrow.apache.org/) memory between R and Rust. It utilizes [extendr](https://extendr.github.io/), the [**`{nanoarrow}`**](https://arrow.apache.org/nanoarrow/0.3.0/r/index.html) R package, and [arrow-rs](https://docs.rs/arrow).\n\n## Versioning\n\nAt present, versions of arrow-rs are not compatible with each other. This means if your crate uses arrow-rs version `48.0.1`, then the arrow-extendr must also use that same version. As such, arrow-extendr uses the same versions as arrow-rs so that it is easy to match the required versions you need.\n\n**Versions**:\n\n- 55.1.0\n- 54.0.0\n- 53.0.0\n- 52.0.0\n- 51.0.0\n- 50.0.0 (compatible with geoarrow-rs 0.1.0)\n- 49.0.0-geoarrow (not available on crates.io but is the current Git version)\n- 48.0.1\n- 49.0.0\n\n### Motivating Example\n\nSay we have the following `DBI` connection which we will send requests to using arrow.\nThe result of `dbGetQueryArrow()` is a `nanoarrow_array_stream`. We want to\ncount the number of rows in each batch of the steam using Rust.\n\n```r\n# adapted from https://github.com/r-dbi/DBI/blob/main/vignettes/DBI-arrow.Rmd\n\nlibrary(DBI)\ncon \u003c- dbConnect(RSQLite::SQLite())\ndata \u003c- data.frame(\n  a = runif(10000, 0, 10),\n  b = rnorm(10000, 4.5),\n  c = sample(letters, 10000, TRUE)\n)\n\ndbWriteTable(con, \"tbl\", data)\n```\n\nWe can write an extendr function which creates an `ArrowArrayStreamReader`\nfrom an `\u0026Robj`. In the function we instantiate a counter to keep track\nof the number of rows per chunk. For each chunk we print the number of rows.\n\n```rust\nuse extendr_api::prelude::*;\nuse arrow_extendr::from::FromArrowRobj;\nuse arrow::ffi_stream::ArrowArrayStreamReader;\n\n#[extendr]\n/// @export\nfn process_stream(stream: Robj) -\u003e i32 {\n    let rb = ArrowArrayStreamReader::from_arrow_robj(\u0026stream)\n        .unwrap();\n\n    let mut n = 0;\n\n    rprintln!(\"Processing `ArrowArrayStreamReader`...\");\n    for chunk in rb {\n        let chunk_rows = chunk.unwrap().num_rows();\n        rprintln!(\"Found {chunk_rows} rows\");\n        n += chunk_rows as i32;\n    }\n\n    n\n}\n```\n\nWith this function we can use it on the output of `dbGetQueryArrow()` or other Arrow\nrelated DBI functions.\n\n```r\nquery \u003c- dbGetQueryArrow(con, \"SELECT * FROM tbl WHERE a \u003c 3\")\nprocess_stream(query)\n#\u003e Processing `ArrowArrayStreamReader`...\n#\u003e Found 256 rows\n#\u003e Found 256 rows\n#\u003e Found 256 rows\n#\u003e ... truncated ...\n#\u003e Found 256 rows\n#\u003e Found 256 rows\n#\u003e Found 143 rows\n#\u003e [1] 2959\n```\n\n## Using arrow-extendr in a package\n\nTo use arrow-extendr in an R package first create an R package and make it an extendr package with:\n\n```r\nusethis::create_package(\"my_package\")\nrextendr::use_extendr();\n```\n\nNext, you have to ensure that `nanoarrow` is a dependency of the package since arrow-extendr will call functions from nanoarrow to convert between R and Arrow memory. To do this run `usethis::use_package(\"nanoarrow\")` to add it to your Imports field in the DESCRIPTION file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fextendr%2Farrow-extendr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fextendr%2Farrow-extendr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fextendr%2Farrow-extendr/lists"}