{"id":31936219,"url":"https://github.com/bbuchsbaum/delarr","last_synced_at":"2025-10-14T07:25:18.495Z","repository":{"id":316682406,"uuid":"1064427294","full_name":"bbuchsbaum/delarr","owner":"bbuchsbaum","description":null,"archived":false,"fork":false,"pushed_at":"2025-09-26T02:57:31.000Z","size":48,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-26T04:30:55.965Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bbuchsbaum.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-26T02:51:47.000Z","updated_at":"2025-09-26T02:57:35.000Z","dependencies_parsed_at":"2025-09-26T04:31:02.845Z","dependency_job_id":"1ca51632-2f7b-4338-abd8-b740f08b6099","html_url":"https://github.com/bbuchsbaum/delarr","commit_stats":null,"previous_names":["bbuchsbaum/delarr"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/bbuchsbaum/delarr","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bbuchsbaum%2Fdelarr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bbuchsbaum%2Fdelarr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bbuchsbaum%2Fdelarr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bbuchsbaum%2Fdelarr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bbuchsbaum","download_url":"https://codeload.github.com/bbuchsbaum/delarr/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bbuchsbaum%2Fdelarr/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279018215,"owners_count":26086303,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-14T07:25:17.522Z","updated_at":"2025-10-14T07:25:18.485Z","avatar_url":"https://github.com/bbuchsbaum.png","language":"R","readme":"# delarr\n\n`delarr` provides a lightweight delayed matrix type for R with a tidy-friendly\nAPI. It keeps the surface area small—one S3 class plus a handful of verbs—while\noffering fused elementwise transforms, reductions, and streaming materialisation\nin column chunks. Column blocks can also be streamed straight to disk via the\nbundled HDF5 writer.\n\n## Installation\n\nThe package is under active development. Clone the repository and use\n`pkgload::load_all()` or `devtools::install()` to experiment with the API.\n\n```r\n# install.packages(\"pkgload\")\npkgload::load_all(\".\")\n```\n\n## Getting started\n\n```r\nlibrary(delarr)\n\nmat \u003c- matrix(rnorm(20), 5, 4)\narr \u003c- delarr(mat)\n\n# Lazy pipeline\nout \u003c- arr |\u003e\n  d_center(dim = \"rows\", na.rm = TRUE) |\u003e\n  d_map(~ .x * 0.5) |\u003e\n  d_reduce(mean, dim = \"rows\")\n\ncollect(out)\n```\n\n### Streaming straight to disk\n\n```r\n# assume `X` lives inside an HDF5 file\nlzy \u003c- delarr_hdf5(\"input.h5\", \"X\")\n\n# Apply a transformation lazily and stream the result into a new dataset\n# (dim(lzy)[2] supplies the total column count for the writer)\nlzy |\u003e\n  d_zscore(dim = \"cols\") |\u003e\n  collect(into = hdf5_writer(\n    path = \"output.h5\",\n    dataset = \"X_zscore\",\n    ncol = dim(lzy)[2],\n    chunk = c(128L, 4096L)\n  ))\n```\n\n## Backends\n\n- `delarr_mem()` wraps any in-memory matrix.\n- `delarr_hdf5()` exposes a dataset through `hdf5r`.\n- `delarr_backend()` lets you create a seed from any `(rows, cols) -\u003e matrix`\npull function.\n- `hdf5_writer()` pairs with `collect(into = ...)` to stream results back to\n  disk without materialising the full matrix in memory (supply `ncol` to size\n  the destination dataset up front).\n\nAdditional backends (e.g., mmap) can be layered on by supplying a compatible\npull function; no extra dependencies ship in the core package.\n\n## Pipelined verbs\n\n- `d_map()/d_map2()` for elementwise transformations.\n- `d_center()/d_scale()/d_zscore()/d_detrend()` for common preprocessing, each\n  with optional `na.rm` handling.\n- `d_reduce()` for row-wise or column-wise reductions, with streaming\n  `na.rm` support for sum/mean/min/max.\n- `d_where()` for masked updates, optionally replacing masked entries via the\n  `fill` argument.\n- `collect()` to realise the data (streamed in chunks), optionally writing to\n  disk with `hdf5_writer()`, and `block_apply()` for chunk-wise computation.\n\nAll verbs return another `delarr`, so pipelines stay lazy until `collect()`\nmaterialises the result.\n\n## Integration with fmridataset\n\n`delarr_backend()` mirrors the contract of the existing DelayedArray seeds in\n*fmridataset*: provide `backend_get_dims()` and `backend_get_data()` style\nfunctions, and wrap them in a simple pull closure. Replacing the S4 seeds with\n`as_delarr()` methods becomes a matter of creating `delarr_backend()` objects\nand updating call sites to rely on matrix-style subsetting, streaming\n`collect()`, and `d_reduce()` for materialisation.\n\n## Testing\n\nThe test suite exercises the core class, slicing, verb fusion, reductions,\nchunk-aware execution, and the HDF5 streaming writer. Run it locally with:\n\n```r\npkgload::load_all(\".\")\ntestthat::test_dir(\"tests/testthat\")\n```\n\n## Roadmap\n\n- Add optional mmap and other backends once their APIs are finalised.\n- Introduce optional sparse adapters and BLAS helpers where it pays off.\n- Expand documentation with vignettes and performance benchmarks.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbbuchsbaum%2Fdelarr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbbuchsbaum%2Fdelarr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbbuchsbaum%2Fdelarr/lists"}