Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/grimbough/rarr

A simple native R reader for Zarr Arrays
https://github.com/grimbough/rarr

ome-zarr on-disk out-of-memory r r-package zarr

Last synced: 8 days ago
JSON representation

A simple native R reader for Zarr Arrays

Host: GitHub
URL: https://github.com/grimbough/rarr
Owner: grimbough
License: mit
Created: 2022-12-05T08:32:33.000Z (almost 2 years ago)
Default Branch: devel
Last Pushed: 2024-10-28T11:25:57.000Z (10 days ago)
Last Synced: 2024-10-28T14:50:29.229Z (10 days ago)
Topics: ome-zarr, on-disk, out-of-memory, r, r-package, zarr
Language: C
Homepage: https://bioconductor.org/packages/Rarr/
Size: 1.43 MB
Stars: 34
Watchers: 7
Forks: 5
Open Issues: 5
Metadata Files:
- Readme: README.Rmd
- Changelog: NEWS.md
- Contributing: CONTRIBUTING.md
- License: LICENSE

Awesome Lists containing this project

README

        ---

title: "Zarr arrays with Rarr" 

author: "Mike L. Smith"

output:   

  github_document:

    toc: true

    toc_depth: 2

---

| GitHub Actions | Bioconductor Build Sysytem | Test Coverage |

|:--------------:|:-------------:|:-----:|

| [![Package Checks](https://github.com/grimbough/Rarr/actions/workflows/main.yml/badge.svg)](https://github.com/grimbough/Rarr/actions/workflows/main.yml) | [![Bioconductor Status](https://bioconductor.org/shields/build/devel/bioc/Rarr.svg)](https://bioconductor.org/checkResults/devel/bioc-LATEST/Rarr/) | [![Codecov test coverage](https://codecov.io/gh/grimbough/Rarr/branch/devel/graph/badge.svg)](https://app.codecov.io/gh/grimbough/Rarr?branch=devel) |

```{r, config, echo = FALSE}

knitr::opts_chunk$set(fig.path="inst/rmd/imgs/", dev = "jpeg")

```

# Introduction to Rarr

The Zarr specification defines a format for chunked, compressed, N-dimensional

arrays.  It's design allows efficient access to subsets of the stored array, and

supports both local and cloud storage systems. Zarr is experiencing increasing

adoption in a number of scientific fields, where multi-dimensional data are

prevalent. 

**Rarr** is intended to be a simple interface to reading and writing individual 

Zarr arrays. It is developed

in R and C with no reliance on external libraries or APIs for interfacing with

the Zarr arrays. Additional compression libraries (e.g. blosc) are bundled with

**Rarr** to provide support for datasets compressed using these tools.

## Limitations with **Rarr**

If you know about Zarr arrays already, you'll probably be aware they can be

stored in hierarchical groups, where additional meta data can explain the

relationship between the arrays.  Currently, **Rarr** is not designed to be

aware of these hierarchical Zarr array collections. However, the component

arrays can be read individually by providing the path to them directly.

Currently, there are also limitations on the Zarr datatypes that can be accessed

using **Rarr**.  For now most numeric types can be read into R, although in some

instances e.g. 64-bit integers there is potential for loss of information.

Writing is more limited with support only for datatypes that are supported

natively in R and only using the column-first representation.

# Quick start guide

```{r, child=c('inst/rmd/quick_start.Rmd')}

```

# Current Status

## Reading and Writing

Reading Zarr arrays is reasonably well supported. Writing is available, but is more limited.  Both aspects are under active development.  

### Data Types

Currently there is only support for reading and writing a subset of the possible datatypes

that can be found in a Zarr array.  In some instances there are also limitations on the 

datatypes natively supported by R, requiring conversion from the Zarr datatype.  The table below summarises the current status of

datatype support.  It will be updated as progress is made.

| Zarr Data Type | Status
(reading / writing) | Notes |

|-----------|:--------------:|-------|

|`boolean`  | ✔ / ❌ | |

|`int8`     | ✔ / ❌ | |

|`uint8` | ✔ / ❌ | |

|`int16` | ✔ / ❌ | |

|`uint16`| ✔ / ❌ | |

|`int32` | ✔ / ✔ | |

|`uint32`| ✔ / ❌ | Values outside the range of `int32` are converted to `NA`.  Future plan is to allow conversion to `double` or use the [bit64](https://cran.r-project.org/package=bit64) package. | 

|`int64` | ✔ / ❌ | Values outside the range of `int32` are converted to `NA`. Future plan is to allow conversion to `double` or use the [bit64](https://cran.r-project.org/package=bit64) package. |

|`uint64`| ✔ / ❌ | Values outside the range of `int32` are converted to `NA`. Future plan is to allow conversion to `double` or use the [bit64](https://cran.r-project.org/package=bit64) package. |

|`half` / `float16`   | ✔ / ❌ | Converted to `double` in R.  No effort is made to assess loss of precision due to conversion.  |

|`single` / `float32` | ✔ / ❌ | Converted to `double` in R.  No effort is made to assess loss of precision due to conversion. |

|`double` / `float64` | ✔ / ✔ | |

|`complex`            | ❌ / ❌ | |

|`timedelta`          | ❌ / ❌ | |

|`datetime`           | ❌ / ❌ | |

|`string`             | ✔ / ✔ | |

|`Unicode`            | ✔ / ✔ | |

|`void *`             | ❌ / ❌ | |

| Structured data types | ❌ / ❌ | |

### Compression Tools

| Data Type   | Status
(reading / writing) | Notes |

|-------------|:-------------------:|-------|

|`zlib / gzip`| ✔ / ✔ | Only system default compression level (normally 6) is enabled for writing. |

|`bzip2`      | ✔ / ✔ | Only compression level 9 is enabled for writing. |

|`blosc`      | ✔ / ✔ | Only `lz4` compression level 5 is enabled for writing. |

|`LZMA `      | ✔ / ✔ | |

|`LZ4`        | ✔ / ✔ | |

|`Zstd`       | ✔ / ✔ | |

Please open an [issue](https://github.com/grimbough/Rarr/issues) if support for a required compression tool is missing.

### Filters

The is currently no support for additional filters.  Please open an [issue](https://github.com/grimbough/Rarr/issues) if you require filter support.

# Required system libraries

To provide support for BLOSC and zstd compression tools Rarr links against libraries providing these tools.  If you have them installed on your system Rarr will attempt to use those versions.  If they are not detected then Rarr will compile and use versions that are distributed with the package.  Either way the functionality will available, however if you are using the system libraries and then later remove them Rarr may fail to work correctly.   

This only concerns users installing the package from source. If you are using the pre-built binaries for Windows or Mac OSX distributed by Bioconductor then this should not be an issue for you.