https://github.com/elianhugh/streams
Flexible data streaming for R
https://github.com/elianhugh/streams
data package r r-package streaming
Last synced: 7 months ago
JSON representation
Flexible data streaming for R
- Host: GitHub
- URL: https://github.com/elianhugh/streams
- Owner: ElianHugh
- License: gpl-3.0
- Created: 2022-06-14T07:18:22.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2022-07-29T07:56:22.000Z (about 3 years ago)
- Last Synced: 2025-01-17T16:37:19.883Z (9 months ago)
- Topics: data, package, r, r-package, streaming
- Language: R
- Homepage: https://elianhugh.github.io/streams/
- Size: 135 KB
- Stars: 3
- Watchers: 1
- Forks: 0
- Open Issues: 7
-
Metadata Files:
- Readme: README.Rmd
- License: LICENSE.md
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
---
output: github_document
---
```{r, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "man/figures/README-",
out.width = "100%"
)
library(streams)
```
# streams
** WORK IN PROGRESS **
[](https://elianhugh.r-universe.dev)
[](https://www.tidyverse.org/lifecycle/#experimental)
[](https://github.com/ElianHugh/streams/actions)
[](https://app.codecov.io/gh/ElianHugh/streams?branch=main)
Data streams are an abstraction for flowing data -- data that is not held in-memory, but is accessed lazily when needed by the consumer. In essence, a data stream is a data-processing object that sacrifices speed for memory optimisation.
Package Aims:
* Simple but extensible data streaming system
* Easy to understand data flow state machine
* Data agnostic
## Installation
Upon release, you can install streams from [my r-universe repo](https://elianhugh.r-universe.dev/ui):
``` r
install.packages("streams", repos = "https://elianhugh.r-universe.dev")
```
Alternatively, you can install the development version of streams from [GitHub](https://github.com/) with:
``` r
# install.packages("devtools")
devtools::install_github("ElianHugh/streams")
```
## Examples
### Stream creation
The simplest form of stream creation comes from the `as_reader` and `as_writer` methods, which coerce a given object to a stream. See below for a *very* brief overview of the stream system.
#### Readable Stream
Let's create a read stream:
```{r}
stream <- as_reader(1:5)
stream
```
In this state, the stream will not output any data. We can attach a listener to cause the stream to flow:
```{r}
stream |>
on("data", print)
#> 1
#> 2
#> 3
#> 4
#> 5
```
```{r, include = FALSE}
rm(stream)
```
#### Writeable Stream
Similarly, we can create a write stream:
```{r}
stream <- as_writer("x")
stream
```
We can then write data to the stream, which will write to the variable `x`:
```{r}
stream$write("Hello world!")
```
```{r}
print(x)
```
```{r, include = FALSE}
rm(x)
```
#### Stream Piping
We can pull all these together with the `%|>%` (stream pipe) command. Not to be confused with the base `|>` or magrittr `%>%` pipe, the stream pipe command is used to chain streams together.
Let's try chaining the previous streams together:
```{r}
as_reader(1L:5L) %|>%
as_writer("x")
```
```{r, include=FALSE}
while (!streams:::no_pending_operations()) {
later::run_now()
}
```
```{r}
print(x)
```
## Inspiration
This package could not have been developed without the existence of data streams in many other programming languages. In particular, the NodeJS and web stream standard were instrumental in the design of {streams}.
- [NodeJS streams](https://nodejs.org/api/stream.html)
- [WHATWG Web Stream standard](https://streams.spec.whatwg.org/)
- [Functional streams](https://www.npmjs.com/package/functional-streams)
- [Stromjs](https://github.com/lewisdiamond/stromjs)
## Code of Conduct
Please note that the streams project is released with a [Contributor Code of Conduct](https://elianhugh.github.io/streams/CODE_OF_CONDUCT.html). By contributing to this project, you agree to abide by its terms.