{"id":16614516,"url":"https://github.com/elianhugh/streams","last_synced_at":"2026-05-26T20:31:19.608Z","repository":{"id":37708612,"uuid":"503250386","full_name":"ElianHugh/streams","owner":"ElianHugh","description":"Flexible data streaming for R","archived":false,"fork":false,"pushed_at":"2022-07-29T07:56:22.000Z","size":138,"stargazers_count":3,"open_issues_count":7,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-18T00:44:37.537Z","etag":null,"topics":["data","package","r","r-package","streaming"],"latest_commit_sha":null,"homepage":"https://elianhugh.github.io/streams/","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ElianHugh.png","metadata":{"files":{"readme":"README.Rmd","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-06-14T07:18:22.000Z","updated_at":"2024-06-22T05:54:38.000Z","dependencies_parsed_at":"2022-08-31T18:21:21.882Z","dependency_job_id":null,"html_url":"https://github.com/ElianHugh/streams","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ElianHugh/streams","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ElianHugh%2Fstreams","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ElianHugh%2Fstreams/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ElianHugh%2Fstreams/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ElianHugh%2Fstreams/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ElianHugh","download_url":"https://codeload.github.com/ElianHugh/streams/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ElianHugh%2Fstreams/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33538659,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"ssl_error","status_checked_at":"2026-05-26T15:22:15.568Z","response_time":63,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data","package","r","r-package","streaming"],"created_at":"2024-10-12T02:06:35.174Z","updated_at":"2026-05-26T20:31:19.592Z","avatar_url":"https://github.com/ElianHugh.png","language":"R","funding_links":[],"categories":[],"sub_categories":[],"readme":"---\noutput: github_document\n---\n\n\u003c!-- README.md is generated from README.Rmd. Please edit that file --\u003e\n\n```{r, include = FALSE}\nknitr::opts_chunk$set(\n  collapse = TRUE,\n  comment = \"#\u003e\",\n  fig.path = \"man/figures/README-\",\n  out.width = \"100%\"\n)\n\nlibrary(streams)\n```\n\n# streams\n\n** WORK IN PROGRESS **\n\n\n\u003c!-- badges: start --\u003e\n[![:name status badge](https://elianhugh.r-universe.dev/badges/:name)](https://elianhugh.r-universe.dev)\n[![Lifecycle: experimental](https://img.shields.io/badge/lifecycle-experimental-orange.svg)](https://www.tidyverse.org/lifecycle/#experimental)\n[![R-CMD-check](https://github.com/ElianHugh/streams/workflows/R-CMD-check/badge.svg)](https://github.com/ElianHugh/streams/actions)\n[![Codecov test coverage](https://codecov.io/gh/ElianHugh/streams/branch/main/graph/badge.svg)](https://app.codecov.io/gh/ElianHugh/streams?branch=main)\n\u003c!-- badges: end --\u003e\n\nData streams are an abstraction for flowing data -- data that is not held in-memory, but is accessed lazily when needed by the consumer. In essence, a data stream is a data-processing object that sacrifices speed for memory optimisation.\n\nPackage Aims:\n\n* Simple but extensible data streaming system\n* Easy to understand data flow state machine\n* Data agnostic\n\n## Installation\n\nUpon release, you can install streams from [my r-universe repo](https://elianhugh.r-universe.dev/ui):\n\n``` r\ninstall.packages(\"streams\", repos = \"https://elianhugh.r-universe.dev\")\n```\n\nAlternatively, you can install the development version of streams from [GitHub](https://github.com/) with:\n\n``` r\n# install.packages(\"devtools\")\ndevtools::install_github(\"ElianHugh/streams\")\n```\n\n## Examples\n\n### Stream creation\n\nThe simplest form of stream creation comes from the `as_reader` and `as_writer` methods, which coerce a given object to a stream. See below for a *very* brief overview of the stream system.\n\n#### Readable Stream\n\nLet's create a read stream:\n\n```{r}\nstream \u003c- as_reader(1:5)\nstream\n```\n\nIn this state, the stream will not output any data. We can attach a listener to cause the stream to flow:\n\n```{r}\nstream |\u003e\n  on(\"data\", print)\n\n#\u003e 1\n#\u003e 2\n#\u003e 3\n#\u003e 4\n#\u003e 5\n```\n\n```{r, include = FALSE}\nrm(stream)\n```\n\n#### Writeable Stream\n\nSimilarly, we can create a write stream:\n\n```{r}\nstream \u003c- as_writer(\"x\")\nstream\n```\n\nWe can then write data to the stream, which will write to the variable `x`:\n\n```{r}\nstream$write(\"Hello world!\")\n```\n\n```{r}\nprint(x)\n```\n\n```{r, include = FALSE}\nrm(x)\n```\n\n#### Stream Piping\n\nWe can pull all these together with the `%|\u003e%` (stream pipe) command. Not to be confused with the base `|\u003e` or magrittr `%\u003e%` pipe, the stream pipe command is used to chain streams together.\n\nLet's try chaining the previous streams together:\n\n```{r}\nas_reader(1L:5L) %|\u003e%\n  as_writer(\"x\")\n```\n\n```{r, include=FALSE}\nwhile (!streams:::no_pending_operations()) {\n  later::run_now()\n}\n```\n\n```{r}\nprint(x)\n```\n\n## Inspiration\n\nThis package could not have been developed without the existence of data streams in many other programming languages. In particular, the NodeJS and web stream standard were instrumental in the design of {streams}.\n\n- [NodeJS streams](https://nodejs.org/api/stream.html)\n- [WHATWG Web Stream standard](https://streams.spec.whatwg.org/)\n- [Functional streams](https://www.npmjs.com/package/functional-streams)\n- [Stromjs](https://github.com/lewisdiamond/stromjs)\n\n## Code of Conduct\n\nPlease note that the streams project is released with a [Contributor Code of Conduct](https://elianhugh.github.io/streams/CODE_OF_CONDUCT.html). By contributing to this project, you agree to abide by its terms.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felianhugh%2Fstreams","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Felianhugh%2Fstreams","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felianhugh%2Fstreams/lists"}