{"id":18610466,"url":"https://github.com/neilotoole/streamcache","last_synced_at":"2025-12-15T05:02:24.508Z","repository":{"id":54291357,"uuid":"342417172","full_name":"neilotoole/streamcache","owner":"neilotoole","description":"Golang in-memory caching stream reader","archived":false,"fork":false,"pushed_at":"2024-03-14T12:30:08.000Z","size":718,"stargazers_count":35,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-11T09:08:21.233Z","etag":null,"topics":["bytes","cache","go","golang","io","multireader","multiwriter","reader","stream"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/neilotoole.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2021-02-26T00:29:42.000Z","updated_at":"2025-04-06T23:23:37.000Z","dependencies_parsed_at":"2024-01-26T20:49:27.266Z","dependency_job_id":"45706ad0-54fe-4c35-8ba4-e8552be640ab","html_url":"https://github.com/neilotoole/streamcache","commit_stats":null,"previous_names":["neilotoole/samplereader"],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/neilotoole/streamcache","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neilotoole%2Fstreamcache","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neilotoole%2Fstreamcache/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neilotoole%2Fstreamcache/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neilotoole%2Fstreamcache/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/neilotoole","download_url":"https://codeload.github.com/neilotoole/streamcache/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neilotoole%2Fstreamcache/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":27744342,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-15T02:00:09.782Z","response_time":96,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bytes","cache","go","golang","io","multireader","multiwriter","reader","stream"],"created_at":"2024-11-07T03:10:06.748Z","updated_at":"2025-12-15T05:02:24.468Z","avatar_url":"https://github.com/neilotoole.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# streamcache: in-memory caching stream reader\n\n[![Go Reference](https://pkg.go.dev/badge/github.com/neilotoole/streamcache.svg)](https://pkg.go.dev/github.com/neilotoole/streamcache)\n[![Go Report Card](https://goreportcard.com/badge/neilotoole/streamcache)](https://goreportcard.com/report/neilotoole/streamcache)\n[![License](https://img.shields.io/badge/License-MIT-blue.svg)](https://github.com/neilotoole/streamcache/blob/master/LICENSE)\n![Workflow](https://github.com/neilotoole/streamcache/actions/workflows/go.yml/badge.svg)\n\n\nPackage [`streamcache`](https://pkg.go.dev/github.com/neilotoole/streamcache)\nimplements a Go in-memory byte cache mechanism that allows multiple callers to\nread some or all of the contents of a source `io.Reader`, while only reading\nfrom the source reader once. When only the final reader remains, the cache is\ndiscarded and the final reader reads directly from the source. This is particularly\nuseful for scenarios where multiple readers may wish to sample the start of a\nstream, but only one reader will read the entire stream.\n\nLet's say we have a program [`typedetect`](./examples/typedetect),\nand we're reading from ``stdin``. For example:\n\n```shell\n$ cat myfile.ext | typedetect  \n```\n\nIn this scenario, `typedetect` wants to detect\nand print the type of data in the file/pipe, and then print the contents.\nThat detection sampling could be done in a separate goroutine per sampler type.\nThe input file could be, let's say, a JSON file, or an XML file.\n\nThe obvious approach is to inspect the first few tokens of the\ninput, and check if the tokens are either valid JSON or valid XML.\nAfter that process, let's say we want to dump out a preview of the file contents.\n\nPackage `streamcache` provides a facility to create a `Stream` from an\nunderlying `io.Reader` (`os.Stdin` in this scenario), and spawn multiple\nreaders, each of which can operate independently, in their own\ngoroutines if desired. The underlying source (again, `os.Stdin` in this\nscenario) will only once be read from, but its data is available to\nmultiple readers, because that data is cached in memory.\n\nThat is, until there's only one final reader left, (after invoking\n`Stream.Seal`), at which point the cache is discarded, and\nthe final `Reader` reads directly from the underlying source.\n\n## Usage\n\nAdd to your `go.mod` via `go get`:\n\n```shell\ngo get github.com/neilotoole/streamcache\n```\n\nHere's a simple [example](./examples/in-out-err) that copies the contents\nof `stdin` to `stdout` and `stderr`, and prints the number of bytes read.\n\n```go\npackage main\n\nimport (\n    \"context\"\n    \"errors\"\n    \"fmt\"\n    \"io\"\n    \"os\"\n\n    \"github.com/neilotoole/streamcache\"\n)\n\n// Write stdin to both stdout and stderr.\n// Some error handling omitted for brevity.\nfunc main() {\n    ctx := context.Background()\n    stream := streamcache.New(os.Stdin)\n\n    r1 := stream.NewReader(ctx)\n    go func() {\n        defer r1.Close()\n        io.Copy(os.Stdout, r1)\n    }()\n\n    r2 := stream.NewReader(ctx)\n    go func() {\n        defer r2.Close()\n        io.Copy(os.Stderr, r2)\n    }()\n    \n    stream.Seal()   // Indicate that there'll be no more readers...\n    \u003c-stream.Done() // Receives when all readers are closed.\n\n    if err := stream.Err(); err != nil \u0026\u0026 !errors.Is(err, io.EOF) {\n        fmt.Fprintln(os.Stderr, \"error:\", err)\n        os.Exit(1)\n    }\n\n    fmt.Fprintf(os.Stdout, \"Read %d bytes from stdin\\n\", stream.Size())\n}\n```\n\nExecuting the above program:\n\n```shell\n$ go install github.com/neilotoole/streamcache/examples/in-out-err\n$ echo \"hello world\" | in-out-err\nhello world\nhello world\nRead 12 bytes from stdin\n```\n\n## Examples\n\n- [`in-out-err`](./examples/in-out-err): copy `stdin` to both `stdout` and `stderr`.\n- [`typedetect`](./examples/typedetect): detect the type of input data, and print the head and tail\n  of the contents.\n  ![streamcache_typedetect.png](examples/typedetect/streamcache_typedetect.png)\n- [`multicase`](./examples/multicase): transform each line of input to upper, lower, and title case.\n  ![streamcache_multicase.png](examples/multicase/streamcache_multicase.png)\n\n## Related work\n\n- [`djherbis/fscache`](https://github.com/djherbis/fscache)\n- [`sq`](https://github.com/neilotoole/sq) uses `streamcache` to stream stdin / HTTP response bodies,\n  allowing `sq` to being processing data on the fly.\n  ![sq streamcache](https://github.com/neilotoole/sq/blob/master/.images/sq_inspect_remote_s3.png)\n- [`fifomu`](https://github.com/neilotoole/fifomu) is a FIFO mutex, used by `streamcache`, which in turn is used by [`sq`](https://github.com/neilotoole/sq).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneilotoole%2Fstreamcache","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fneilotoole%2Fstreamcache","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneilotoole%2Fstreamcache/lists"}