{"id":16841116,"url":"https://github.com/bobheadxi/streamline","last_synced_at":"2025-03-17T04:33:44.177Z","repository":{"id":65194860,"uuid":"586768474","full_name":"bobheadxi/streamline","owner":"bobheadxi","description":"✏️ Transform and handle your data, line by line","archived":false,"fork":false,"pushed_at":"2024-04-05T16:09:57.000Z","size":439,"stargazers_count":10,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-27T18:12:29.905Z","etag":null,"topics":["go","golang","io","streaming"],"latest_commit_sha":null,"homepage":"https://pkg.go.dev/go.bobheadxi.dev/streamline","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bobheadxi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-09T07:23:50.000Z","updated_at":"2024-06-29T13:23:14.000Z","dependencies_parsed_at":"2024-01-25T02:25:22.051Z","dependency_job_id":"f70b9b03-425b-479d-9149-6f87eefcb5ac","html_url":"https://github.com/bobheadxi/streamline","commit_stats":{"total_commits":113,"total_committers":1,"mean_commits":113.0,"dds":0.0,"last_synced_commit":"a612d048b15984ce20645f431a340dd237bc1645"},"previous_names":[],"tags_count":20,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bobheadxi%2Fstreamline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bobheadxi%2Fstreamline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bobheadxi%2Fstreamline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bobheadxi%2Fstreamline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bobheadxi","download_url":"https://codeload.github.com/bobheadxi/streamline/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243846976,"owners_count":20357294,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","io","streaming"],"created_at":"2024-10-13T12:40:21.344Z","updated_at":"2025-03-17T04:33:43.879Z","avatar_url":"https://github.com/bobheadxi.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# streamline [![go reference](https://pkg.go.dev/badge/go.bobheadxi.dev/streamline.svg)](https://pkg.go.dev/go.bobheadxi.dev/streamline) [![Sourcegraph](https://img.shields.io/badge/view%20on-sourcegraph-A112FE?logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAADIAAAAyCAYAAAAeP4ixAAAEZklEQVRoQ+2aXWgUZxSG3292sxtNN43BhBakFPyhxSujRSxiU1pr7SaGXqgUxOIEW0IFkeYighYUxAuLUlq0lrq2iCDpjWtmFVtoG6QVNOCFVShVLyxIk0DVjZLMxt3xTGTccd2ZOd/8JBHci0CY9zvnPPN+/7sCIXwKavOwAcy2QgngQiIztDSE0OwQlDPYR1ebiaH6J5kZChyfW12gRG4QVgGTBfMchMbFP9Sn5nlZL2D0JjLD6710lc+z0NfqSGTXQRQ4bX07Mq423yoBL3OSyHSvUxirMuaEvgbJWrdcvkHMoJwxYuq4INUhyuWvQa1jvdMGxAvCxJlyEC9XOBCWL04wwRzpbDoDQ7wfZJzIQLi5Eggk6DiRhZgWIAbE3NrM4A3LPT8Q7UgqAqLqTmLSHLGPkyzG/qXEczhd0q6RH+zaSBfaUoc4iQx19pIClIscrTkNZzG6gd7qMY6eC2Hqyo705ZfTf+eqJmhMzcSbYtQpOXc92ZsZjLVAL4YNUQbJ5Ttg4CQrQdGYj44Xr9m1XJCzmZusFDJOWNpHjmh5x624a2ZFtOKDVL+uNo2TuXE3bZQQZUf8gtgqP31uI94Z/rMqix+IGiRfWw3xN9dCgVx+L3WrHm4Dju6PXz/EkjuXJ6R+IGgyOE1TbZqTq9y1eo0EZo7oMo1ktPu3xjHvuiLT5AFNszUyDULtWpzE2/fEsey8O5TbWuGWwxrs5rS7nFNMWJrNh2No74s9Ec4vRNmRRzPXMP19fBMSVsGcOJ98G8N3Wl2gXcbTjbX7vUBxLaeASDQCm5Cu/0E2tvtb0Ea+BowtskFD0wvlc6Rf2M+Jx7dTu7ubFr2dnKDRaMQe2v/tcIrNB7FH0O50AcrBaApmRDVwFO31ql3pD8QW4dP0feNwl/Q+kFEtRyIGyaWXnpy1OO0qNJWHo1y6iCmAGkBb/Ru+HenDWIF2mo4r8G+tRRzoniSn2uqFLxANhe9LKHVyTbz6egk9+x5w5fK6ulSNNMhZ/Feno+GebLZV6isTTa6k5qNl5RnZ5u56Ib6SBvFzaWBBVFZzvnERWlt/Cg4l27XChLCqFyLekjhy6xJyoytgjPf7opIB8QPx7sYFiMXHPGt76m741MhCKMZfng0nBOIjmoJPsLqWHwgFpe6V6qtfcopxveR2Oy+J0ntIN/zCWkf8QNAJ7y6d8Bq4lxLc2/qJl5K7t432XwcqX5CrI34gzATWuYILQtdQPyePDK3iuOekCR3Efjhig1B1Uq5UoXEEoZX7d1q535J5S9VOeFyYyEBku5XTMXXKQTToX5Rg7OI44nbW5oKYeYK4EniMeF0YFNSmb+grhc84LyRCEP1/OurOcipCQbKxDeK2V5FcVyIDMQvsgz5gwFhcWWwKyRlvQ3gv29RwWoDYAbIofNyBxI9eDlQ+n3YgsgCWnr4MStGXQXmv9pF2La/k3OccV54JEBM4yp9EsXa/3LfO0dGPcYq0Y7DfZB8nJzZw2rppHgKgVHs8L5wvRwAAAABJRU5ErkJggg==)](https://sourcegraph.com/github.com/bobheadxi/streamline)\n\n[![pipeline](https://github.com/bobheadxi/streamline/actions/workflows/pipeline.yaml/badge.svg)](https://github.com/bobheadxi/streamline/actions/workflows/pipeline.yaml)\n[![codecov](https://codecov.io/gh/bobheadxi/streamline/branch/main/graph/badge.svg?token=f1VZULSJsT)](https://codecov.io/gh/bobheadxi/streamline)\n[![Go Report Card](https://goreportcard.com/badge/go.bobheadxi.dev/streamline)](https://goreportcard.com/report/go.bobheadxi.dev/streamline)\n[![benchmarks](https://img.shields.io/website/https/bobheadxi.dev/streamline.svg?down_color=red\u0026down_message=offline\u0026label=benchmarks\u0026up_message=live)](https://bobheadxi.dev/streamline)\n\nTransform and handle your data, line by line.\n\n```sh\ngo get go.bobheadxi.dev/streamline\n```\n\n## Overview\n\n[`streamline`](https://pkg.go.dev/go.bobheadxi.dev/streamline) offers a variety of primitives to make working with data line by line a breeze:\n\n- [`streamline.Stream`](https://pkg.go.dev/go.bobheadxi.dev/streamline#Stream) offers the ability to add hooks that handle an `io.Reader` line-by-line with `(*Stream).Stream`, `(*Stream).StreamBytes`, and other utilities.\n- [`pipeline.Pipeline`](https://pkg.go.dev/go.bobheadxi.dev/streamline/pipeline#Pipeline) offers a way to build pipelines that transform the data in a `streamline.Stream`, such as cleaning, filtering, mapping, or sampling data.\n  - [`jq.Pipeline`](https://pkg.go.dev/go.bobheadxi.dev/streamline/jq#Pipeline) can be used to map every line to the output of a JQ query, for example.\n  - [`streamline.Stream` implements standard `io` interfaces like `io.Reader`](https://pkg.go.dev/go.bobheadxi.dev/streamline#Stream.Read), so `pipeline.Pipeline` can be used for general-purpose data manipulation as well.\n- [`pipe.NewStream`](https://pkg.go.dev/go.bobheadxi.dev/streamline/pipe#NewStream) offers a way to create a buffered pipe between a writer and a `Stream`.\n  - [`streamexec.Start`](https://pkg.go.dev/go.bobheadxi.dev/streamline/streamexec#Start) uses this to attach a `Stream` to an `exec.Cmd` to work with command output.\n\nWhen working with data streams in Go, you typically get an `io.Reader`, which is great for arbitrary data - but in many cases, especially when scripting, it's common to either end up with data and outputs that are structured line by line, or want to handle data line by line, for example to send to a structured logging library. You can set up a `bufio.Reader` or `bufio.Scanner` to do this, but for cases like `exec.Cmd` you will also need boilerplate to configure the command and set up pipes, and for additional functionality like transforming, filtering, or sampling output you will need to write your own additional handlers. `streamline` aims to provide succint ways to do all of the above and more.\n\n### Add prefixes to command output\n\n\u003ctable\u003e\n\u003ctr\u003e\n  \u003cth\u003e\u003ccode\u003ebufio.Scanner\u003c/code\u003e\u003c/th\u003e\n  \u003cth\u003e\u003ccode\u003estreamline/streamexec\u003c/code\u003e\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\n```go\nfunc PrefixOutput(cmd *exec.Cmd) error {\n    reader, writer := io.Pipe()\n    cmd.Stdout = writer\n    cmd.Stderr = writer\n    if err := cmd.Start(); err != nil {\n        return err\n    }\n    errC := make(chan error)\n    go func() {\n        err := cmd.Wait()\n        writer.Close()\n        errC \u003c- err\n    }()\n    s := bufio.NewScanner(reader)\n    for s.Scan() {\n        println(\"PREFIX: \", s.Text())\n    }\n    if err := s.Err(); err != nil {\n        return err\n    }\n    return \u003c-errC\n}\n```\n\n\u003c/td\u003e\n\u003ctd\u003e\n\n```go\nfunc PrefixOutput(cmd *exec.Cmd) error {\n    stream, err := streamexec.Start(cmd)\n    if err != nil {\n        return err\n    }\n    return stream.Stream(func(line string) {\n        println(\"PREFIX: \", line)\n    })\n}\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n### Process JSON on the fly\n\n\u003ctable\u003e\n\u003ctr\u003e\n  \u003cth\u003e\u003ccode\u003ebufio.Scanner\u003c/code\u003e\u003c/th\u003e\n  \u003cth\u003e\u003ccode\u003estreamline\u003c/code\u003e\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\n```go\nfunc GetMessages(r io.Reader) error {\n    s := bufio.NewScanner(r)\n    for s.Scan() {\n        var result bytes.Buffer\n        cmd := exec.Command(\"jq\", \".msg\")\n        cmd.Stdin = bytes.NewReader(s.Bytes())\n        cmd.Stdout = \u0026result\n        if err := cmd.Run(); err != nil {\n            return err\n        }\n        print(result.String())\n    }\n    return s.Err()\n}\n```\n\n\u003c/td\u003e\n\n\u003ctd\u003e\n\n```go\nfunc GetMessages(r io.Reader) error {\n    return streamline.New(r).\n        WithPipeline(jq.Pipeline(\".msg\")).\n        Stream(func(line string) {\n            println(line)\n        })\n}\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n### Sample noisy output\n\n\u003ctable\u003e\n\u003ctr\u003e\n  \u003cth\u003e\u003ccode\u003ebufio.Scanner\u003c/code\u003e\u003c/th\u003e\n  \u003cth\u003e\u003ccode\u003estreamline\u003c/code\u003e\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\n```go\nfunc PrintEvery10th(r io.Reader) error {\n    s := bufio.NewScanner(r)\n    var count int\n    for s.Scan() {\n        count++\n        if count%10 != 0 {\n            continue\n        }\n        println(s.Text())\n    }\n    return s.Err()\n}\n```\n\n\u003c/td\u003e\n\n\u003ctd\u003e\n\n```go\nfunc PrintEvery10th(r io.Reader) error {\n    return streamline.New(r).\n        WithPipeline(pipeline.Sample(10)).\n        Stream(func(line string) {\n            println(line)\n        })\n}\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n### Transform specific lines\n\nThis particular example is a somewhat realistic one - [GCP Cloud SQL cannot accept `pgdump` output that contains certain `EXTENSION`-related statements](https://cloud.google.com/sql/docs/postgres/import-export/import-export-dmp#external-server), so to `pgdump` a PostgreSQL database and upload the dump in a bucket for import into Cloud SQL, one must pre-process their dumps to remove offending statements.\n\n\u003ctable\u003e\n\u003ctr\u003e\n  \u003cth\u003e\u003ccode\u003ebufio.Scanner\u003c/code\u003e\u003c/th\u003e\n  \u003cth\u003e\u003ccode\u003estreamline\u003c/code\u003e\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\n```go\nvar unwanted = []byte(\"COMMENT ON EXTENSION\")\n\nfunc Upload(pgdump *os.File, dst io.Writer) error {\n    s := bufio.NewScanner(pgdump)\n    for s.Scan() {\n        line := s.Bytes()\n        var err error\n        if bytes.Contains(line, unwanted) {\n            _, err = dst.Write(\n                // comment out this line\n                append([]byte(\"-- \"), line...))\n        } else {\n            _, err = dst.Write(line)\n        }\n        if err != nil {\n            return err\n        }\n    }\n    return s.Err()\n}\n```\n\n\u003c/td\u003e\n\n\u003ctd\u003e\n\n```go\nvar unwanted = []byte(\"COMMENT ON EXTENSION\")\n\nfunc Upload(pgdump *os.File, dst io.Writer) error {\n    _, err := streamline.New(pgdump).\n        WithPipeline(pipeline.Map(func(line []byte) []byte {\n            if bytes.Contains(line, unwanted) {\n                // comment out this line\n                return append([]byte(\"-- \"), line...)\n            }\n            return line\n        })).\n        WriteTo(dst)\n    return err\n}\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n## Background\n\nSome of the ideas in this package started in [`sourcegraph/run`](https://github.com/sourcegraph/run), which started as a project trying to build utilities that [made it easier to write bash-esque scripts using Go](https://github.com/sourcegraph/sourcegraph/blob/main/doc/dev/adr/1652433602-use-go-for-scripting.md) - namely being able to do things you would often to in scripts such as grepping and iterating over lines. `streamline` generalizes on the ideas used in `sourcegraph/run` for working with command output to work on arbitrary inputs, and `sourcegraph/run` now uses `streamline` internally.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbobheadxi%2Fstreamline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbobheadxi%2Fstreamline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbobheadxi%2Fstreamline/lists"}