{"id":17158224,"url":"https://github.com/msoap/byline","last_synced_at":"2025-07-14T11:09:23.364Z","repository":{"id":57480874,"uuid":"83917848","full_name":"msoap/byline","owner":"msoap","description":"Go library for convert io.Reader to line-by-line Reader","archived":false,"fork":false,"pushed_at":"2024-09-30T21:05:06.000Z","size":77,"stargazers_count":91,"open_issues_count":0,"forks_count":4,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-05T12:04:24.104Z","etag":null,"topics":["go","golang","library","reader"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/msoap.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-03-04T19:08:36.000Z","updated_at":"2025-03-13T01:53:46.000Z","dependencies_parsed_at":"2024-11-07T21:00:47.550Z","dependency_job_id":"426e9ae3-a1e4-4cef-a6c7-9f29ec94ed09","html_url":"https://github.com/msoap/byline","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/msoap/byline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msoap%2Fbyline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msoap%2Fbyline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msoap%2Fbyline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msoap%2Fbyline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/msoap","download_url":"https://codeload.github.com/msoap/byline/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msoap%2Fbyline/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265281303,"owners_count":23739873,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","library","reader"],"created_at":"2024-10-14T22:10:52.830Z","updated_at":"2025-07-14T11:09:23.314Z","avatar_url":"https://github.com/msoap.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# byline Reader [![Go Reference](https://pkg.go.dev/badge/github.com/msoap/byline.svg)](https://pkg.go.dev/github.com/msoap/byline) [![Go](https://github.com/msoap/byline/actions/workflows/go.yml/badge.svg)](https://github.com/msoap/byline/actions/workflows/go.yml) [![Coverage Status](https://coveralls.io/repos/github/msoap/byline/badge.svg?branch=master)](https://coveralls.io/github/msoap/byline?branch=master) [![Sourcegraph](https://sourcegraph.com/github.com/msoap/byline/-/badge.svg)](https://sourcegraph.com/github.com/msoap/byline?badge) [![Report Card](https://goreportcard.com/badge/github.com/msoap/byline)](https://goreportcard.com/report/github.com/msoap/byline)\n\nGo-library for reading and processing data from a `io.Reader` line by line. Now you can add UNIX text processing principles to its Reader (like with awk, grep, sed ...).\n\n## Install\n\n`go get -u github.com/msoap/byline`\n\n## Usage\n\n```Go\nimport \"github.com/msoap/byline\"\n\n// Create new line-by-line Reader from io.Reader:\nlr := byline.NewReader(reader)\n\n// Add to the Reader stack of a filter functions:\nlr.MapString(func(line string) string {return \"prefix_\" + line}).GrepByRegexp(regexp.MustCompile(\"only this\"))\n\n// Read all content\nresult, err := lr.ReadAll()\n\n// Use everywhere instead of io.Reader\n_, err := io.Copy(os.Stdout, lr)\n\n// Or in one place\nresult, err := byline.NewReader(reader).MapString(func(line string) string {return \"prefix_\" + line}).ReadAll()\n```\n\n## Filter functions\n\n  * `Map(func([]byte) []byte)` - processing of each line as `[]byte`.\n  * `MapErr(func([]byte) ([]byte, error))` - processing of each line as `[]byte`, and you can return error, `io.EOF` or custom error.\n  * `MapString(func(string) string)` - processing of each line as `string`.\n  * `MapStringErr(func(string) (string, error))` - processing of each line as `string`, and you can return error.\n  * `Each(func([]byte))` - processing each line without changing the line\n  * `EachString(func(string))` - processing each line as string without changing the line\n  * `Grep(func([]byte) bool)` - filtering lines by function.\n  * `GrepString(func(string) bool)` - filtering lines as `string` by function.\n  * `GrepByRegexp(re *regexp.Regexp)` - filtering lines by regexp.\n  * `AWKMode(func(line string, fields []string, vars AWKVars) (string, error))` - processing of each line in AWK mode.\n    In addition to current line, `filterFn` gets slice with fields splitted by separator (default is `/\\s+/`) and vars releated to awk (`NR`, `NF`, `RS`, `FS`).\n    Attention! Use `AWKMode()` with caution on large data sets, see [Overheads](#overheads) below.\n\n`Map*Err`, `AWKMode` methods can return `byline.ErrOmitLine` - error for discard processing of current line.\n\n## Helper methods\n\n  * `SetRS(rs byte)` - set line (record) separator, default is newline - `\\n`.\n  * `SetFS(fs *regexp.Regexp)` - set field separator for AWK mode, default is `\\s+`.\n  * `Discard()` - discard all content from Reader only for side effect of filter functions.\n  * `ReadAll() ([]byte, error)` - return all content as slice of bytes.\n  * `ReadAllSlice() ([][]byte, error)` - return all content by lines as `[][]byte`.\n  * `ReadAllString() (string, error)` - return all content as string.\n  * `ReadAllSliceString() ([]string, error)` - return all content by lines as slice of strings.\n\n## Examples\n\nAdd line number to each line and add suffix at the end of line:\n\n```Go\nreader := strings.NewReader(\"111\\n222\\n333\")\n// or read file\nreader, err := os.Open(\"file.txt\")\n// or process response from HTTP client\nreader := httpResponse.Body\n\ni := 0\nblr := byline.NewReader(reader).MapString(func(line string) string {\n\ti++\n\treturn fmt.Sprintf(\"(%d) %s\", i, string(line))\n}).Map(func(line []byte) []byte {\n\treturn regexp.MustCompile(`\\n?$`).ReplaceAll(line, []byte(\" suf\\n\"))\n})\n\nresult, err := blr.ReadAll()\n```\n\n\u003cdetails\u003e\u003csummary\u003eSelect all types from the Go-source:\u003c/summary\u003e\n\n```Go\ntype StateMachine struct {\n\tbeginRe *regexp.Regexp\n\tendRe   *regexp.Regexp\n\tinBlock bool\n}\n\nfunc (sm *StateMachine) SMFilter(line []byte) bool {\n\tswitch {\n\tcase sm.beginRe.Match(line):\n\t\tsm.inBlock = true\n\t\treturn true\n\tcase sm.inBlock \u0026\u0026 sm.endRe.Match(line):\n\t\tsm.inBlock = false\n\t\treturn true\n\tdefault:\n\t\treturn sm.inBlock\n\t}\n}\n\nfunc ExampleReader_Grep() {\n\tfile, err := os.Open(\"byline.go\")\n\tif err != nil {\n\t\tfmt.Println(err)\n\t\treturn\n\t}\n\n\t// get all lines between \"^type...\" and \"^}\"\n\tsm := StateMachine{\n\t\tbeginRe: regexp.MustCompile(`^type `),\n\t\tendRe:   regexp.MustCompile(`^}\\s+$`),\n\t}\n\n\tblr := byline.NewReader(file).Grep(sm.SMFilter).Map(func(line []byte) []byte {\n\t\t// and remove comments\n\t\treturn regexp.MustCompile(`\\s+//.+`).ReplaceAll(line, []byte{})\n\t})\n\n\tresult, err := blr.ReadAllString()\n\tif err != nil {\n\t\tfmt.Println(err)\n\t\treturn\n\t}\n\n\tfmt.Print(result)\n}\n```\nOutput:\n```\ntype Reader struct {\n\tscanner     *bufio.Scanner\n\tbuffer      bytes.Buffer\n\texistsData  bool\n\tfilterFuncs []func(line []byte) ([]byte, error)\n\tawkVars     AWKVars\n}\ntype AWKVars struct {\n\tNR int\n\tNF int\n\tRS byte\n\tFS *regexp.Regexp\n}\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\u003csummary\u003eExample of AWK mode, sum the third column with the filter (\u003e10.0):\u003c/summary\u003e\n\n```Go\n// CSV with \"#\" instead of \"\\n\"\nreader := strings.NewReader(`1,name one,12.3#2,second row;7.1#3,three row;15.51`)\n\nsum := 0.0\nerr := byline.NewReader(reader).\n\tSetRS('#').\n\tSetFS(regexp.MustCompile(`[,;]`)).\n\tAWKMode(func(line string, fields []string, vars byline.AWKVars) (string, error) {\n\t\tif vars.NF \u003c 3 {\n\t\t\treturn \"\", fmt.Errorf(\"csv parse failed for %q\", line)\n\t\t}\n\n\t\tif price, err := strconv.ParseFloat(fields[2], 10); err != nil {\n\t\t\treturn \"\", err\n\t\t} else if price \u003c 10 {\n\t\t\treturn \"\", byline.ErrOmitLine\n\t\t} else {\n\t\t\tsum += price\n\t\t\treturn \"\", nil\n\t\t}\n\t}).Discard()\n\nif err != nil {\n\tfmt.Println(\"Price sum:\", sum)\n}\n\n```\nOutput:\n```\nPrice sum: 27.81\n```\n\u003c/details\u003e\n\n## Overheads\n\nAn example in which we get odd lines (for `io.Reader` with 10000 lines):\n\n    ❯ make benchmark\n    go test -benchtime 5s -benchmem -bench .\n    Benchmark_NativeScannerBytes-4       \t   20000\t    312502 ns/op\t  215080 B/op\t      24 allocs/op\n    Benchmark_NativeScannerOnlyCount-4   \t   30000\t    217491 ns/op\t    4160 B/op\t       4 allocs/op\n    Benchmark_MapBytes-4                 \t   10000\t    567421 ns/op\t  135184 B/op\t      17 allocs/op\n    Benchmark_MapString-4                \t    5000\t   1408956 ns/op\t  374000 B/op\t   15018 allocs/op\n    Benchmark_Grep-4                     \t   10000\t    592100 ns/op\t  135200 B/op\t      18 allocs/op\n    Benchmark_GrepString-4               \t    5000\t   1151309 ns/op\t  294416 B/op\t   10019 allocs/op\n    Benchmark_Each-4                     \t   10000\t    562337 ns/op\t    6201 B/op\t      13 allocs/op\n    Benchmark_EachString-4               \t   10000\t    991528 ns/op\t  165427 B/op\t   10013 allocs/op\n    Benchmark_AWKMode-4                  \t     500\t  11865482 ns/op\t 3410392 B/op\t   55466 allocs/op\n    PASS\n\nSee `benchmark_test.go` for benchmark code\n\n## See also\n\n  * [io](https://pkg.go.dev/io), [bufio](https://pkg.go.dev/bufio) - Go packages for work with Readers.\n  * [go-linereader](https://github.com/mitchellh/go-linereader) - package that reads lines from an io.Reader and puts them onto a channel.\n  * [AWK](https://en.wikipedia.org/wiki/AWK) - programming language and great UNIX tool.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmsoap%2Fbyline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmsoap%2Fbyline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmsoap%2Fbyline/lists"}