{"id":15914391,"url":"https://github.com/bbengfort/iterfile","last_synced_at":"2025-07-14T17:42:50.471Z","repository":{"id":57551216,"uuid":"77152833","full_name":"bbengfort/iterfile","owner":"bbengfort","description":"Benchmarking for various file iteration utilities","archived":false,"fork":false,"pushed_at":"2016-12-23T15:42:00.000Z","size":1450,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-08T17:28:21.528Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bbengfort.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-12-22T14:56:00.000Z","updated_at":"2021-07-06T01:16:08.000Z","dependencies_parsed_at":"2022-09-26T18:41:22.754Z","dependency_job_id":null,"html_url":"https://github.com/bbengfort/iterfile","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bbengfort%2Fiterfile","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bbengfort%2Fiterfile/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bbengfort%2Fiterfile/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bbengfort%2Fiterfile/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bbengfort","download_url":"https://codeload.github.com/bbengfort/iterfile/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246930053,"owners_count":20856533,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-06T17:02:10.418Z","updated_at":"2025-04-03T03:26:20.191Z","avatar_url":"https://github.com/bbengfort.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# File Iteration Benchmarks  \n\n[![Build Status](https://travis-ci.org/bbengfort/iterfile.svg?branch=master)](https://travis-ci.org/bbengfort/iterfile)\n[![Go Report Card](https://goreportcard.com/badge/github.com/bbengfort/iterfile)](https://goreportcard.com/report/github.com/bbengfort/iterfile)\n[![GoDoc](https://godoc.org/github.com/bbengfort/iterfile?status.svg)](https://godoc.org/github.com/bbengfort/iterfile)\n\n[![Lines \u0026 Curves](fixtures/lines.jpg)](https://flic.kr/p/iaVByW)\n\n**Benchmarking for various file iteration utilities**\n\nThis small library provides various mechanisms for reading a file one line at a time. These utilities aren't necessarily meant to be used as a library for use in production code  (though you're more than welcome to) but rather to profile and benchmark various iteration constructs. See [Benchmarking Readline Iterators](http://bbengfort.github.io/programmer/2016/12/23/benchmarking-readlines.html) for a quick writeup about this repository.\n\n\u003e Read more at [Benchmarking Readline Iterators](http://bbengfort.github.io/programmer/2016/12/23/benchmarking-readlines.html) and [Yielding Functions for Iteration in Go](http://bbengfort.github.io/snippets/2016/12/22/yielding-functions-for-iteration-golang.html).\n\n## Usage\n\nAll of the functions in this library are `Readlines` functions; that is they take as input at least the path to a file, and then provide some iterable context with which to handle one line of the file at a time. The examples for usage here will simply be a line character count (less the newlines), the testing methodology uses line, word, and character counts. Currently we have implemented:\n\n- `ChanReadlines`: returns a channel to `range` on.\n- `CallbackReadlines`: accepts a per-line callback function.\n- `IteratorReadlines`: returns a stateful iterator.\n\n### Channel Readlines\n\nUse the channel based readlines iterator as follows:\n\n```go\n// construct the reader and the line count.\nvar chars int\nreader, err := ChanReadlines(\"fixtures/small.txt\")\n\n// check if there was an error opening the file or scanning.\nif err != nil {\n    log.Fatal(err)\n}\n\n// iterate over the lines using range\nfor line := range reader {\n    chars += len(line)\n}\n```\n\nVariants of this reader would not require the error checking at the beginning, but would rather yield errors in iteration along with the line.\n\n### Callback Readlines\n\nUse the callback-style readlines iterator as follows:\n\n```go\nvar chars int\n\n// Define the callback function\ncb := func(line string) error {\n    chars += len(line)\n    return nil\n}\n\n// Pass the callback to the iterator\nerr := CallbackReadlines(\"fixtures/small.txt\", cb)\nif err != nil {\n    log.Fatal(err)\n}\n```\n\nNote that in this mechanism, you can `break` out of the loop by returning an\nerror from the callback, which will cause the calling iterator to return and\nhopefully also close the file and be done!\n\n## Iterator Readlines\n\nUse the stateful iterator returned by the readlines iterator as follows:\n\n```go\nvar chars int\nreader, err := IteratorReadlines(\"fixtures/small.txt\")\n\n// check if there was an error opening the file or scanning.\nif err != nil {\n    log.Fatal(err)\n}\n\n// iterate over the stateful LineIterator that has been returned.\nfor reader.Next() {\n    chars += len(reader.Line())\n}\n```\n\n## Benchmarks\n\nBenchmarks can be run with the `go test -bench=.` command. The current benchmarks are as follows:\n\n```\nBenchmarkChanReadlinesSmall-8         \t   20000\t     74958 ns/op\nBenchmarkChallbackReadlinesSmall-8    \t   50000\t     28836 ns/op\nBenchmarkIteratorReadlinesSmall-8     \t   50000\t     29006 ns/op\n\nBenchmarkChanReadlinesMedium-8        \t    2000\t    621716 ns/op\nBenchmarkChallbackReadlinesMedium-8   \t   10000\t    216734 ns/op\nBenchmarkIteratorReadlinesMedium-8    \t   10000\t    219842 ns/op\n\nBenchmarkChanReadlinesLarge-8         \t     200\t   6250004 ns/op\nBenchmarkChallbackReadlinesLarge-8    \t    1000\t   2198904 ns/op\nBenchmarkIteratorReadlinesLarge-8     \t    1000\t   2229104 ns/op\n```\n\nWe benchmark each word count function on small (100 lines), medium (1000 lines) and large (10000 lines) text files.  \n\n## Profiling\n\nMemory usage is just as critical as time performance, so I profiled memory usage using the [mprof](https://pypi.python.org/pypi/memory_profiler) utility by [Fabian Pedregosa](https://github.com/fabianp) and [Philippe Gervais](https://github.com/pgervais). The profiler ran a command-line script in `cmd/readline.go` that allows you to select an iteration function as an argument. For comparison, I also created a Python script that implemented the same functionality. All iterators are counting characters from a 3.9GB, 900,002 line file filled with \"fizz buzz\" text.\n\n![Memory Profiling of Readlines Iteration for a 3.9G Text File](profile/profile.png)\n\nInterestingly, while the channel readlines implementation took almost as long as Python, it used the least amount of memory. Both the iterator and callback implementations used slightly more memory, probably due to the state tracking each method was required to perform. These methods both took approximately the same time to complete, significantly faster than the channel method.\n\n## Help Wanted!\n\nHave a method or mechanism for line-by-line reading of a file, submit it with a pull-request and add it to the list of benchmarks! In particular, I couldn't get the closure-style of read-by-line iterator work:\n\n```go\nfor gen, next, err := GeneratorReadlines(\"myfile.txt\"); next; line, next, err = gen() {\n    // do something with the line\n}\n```\n\nI was either not getting all the lines or I was getting a final line that was simply the empty string, making all my counts incorrect. If you're interested in this problem, take a look at the current implementation and tests. Submit an issue if you'd like to discuss it!\n\n## About\n\nLearning a new programming language often means that you want to explore everything as completely as possible. That's what this small repository is about for me, learning to write benchmarking code and to write quality iterators that are Go idiomatic. Of course, then the repository gets out of control with Repo images, etc. But hey - if you're not having fun, why are you programming?\n\n### Acknowledgements\n\nMost of the iterators were implemented based on Ewan Cheslack-Postava's [Iterators in Go](https://ewencp.org/blog/golang-iterators/) blob post. Table based testing inspired by Dave Chaney's [Writing table driven tests in Go](https://dave.cheney.net/2013/06/09/writing-table-driven-tests-in-go) blog post. Benchmarking was similarly inspired by [How to write benchmarks in Go](https://dave.cheney.net/2013/06/30/how-to-write-benchmarks-in-go). Check those posts out if you haven't already.\n\nThe banner image used in this README, [\u0026ldquo;lines \u0026 curves\u0026rdquo;](https://flic.kr/p/iaVByW) by [Josef Stuefer](https://www.flickr.com/photos/josefstuefer/) is used by a Creative Commons [BY-NC-ND](https://creativecommons.org/licenses/by-nc-nd/2.0/) license.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbbengfort%2Fiterfile","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbbengfort%2Fiterfile","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbbengfort%2Fiterfile/lists"}