{"id":17785075,"url":"https://github.com/eun/go-batch-iterator","last_synced_at":"2026-02-16T10:04:06.306Z","repository":{"id":64521746,"uuid":"576430195","full_name":"Eun/go-batch-iterator","owner":"Eun","description":"iterator to sequentially iterate over datasources utilizing a batch approach","archived":false,"fork":false,"pushed_at":"2025-02-18T14:17:36.000Z","size":20258,"stargazers_count":2,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-27T05:11:15.470Z","etag":null,"topics":["go","golang","library"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Eun.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["Eun"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":"Eun","issuehunt":"eun","otechie":null,"custom":null}},"created_at":"2022-12-09T20:58:48.000Z","updated_at":"2025-02-18T14:17:39.000Z","dependencies_parsed_at":"2025-02-06T16:36:48.208Z","dependency_job_id":null,"html_url":"https://github.com/Eun/go-batch-iterator","commit_stats":{"total_commits":6,"total_committers":2,"mean_commits":3.0,"dds":"0.16666666666666663","last_synced_commit":"b1a9459d0b78ccd233c3b7069f864a8feee65aee"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Eun%2Fgo-batch-iterator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Eun%2Fgo-batch-iterator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Eun%2Fgo-batch-iterator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Eun%2Fgo-batch-iterator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Eun","download_url":"https://codeload.github.com/Eun/go-batch-iterator/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243806122,"owners_count":20350773,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","library"],"created_at":"2024-10-27T08:23:16.578Z","updated_at":"2026-02-16T10:04:06.273Z","avatar_url":"https://github.com/Eun.png","language":"Go","funding_links":["https://github.com/sponsors/Eun","https://liberapay.com/Eun","https://issuehunt.io/r/eun"],"categories":[],"sub_categories":[],"readme":"# go-batch-iterator\n[![Actions Status](https://github.com/Eun/go-batch-iterator/workflows/push/badge.svg)](https://github.com/Eun/go-batch-iterator/actions)\n[![Coverage Status](https://coveralls.io/repos/github/Eun/go-batch-iterator/badge.svg?branch=master)](https://coveralls.io/github/Eun/go-batch-iterator?branch=master)\n[![PkgGoDev](https://img.shields.io/badge/pkg.go.dev-reference-blue)](https://pkg.go.dev/github.com/Eun/go-batch-iterator)\n[![GoDoc](https://godoc.org/github.com/Eun/go-batch-iterator?status.svg)](https://godoc.org/github.com/Eun/go-batch-iterator)\n[![go-report](https://goreportcard.com/badge/github.com/Eun/go-batch-iterator)](https://goreportcard.com/report/github.com/Eun/go-batch-iterator)\n[![go1.18](https://img.shields.io/badge/go-1.18-blue)](#)\n---\n*go-batch-iterator* provides an iterator to sequentially iterate over datasources utilizing a\nbatch approach.  \nIt utilizes generics to achieve an independence from underlying datastructures.\nTherefore, it is ready to serve rows from a database or objects from an api endpoint with pagination.\nAll the user has to do is provide the logic to fetch the next batch from their datasource.\n\n\u003e go get -u github.com/Eun/go-batch-iterator\n\n### Example Usage\n```go\npackage batchiterator_test\n\nimport (\n\t\"context\"\n\t\"encoding/json\"\n\t\"errors\"\n\t\"log\"\n\t\"net/http\"\n\t\"time\"\n\n\tbatchiterator \"github.com/Eun/go-batch-iterator\"\n\t\"golang.org/x/time/rate\"\n)\n\nfunc httpServer() *http.Server {\n\tmux := http.NewServeMux()\n\tserver := http.Server{\n\t\tAddr:    \"127.0.0.1:8080\",\n\t\tHandler: mux,\n\t}\n\tmux.HandleFunc(\"/users\", func(w http.ResponseWriter, r *http.Request) {\n\t\tswitch r.Header.Get(\"X-NextPageToken\") {\n\t\tcase \"\":\n\t\t\tw.Header().Set(\"X-NextPageToken\", \"A\")\n\t\t\tw.WriteHeader(http.StatusOK)\n\t\t\tjson.NewEncoder(w).Encode([]string{\"Alice\", \"Bob\"})\n\t\t\treturn\n\t\tcase \"A\":\n\t\t\tw.Header().Set(\"X-NextPageToken\", \"B\")\n\t\t\tw.WriteHeader(http.StatusOK)\n\t\t\tjson.NewEncoder(w).Encode([]string{\"Charlie\"})\n\t\t\treturn\n\t\tdefault:\n\t\t\tw.WriteHeader(http.StatusNoContent)\n\t\t\treturn\n\t\t}\n\t})\n\tgo func() {\n\t\tif err := server.ListenAndServe(); err != nil {\n\t\t\tif !errors.Is(err, http.ErrServerClosed) {\n\t\t\t\tlog.Panicln(err)\n\t\t\t}\n\t\t}\n\t}()\n\treturn \u0026server\n}\n\nfunc getNextUsersFromAPI(nextPageToken *string) func(ctx context.Context) (items []string, hasMoreItems bool, err error) {\n\treturn func(ctx context.Context) (items []string, hasMoreItems bool, err error) {\n\t\treq, err := http.NewRequestWithContext(ctx, http.MethodGet, \"http://127.0.0.1:8080/users\", http.NoBody)\n\t\tif err != nil {\n\t\t\treturn nil, false, err\n\t\t}\n\t\treq.Header.Set(\"X-NextPageToken\", *nextPageToken)\n\t\tresp, err := http.DefaultClient.Do(req)\n\t\tif err != nil {\n\t\t\treturn nil, false, err\n\t\t}\n\t\tdefer resp.Body.Close()\n\n\t\tswitch resp.StatusCode {\n\t\tcase http.StatusTooManyRequests:\n\t\t\t// too many requests retry later\n\t\t\treturn nil, true, err\n\t\tcase http.StatusNoContent:\n\t\t\t// no more items\n\t\t\treturn nil, false, nil\n\t\tcase http.StatusOK:\n\t\t\tvar users []string\n\t\t\t*nextPageToken = resp.Header.Get(\"X-NextPageToken\")\n\t\t\terr := json.NewDecoder(resp.Body).Decode(\u0026users)\n\t\t\treturn users, true, err\n\t\tdefault:\n\t\t\treturn nil, false, errors.New(\"unknown status code\")\n\t\t}\n\t}\n}\n\nfunc ExampleIterator_pagination() {\n\tserver := httpServer()\n\tdefer server.Shutdown(context.Background())\n\n\tvar nextPageToken string\n\titer := batchiterator.Iterator[string]{\n\t\tNextBatchFunc: batchiterator.RateLimit(\n\t\t\trate.NewLimiter(rate.Every(time.Second), 1),\n\t\t\tgetNextUsersFromAPI(\u0026nextPageToken),\n\t\t),\n\t}\n\tfor iter.Next(context.Background()) {\n\t\tlog.Print(*iter.Value())\n\t}\n\tif err := iter.Error(); err != nil {\n\t\tlog.Print(err)\n\t\treturn\n\t}\n}\n```\n[example_pagination_test.go](example_pagination_test.go)\n\n### Helpers\n[**StaticSlice**](option.go:#L11)  \nSometimes a developer wants to use the iterator but the items are already fetched.\n```go\niter := batchiterator.Iterator[string]{\n    NextBatchFunc: batchiterator.StaticSlice([]string{\"A\", \"B\", \"C\"}),\n}\n```\n\n\n[**RateLimit**](option.go:#L18)  \n`RateLimit` takes a `rate.Limiter` and wraps the `NextBatchFunc` with this limit.\n```go\niter := batchiterator.Iterator[string]{\n    NextBatchFunc: batchiterator.RateLimit(\n\t\trate.NewLimiter(rate.Every(time.Second), 1), \n\t\tfunc (ctx context.Context) (items []string, hasMoreItems bool, err error) {\n\t\t\tpanic(\"not implemented\")\n\t\t}, \n\t),\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feun%2Fgo-batch-iterator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feun%2Fgo-batch-iterator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feun%2Fgo-batch-iterator/lists"}