{"id":20578331,"url":"https://github.com/mikestefanello/batcher","last_synced_at":"2026-02-15T22:31:47.257Z","repository":{"id":176681856,"uuid":"618989724","full_name":"mikestefanello/batcher","owner":"mikestefanello","description":"Type-safe, automatic, asynchronous batch processing.","archived":false,"fork":false,"pushed_at":"2024-06-17T21:30:05.000Z","size":14,"stargazers_count":18,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-09-26T01:35:58.105Z","etag":null,"topics":["batch","batch-processing","concurrency","data","goroutines"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mikestefanello.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-03-25T23:36:59.000Z","updated_at":"2025-03-15T23:21:46.000Z","dependencies_parsed_at":null,"dependency_job_id":"a2a79f27-1727-4622-979c-fcee4e18bcb3","html_url":"https://github.com/mikestefanello/batcher","commit_stats":null,"previous_names":["mikestefanello/batcher"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/mikestefanello/batcher","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikestefanello%2Fbatcher","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikestefanello%2Fbatcher/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikestefanello%2Fbatcher/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikestefanello%2Fbatcher/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mikestefanello","download_url":"https://codeload.github.com/mikestefanello/batcher/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikestefanello%2Fbatcher/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29490870,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-15T19:29:10.908Z","status":"ssl_error","status_checked_at":"2026-02-15T19:29:10.419Z","response_time":118,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch","batch-processing","concurrency","data","goroutines"],"created_at":"2024-11-16T06:12:26.947Z","updated_at":"2026-02-15T22:31:47.241Z","avatar_url":"https://github.com/mikestefanello.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Batcher\n\n[![Go Report Card](https://goreportcard.com/badge/github.com/mikestefanello/batcher)](https://goreportcard.com/report/github.com/mikestefanello/batcher)\n[![Test](https://github.com/mikestefanello/batcher/actions/workflows/test.yml/badge.svg)](https://github.com/mikestefanello/batcher/actions/workflows/test.yml)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Go Reference](https://pkg.go.dev/badge/github.com/mikestefanello/batcher.svg)](https://pkg.go.dev/github.com/mikestefanello/batcher)\n[![GoT](https://img.shields.io/badge/Made%20with-Go-1f425f.svg)](https://go.dev)\n\n## Overview\n\n_Batcher_ is a Go library that provides a type-safe, easy way to batch together arbitrary groups of items to be automatically and asynchronously processed. An item can be of any type that you want to pass to your processor. Items are queued within groups (using a _string_ as a name). If you do not have a need to separately group items, you can specify the same, or empty, group name for all items. Each group of items is separately sent to the processor callback that you specify.\n\nThe queue can be configured to automatically execute the batch operation under any of the following circumstances:\n1) A specified duration has elapsed since the last process executed\n2) The amount of total items queued (across all groups) has exceeded a given threshold\n3) The amount of _groups_ of items has exceeded a given threshold\n\n## Installation\n\n`go get github.com/mikestefanello/batcher`\n\n## Usage\n\nAs an example, we'll create a batcher to process items of type `Log` which will be grouped together via their `Level` then written in bulk.\n\n```go\nb, err := batcher.NewBatcher[Log](Config[Log]{\n    // Process the batch if we've queued 5 different Level values\n    GroupCountThreshold: 5,\n    // Process the batch if we've queued 100 Log items\n    ItemCountThreshold:  100,\n    // Process the queue every 30 seconds\n    DelayThreshold:      30 * time.Second,\n    // Use 3 Goroutines to process the queue groups\n    NumGoroutines:       3,\n    // Execute this func for each queued group\n    Processor:           func(group string, log []Log) {\n        writeLogs(logs)\n    },\n})\n```\n\nAdd a `Log` to the batch queue.\n\n```go\nb.Add(log.Level, log)\n```\n\n## Origin\n\nThis concept was originally devised to handle some challenges faced with PubSub messages but was made entirely generic for this library so it could be used for any purpose.\n1) Group incoming messages by a unique identifier, ie, a user ID, to deduplicate requests and ensure a given pod was only ever executing operations for a given ID in isolation.\n2) Group all incoming messages to bulk-export to persistent storage for archiving.\n\nTo _roughly_ illustrate an example in code:\n\n```go\nfunc main() {\n    b, err := batcher.NewBatcher[*pubsub.Message](Config[*pubsub.Message]{\n        GroupCountThreshold: 10,\n        ItemCountThreshold:  100,\n        DelayThreshold:      10 * time.Second,\n        NumGoroutines:       3,\n        Processor:           func(group string, items []*pubsub.Message) {\n            err := doUserOperation(group)\n\n            for _, m := range items {\n                if err != nil {\n                    m.Nack()\n                } else {\n                    m.Ack()\n                }\n            }\n        },\n    })\n\n\n    // Consume the messages from PubSub\n    err = subscription.Receive(ctx, func(ctx context.Context, message *pubsub.Message) {\n        // For this example, Data is the user ID\n        b.Add(string(message.Data), message)\n    })\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmikestefanello%2Fbatcher","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmikestefanello%2Fbatcher","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmikestefanello%2Fbatcher/lists"}