{"id":18635395,"url":"https://github.com/naughtygopher/nibbler","last_synced_at":"2025-05-07T20:38:10.611Z","repository":{"id":261034946,"uuid":"882330202","full_name":"naughtygopher/nibbler","owner":"naughtygopher","description":"A package for micro batch processing","archived":false,"fork":false,"pushed_at":"2024-11-11T17:01:08.000Z","size":33,"stargazers_count":10,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-07T20:38:02.925Z","etag":null,"topics":["batch","batch-processing","event-processing","go","go-library","go-package","golang","micro-batches","microbatch","stream","stream-processing"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/naughtygopher.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-02T14:38:12.000Z","updated_at":"2025-03-12T03:01:44.000Z","dependencies_parsed_at":"2024-11-04T11:18:27.683Z","dependency_job_id":"c0529a72-38a1-40ca-92cc-dd7cabcdee97","html_url":"https://github.com/naughtygopher/nibbler","commit_stats":null,"previous_names":["naughtygopher/nibbler"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/naughtygopher%2Fnibbler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/naughtygopher%2Fnibbler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/naughtygopher%2Fnibbler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/naughtygopher%2Fnibbler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/naughtygopher","download_url":"https://codeload.github.com/naughtygopher/nibbler/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252954125,"owners_count":21830892,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch","batch-processing","event-processing","go","go-library","go-package","golang","micro-batches","microbatch","stream","stream-processing"],"created_at":"2024-11-07T05:24:49.336Z","updated_at":"2025-05-07T20:38:10.589Z","avatar_url":"https://github.com/naughtygopher.png","language":"Go","funding_links":[],"categories":["Stream Processing","流处理"],"sub_categories":["HTTP Clients","HTTP客户端"],"readme":"\u003cp align=\"center\"\u003e\u003cimg src=\"https://github.com/user-attachments/assets/1b34c21a-8031-43d3-a172-44e039b58190\" alt=\"nibbler gopher\" width=\"256px\"/\u003e\u003c/p\u003e\n\n[![](https://github.com/naughtygopher/nibbler/actions/workflows/go.yml/badge.svg?branch=main)](https://github.com/naughtygopher/nibbler/actions)\n[![Go Reference](https://pkg.go.dev/badge/github.com/naughtygopher/nibbler.svg)](https://pkg.go.dev/github.com/naughtygopher/nibbler)\n[![Go Report Card](https://goreportcard.com/badge/github.com/naughtygopher/nibbler?cache_invalidate=1)](https://goreportcard.com/report/github.com/naughtygopher/nibbler)\n[![Coverage Status](https://coveralls.io/repos/github/naughtygopher/nibbler/badge.svg?branch=main\u0026cache_invalidate=1)](https://coveralls.io/github/naughtygopher/nibbler?branch=main)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/creativecreature/sturdyc/blob/master/LICENSE)\n\n# Nibbler\n\nNibbler is a resilient, minimal, package which helps you implement micro-batch processing, within an application. Nibbler remains minimal with its 0 external dependencies and remains resilient within the context of the application by gracefully handling errors and panics.\n\nIMPORTANT: This is not a general purpose distributed task queue.\n\n## What is Micro-batch Processing?\n\nMicro-batch processing is a way to handle data by breaking a big task into smaller pieces and processing them one by one. This method is useful in real-time data or streaming situations, wher ,the incoming data is split into \"micro-batches\" and processed quickly, rather than waiting to collect all data at once.\n\nThe same concept can also be extended to handle events processing. So, we have a queue subscriber, and instead of processing the events individually, we create micro batches and process them.\n\nThe processing of a single micro batch can be triggered in two ways, based on a time ticker or if the micro batch size is full. i.e. process a non empty batch if duration X has passed or if the batch size is full\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/user-attachments/assets/0a7df1c0-2d23-475e-9cc3-205f3f9bf4c4\" alt=\"nibbler\" width=\"384px\"/\u003e\n\u003c/p\u003e\n\n## Why use nibbler?\n\nIn any high throughput event/stream processing, it is imperative to process them in batches instead of individually. Processing events in batches when done properly optimizes usage of the downstream dependencies like databases, external systems (if they support) etc by significantly reducing [IOPS](https://en.wikipedia.org/wiki/IOPS). When deciding on how to process batches, it is important to still be able to process them realtime or near realtime. So, if we wait for a batch to be \"full\", and for any reason if the batch is not full fast enough, then processing would be indefinitely delayed. Hence the batches have to be flushed periodically, based on an acceptable tradeoff. The tradeoff in this case is, when the batch is not filled very fast, then we lose near realtime processing, rather would only be processed every N seconds/minute/duration.\n\n### Config\n\n```golang\ntype BatchProcessor[T any] func(ctx context.Context, trigger trigger, batch []T) error\n\ntype Config[T any] struct {\n    // ProcessingTimeout is context timeout for processing a single batch\n    ProcessingTimeout time.Duration\n    // TickerDuration is the ticker duration, for when a non empty batch would be processed\n    TickerDuration    time.Duration\n    // Size is the micro batch size\n    Size uint\n\n    // Processor is the function which processes a single batch\n    Processor BatchProcessor[T]\n\n    // ResumeAfterErr if true will continue listening and keep processing if the processor returns\n    // an error, or if processor panics. In both cases, ProcessorErr would be executed\n    ResumeAfterErr bool\n    // ProcessorErr is executed if the processor returns erorr or panics\n    ProcessorErr   func(failedBatch []T, err error)\n}\n```\n\n## How to use nibbler?\n\nBelow is an example showing how batching is used for a \"banking\" app which bulk processes account statements.\n\n```golang\npackage main\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"sync\"\n\t\"time\"\n\n\t\"github.com/naughtygopher/nibbler\"\n)\n\ntype db struct {\n\tdata         sync.Map\n\ttotalBalance int\n}\n\nfunc (d *db) BulkAddAccountsAndBalance(pp []AccStatement) error {\n\t// assume we are doing a bulk insert/update into the database instead of inserting one by one.\n\t// Bulk operations reduce the number of I/O required between your application and the database.\n\t// Thereby making it better in most cases.\n\tfor _, p := range pp {\n\t\td.data.Store(p.AccountID, p.Balance)\n\t\td.totalBalance += p.Balance\n\t}\n\treturn nil\n}\n\ntype Bank struct {\n\tdb *db\n}\n\nfunc (bnk *Bank) ProcessAccountsBatch(\n\tctx context.Context,\n\ttrigger nibbler.Trigger,\n\tbatch []AccStatement,\n) error {\n\terr := bnk.db.BulkAddAccountsAndBalance(batch)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\treturn nil\n}\n\nfunc (bnk *Bank) TotalBalance() int {\n\treturn bnk.db.totalBalance\n}\n\nfunc (bnk *Bank) TotalAccounts() int {\n\tcounter := 0\n\tbnk.db.data.Range(func(key, value any) bool {\n\t\tcounter++\n\t\treturn true\n\t})\n\treturn counter\n}\n\ntype AccStatement struct {\n\tAccountID string\n\tBalance   int\n}\n\nfunc main() {\n\tbnk := Bank{\n\t\tdb: \u0026db{\n\t\t\tdata: sync.Map{},\n\t\t},\n\t}\n\n\tnib, err := nibbler.Start(\u0026nibbler.Config[AccStatement]{\n\t\tSize:           10,\n\t\tTickerDuration: time.Second,\n\t\tProcessor:      bnk.ProcessAccountsBatch,\n\t})\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\n\treceiver := nib.Receiver()\n\tfor i := range 100 {\n\t\taccID := fmt.Sprintf(\"account_id_%d\", i)\n\t\treceiver \u003c- AccStatement{\n\t\t\tAccountID: accID,\n\t\t\tBalance:   50000 / (i + 1),\n\t\t}\n\t}\n\n\t// wait for batches to be processed. Ideally this wouldn't be required as our application\n\t// would not exit, instead just keep listening to the events stream.\n\ttime.Sleep(time.Second)\n\n\tfmt.Printf(\n\t\t\"Number of accounts %d, total balance: %d\\n\",\n\t\tbnk.TotalAccounts(),\n\t\tbnk.TotalBalance(),\n\t)\n}\n```\n\nYou can find all usage details in the tests.\n\n## The gopher\n\nThe gopher used here was created using [Gopherize.me](https://gopherize.me/). Nibbler is out there eating your events/streams\none bite at a time.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnaughtygopher%2Fnibbler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnaughtygopher%2Fnibbler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnaughtygopher%2Fnibbler/lists"}