{"id":18712439,"url":"https://github.com/serkodev/aggregator","last_synced_at":"2025-08-09T00:15:11.434Z","repository":{"id":116427783,"uuid":"459980454","full_name":"serkodev/aggregator","owner":"serkodev","description":"Batch processing library for Go supports generics \u0026 values returning","archived":false,"fork":false,"pushed_at":"2022-03-16T07:13:11.000Z","size":39,"stargazers_count":9,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-08-08T21:10:51.341Z","etag":null,"topics":["aggregator","batch","concurrency","generics","go","golang","performance","worker"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/serkodev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-02-16T11:45:09.000Z","updated_at":"2025-05-28T04:26:45.000Z","dependencies_parsed_at":null,"dependency_job_id":"2c0eb4bd-46de-4794-b4fa-a055d48f7f05","html_url":"https://github.com/serkodev/aggregator","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/serkodev/aggregator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serkodev%2Faggregator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serkodev%2Faggregator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serkodev%2Faggregator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serkodev%2Faggregator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/serkodev","download_url":"https://codeload.github.com/serkodev/aggregator/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serkodev%2Faggregator/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269509123,"owners_count":24428833,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-08T02:00:09.200Z","response_time":72,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aggregator","batch","concurrency","generics","go","golang","performance","worker"],"created_at":"2024-11-07T12:42:49.415Z","updated_at":"2025-08-09T00:15:11.363Z","avatar_url":"https://github.com/serkodev.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Aggregator\n\n[![Go Reference](https://pkg.go.dev/badge/github.com/serkodev/aggregator.svg)](https://pkg.go.dev/github.com/serkodev/aggregator)\n\nAggregator is a batch processing library for Go supports returning values. You can group up and process batch of tasks with keys in a single callback. Using it for grouping up database query or cache can help you to reduce loading of database and network.\n\n### THIS PROJECT IS IN BETA\n\nThis project may contain bugs and have not being tested at all. Use under your own risk, but feel free to test, make pull request and improve this project.\n\n## Features\n\n- Support multi Aggregators (using `AggregatorList`) for fallback.\n- Support multi workers to flush tasks.\n- Support Go generics for query keys and result values.\n- Support timeout-only or tasks limit-only.\n- Support singleflight (using [singleflight-any](https://github.com/serkodev/singleflight-any)).\n\n## Install\n\nCurrently Go 1.18+ is required (for go generics), backward compatible is planned.\n\n```bash\ngo get github.com/serkodev/aggregator@latest\n```\n\n## Example\n\n```go\ncallback := func(keys []string) (map[string]Book, error) {\n    results := db.Query(`SELECT * FROM books WHERE name IN ?`, keys)\n    return results, nil\n}\nagg, _ := aggregator.New(callback, 100*time.Millisecond, 5).Run()\n\nfor _, name := range []string{\"foo\", \"foo\", \"bar\", \"baz\", \"baz\"} {\n    go func(n string) {\n        book, err := agg.Query(n).Get()\n        if err == nil {\n            print(book.Name + \":\" + book.Price, \" \")\n        }\n    }(name)\n}\n\n// foo:10 foo:10 bar:25 baz:30 baz:30\n```\n\n## How it works\n\n```mermaid\nflowchart LR;\n    subgraph A [Aggregator]\n        direction TB\n        subgraph cb [\"Customize Process (example)\"]\n        direction TB\n            input(\"Input\n            []string{#quot;foo#quot;, #quot;bar#quot;, #quot;baz#quot;}\")\n            db[(\"Database\n\n            SELECT price FROM books\u003cbr /\u003eWHERE name\n            IN ('foo', 'bar', 'baz')\")]\n            output(\"return map[string]int{\n                \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;#quot;foo#quot;: 10,\n                \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;#quot;bar#quot;: 25,\n                \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;#quot;baz#quot;: 30,\n            }\")\n            input --\u003e db --\u003e output\n            style output text-align:left\n        end\n\n        Wait -- Reach tasks limit / Timeout --\u003e\n        cb --\u003e rt(\"Return value to each Request\")\n    end\n\n    req1[Request 1] --\u003e q_foo_(\"Query(#quot;foo#quot;)\"):::bgFoo --\u003e A\n    req2[Request 2] --\u003e q_foo2(\"Query(#quot;foo#quot;)\"):::bgFoo --\u003e A\n    req3[Request 3] --\u003e q_bar_(\"Query(#quot;bar#quot;)\"):::bgBar --\u003e A\n    req4[Request 4] --\u003e q_baz_(\"Query(#quot;baz#quot;)\"):::bgBaz --\u003e A\n    req5[Request 5] --\u003e q_baz2(\"Query(#quot;baz#quot;)\"):::bgBaz --\u003e A\n\n    A --- rtn1(\"return 10\"):::bgFoo --\u003e req1_[Request 1]\n    A --- rtn2(\"return 10\"):::bgFoo --\u003e req2_[Request 2]\n    A --- rtn3(\"return 25\"):::bgBar --\u003e req3_[Request 3]\n    A --- rtn4(\"return 30\"):::bgBaz --\u003e req4_[Request 4]\n    A --- rtn5(\"return 30\"):::bgBaz --\u003e req5_[Request 5]\n\n    classDef bgFoo fill:green;\n    classDef bgBar fill:blue;\n    classDef bgBaz fill:purple;\n```\n\n## Advance\n\n### AggregatorList\n\n`AggregatorList` contains a slice of `Aggregator`, you can create it by `aggregator.NewList(...)`. If the prior order aggregator cannot return data for any keys. Then `AggregatorList` will query data from the next aggregator for fallback.\n\nFor example, you create an `AggregatorList` with cache and database aggregator, when the data has not been cached, it will auto query from database.\n\n```go\ncacheAgg := aggregator.New(func(k []string) (map[string]string, error) {\n    fmt.Println(\"fetch from cache...\", k)\n    return map[string]string{\n        \"key1\": \"val1\",\n        \"key2\": \"val2\",\n    }, nil\n}, 50*time.Millisecond, 10)\n\ndatabaseAgg := aggregator.New(func(k []string) (map[string]string, error) {\n    fmt.Println(\"fetch from database...\", k)\n    return map[string]string{\n        \"key1\": \"val1\",\n        \"key2\": \"val2\",\n        \"key3\": \"val3\",\n        \"key4\": \"val4\",\n    }, nil\n}, 50*time.Millisecond, 10)\n\nlist := aggregator.NewList(cacheAgg, databaseAgg).Run()\nresults := list.QueryMulti([]string{\"key1\", \"key2\", \"key3\", \"key4\"})\n\n// fetch from cache... [\"key1\", \"key2\", \"key3\", \"key4\"]\n// fetch from database... [\"key3\", \"key4\"]\n// results: [\"val1\", \"val2\", \"val3\", \"val4\"]\n```\n\n### singleflight\n\nIn some use case you may need to prevent cache breakdown. Aggregator works with singleflight by using [singleflight-any](https://github.com/serkodev/singleflight-any) (supports Go generics).\n\n## Inspiration\n\n- [API Performance Tunning Story when Goalng meet with GraphQL](https://hackmd.io/zvmgdunRR8mjAjVIMx0eDA?both) - Kane Wang\n- [gobatch](https://github.com/herryg91/gobatch)\n\n## LICENSE\n\nMIT License\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fserkodev%2Faggregator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fserkodev%2Faggregator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fserkodev%2Faggregator/lists"}