{"id":13714007,"url":"https://github.com/veggiemonk/batch","last_synced_at":"2025-03-16T09:33:38.827Z","repository":{"id":61629182,"uuid":"552107850","full_name":"veggiemonk/batch","owner":"veggiemonk","description":"Split an array/slice into n evenly chunks. Spread load evenly across workers","archived":false,"fork":false,"pushed_at":"2024-02-11T22:50:22.000Z","size":19,"stargazers_count":5,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-02-27T07:12:16.867Z","etag":null,"topics":["batch","cloud-run","cloud-run-jobs","go","go-generics","golang","jobs"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/veggiemonk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-15T20:46:29.000Z","updated_at":"2023-06-08T05:51:23.000Z","dependencies_parsed_at":"2024-02-11T23:32:01.344Z","dependency_job_id":"bf72991c-73ff-4c51-a346-98636d3e4c98","html_url":"https://github.com/veggiemonk/batch","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/veggiemonk%2Fbatch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/veggiemonk%2Fbatch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/veggiemonk%2Fbatch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/veggiemonk%2Fbatch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/veggiemonk","download_url":"https://codeload.github.com/veggiemonk/batch/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243809875,"owners_count":20351407,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch","cloud-run","cloud-run-jobs","go","go-generics","golang","jobs"],"created_at":"2024-08-02T23:01:49.937Z","updated_at":"2025-03-16T09:33:38.359Z","avatar_url":"https://github.com/veggiemonk.png","language":"Go","funding_links":[],"categories":["Repositories"],"sub_categories":[],"readme":"# Batch\n\n[![Go Reference](https://pkg.go.dev/badge/github.com/veggiemonk/batch.svg)](https://pkg.go.dev/github.com/veggiemonk/batch)\n\nSplit an array/slice into `n` evenly chunks.\n\nInspired from the blog post by [Paul Di Gian](https://github.com/PaulDiGian) on his blog:\n[Split a slice or array in a defined number of chunks in golang](https://pauldigian.com/split-a-slice-or-array-in-a-defined-number-of-chunks-in-golang-but-any-language-really)\n\n**Note**: you might better off just copying the function into your codebase.\nIt has little code.\n\nSee [Go Proverbs](https://go-proverbs.github.io/) for more details.\n\n\u003e A little copying is better than a little dependency.\n\nThis library isn't really meant to be imported.\nJust copy the one function and adapt it to your needs.\nLook at the [tests](batch_test.go) for edge cases.\nThe benchmarks and fuzzing are just for me to learn and have a playground\nto try things out.\n\n\u003c!-- TOC --\u003e\n\n-   [Batch](#batch)\n    -   [Installation](#installation)\n    -   [Usage](#usage)\n    -   [Usage with Cloud Run Jobs](#usage-with-cloud-run-jobs)\n    -   [Rationale](#rationale)\n    -   [Links](#links)\n\n## Installation\n\nRequires Go 1.18 or later.\n\nJust copy the function in [batch.go](batch.go)\n\n## Usage\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\n\t\"github.com/veggiemonk/batch\"\n)\n\nfunc main() {\n    s :=  []int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}\n\n    // Split the slice into 3 even parts\n    chunks := batch.Slice(s, 3)\n\n    fmt.Println(chunks)\n    // length      3       3        4\n    // output: [[1 2 3] [4 5 6] [7 8 9 10]]\n    // the size of each batch has variation of max 1 item\n    // this can spread the load evenly amongst workers\n}\n```\n\n## Usage with Cloud Run Jobs\n\n```go\nbatchID = uuid.New().String()\ntaskCount, _ = strconv.Atoi(os.Getenv(\"CLOUD_RUN_TASK_COUNT\"))\ntaskIndex, _ = strconv.Atoi(os.Getenv(\"CLOUD_RUN_TASK_INDEX\"))\n\ntt, _ := requestToTasks(request)\n\nbatches := batch.Slice(tt, taskCount)\nif taskIndex \u003e= len(batches) || taskIndex \u003c 0 {\n\treturn fmt.Errorf(\"index (%d) out of bounds (max: %d), (id:%s): %w\", taskIndex, len(batches), batchID, ErrTaskIndexOutOfBounds)\n}\n\nb := batches[taskIndex]\nif err := process(b); err != nil {\n    return fmt.Errorf(\"failed to process batch (id:%s): %w\", batchID, err)\n}\n```\n\n## Rationale\n\nHaving (almost) same sized batch is useful when you want to distribute the workload evenly across multiple workers.\n\nAs opposed to defining the _size of each batch_, we define the _number of batch we want_ to have.\n\nHere a **counter** example:\n\n```go\nactions := []int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}\nbatchSize := 3\nbatches := make([][]int, 0, (len(actions) + batchSize - 1) / batchSize)\n\nfor batchSize \u003c len(actions) {\n    actions, batches = actions[batchSize:], append(batches, actions[0:batchSize:batchSize])\n}\nbatches = append(batches, actions)\nfmt.Println(result)\n// length       4    |    4    |  2\n// output: [[1 2 3 4] [5 6 7 8] [9 10]]\n// 2 workers will do double the work of the last worker.\n// --\u003e Not what we want.\n}\n```\n\nThis is not ideal when you want to distribute the workload evenly across multiple workers.\n\nThe code was taken from [Go wiki - Slice Tricks](https://go.dev/wiki/SliceTricks#batching-with-minimal-allocation).\n\n[//]: # \"can be played with here: https://go.dev/play/p/-ULiql4tOTc\"\n\n## Links\n\n-   [Split a slice or array in a defined number of chunks in golang](https://pauldigian.com/split-a-slice-or-array-in-a-defined-number-of-chunks-in-golang-but-any-language-really)\n-   [Go wiki - Slice Tricks](https://go.dev/wiki/SliceTricks#batching-with-minimal-allocation)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fveggiemonk%2Fbatch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fveggiemonk%2Fbatch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fveggiemonk%2Fbatch/lists"}