{"id":21405781,"url":"https://github.com/ovhemert/batch2","last_synced_at":"2025-07-30T12:05:55.607Z","repository":{"id":34253383,"uuid":"173186531","full_name":"ovhemert/batch2","owner":"ovhemert","description":"A transform stream that collects chunks and passes them on as batches.","archived":false,"fork":false,"pushed_at":"2023-01-07T00:29:04.000Z","size":400,"stargazers_count":3,"open_issues_count":2,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-07-02T03:48:43.449Z","etag":null,"topics":["batch","bulk","chunk","stream","streams2","through","through2","transform"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ovhemert.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":".github/SECURITY.md","support":null},"funding":{"github":["ovhemert"],"patreon":"ovhemert","ko_fi":"ovhemert","custom":"ovhemert.dev/donate"}},"created_at":"2019-02-28T20:59:16.000Z","updated_at":"2022-04-11T18:47:49.000Z","dependencies_parsed_at":"2023-01-15T05:41:31.130Z","dependency_job_id":null,"html_url":"https://github.com/ovhemert/batch2","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/ovhemert/batch2","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovhemert%2Fbatch2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovhemert%2Fbatch2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovhemert%2Fbatch2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovhemert%2Fbatch2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ovhemert","download_url":"https://codeload.github.com/ovhemert/batch2/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovhemert%2Fbatch2/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265220096,"owners_count":23729761,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch","bulk","chunk","stream","streams2","through","through2","transform"],"created_at":"2024-11-22T16:28:19.497Z","updated_at":"2025-07-13T23:30:45.130Z","avatar_url":"https://github.com/ovhemert.png","language":"JavaScript","funding_links":["https://github.com/sponsors/ovhemert","https://patreon.com/ovhemert","https://ko-fi.com/ovhemert","ovhemert.dev/donate"],"categories":[],"sub_categories":[],"readme":"# batch2\n\n[![CI](https://github.com/ovhemert/batch2/workflows/CI/badge.svg)](https://github.com/ovhemert/batch2/actions)\n[![Codacy Badge](https://api.codacy.com/project/badge/Grade/706df16ae6124bb782e7e4a78a0bcfc3)](https://www.codacy.com/app/ovhemert/batch2?utm_source=github.com\u0026amp;utm_medium=referral\u0026amp;utm_content=ovhemert/batch2\u0026amp;utm_campaign=Badge_Grade)\n[![Known Vulnerabilities](https://snyk.io/test/npm/batch2/badge.svg)](https://snyk.io/test/npm/batch2)\n[![Coverage Status](https://coveralls.io/repos/github/ovhemert/batch2/badge.svg?branch=master)](https://coveralls.io/github/ovhemert/batch2?branch=master)\n[![js-standard-style](https://img.shields.io/badge/code%20style-standard-brightgreen.svg?style=flat)](http://standardjs.com/)\n\nA transform stream that collects chunks and passes them on as batches.\n\n## Example\n\nLet's say you want to stream documents from a MongoDB collection to ElasticSearch. For every document in the collection, the write stream will emit an event that you use to call ElasticSearch index function. That means that 100.000 documents will result in 100.000 API calls.\n\nThe ElasticSearch library has a function to [bulk](https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/api-reference.html#api-bulk) index, but since the stream emits a write for each document, we cannot group multiple index operations together.\n\nThe batch2 transform stream can help by buffering the chunks/docs and passing them on as batches. For example, we can now create batches of 500 docs each and reduce the number of API calls to ElasticSearch from 100.000 to 200, which will improve speed.\n\n```js\n  mongoReadStream()\n  .pipe(batch2({ size: 5 }) // transforms multiple chunks (mongo docs) to [chunk, chunk, chunk, chunk, chunk]\n  .pipe(transformToElasticBulkOperation())\n  .pipe(elasticWriteStream())\n```\n\n## Installation\n\n```bash\n$ npm install batch2\n```\n\n## API\n\n\u003cb\u003e\u003ccode\u003ebatch2(\\[options\\], \\[transformFunction\\])\u003c/code\u003e\u003c/b\u003e\n\nConsult the **[stream.Transform](http://nodejs.org/docs/latest/api/stream.html#stream_class_stream_transform)** documentation for the exact rules of the `transformFunction` (i.e. `this._transform`) and the optional `flushFunction` (i.e. `this._flush`).\n\n### options\n\nThe options argument is optional and is passed straight through to `stream.Transform`. So you can use `objectMode:true` if you are processing non-binary streams (or just use `batch2.obj()`).\n\n### transformFunction\n\nThe `transformFunction` must have the following signature: `function (chunk, encoding, callback) {}`. A minimal implementation should call the `callback` function to indicate that the transformation is done, even if that transformation means discarding the chunk.\n\nTo queue a new chunk, call `this.push(chunk)`\u0026mdash;this can be called as many times as required before the `callback()` if you have multiple pieces to send on.\n\nAlternatively, you may use `callback(err, chunk)` as shorthand for emitting a single chunk or an error.\n\nIf you **do not provide a `transformFunction`** then you will get a simple pass-through stream.\n\n## Maintainers\n\nOsmond van Hemert\n[![Github](https://img.shields.io/badge/-website.svg?style=social\u0026logoColor=333\u0026logo=github)](https://github.com/ovhemert)\n[![Web](https://img.shields.io/badge/-website.svg?style=social\u0026logoColor=333\u0026logo=nextdoor)](https://ovhemert.dev)\n\n## Contributing\n\nIf you would like to help out with some code, check the [details](./.github/CONTRIBUTING.md).\n\nNot a coder, but still want to support? Have a look at the options available to [donate](https://ovhemert.dev/donate).\n\n## License\n\nLicensed under [MIT](./LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fovhemert%2Fbatch2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fovhemert%2Fbatch2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fovhemert%2Fbatch2/lists"}