{"id":21405775,"url":"https://github.com/ovhemert/elasticsearch-batch-stream","last_synced_at":"2025-07-13T23:30:46.801Z","repository":{"id":34307559,"uuid":"177102410","full_name":"ovhemert/elasticsearch-batch-stream","owner":"ovhemert","description":"A write stream that creates batches of elasticsearch bulk operations","archived":false,"fork":false,"pushed_at":"2023-03-02T21:14:04.000Z","size":399,"stargazers_count":5,"open_issues_count":3,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-07-09T17:13:20.444Z","etag":null,"topics":["batch","bulk","chunk","elastic","elasticsearch","stream"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ovhemert.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"docs/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["ovhemert"],"patreon":"ovhemert","ko_fi":"ovhemert","custom":"ovhemert.dev/donate"}},"created_at":"2019-03-22T08:35:59.000Z","updated_at":"2022-02-21T19:40:36.000Z","dependencies_parsed_at":"2024-11-22T16:40:27.090Z","dependency_job_id":null,"html_url":"https://github.com/ovhemert/elasticsearch-batch-stream","commit_stats":{"total_commits":56,"total_committers":6,"mean_commits":9.333333333333334,"dds":0.3214285714285714,"last_synced_commit":"2a05a24436ef3067a675a23f3d7ac6804fb4bf01"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/ovhemert/elasticsearch-batch-stream","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovhemert%2Felasticsearch-batch-stream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovhemert%2Felasticsearch-batch-stream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovhemert%2Felasticsearch-batch-stream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovhemert%2Felasticsearch-batch-stream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ovhemert","download_url":"https://codeload.github.com/ovhemert/elasticsearch-batch-stream/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovhemert%2Felasticsearch-batch-stream/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265220097,"owners_count":23729761,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch","bulk","chunk","elastic","elasticsearch","stream"],"created_at":"2024-11-22T16:28:17.629Z","updated_at":"2025-07-13T23:30:46.472Z","avatar_url":"https://github.com/ovhemert.png","language":"JavaScript","funding_links":["https://github.com/sponsors/ovhemert","https://patreon.com/ovhemert","https://ko-fi.com/ovhemert","ovhemert.dev/donate"],"categories":[],"sub_categories":[],"readme":"# elasticsearch-batch-stream\n\n[![Travis](https://img.shields.io/travis/com/ovhemert/elasticsearch-batch-stream.svg?branch=master\u0026logo=travis)](https://travis-ci.com/ovhemert/elasticsearch-batch-stream)\n[![Codacy Badge](https://api.codacy.com/project/badge/Grade/2b7f2dae5ec947d8a46362314bd90e53)](https://www.codacy.com/app/ovhemert/elasticsearch-batch-stream?utm_source=github.com\u0026amp;utm_medium=referral\u0026amp;utm_content=ovhemert/elasticsearch-batch-stream\u0026amp;utm_campaign=Badge_Grade)\n[![Known Vulnerabilities](https://snyk.io/test/npm/elasticsearch-batch-stream/badge.svg)](https://snyk.io/test/npm/elasticsearch-batch-stream)\n[![Coverage Status](https://coveralls.io/repos/github/ovhemert/elasticsearch-batch-stream/badge.svg?branch=master)](https://coveralls.io/github/ovhemert/elasticsearch-batch-stream?branch=master)\n[![js-standard-style](https://img.shields.io/badge/code%20style-standard-brightgreen.svg?style=flat)](http://standardjs.com/)\n\nA write stream that creates batches of elasticsearch bulk operations.\n\n## Example\n\nThe ElasticSearch library has a function to [bulk](https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/api-reference.html#api-bulk) write documents, but since a stream emits a write for each document, we cannot group multiple operations together.\n\nThis package wraps the `bulk` function in a writestream to help buffer the operations and passing them on as batches to the bulk function. For example, we can now create batches of 500 docs each and reduce the number of API calls to ElasticSearch from 100.000 to 200, which will improve speed.\n\n```js\n  const docTransformStream = through2.obj(function (chunk, enc, callback) {\n    // convert chunk =\u003e doc\n    const doc = { index: 'myindex', type: 'mytype', id: '12345', action: 'index', doc: { name: 'test' } }\n    callback(null, doc)\n  })\n\n  sourceReadStream().pipe(docTransformStream()).pipe(bulkWriteStream({ client, size: 500 }))\n```\n\n## Installation\n\n```bash\n$ npm install elasticsearch-batch-stream\n```\n\n## API\n\n\u003cb\u003e\u003ccode\u003ebulkWriteStream(options = { client, size })\u003c/code\u003e\u003c/b\u003e\n\nCreates the write stream to ElasticSearch.\n\n### options\n\nThe options object argument is required and should at least include the ElasticSearch client object.\n\n#### client\n\nAn instance of the ElasticSearch client i.e. `new elasticsearch.Client()`\n\n#### size\n\nNumber of stream operations to group together in the bulk command (default = 100).\n\n## Maintainers\n\nOsmond van Hemert\n[![Github](https://img.shields.io/badge/-website.svg?style=social\u0026logoColor=333\u0026logo=github)](https://github.com/ovhemert)\n[![Web](https://img.shields.io/badge/-website.svg?style=social\u0026logoColor=333\u0026logo=nextdoor)](https://ovhemert.dev)\n\n## Contributing\n\nIf you would like to help out with some code, check the [details](./docs/CONTRIBUTING.md).\n\nNot a coder, but still want to support? Have a look at the options available to [donate](https://ovhemert.dev/donate).\n\n## License\n\nLicensed under [MIT](./LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fovhemert%2Felasticsearch-batch-stream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fovhemert%2Felasticsearch-batch-stream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fovhemert%2Felasticsearch-batch-stream/lists"}