{"id":35244757,"url":"https://github.com/walterra/node-es-transformer","last_synced_at":"2026-03-04T01:24:28.042Z","repository":{"id":48028568,"uuid":"134525478","full_name":"walterra/node-es-transformer","owner":"walterra","description":"nodejs based (re)index and data transformation library for Elasticsearch.","archived":false,"fork":false,"pushed_at":"2024-12-31T14:48:24.000Z","size":815,"stargazers_count":7,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-02-08T13:09:44.549Z","etag":null,"topics":["elasticsearch","elasticsearch-ingestion","javascript","javascript-library","reindex"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/walterra.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-05-23T06:43:42.000Z","updated_at":"2025-09-23T07:28:40.000Z","dependencies_parsed_at":"2023-11-07T13:29:10.585Z","dependency_job_id":"3dea3194-a6b5-4f06-8588-943c457a642e","html_url":"https://github.com/walterra/node-es-transformer","commit_stats":{"total_commits":35,"total_committers":1,"mean_commits":35.0,"dds":0.0,"last_synced_commit":"dd2ea3cf50a119a60b180ca2781df8ccf0b55582"},"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"purl":"pkg:github/walterra/node-es-transformer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/walterra%2Fnode-es-transformer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/walterra%2Fnode-es-transformer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/walterra%2Fnode-es-transformer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/walterra%2Fnode-es-transformer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/walterra","download_url":"https://codeload.github.com/walterra/node-es-transformer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/walterra%2Fnode-es-transformer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29441096,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-14T07:24:13.446Z","status":"ssl_error","status_checked_at":"2026-02-14T07:23:58.969Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","elasticsearch-ingestion","javascript","javascript-library","reindex"],"created_at":"2025-12-30T05:56:39.341Z","updated_at":"2026-03-04T01:24:28.033Z","avatar_url":"https://github.com/walterra.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![npm version](https://img.shields.io/npm/v/node-es-transformer.svg)](https://www.npmjs.com/package/node-es-transformer)\n[![npm downloads](https://img.shields.io/npm/dt/node-es-transformer.svg)](https://www.npmjs.com/package/node-es-transformer)\n[![license](https://img.shields.io/npm/l/node-es-transformer.svg)](https://github.com/walterra/node-es-transformer/blob/main/LICENSE)\n[![Node.js version](https://img.shields.io/badge/node-%3E%3D22-brightgreen.svg)](https://nodejs.org/)\n[![CI](https://github.com/walterra/node-es-transformer/actions/workflows/ci.yml/badge.svg)](https://github.com/walterra/node-es-transformer/actions)\n[![TypeScript](https://img.shields.io/badge/TypeScript-definitions-blue.svg)](https://github.com/walterra/node-es-transformer/blob/main/index.d.ts)\n[![Elasticsearch](https://img.shields.io/badge/Elasticsearch-8.x%20%7C%209.x-005571.svg)](https://www.elastic.co/elasticsearch/)\n\n# node-es-transformer\n\nStream-based library for ingesting and transforming large data files (NDJSON/CSV/Parquet/Arrow IPC) into Elasticsearch indices.\n\n## Quick Start\n\n```bash\nnpm install node-es-transformer\n```\n\n```javascript\nconst transformer = require('node-es-transformer');\n\n// Ingest a large JSON file\nawait transformer({\n  fileName: 'data.json',\n  targetIndexName: 'my-index',\n  mappings: {\n    properties: {\n      '@timestamp': { type: 'date' },\n      'message': { type: 'text' }\n    }\n  }\n});\n```\n\nSee [Usage](#usage) for more examples.\n\n## Why Use This?\n\nIf you need to ingest large NDJSON/CSV/Parquet/Arrow IPC files (GigaBytes) into Elasticsearch without running out of memory, this is the tool for you. Other solutions often run out of JS heap, hammer ES with too many requests, time out, or try to do everything in a single bulk request.\n\n**When to use this:**\n- Large file ingestion (20-30 GB tested)\n- Custom JavaScript transformations\n- Cross-version migration (ES 8.x → 9.x)\n- Developer-friendly Node.js workflow\n\n**When to use alternatives:**\n- [Logstash](https://www.elastic.co/products/logstash) - Enterprise ingestion pipelines\n- [Filebeat](https://www.elastic.co/products/beats/filebeat) - Log file shipping\n- [Elastic Agent](https://www.elastic.co/guide/en/fleet/current/fleet-overview.html) - Modern unified agent\n- [Elasticsearch Transforms](https://www.elastic.co/guide/en/elasticsearch/reference/current/transforms.html) - Built-in data transformation\n\n## Table of Contents\n\n- [Features](#features)\n- [Quick Start](#quick-start)\n- [Installation](#installation)\n- [Version Compatibility](#version-compatibility)\n- [Usage](#usage)\n  - [Read from a file](#read-from-a-file)\n  - [Read from another index](#read-from-another-index)\n  - [Reindex from ES 8.x to 9.x](#reindex-from-elasticsearch-8x-to-9x)\n- [API Reference](#api-reference)\n- [Documentation](#documentation)\n- [Contributing](#contributing)\n- [License](#license)\n\n## Features\n\n- **Streaming and buffering**: Files are read using streams and Elasticsearch ingestion uses buffered bulk indexing. Handles very large files (20-30 GB tested) without running out of memory.\n- **High throughput**: Up to 20k documents/second on a single machine (2.9 GHz Intel Core i7, 16GB RAM, SSD), depending on document size. See [PERFORMANCE.md](PERFORMANCE.md) for benchmarks and tuning guidance.\n- **Wildcard support**: Ingest multiple files matching a pattern (e.g., `logs/*.json`).\n- **Flexible sources**: Read from files, Elasticsearch indices, or Node.js streams.\n- **Reindexing with transforms**: Fetch documents from existing indices and transform them using JavaScript.\n- **Document splitting**: Transform one source document into multiple target documents (e.g., tweets → hashtags).\n- **Cross-version support**: Seamlessly reindex between Elasticsearch 8.x and 9.x.\n\n## Version Compatibility\n\n| node-es-transformer     | Elasticsearch Client | Elasticsearch Server | Node.js |\n| ----------------------- | -------------------- | -------------------- | ------- |\n| 1.0.0+                  | 8.x and 9.x          | 8.x and 9.x          | 22+     |\n| 1.0.0-beta7 and earlier | 8.x                  | 8.x                  | 18-20   |\n\n**Multi-Version Support**: Starting with v1.0.0, the library supports both Elasticsearch 8.x and 9.x through automatic version detection and client aliasing. This enables seamless reindexing between major versions (e.g., migrating from ES 8.x to 9.x). All functionality is tested in CI against multiple ES versions including cross-version reindexing scenarios.\n\n**Upgrading?** See [MIGRATION.md](MIGRATION.md) for upgrade guidance from beta versions to v1.0.0.\n\n## Installation\n\n```bash\nnpm install node-es-transformer\n# or\nyarn add node-es-transformer\n```\n\n## Usage\n\n### Read NDJSON from a file\n\n```javascript\nconst transformer = require('node-es-transformer');\n\ntransformer({\n  fileName: 'filename.json',\n  targetIndexName: 'my-index',\n  mappings: {\n    properties: {\n      '@timestamp': {\n        type: 'date'\n      },\n      'first_name': {\n        type: 'keyword'\n      },\n      'last_name': {\n        type: 'keyword'\n      }\n      'full_name': {\n        type: 'keyword'\n      }\n    }\n  },\n  transform(line) {\n    return {\n      ...line,\n      full_name: `${line.first_name} ${line.last_name}`\n    }\n  }\n});\n```\n\n### Read CSV from a file\n\n```javascript\nconst transformer = require('node-es-transformer');\n\ntransformer({\n  fileName: 'users.csv',\n  sourceFormat: 'csv',\n  targetIndexName: 'users-index',\n  mappings: {\n    properties: {\n      id: { type: 'integer' },\n      first_name: { type: 'keyword' },\n      last_name: { type: 'keyword' },\n      full_name: { type: 'keyword' },\n    },\n  },\n  transform(row) {\n    return {\n      ...row,\n      id: Number(row.id),\n      full_name: `${row.first_name} ${row.last_name}`,\n    };\n  },\n});\n```\n\n### Read Parquet from a file\n\n```javascript\nconst transformer = require('node-es-transformer');\n\ntransformer({\n  fileName: 'users.parquet',\n  sourceFormat: 'parquet',\n  targetIndexName: 'users-index',\n  mappings: {\n    properties: {\n      id: { type: 'integer' },\n      first_name: { type: 'keyword' },\n      last_name: { type: 'keyword' },\n      full_name: { type: 'keyword' },\n    },\n  },\n  transform(row) {\n    return {\n      ...row,\n      id: Number(row.id),\n      full_name: `${row.first_name} ${row.last_name}`,\n    };\n  },\n});\n```\n\n### Read Arrow IPC from a file\n\n```javascript\nconst transformer = require('node-es-transformer');\n\ntransformer({\n  fileName: 'users.arrow',\n  sourceFormat: 'arrow',\n  targetIndexName: 'users-index',\n  mappings: {\n    properties: {\n      id: { type: 'integer' },\n      first_name: { type: 'keyword' },\n      last_name: { type: 'keyword' },\n    },\n  },\n  transform(row) {\n    return {\n      ...row,\n      id: Number(row.id),\n    };\n  },\n});\n```\n\n### Infer mappings from CSV sample\n\n```javascript\nconst transformer = require('node-es-transformer');\n\ntransformer({\n  fileName: 'users.csv',\n  sourceFormat: 'csv',\n  targetIndexName: 'users-index',\n  inferMappings: true,\n  inferMappingsOptions: {\n    sampleBytes: 200000,\n    lines_to_sample: 2000,\n  },\n});\n```\n\n### Read from another index\n\n```javascript\nconst transformer = require('node-es-transformer');\n\ntransformer({\n  sourceIndexName: 'my-source-index',\n  targetIndexName: 'my-target-index',\n  // optional, if you skip mappings, they will be fetched from the source index.\n  mappings: {\n    properties: {\n      '@timestamp': {\n        type: 'date'\n      },\n      'first_name': {\n        type: 'keyword'\n      },\n      'last_name': {\n        type: 'keyword'\n      }\n      'full_name': {\n        type: 'keyword'\n      }\n    }\n  },\n  transform(doc) {\n    return {\n      ...doc,\n      full_name: `${line.first_name} ${line.last_name}`\n    }\n  }\n});\n```\n\n### Reindex from Elasticsearch 8.x to 9.x\n\nThe library automatically detects the Elasticsearch version and uses the appropriate client. This enables seamless reindexing between major versions:\n\n```javascript\nconst transformer = require('node-es-transformer');\n\n// Auto-detection (recommended)\ntransformer({\n  sourceClientConfig: {\n    node: 'https://es8-cluster.example.com:9200',\n    auth: { apiKey: 'your-es8-api-key' },\n  },\n  targetClientConfig: {\n    node: 'https://es9-cluster.example.com:9200',\n    auth: { apiKey: 'your-es9-api-key' },\n  },\n  sourceIndexName: 'my-source-index',\n  targetIndexName: 'my-target-index',\n  transform(doc) {\n    // Optional transformation during reindexing\n    return doc;\n  },\n});\n\n// Explicit version specification (if auto-detection fails)\ntransformer({\n  sourceClientConfig: {\n    /* ... */\n  },\n  targetClientConfig: {\n    /* ... */\n  },\n  sourceClientVersion: 8, // Force ES 8.x client\n  targetClientVersion: 9, // Force ES 9.x client\n  sourceIndexName: 'my-source-index',\n  targetIndexName: 'my-target-index',\n});\n\n// Using pre-instantiated clients (advanced)\nconst { Client: Client8 } = require('es8');\nconst { Client: Client9 } = require('es9');\n\nconst sourceClient = new Client8({\n  node: 'https://es8-cluster.example.com:9200',\n});\nconst targetClient = new Client9({\n  node: 'https://es9-cluster.example.com:9200',\n});\n\ntransformer({\n  sourceClient,\n  targetClient,\n  sourceIndexName: 'my-source-index',\n  targetIndexName: 'my-target-index',\n});\n```\n\n**Note**: To use pre-instantiated clients with different ES versions, install both client versions:\n\n```bash\nnpm install es9@npm:@elastic/elasticsearch@^9.2.0\nnpm install es8@npm:@elastic/elasticsearch@^8.17.0\n```\n\n## API Reference\n\n### Configuration Options\n\nAll options are passed to the main `transformer()` function.\n\n#### Required Options\n\n- **`targetIndexName`** (string): The target Elasticsearch index where documents will be indexed.\n\n#### Source Options\n\nChoose **one** of these sources:\n\n- **`fileName`** (string): Source filename to ingest. Supports wildcards (e.g., `logs/*.json`, `data/*.csv`, `data/*.parquet`, `data/*.arrow`).\n- **`sourceIndexName`** (string): Source Elasticsearch index to reindex from.\n- **`stream`** (Readable): Node.js readable stream to ingest from.\n- **`sourceFormat`** (`'ndjson' | 'csv' | 'parquet' | 'arrow'`): Format for file/stream sources. Default: `'ndjson'`.\n  - `arrow` expects Arrow IPC file/stream payloads.\n  - `parquet` stream sources are currently buffered in memory before row iteration (file sources remain streaming by row cursor).\n  - `parquet` supports ZSTD-compressed files when running on Node.js 22+ (uses the built-in zlib zstd implementation).\n  - `parquet` INT64 values are normalized for JSON: safe-range values become numbers, larger values become strings.\n- **`csvOptions`** (object): CSV parser options (delimiter, quote, columns, etc.) used when `sourceFormat: 'csv'`.\n\n#### Client Configuration\n\n- **`sourceClient`** (Client): Pre-instantiated Elasticsearch client for source operations. If provided, `sourceClientConfig` is ignored.\n- **`targetClient`** (Client): Pre-instantiated Elasticsearch client for target operations. If not provided, uses `sourceClient` or creates from config.\n- **`sourceClientConfig`** (object): Elasticsearch client configuration for source. Default: `{ node: 'http://localhost:9200' }`. Ignored if `sourceClient` is provided.\n- **`targetClientConfig`** (object): Elasticsearch client configuration for target. If not provided, uses `sourceClientConfig`. Ignored if `targetClient` is provided.\n- **`sourceClientVersion`** (8 | 9): Force specific ES client version for source. Auto-detected if not specified.\n- **`targetClientVersion`** (8 | 9): Force specific ES client version for target. Auto-detected if not specified.\n\n#### Index Configuration\n\n- **`mappings`** (object): Elasticsearch document mappings for target index. If reindexing and not provided, mappings are copied from source index.\n- **`mappingsOverride`** (boolean): When reindexing, apply `mappings` on top of source index mappings. Default: `false`.\n- **`inferMappings`** (boolean): Infer mappings for `fileName` sources via `/_text_structure/find_structure`. Supported for `sourceFormat: 'ndjson'` and `sourceFormat: 'csv'` only. Ignored when `mappings` is provided. If inference returns `ingest_pipeline`, it is created as `\u003ctargetIndexName\u003e-inferred-pipeline` and applied as the index default pipeline (unless `pipeline` is explicitly set). Default: `false`.\n- **`inferMappingsOptions`** (object): Options for `/_text_structure/find_structure` (for example `sampleBytes`, `lines_to_sample`, `delimiter`, `quote`, `has_header_row`, `timeout`).\n- **`deleteIndex`** (boolean): Delete target index if it exists before starting. Default: `false`.\n- **`indexMappingTotalFieldsLimit`** (number): Field limit for target index (`index.mapping.total_fields.limit` setting).\n- **`pipeline`** (string): Elasticsearch ingest pipeline name to use during indexing.\n\nWhen `inferMappings` is enabled, the target cluster must allow `/_text_structure/find_structure` (cluster privilege: `monitor_text_structure`). If inferred ingest pipelines are used, the target cluster must also allow creating ingest pipelines (`_ingest/pipeline`).\n\n#### Performance Options\n\n- **`bufferSize`** (number): Buffer size threshold in KBytes for bulk indexing. Default: `5120` (5 MB).\n- **`searchSize`** (number): Number of documents to fetch per search request when reindexing. Default: `100`.\n- **`populatedFields`** (boolean): Detect which fields are actually populated in documents. Useful for optimizing indices with many mapped but unused fields. Default: `false`.\n\n#### Processing Options\n\n- **`transform`** (function): Callback to transform documents. Signature: `(doc, context?) =\u003e doc | doc[] | null | undefined`.\n  - Return transformed document\n  - Return array of documents to split one source into multiple targets\n  - Return `null`/`undefined` to skip document\n- **`query`** (object): Elasticsearch [DSL query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html) to filter source documents.\n- **`splitRegex`** (RegExp): Line split regex for file/stream sources when `sourceFormat` is `'ndjson'`. Default: `/\\n/`.\n- **`skipHeader`** (boolean): Header skipping for file/stream sources.\n  - NDJSON: skips the first non-empty line\n  - CSV: skips the first data line only when `csvOptions.columns` does not consume headers\n  - Parquet/Arrow: ignored\n  - Default: `false`\n  - Applies only to `fileName`/`stream` sources\n- **`verbose`** (boolean): Enable verbose logging and progress bars when using the built-in logger. Default: `true`.\n- **`logger`** (object): Optional custom Pino-compatible logger. If omitted, the library creates an internal Pino logger (`name: node-es-transformer`) and uses `LOG_LEVEL` (if set) or `info`/`error` based on `verbose`.\n\n### Return Value\n\nThe `transformer()` function returns a Promise that resolves to an object with:\n\n- **`events`** (EventEmitter): Event emitter for monitoring progress.\n  - `'queued'`: Document added to queue\n  - `'indexed'`: Document successfully indexed\n  - `'complete'`: All documents processed\n  - `'error'`: Error occurred\n\n```javascript\nconst pino = require('pino');\nconst logger = pino({ name: 'my-app', level: process.env.LOG_LEVEL || 'info' });\n\nconst result = await transformer({\n  /* options */\n});\n\nresult.events.on('complete', () =\u003e {\n  logger.info('Ingestion complete');\n});\n\nresult.events.on('error', err =\u003e {\n  logger.error({ err }, 'Ingestion failed');\n});\n```\n\n### TypeScript Support\n\nFull TypeScript definitions are included. Import types for type-safe configuration:\n\n```typescript\nimport transformer, { TransformerOptions } from 'node-es-transformer';\n\nconst options: TransformerOptions = {\n  fileName: 'data.json',\n  targetIndexName: 'my-index',\n};\n```\n\nSee [examples/typescript-example.ts](examples/typescript-example.ts) for more examples.\n\n## Documentation\n\n- **[README.md](README.md)** - Getting started and API reference (you are here)\n- **[examples/](examples/)** - Practical code samples for common use cases\n- **[VERSIONING.md](VERSIONING.md)** - API stability guarantees and versioning policy\n- **[PERFORMANCE.md](PERFORMANCE.md)** - Benchmarks, tuning, and optimization guide\n- **[TESTING.md](TESTING.md)** - Test coverage, approach, and how to run tests\n- **[DEPENDENCIES.md](DEPENDENCIES.md)** - Dependency audit and update tracking\n- **[MIGRATION.md](MIGRATION.md)** - Upgrading from beta to v1.0.0\n- **[CONTRIBUTING.md](CONTRIBUTING.md)** - How to contribute (open an issue first!)\n- **[DEVELOPMENT.md](DEVELOPMENT.md)** - Development setup and testing\n- **[RELEASE.md](RELEASE.md)** - Complete release process and troubleshooting\n- **[SECURITY.md](SECURITY.md)** - Security policy and vulnerability reporting\n\n### Error Handling\n\nAlways handle errors when using the library:\n\n```javascript\nconst pino = require('pino');\nconst logger = pino({ name: 'my-app', level: process.env.LOG_LEVEL || 'info' });\n\ntransformer({\n  /* options */\n})\n  .then(() =\u003e logger.info('Success'))\n  .catch(err =\u003e logger.error({ err }, 'Transformer failed'));\n\n// Or with async/await\ntry {\n  await transformer({\n    /* options */\n  });\n  logger.info('Success');\n} catch (err) {\n  logger.error({ err }, 'Transformer failed');\n}\n```\n\n### More Examples\n\nSee the [examples/](examples/) directory for practical code samples covering:\n\n- Basic file ingestion\n- Reindexing with transformations\n- Cross-version migration (ES 8.x → 9.x)\n- Document splitting\n- Wildcard file processing\n- Stream-based ingestion\n\n## Contributing\n\nContributions are welcome! Before starting work on a PR, please open an issue to discuss your proposed changes.\n\n- [CONTRIBUTING.md](CONTRIBUTING.md) - Contribution guidelines and PR process\n- [DEVELOPMENT.md](DEVELOPMENT.md) - Development setup, testing, and release process\n- [SECURITY.md](SECURITY.md) - Security policy and vulnerability reporting\n\n## Support\n\nThis is a single-person best-effort project. While I aim to address issues and maintain the library, response times may vary. See [VERSIONING.md](VERSIONING.md) for details on API stability and support expectations.\n\n**Getting help:**\n- Check the [documentation](#documentation) first\n- Review [examples/](examples/) for practical code samples\n- Search [existing issues](https://github.com/walterra/node-es-transformer/issues)\n- Open a new issue with details (version, steps to reproduce, expected vs actual behavior)\n\n## License\n\n[Apache 2.0](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwalterra%2Fnode-es-transformer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwalterra%2Fnode-es-transformer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwalterra%2Fnode-es-transformer/lists"}