{"id":19542749,"url":"https://github.com/opensearch-project/opensearch-migrations","last_synced_at":"2026-05-06T06:03:30.863Z","repository":{"id":64123671,"uuid":"564884242","full_name":"opensearch-project/opensearch-migrations","owner":"opensearch-project","description":"Migrate, upgrade, compare, and replicate OpenSearch clusters with ease.","archived":false,"fork":false,"pushed_at":"2025-04-25T17:11:37.000Z","size":13355,"stargazers_count":50,"open_issues_count":38,"forks_count":33,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-04-25T17:33:21.142Z","etag":null,"topics":["ab-testing","backfill","disaster-recovery","elasticsearch","livecapture","migration","opensearch","performance-testing"],"latest_commit_sha":null,"homepage":"https://aws.amazon.com/solutions/implementations/migration-assistant-for-amazon-opensearch-service/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/opensearch-project.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-11-11T18:22:44.000Z","updated_at":"2025-04-25T17:11:42.000Z","dependencies_parsed_at":"2023-09-27T20:25:12.666Z","dependency_job_id":"3d986647-7931-4e64-b535-0b78d130138b","html_url":"https://github.com/opensearch-project/opensearch-migrations","commit_stats":null,"previous_names":[],"tags_count":58,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opensearch-project%2Fopensearch-migrations","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opensearch-project%2Fopensearch-migrations/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opensearch-project%2Fopensearch-migrations/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opensearch-project%2Fopensearch-migrations/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/opensearch-project","download_url":"https://codeload.github.com/opensearch-project/opensearch-migrations/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251025811,"owners_count":21524866,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ab-testing","backfill","disaster-recovery","elasticsearch","livecapture","migration","opensearch","performance-testing"],"created_at":"2024-11-11T03:15:52.383Z","updated_at":"2026-05-06T06:03:30.829Z","avatar_url":"https://github.com/opensearch-project.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OpenSearch Migration Assistant\n\n[![codecov](https://codecov.io/gh/opensearch-project/opensearch-migrations/graph/badge.svg)](https://codecov.io/gh/opensearch-project/opensearch-migrations)\n\n## Table of Contents\n- [OpenSearch Migration Assistant](#opensearch-migration-assistant)\n  - [Table of Contents](#table-of-contents)\n  - [Overview](#overview)\n  - [Key Features](#key-features)\n  - [Supported Migration Paths and Platforms](#supported-migration-paths-and-platforms)\n    - [Migration Paths](#migration-paths)\n    - [Platforms](#platforms)\n    - [Performance Limitations](#performance-limitations)\n      - [Test Results](#test-results)\n  - [Issue Tracking](#issue-tracking)\n  - [Roadmap](#roadmap)\n  - [User Guide Documentation](#user-guide-documentation)\n  - [Getting Started](#getting-started)\n    - [Local Deployment](#local-deployment)\n    - [AWS Deployment](#aws-deployment)\n  - [Continuous Integration and Deployment](#continuous-integration-and-deployment)\n  - [Contributing](#contributing)\n  - [Security](#security)\n  - [License](#license)\n  - [Acknowledgments](#acknowledgments)\n\n\n## Overview\n\nOpenSearch Migration Assistant is a comprehensive set of tools designed to facilitate upgrades, migrations, and comparisons for OpenSearch and Elasticsearch clusters. This project aims to simplify the process of moving between different versions and platforms while ensuring data integrity and performance.\n\n## Key Features\n\n- **Upgrade and Migration Support**: Provides tools for migrating between different versions of Elasticsearch and OpenSearch.\n  - **[Metadata Migration](MetadataMigration/README.md)**: Migrate essential cluster components such as configuration, settings, templates, and aliases.\n  - **Multi-Version Upgrade**: Easily migrate across major versions (e.g., from Elasticsearch 6.8 to OpenSearch 2.15), skipping intermediate upgrades and reducing time and risk.\n  - **Downgrade Support**: Downgrade to an earlier version if needed (e.g., from Elasticsearch 7.17 to 7.10.2).\n  - **Existing Data Migration with [Reindex-from-Snapshot](RFS/docs/DESIGN.md)**: Migrate indices and documents using snapshots, updating your data to the latest Lucene version quickly without impacting the target cluster.\n  - **Live Traffic Capture with [Capture-and-Replay](docs/TrafficCaptureAndReplayDesign.md)**: Capture live traffic from the source cluster and replay it on the target cluster for validation. This ensures the target cluster can handle real-world traffic patterns before fully migrating.\n  \n- **Zero-Downtime Migration with [Live Traffic Routing](docs/ClientTrafficSwinging.md)**: Tools to seamlessly switch client traffic between clusters while keeping services fully operational.\n\n- **Migration Rollback**: Keep your source cluster synchronized during the migration, allowing you to monitor the target cluster's performance before fully committing to the switch. You can safely revert if needed.\n\n- **User-Friendly Interface via [Migration Console](https://github.com/opensearch-project/opensearch-migrations/blob/main/docs/migration-console.md)**: Command Line Interface (CLI) that guides you through each migration step.\n\n- **AI-Assisted Migration with [Kiro](kiro-cli/README.md)**: An AI agent that can guide you through the entire migration process — from source discovery to target provisioning to data migration. See [`agent-sops/`](agent-sops/) for the standard operating procedures and [`kiro-cli/`](kiro-cli/) for configuration.\n\n- **Flexible Deployment Options**:\n  - **[AWS Deployment](https://aws.amazon.com/solutions/implementations/migration-assistant-for-amazon-opensearch-service/)**: Fully automated deployment to AWS.\n  - **[Local Docker Deployment](./TrafficCapture/dockerSolution/README.md)**: Run the solution locally in a container for testing and development.\n  - Contribute to add more deployment options.\n\n## Supported Migration Paths and Platforms\n\n### Migration Paths\n\n\u003ctable\u003e\n\u003ctr\u003e\n  \u003cth rowspan=\"2\"\u003eSource Cluster\u003c/th\u003e\n  \u003cth colspan=\"3\"\u003eTarget Cluster\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003cth\u003eOpenSearch 1.x\u003c/th\u003e\n  \u003cth\u003eOpenSearch 2.x\u003c/th\u003e\n  \u003cth\u003eOpenSearch 3.x\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003eElasticsearch 1.x\u003c/td\u003e\u003ctd\u003e✅\u003c/td\u003e\u003ctd\u003e✅\u003c/td\u003e\u003ctd\u003e✅\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003eElasticsearch 2.x\u003c/td\u003e\u003ctd\u003e✅\u003c/td\u003e\u003ctd\u003e✅\u003c/td\u003e\u003ctd\u003e✅\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003eElasticsearch 5.x\u003c/td\u003e\u003ctd\u003e✅\u003c/td\u003e\u003ctd\u003e✅\u003c/td\u003e\u003ctd\u003e✅\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003eElasticsearch 6.x\u003c/td\u003e\u003ctd\u003e✅\u003c/td\u003e\u003ctd\u003e✅\u003c/td\u003e\u003ctd\u003e✅\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003eElasticsearch 7.x\u003c/td\u003e\u003ctd\u003e✅\u003c/td\u003e\u003ctd\u003e✅\u003c/td\u003e\u003ctd\u003e✅\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003eElasticsearch 8.x\u003c/td\u003e\u003ctd\u003e\u003c/td\u003e\u003ctd\u003e✅\u003c/td\u003e\u003ctd\u003e✅\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003eOpenSearch 1.x\u003c/td\u003e\u003ctd\u003e✅\u003c/td\u003e\u003ctd\u003e✅\u003c/td\u003e\u003ctd\u003e✅\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003eOpenSearch 2.x\u003c/td\u003e\u003ctd\u003e\u003c/td\u003e\u003ctd\u003e✅\u003c/td\u003e\u003ctd\u003e✅\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003eOpenSearch 3.x\u003c/td\u003e\u003ctd\u003e\u003c/td\u003e\u003ctd\u003e\u003c/td\u003e\u003ctd\u003e🔜 \u003ca href=\"https://github.com/orgs/opensearch-project/projects/229?pane=issue\u0026itemId=117495207\"\u003elink\u003c/a\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003eApache Solr 8.x–9.x*\u003c/td\u003e\u003ctd\u003e\u003c/td\u003e\u003ctd\u003e\u003c/td\u003e\u003ctd\u003e✅\u003c/td\u003e\u003c/tr\u003e\n\u003c/table\u003e\n\n\\* Backfill only — Capture and Replay is not supported for Solr sources.\n\nNote that testing is done on specific minor versions, but any minor versions within a listed major version are expected to work.\n\n### Platforms\n  - Self-managed (cloud provider hosted)\n  - Self-managed (on-premises)\n  - Managed cloud offerings (e.g., Amazon OpenSearch)\n  - Managed serverless cloud offerings (e.g. Amazon OpenSearch Serverless)\n\n### Performance Limitations\nA performance test was performed on 03/10/25 alongside [PR 1337](https://github.com/opensearch-project/opensearch-migrations/pull/1337)\nto identify general indexing throughput with Reindex-From-Snapshot and Traffic Replay. While Reindex-From-Snapshot\nincludes periods of ingestion inactivity (e.g. while downloading the shard data), this was not factored into the ingestion rates.\nTest used docs with an average uncompressed size 2.39 KB (source doc and index command) which corresponded to 1.54 KB of primary shard size per doc on OS 2.17 with default settings\n\nFor real-word use, throughput can be increased by vertically scaling Reindex-From-Snapshot and Traffic Replay instances or horizontally scaling\nthe Reindex-From-Snapshot workers that are running on AWS Fargate. Outside the exception indicated, all cases are CPU limited for throughput.\nAll cases were using uncompressed network traffic generated by the applications, results will vary if using client compression.\n\nThroughput here is measured by the rate of increase in primary shard data with bulk ingestion on the target cluster alongside\nthe uncompressed size of the source data ingested.\n\nTests were ran with and without the [Type Mapping Sanitization Transformer](./transformation/transformationPlugins/jsonMessageTransformers/jsonTypeMappingsSanitizationTransformer/README.md).\n\n#### Test Results\n| Service               | vCPU | Memory (GB) | Type Mapping Sanitization | Peak Docs Ingested per minute | Primary Shard Data Ingestion Rate (MBps) | Uncompressed Source Data Ingestion Rate (MBps) |\n| --------------------- | ---- | ----------- | ------------------------- | ----------------------------- | ---------------------------------------- | ---------------------------------------------- |\n| Reindex-From-Snapshot | 2    | 4           | Disabled                  | 590,000                       | 15.1                                     | 23.5                                           |\n| Reindex-From-Snapshot | 2    | 4           | Enabled                   | 546,000                       | 14.0                                     | 21.7                                           |\n| Traffic Replay        | 8    | 48          | Disabled                  | 1,694,000 *[1]*               | 43.5  *[1]*                              | 67.5   *[1]*                                   |\n| Traffic Replay        | 8    | 48          | Enabled                   | 1,645,000                     | 42.2                                     | 65.5                                           |\n\n**[1] Network Bandwidth Limitations Observed**\n\n## Issue Tracking\n\nWe encourage users to open bugs and feature requests in this GitHub repository. \n\n**Encountering a compatibility issue or missing feature?**\n\n- [Search existing issues](https://github.com/opensearch-project/opensearch-migrations/issues) to see if it’s already reported. If it is, feel free to **upvote** and **comment**.\n- Can’t find it? [Create a new issue](https://github.com/opensearch-project/opensearch-migrations/issues/new/choose) to let us know.\n\nFor issue prioritization and management, the migrations team uses Jira, but uses GitHub issues for community intake:\n\nhttps://opensearch.atlassian.net/\n\n## Roadmap\n\nFor upcoming features, enhancements, and priorities, check out the [OpenSearch Migrations Project Board](https://github.com/orgs/opensearch-project/projects/229/views/1).\n\n## User Guide Documentation\n\nUser guide documentation is available in the [OpenSearch Migration Assistant documentation](https://docs.opensearch.org/latest/migration-assistant/).\n\n## Getting Started\n\n### Local Deployment\n\n Refer to the [Development Guide](DEVELOPER_GUIDE.md) for more details.\n\n### AWS Deployment\n\nTo deploy the solution on AWS, follow the steps outlined in [Migration Assistant for Amazon OpenSearch Service](https://aws.amazon.com/solutions/implementations/migration-assistant-for-amazon-opensearch-service/), specifically [deploying the solution](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/deploy-the-solution.html).\n\n\n## Continuous Integration and Deployment\nWe use a combination of GitHub actions and Jenkins so that we can publish releases on a weekly basis and allow users to provide attestation for migration tooling.\n\nJenkins pipelines are available [here](https://migrations.ci.opensearch.org/)\n\n## Contributing\n\nPlease read [CONTRIBUTING.md](CONTRIBUTING.md) for details on our code of conduct and the process for submitting pull requests.\n\nPlease refer to the [Development Guide](DEVELOPER_GUIDE.md) for details on building and deploying.\n\n## Security\n\nSee [SECURITY.md](SECURITY.md) for information about reporting security vulnerabilities.\n\n## License\n\nThis project is licensed under the Apache-2.0 License - see the [LICENSE](LICENSE) file for details.\n\n## Acknowledgments\n\n- OpenSearch Community\n- Contributors and maintainers\n\nFor more detailed information about specific components, please refer to the README files in the respective directories.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopensearch-project%2Fopensearch-migrations","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopensearch-project%2Fopensearch-migrations","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopensearch-project%2Fopensearch-migrations/lists"}