{"id":28466144,"url":"https://github.com/zilliztech/milvus-backup","last_synced_at":"2025-06-29T21:32:35.910Z","repository":{"id":49870743,"uuid":"518393338","full_name":"zilliztech/milvus-backup","owner":"zilliztech","description":" Backup and restore tool for Milvus","archived":false,"fork":false,"pushed_at":"2025-06-20T03:16:33.000Z","size":12413,"stargazers_count":171,"open_issues_count":41,"forks_count":56,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-06-20T04:26:07.472Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zilliztech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-07-27T09:30:19.000Z","updated_at":"2025-06-20T03:16:37.000Z","dependencies_parsed_at":"2023-10-11T15:38:55.550Z","dependency_job_id":"b0389e2a-a42e-4af4-9427-fbe44c6a4ac2","html_url":"https://github.com/zilliztech/milvus-backup","commit_stats":{"total_commits":121,"total_committers":7,"mean_commits":"17.285714285714285","dds":0.2644628099173554,"last_synced_commit":"6237aa42b58a89dd515553fa3ea38fcb7c3b4403"},"previous_names":[],"tags_count":63,"template":false,"template_full_name":null,"purl":"pkg:github/zilliztech/milvus-backup","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zilliztech%2Fmilvus-backup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zilliztech%2Fmilvus-backup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zilliztech%2Fmilvus-backup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zilliztech%2Fmilvus-backup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zilliztech","download_url":"https://codeload.github.com/zilliztech/milvus-backup/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zilliztech%2Fmilvus-backup/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262671701,"owners_count":23346453,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-07T06:08:38.253Z","updated_at":"2025-06-29T21:32:35.873Z","avatar_url":"https://github.com/zilliztech.png","language":"Go","readme":"# Milvus-backup\n\u003cdiv class=\"column\" align=\"left\"\u003e\n  \u003ca href=\"https://discord.com/invite/8uyFbECzPX\"\u003e\u003cimg height=\"20\" src=\"https://img.shields.io/badge/Discord-%235865F2.svg?style=for-the-badge\u0026logo=discord\u0026logoColor=white\" alt=\"license\"/\u003e\u003c/a\u003e\n  \u003cimg src=\"https://img.shields.io/github/license/milvus-io/milvus\" alt=\"license\"/\u003e\n\u003c/div\u003e\n\n\nMilvus-Backup is a tool that allows users to backup and restore Milvus data. This tool can be utilized either through the command line or an API server.\n\nThe Milvus-backup process has negligible impact on the performance of Milvus. Milvus cluster is fully functional and can operate normally while backup and restoration are in progress.\n\n## Installation\n\n* Download binary from [release page](https://github.com/zilliztech/milvus-backup/releases). Usually the latest is recommended.\n\n* Use [homebrew](https://brew.sh/) to install on Mac\n  ```shell\n  brew install zilliztech/tap/milvus-backup\n  ```\n\n## Usage\n\nMilvus-backup provides command line and API server for usage.\n\n### Configuration\nIn order to use Milvus-Backup, access to Milvus proxy and Minio cluster is required. Configuration settings related to this access can be edited in [backup.yaml](configs/backup.yaml).\n\n\u003e [!NOTE]\n\u003e\n\u003e Please ensure that the configuration settings for Minio are accurate. There may be variations in the default value of Minio's configuration depending on how Milvus is deployed, either by docker-compose or k8s.\n\u003e |field|docker-compose |helm|\n\u003e |---|---|---|\n\u003e |bucketName|a-bucket|milvus-bucket|\n\u003e |rootPath|files|file|\n\n### Command Line\n\nMilvus-backup establish CLI based on cobra. Use the following command to see all the usage.\n\n```\nmilvus-backup is a backup\u0026restore tool for milvus.\n\nUsage:\n  milvus-backup [flags]\n  milvus-backup [command]\n\nAvailable Commands:\n  check       check if the connects is right.\n  create      create subcommand create a backup.\n  delete      delete subcommand delete backup by name.\n  get         get subcommand get backup by name.\n  help        Help about any command\n  list        list subcommand shows all backup in the cluster.\n  restore     restore subcommand restore a backup.\n  server      server subcommand start milvus-backup RESTAPI server.\n\nFlags:\n      --config string   config YAML file of milvus (default \"backup.yaml\")\n  -h, --help            help for milvus-backup\n\nUse \"milvus-backup [command] --help\" for more information about a command.\n```\n\nHere is a [demo](docs/user_guide/e2e_demo_cli.md) for a complete backup and restore process.\n\n### API Server\n\nTo start the RESTAPI server, use the following command after building:\n\n```shell\n./milvus-backup server\n```\n\nThe server will listen on port 8080 by default. However, you can change it by using the `-p` parameter as shown below:\n\n```shell\n./milvus-backup server -p 443\n```\n\nWe offer a [demo](docs/user_guide/api_demo.md) of the key APIs; however, please refer to the Swagger UI for the most up-to-date usage details, as the demo may occasionally become outdated.\n\n#### swagger UI\n\nWe offer access to our Swagger UI, which displays comprehensive information for our APIs. To view it, simply go to\n\n```\nhttp://localhost:8080/api/v1/docs/index.html\n```\n---\n\n## Backup.yaml Configurations\n\nBelow is a summary of the configurations supported in `backup.yaml`:\n\n| **Section**       | **Field**                       | **Description**                                                                                              | **Default/Example**     |\n|--------------------|---------------------------------|--------------------------------------------------------------------------------------------------------------|-------------------------|\n| `log`             | `level`                         | Logging level. Supported: `debug`, `info`, `warn`, `error`, `panic`, `fatal`.                                | `info`                  |\n|                   | `console`                       | Whether to print logs to the console.                                                                        | `true`                  |\n|                   | `file.rootPath`                 | Path to the log file.                                                                                        | `logs/backup.log`       |\n| `http`            | `simpleResponse`                | Whether to enable simple HTTP responses.                                                                     | `true`                  |\n| `milvus`          | `address`                       | Milvus proxy address.                                                                                        | `localhost`             |\n|                   | `port`                          | Milvus proxy port.                                                                                           | `19530`                 |\n|                   | `user`                          | Username for Milvus.                                                                                         | `root`                  |\n|                   | `password`                      | Password for Milvus.                                                                                         | `Milvus`                |\n|                   | `tlsMode`                       | TLS mode (0: none, 1: one-way, 2: two-way/mtls).                                                             | `0`                     |\n|                   | `caCertPath`                    | Path to your ca certificate file                                                                             | `/path/to/certificate`  |\n|                   | `serverName `                   | Server name                                                                                                  | `localhost`             |\n|                   | `mtlsCertPath`                  | Path to your mtls certificate file                                                                           | `/path/to/certificate`  |\n|                   | `mtlsKeyPath `                  | Path to your mtls key file                                                                                   | `/path/to/key`          |\n| `minio`           | `storageType`                   | Storage type for Milvus (e.g., `local`, `minio`, `s3`, `aws`, `gcp`, `ali(aliyun)`, `azure`, `tc(tencent)`, `gcpnative`). Use `gcpnative` for the Google Cloud Platform provider.          | `minio`                 |\n|                   | `address`                       | MinIO/S3 address.                                                                                            | `localhost`             |\n|                   | `port`                          | MinIO/S3 port.                                                                                               | `9000`                  |\n|                   | `accessKeyID`                   | MinIO/S3 access key ID.                                                                                      | `minioadmin`            |\n|                   | `secretAccessKey`               | MinIO/S3 secret access key.                                                                                  | `minioadmin`            |\n|                   | `gcpCredentialJSON`             | Path to your GCP credential JSON key file. Used only for the \"gcpnative\" cloud provider.                     | `/path/to/json-key-file`       |\n|                   | `useSSL`                        | Whether to use SSL for MinIO/S3.                                                                             | `false`                 |\n|                   | `bucketName`                    | Bucket name in MinIO/S3.                                                                                     | `a-bucket`              |\n|                   | `rootPath`                      | Storage root path in MinIO/S3.                                                                               | `files`                 |\n| `minio (backup)`  | `backupStorageType`             | Backup storage type (e.g., `local`, `minio`, `s3`, `aws`, `gcp`, `ali(aliyun)`, `azure`, `tc(tencent)`, `gcpnative`). Use `gcpnative` for the Google Cloud Platform provider.                       | `minio`                 |\n|                   | `backupAddress`                 | Address of backup storagße.                                                                                   | `localhost`             |\n|                   | `backupPort`                    | Port of backup storage.                                                                                      | `9000`                  |\n|                   | `backupUseSSL`                  | Whether to use SSL for backup storage.                                                                       | `false`                 |\n|                   | `backupAccessKeyID`             | Backup storage access key ID.                                                                                | `minioadmin`            |\n|                   | `backupSecretAccessKey`         | Backup storage secret access key.                                                                            | `minioadmin`            |\n|                   | `backupGcpCredentialJSON`       | Path to your GCP credential JSON key file. Used only for the \"gcpnative\" cloud provider.                     | `/path/to/json-key-file`       |\n|                   | `backupBucketName`              | Bucket name for backups.                                                                                     | `a-bucket`              |\n|                   | `backupRootPath`                | Root path to store backup data.                                                                              | `backup`                |\n|                   | `crossStorage`                  | Enable cross-storage backups (e.g., MinIO to AWS S3).                                                        | `false`                 |\n| `backup`          | `maxSegmentGroupSize`           | Maximum segment group size for backups.                                                                      | `2G`                    |\n|                   | `parallelism.backupCollection`  | Collection-level parallelism for backup.                                                                     | `4`                     |\n|                   | `parallelism.copydata`          | Thread pool size for copying data.                                                                           | `128`                   |\n|                   | `parallelism.restoreCollection` | Collection-level parallelism for restore.                                                                    | `2`                     |\n|                   | `keepTempFiles`                 | Whether to keep temporary files during restore (for debugging).                                              | `false`                 |\n|                   | `gcPause.enable`                | Pause Milvus garbage collection during backup.                                                               | `true`                  |\n|                   | `gcPause.seconds`               | Duration to pause garbage collection (in seconds).                                                           | `7200`                  |\n|                   | `gcPause.address`               | Address for Milvus garbage collection API.                                                                   | `http://localhost:9091` |\n\nFor more details, refer to the [backup.yaml](configs/backup.yaml) configuration file.\n\n### Advanced feature\n\n1. [Cross Storage Backup](docs/user_guide/cross_storage.md): Data is read from the source storage and written to a different storage through the Milvus-backup service. Such as, S3 -\u003e local, S3 a -\u003e S3 b. \n\n2. [RBAC Backup\u0026Restore](docs/user_guide/rbac.md): Enable backup and restore RBAC meta with extra parameter.\n\n## Examples\n\n### Syncing Minio Backups to an AWS S3 Bucket\n\n\u003e **NOTE:** The following configuration is an example only. Replace the placeholders with your actual AWS and Minio settings.\n\nTo back up Milvus data to an AWS S3 bucket, you need to configure the `backup.yaml` file with the following settings:\n\n```yaml\n# Backup storage configs: Configure the storage where you want to save the backup data\nbackupStorageType: \"aws\"                     # Specifies the storage type as AWS S3\nbackupAddress: s3.{your-aws-region}.amazonaws.com  # Address of AWS S3 (replace {your-aws-region} with your bucket's region)\nbackupPort: 443                              # Default port for AWS S3\nbackupAccessKeyID: \u003cyour-access-key-id\u003e      # Access key ID for your AWS S3\nbackupSecretAccessKey: \u003cyour-secret-key\u003e     # Secret access key for AWS S3\nbackupGcpCredentialJSON: \"/path/to/file\"     # Path to your GCP credential JSON key file. Used only for the \"gcpnative\" cloud provider.\nbackupBucketName: \"your-bucket-name\"         # Bucket name where the backups will be stored\nbackupRootPath: \"backups\"                    # Root path inside the bucket to store backups\nbackupUseSSL: true                           # Use SSL for secure connections (Required)\ncrossStorage: \"true\"                         # Required for minio to S3 backups\n```\n\n\n## Development\n\n### Build\n\nFor developers, Milvus-backup is easy to contribute to.\n\nExecute `make all` will generate an executable binary `milvus-backup` in the `{project_path}/bin` directory.\n\n### Test\n\nDevelopers can also test it using an IDE. You can test it using the command line interface:\n\n```shell\nmake test\n```\n\n\n## License\nmilvus-backup is licensed under the Apache License, Version 2.0.\n","funding_links":[],"categories":["Data Integration \u0026 Migration"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzilliztech%2Fmilvus-backup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzilliztech%2Fmilvus-backup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzilliztech%2Fmilvus-backup/lists"}