{"id":13819116,"url":"https://github.com/mgtv-tech/redis-GunYu","last_synced_at":"2025-05-16T04:32:39.072Z","repository":{"id":228750528,"uuid":"774808537","full_name":"mgtv-tech/redis-GunYu","owner":"mgtv-tech","description":null,"archived":false,"fork":false,"pushed_at":"2024-05-21T01:59:39.000Z","size":829,"stargazers_count":66,"open_issues_count":0,"forks_count":13,"subscribers_count":0,"default_branch":"master","last_synced_at":"2024-05-21T07:31:12.900Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mgtv-tech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-20T08:20:53.000Z","updated_at":"2024-05-31T10:25:52.931Z","dependencies_parsed_at":"2024-05-31T10:25:48.034Z","dependency_job_id":"faf22e41-0f08-4f8b-bf3c-9d30e5b41b0e","html_url":"https://github.com/mgtv-tech/redis-GunYu","commit_stats":null,"previous_names":["mgtv-tech/redis-gunyu"],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgtv-tech%2Fredis-GunYu","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgtv-tech%2Fredis-GunYu/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgtv-tech%2Fredis-GunYu/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgtv-tech%2Fredis-GunYu/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mgtv-tech","download_url":"https://codeload.github.com/mgtv-tech/redis-GunYu/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254469538,"owners_count":22076514,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-04T08:00:40.480Z","updated_at":"2025-05-16T04:32:34.062Z","avatar_url":"https://github.com/mgtv-tech.png","language":"Go","funding_links":[],"categories":["Go","Redis-Manage"],"sub_categories":[],"readme":"# Redis GunYu\n\n[![CI](https://github.com/mgtv-tech/redis-GunYu/workflows/goci/badge.svg)](https://github.com/mgtv-tech/redis-GunYu/actions/workflows/goci.yml)\n[![LICENSE](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/mgtv-tech/redis-GunYu/blob/master/LICENSE)\n[![release](https://img.shields.io/github/release/mgtv-tech/redis-GunYu)](https://github.com/mgtv-tech/redis-GunYu/releases)\n\n\nTranslations: [English](README.md) | [简体中文](README_ZH.md)\n\n\n- [Redis GunYu](#redis-gunyu)\n  - [Overview](#overview)\n  - [Features](#features)\n    - [Real-time Data Synchronization](#real-time-data-synchronization)\n    - [Load RDB into redis](#load-rdb-into-redis)\n    - [Other Features](#other-features)\n  - [Product Comparison](#product-comparison)\n  - [Technical Implementation](#technical-implementation)\n  - [Quick Start](#quick-start)\n    - [Installation](#installation)\n    - [Usage](#usage)\n    - [Running the Demo](#running-the-demo)\n  - [Documentation](#documentation)\n  - [Contributing](#contributing)\n  - [License](#license)\n  - [Contact](#contact)\n\n## Overview\n\n`redis-GunYu` is a Redis data management tool capable of real-time data synchronization, data migration, backup, verification and recovery, data analysis, and more.\n\n## Features\n\n### Real-time Data Synchronization\n\nThe feature matrix of `redis-GunYu` for real-time synchronization\n\n| Feature | Supported |\n| :- | :- |\n| Resuming from Breakpoints | Yes | \n| Inconsistent slots between source and target clusters | Yes | \n| Topology changes in source or target clusters (scaling, migration, etc.) | Yes | \n| High availability | Yes | \n| Data filtering | Yes | \n| Data consistency model | Eventual/Weak | \n\n`redis-GunYu` has additional advantages:\n- Minimal impact on stability\n  - Ingest source: Specify whether to sync data from a slave, master or prefer slave\n  - Local cache + resuming from breakpoints: Minimizes the impact on the source Redis\n  - Splits big keys of RDB to synchronize them\n  - Lower replication latency: Sync data concurrently, see [sync latency metrics](docs/deployment_en.md#Monitor)\n- Data security and high availability\n  - Local cache supports data verification\n  - High availability of the tool: Supports master-slave mode, self-election based on the latest records, automatic and manual failover; the tool is P2P architecture, minimizing downtime impact\n- Fewer restrictions on Redis\n  - Supports different deployment modes of Redis on the source and target, such as cluster or standalone instances\n  - Compatible with different versions of Redis on the source and target, supports from Redis 4.0 to Redis 7.2, see [testing](docs/test_en.md#Compatibility)\n- More flexible data consistency strategies, automatic switching\n  - When the shards distribution of the source and target is the same, batch writes in pseudo-transaction mode, and offsets are updated in real-time, maximizing inconsistent\n  - When the shard distribution of the source and target is different, offsets are updated periodically\n- User-friendly for dev-ops\n  - API: supports HTTP API, such as full sync, checking synchronization status, pausing synchronization, etc.\n  - Monitoring: Rich monitoring metrics, such as replication latency metrics in time and space dimensions\n  - Data filtering: Filter by certain regular keys, databases, commands, etc.\n  - Redis topology: Real-time monitoring of topology changes in the source and target Redis (e.g., adding/removing nodes, master-slave switch, etc.), to change consistency strategies and adjust other functional strategies\n\n\n### Load RDB into redis\n\nLoad a RDB file to a running redis server or cluster. Refers to [RDB](rdb_en.md)\n\n\n### Other Features\n\nAdditional features are currently under development.\n\n## Product Comparison\n\nComparison of redis-GunYu with several top-tranking tools based on product requirements\n\n| Feature | redis-shake/v2 | DTS | xpipe | redis-GunYu |\n| -- | -- | -- | -- | -- |\n| Resuming from Breakpoints | Yes (no local cache) | Yes | Yes | Yes |\n| Supports different sharding between source and target | No | Yes | No | Yes |\n| Topology changes | No | No | No | Yes |\n| High availability | No | No | Yes | Yes |\n| Data consistency | Eventual | Weak | Weak | Eventual (same sharding) + Weak (different sharding) |\n\n## Technical Implementation\n\nThe technical implementation of `redis-GunYu` is illustrated in the diagram below. For detailed technical principles, see [Technical Implementation](docs/tech_en.md)\n\n\u003cimg src=\"docs/imgs/sync_en.png\" width = \"400\" height = \"150\" alt=\"Architecture Diagram\" align=center /\u003e\n\n## Quick Start\n\n### Installation\n\nYou can compile it yourself or run it directly in a container\n\n**Download Binary**\n\n**Compile Source Code**\n\nMake sure Go language is installed and environment variables are configured\n\n```\ngit clone https://github.com/mgtv-tech/redis-gunyu.git\ncd redis-GunYu\n\n## Add proxy if needed\nexport GOPROXY=https://goproxy.io,direct\n\nmake\n```\n\nThis generates the `redisGunYu` binary file locally.\n\n### Usage\n\n**Start with configuration file**\n\n```\n./redisGunYu -conf ./config.yaml\n```\nDefault command is sync, you can run other command with `-cmd=command name`.\n\n\n**Start with command line arguments**\n``` \n./redisGunYu --sync.input.redis.addresses=127.0.0.1:6379 --sync.output.redis.addresses=127.0.0.1:16379\n```\n\n\n**Run in docker**\n\n```\ndocker run mgtvtech/redisgunyu:latest --sync.input.redis.addresses=172.10.10.10:6379 --sync.output.redis.addresses=172.10.10.11:6379\n\n\n# For local testing, start the docker in host network mode --network=host, so redisGunYu can communicate with local Redis\ndocker run --network=host mgtvtech/redisgunyu:latest --sync.input.redis.addresses=127.0.0.1:6700 --sync.output.redis.addresses=127.0.0.1:6710\n```\n\n### Running the Demo\n\n**Start demo service**\n```\ndocker run --rm -p 16379:16379 -p 26379:26379 -p 18001:18001 mgtvtech/redisgunyudemo:latest\n```\n- Source Redis: port 16379\n- Target Redis: port 26379\n- Synchronization tool: port 18001\n\n**Target Redis**\n\n```\nredis-cli -p 26379\n127.0.0.1:26379\u003e monitor\n```\nInput `monitor` in the target Redis CLI\n\n\n**Source Redis**\n\nConnect to the source Redis and set a key. The synchronization tool will synchronize the command to the target Redis. Check the output of the target Redis connected by the Redis CLI\n\n```\nredis-cli -p 16300\n127.0.0.1:16379\u003e set a 1\n```\n\n\n\n**Check Status**\n```\ncurl http://localhost:18001/syncer/status\n```\nCheck the status of the synchronization tool\n\n## Documentation\n\n- [Configuration](docs/configuration_en.md)\n- [Deployment](docs/deployment_en.md)\n- [API](docs/API_en.md)\n- [Testing](docs/test_en.md)\n- [Attentions](docs/attentions_en.md)\n\n## Contributing\n\nEveryone is welcome to help improve redis-GunYu. If you have any questions, suggestions, or want to add other features, please submit an issue or PR directly.\n\nPlease follow these steps to submit a PR:\n- Clone the repository\n- Create a new branch: name it `feature-xxx` for new features or `bug-xxx` for bug fixes\n- Describe the changes in detail in the PR\n\n## License\n\n`redis-GunYu` is licensed under Apache 2.0, see [LICENSE](LICENSE).\n\n## Contact\n\nIf you have any questions, please contact `ikenchina@gmail.com`.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmgtv-tech%2Fredis-GunYu","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmgtv-tech%2Fredis-GunYu","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmgtv-tech%2Fredis-GunYu/lists"}