Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/leonchen83/redis-rdb-cli

Redis rdb CLI : A CLI tool that can parse, filter, split, merge rdb and analyze memory usage offline. It can also sync 2 redis data and allow user define their own sink service to migrate redis data to somewhere.
https://github.com/leonchen83/redis-rdb-cli

analyze cli dashboard memory migrate rdb redis

Last synced: about 2 months ago
JSON representation

Redis rdb CLI : A CLI tool that can parse, filter, split, merge rdb and analyze memory usage offline. It can also sync 2 redis data and allow user define their own sink service to migrate redis data to somewhere.

Awesome Lists containing this project

README 

        
# redis-rdb-cli



Buy Me A Coffee



A tool that can parse, filter, split, merge rdb and analyze memory usage offline. It can also sync 2 redis data and allow user define there own sink service to migrate redis data to somewhere.



[![Java CI](https://github.com/leonchen83/redis-rdb-cli/actions/workflows/maven.yml/badge.svg)](https://github.com/leonchen83/redis-rdb-cli/actions/workflows/maven.yml)

[![Gitter](https://badges.gitter.im/leonchen83/redis-rdb-cli.svg)](https://gitter.im/leonchen83/redis-rdb-cli?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)

[![Hex.pm](https://img.shields.io/hexpm/l/plug.svg?maxAge=2592000)](https://github.com/leonchen83/redis-rdb-cli/blob/master/LICENSE)  

  

## Chat with author  

  

[![Join the chat at https://gitter.im/leonchen83/redis-rdb-cli](https://badges.gitter.im/leonchen83/redis-rdb-cli.svg)](https://gitter.im/leonchen83/redis-rdb-cli?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)  

  

## Contract the author

  

**[email protected]**  

  

## Binary release



[binary releases](https://github.com/leonchen83/redis-rdb-cli/releases)



## Runtime requirement



```text

jdk 1.8+

```



## Install



```shell

$ wget https://github.com/leonchen83/redis-rdb-cli/releases/download/${version}/redis-rdb-cli-release.zip

$ unzip redis-rdb-cli-release.zip

$ cd ./redis-rdb-cli/bin

$ ./rct -h

```



## Compile requirement



```text



jdk 1.8+

maven-3.3.1+



```



## Compile & run



```shell

$ git clone https://github.com/leonchen83/redis-rdb-cli.git

$ cd redis-rdb-cli

$ mvn clean install -Dmaven.test.skip=true

$ cd target/redis-rdb-cli-release/redis-rdb-cli/bin

$ ./rct -h 

```



## Run in docker



```shell

# run with jvm

$ docker run -it --rm redisrdbcli/redis-rdb-cli:latest

$ rct -V



# run without jvm

$ docker run -it --rm redisrdbcli/redis-rdb-cli:latest-native

$ rct -V

```



## Build native image via graalvm in docker

```shell

$ docker build -m 8g -f DockerfileNative -t redisrdbcli:redis-rdb-cli .

$ docker run -it redisrdbcli:redis-rdb-cli bash

$ bash-5.1# rct -V

```



## Windows Environment Variables

  

Add `/path/to/redis-rdb-cli/bin` to `Path` environment variable  

  

## Usage



### Redis mass insertion



```shell



$ rct -f dump -s /path/to/dump.rdb -o /path/to/dump.aof -r

$ cat /path/to/dump.aof | /redis/src/redis-cli -p 6379 --pipe



```



### Convert rdb to dump format



```shell

$ rct -f dump -s /path/to/dump.rdb -o /path/to/dump.aof

```



### Convert rdb to json format



```shell

$ rct -f json -s /path/to/dump.rdb -o /path/to/dump.json

```



### Numbers of key in rdb



```shell

$ rct -f count -s /path/to/dump.rdb -o /path/to/dump.csv

```



### Find top 50 largest keys



```shell

$ rct -f mem -s /path/to/dump.rdb -o /path/to/dump.mem -l 50

```



### Diff rdb



```shell

$ rct -f diff -s /path/to/dump1.rdb -o /path/to/dump1.diff

$ rct -f diff -s /path/to/dump2.rdb -o /path/to/dump2.diff

$ diff /path/to/dump1.diff /path/to/dump2.diff

```



### Convert rdb to RESP



```shell

$ rct -f resp -s /path/to/dump.rdb -o /path/to/appendonly.aof

```



### Sync with 2 redis

```shell

$ rst -s redis://127.0.0.1:6379 -m redis://127.0.0.1:6380 -r

```



### Sync single redis to redis cluster

```shell

$ rst -s redis://127.0.0.1:6379 -m redis://127.0.0.1:30001 -r -d 0

```



### Handle infinite loop in rst command



```shell

# set client-output-buffer-limit in source redis

$ redis-cli config set client-output-buffer-limit "slave 0 0 0"

$ rst -s redis://127.0.0.1:6379 -m redis://127.0.0.1:6380 -r

```



### Migrate rdb to remote redis



```shell

$ rmt -s /path/to/dump.rdb -m redis://192.168.1.105:6379 -r

```



### Downgrade migration



```shell



# Migrate data from redis-7 to redis-6

# About dump_rdb_version please see comment in redis-rdb-cli.conf

$ sed -i 's/dump_rdb_version=-1/dump_rdb_version=9/g' /path/to/redis-rdb-cli/conf/redis-rdb-cli.conf

$ rmt -s redis://com.redis7:6379 -m redis://com.redis6:6379 -r

```



### Handle big key in migration

```shell



# set proto-max-bulk-len in target redis

$ redis-cli -h ${host} -p 6380 -a ${pwd} config set proto-max-bulk-len 2048mb



# set Xms Xmx in redis-rdb-cli node

$ export JAVA_TOOL_OPTIONS="-Xms8g -Xmx8g"



# execute migration

$ rmt -s redis://127.0.0.1:6379 -m redis://127.0.0.1:6380 -r



```



### Migrate rdb to remote redis cluster



```shell

$ rmt -s /path/to/dump.rdb -c ./nodes-30001.conf -r

```



or simply use following cmd without `nodes-30001.conf`



```shell

$ rmt -s /path/to/dump.rdb -m redis://127.0.0.1:30001 -r

```



### Backup remote rdb



```shell

$ rdt -b redis://192.168.1.105:6379 -o /path/to/dump.rdb

```



### Backup remote rdb and convert db to dest db



```shell

$ rdt -b redis://192.168.1.105:6379 -o /path/to/dump.rdb --goal 3

```



### Filter rdb



```shell

$ rdt -b /path/to/dump.rdb -o /path/to/filtered-dump.rdb -d 0 -t string

```



### Split rdb via cluster's nodes.conf



```shell

$ rdt -s ./dump.rdb -c ./nodes.conf -o /path/to/folder -d 0

```



### Merge multi rdb to one



```shell

$ rdt -m ./dump1.rdb ./dump2.rdb -o ./dump.rdb -t hash

```



### Cut aof-use-rdb-preamble file to rdb file and aof file



```shell

$ rcut -s ./aof-use-rdb-preamble.aof -r ./dump.rdb -a ./appendonly.aof

```



### Other parameter



More configurable parameter can be modified in `/path/to/redis-rdb-cli/conf/redis-rdb-cli.conf`



### Filter



1. `rct`, `rdt` and `rmt` these 3 commands support data filter by `type`,`db` and `key` RegEx(Java style).  

2. `rst` this command only support data filter by `db`.  

  

For example:



```shell



$ rct -f dump -s /path/to/dump.rdb -o /path/to/dump.aof -d 0

$ rct -f dump -s /path/to/dump.rdb -o /path/to/dump.aof -t string hash

$ rmt -s /path/to/dump.rdb -m redis://192.168.1.105:6379 -r -d 0 1 -t list

$ rst -s redis://127.0.0.1:6379 -m redis://127.0.0.1:6380 -d 0

```



### Monitor redis server



```shell

# step1 

# open file `/path/to/redis-rdb-cli/conf/redis-rdb-cli.conf`

# change property `metric_gateway from `none` to `influxdb`

#

# step2

$ cd /path/to/redis-rdb-cli/dashboard

$ docker-compose up -d

#

# step3

$ rmonitor -s redis://127.0.0.1:6379 -n standalone

$ rmonitor -s redis://127.0.0.1:30001 -n cluster

$ rmonitor -s redis-sentinel://sntnl-usr:[email protected]:26379?master=mymaster&authUser=usr&authPassword=pwd -n sentinel

#

# step4

# open url `http://localhost:3000/d/monitor/monitor`, login grafana use `admin`, `admin` and check monitor result.

```



![monitor](./images/monitor.png)



## Difference between rmt and rst



1. When `rmt` started. source redis first do `BGSAVE` and generate a snapshot rdb file. `rmt` command migrate this snapshot file to target redis. after this process done, `rmt` terminated.  

2. `rst` not only migrate snapshot rdb file but also incremental data from source redis. so `rst` never terminated except type `CTRL+C`. `rst` only support `db` filter more details please refer to [Limitation of migration](#limitation-of-migration) 



## Dashboard



Since `v0.1.9`, the `rct -f mem` support showing result in grafana dashboard like the following:  

![memory](./images/memory.png)  



If you want to turn it on. you **MUST** install `docker` and `docker-compose` first, the installation please refer to [docker](https://docs.docker.com/install/)  

Then run the following command:  



```shell

$ cd /path/to/redis-rdb-cli/dashboard



# start

$ docker-compose up -d



# stop

$ docker-compose down

```

  

`cd /path/to/redis-rdb-cli/conf/redis-rdb-cli.conf`  

Then change parameter [metric_gateway](https://github.com/leonchen83/redis-rdb-cli/blob/master/src/main/resources/redis-rdb-cli.conf#L169) from `none` to `influxdb`.  

  

Open `http://localhost:3000` to check the `rct -f mem`'s result.  

  

If you deployed this tool in multi instance, you need to change parameter [metric_instance](https://github.com/leonchen83/redis-rdb-cli/blob/master/src/main/resources/redis-rdb-cli.conf#L215) to make sure unique between instances.  

  

## Redis 6

  

### Redis 6 SSL

  

1. use openssl to generate keystore

  

```shell



$ cd /path/to/redis-6.0-rc1

$ ./utils/gen-test-certs.sh

$ cd tests/tls

$ openssl pkcs12 -export -CAfile ca.crt -in redis.crt -inkey redis.key -out redis.p12



```

  

2. If source redis and target redis use the same keystore. then config following parameters  

[source_keystore_path](https://github.com/leonchen83/redis-rdb-cli/blob/master/src/main/resources/redis-rdb-cli.conf#L230) and [target_keystore_path](https://github.com/leonchen83/redis-rdb-cli/blob/master/src/main/resources/redis-rdb-cli.conf#L259) to point to `/path/to/redis-6.0-rc1/tests/tls/redis.p12`  

set [source_keystore_pass](https://github.com/leonchen83/redis-rdb-cli/blob/master/src/main/resources/redis-rdb-cli.conf#L238) and [target_keystore_pass](https://github.com/leonchen83/redis-rdb-cli/blob/master/src/main/resources/redis-rdb-cli.conf#L267)  

  

3. after config ssl parameters use `rediss://host:port` in your command to open ssl, for example: `rst -s rediss://127.0.0.1:6379 -m rediss://127.0.0.1:30001 -r -d 0`

  

### Redis 6 ACL



1. use following URI to open redis ACL support  

  

```shell

$ rst -s redis://user:[email protected]:6379 -m redis://user:[email protected]:6380 -r -d 0

```

  

2. `user` **MUST** have `+@all` permission to handle commands

  

## Hack rmt



### Rmt threading model



The `rmt` command use the following 4 parameters([redis-rdb-cli.conf](https://github.com/leonchen83/redis-rdb-cli/blob/master/src/main/resources/redis-rdb-cli.conf)) to migrate data to remote.  

  

```properties

migrate_batch_size=4096

migrate_threads=4

migrate_flush=yes

migrate_retries=1

```



The most important parameter is `migrate_threads=4`. this means we use the following threading model to migrate data.  



```text



single redis ----> single redis



+--------------+         +----------+     thread 1      +--------------+

|              |    +----| Endpoint |-------------------|              |

|              |    |    +----------+                   |              |

|              |    |                                   |              |

|              |    |    +----------+     thread 2      |              |

|              |    |----| Endpoint |-------------------|              |

|              |    |    +----------+                   |              |

| Source Redis |----|                                   | Target Redis |

|              |    |    +----------+     thread 3      |              |

|              |    |----| Endpoint |-------------------|              |

|              |    |    +----------+                   |              |

|              |    |                                   |              |

|              |    |    +----------+     thread 4      |              |

|              |    +----| Endpoint |-------------------|              |

+--------------+         +----------+                   +--------------+



``` 



```text



single redis ----> redis cluster



+--------------+         +----------+     thread 1      +--------------+

|              |    +----| Endpoints|-------------------|              |

|              |    |    +----------+                   |              |

|              |    |                                   |              |

|              |    |    +----------+     thread 2      |              |

|              |    |----| Endpoints|-------------------|              |

|              |    |    +----------+                   |              |

| Source Redis |----|                                   | Redis cluster|

|              |    |    +----------+     thread 3      |              |

|              |    |----| Endpoints|-------------------|              |

|              |    |    +----------+                   |              |

|              |    |                                   |              |

|              |    |    +----------+     thread 4      |              |

|              |    +----| Endpoints|-------------------|              |

+--------------+         +----------+                   +--------------+



``` 



The difference between cluster migration and single migration is `Endpoint` and `Endpoints`. In cluster migration the `Endpoints` contains multi `Endpoint` to point to every `master` instance in cluster. For example:  

  

3 masters 3 replicas redis cluster. if `migrate_threads=4` then we have `3 * 4 = 12` connections that connected with `master` instance. 



### Migration performance



The following 3 parameters affect migration performance  

  

```properties

migrate_batch_size=4096

migrate_retries=1

migrate_flush=yes

```



1. `migrate_batch_size`: By default we use redis `pipeline` to migrate data to remote. the `migrate_batch_size` is the `pipeline` batch size. if `migrate_batch_size=1` then the `pipeline` devolved into 1 single command to sent and wait the response from remote.  

2. `migrate_retries`: The `migrate_retries=1` means if socket error occurred. we recreate a new socket and retry to send that failed command to target redis with `migrate_retries` times.  

3. `migrate_flush`: The `migrate_flush=yes` means we write every 1 command to socket. then we invoke `SocketOutputStream.flush()` immediately. if `migrate_flush=no` we invoke `SocketOutputStream.flush()` when write to socket every 64KB. notice that this parameter also affect `migrate_retries`. the `migrate_retries` only take effect when `migrate_flush=yes`.  



### Migration principle



```text



+---------------+             +-------------------+    restore      +---------------+ 

|               |             | redis dump format |---------------->|               |

|               |             |-------------------|    restore      |               |

|               |   convert   | redis dump format |---------------->|               |

|    Dump rdb   |------------>|-------------------|    restore      |  Targe Redis  |

|               |             | redis dump format |---------------->|               |

|               |             |-------------------|    restore      |               |

|               |             | redis dump format |---------------->|               |

+---------------+             +-------------------+                 +---------------+

```



## Limitation of migration



1. We use cluster's `nodes.conf` to migrate data to cluster. because of we didn't handle the `MOVED` `ASK` redirection. so limitation of cluster migration is that the cluster **MUST** in stable state during the migration. this means the cluster **MUST** have no `migrating`, `importing` slot and no switch slave to master. 

2. If use `rst` migrate data to cluster. the following commands not supported `PUBLISH,SWAPDB,MOVE,FLUSHALL,FLUSHDB,MULTI,EXEC,SCRIPT FLUSH,SCRIPT LOAD,EVAL,EVALSHA`. and the following commands `RPOPLPUSH,SDIFFSTORE,SINTERSTORE,SMOVE,ZINTERSTORE,ZUNIONSTORE,DEL,UNLINK,RENAME,RENAMENX,PFMERGE,PFCOUNT,MSETNX,BRPOPLPUSH,BITOP,MSET,COPY,BLMOVE,LMOVE,ZDIFFSTORE,GEOSEARCHSTORE` **ONLY SUPPORT WHEN THESE COMMAND KEYS IN THE SAME SLOT**(eg: `del {user}:1 {user}:2`)



## Hack ret



### What ret command do



1. `ret` command that allow user define there own sink service like sink redis data to `mysql` or `mongodb`.

2. `ret` command using Java SPI extension to do this job.



### How to implement a sink service



User should follow the steps below to implement a sink service.  



1. create a java project using maven pom.xml



```xml



    4.0.0

    

    com.your.company

    your-sink-service

    1.0.0

    

    

        UTF-8

        1.8

        1.8

    



    

        

            com.moilioncircle

            redis-rdb-cli-api

            1.8.0

            provided

        

        

            com.moilioncircle

            redis-replicator

            [3.6.4, )

            provided

        

        

            org.slf4j

            slf4j-api

            1.7.25

            provided

        

        

        

        

    

    

    

        

            

                maven-assembly-plugin

                3.1.0

                

                    

                        jar-with-dependencies

                    

                

                

                    

                        make-assembly

                        package

                        

                            single

                        

                    

                

            

            

                org.apache.maven.plugins

                maven-compiler-plugin

                3.8.1

                

                    ${maven.compiler.source}

                    ${maven.compiler.target}

                    ${project.build.sourceEncoding}

                

            

        

    



```



2. implement `SinkService` interface



```java



public class YourSinkService implements SinkService {



    @Override

    public String sink() {

        return "your-sink-service";

    }



    @Override

    public void init(File config) throws IOException {

        // parse your external sink config

    }



    @Override

    public void onEvent(Replicator replicator, Event event) {

        // your sink business

    }

}



```

3. register this service using Java SPI



```text

# create com.moilioncircle.redis.rdb.cli.api.sink.SinkService file in src/main/resources/META-INF/services/



|-src

|____main

| |____resources

| | |____META-INF

| | | |____services

| | | | |____com.moilioncircle.redis.rdb.cli.api.sink.SinkService



# add following content in com.moilioncircle.redis.rdb.cli.api.sink.SinkService



your.package.YourSinkService



```



4. package and deploy



```shell



$ mvn clean install



$ cp ./target/your-sink-service-1.0.0-jar-with-dependencies.jar /path/to/redis-rdb-cli/lib

```

5. run your sink service



```shell



$ ret -s redis://127.0.0.1:6379 -c config.conf -n your-sink-service

```



6. debug your sink service



```java  



    public static void main(String[] args) throws Exception {

        Replicator replicator = new RedisReplicator("redis://127.0.0.1:6379");

        Runtime.getRuntime().addShutdownHook(new Thread(() -> {

            Replicators.closeQuietly(replicator);

        }));

        replicator.addExceptionListener((rep, tx, e) -> {

            throw new RuntimeException(tx.getMessage(), tx);

        });

        SinkService sink = new YourSinkService();

        sink.init(new File("/path/to/your-sink.conf"));

        replicator.addEventListener(new AsyncEventListener(sink, replicator, 4, Executors.defaultThreadFactory()));

        replicator.open();

    }



```



### How to implement a formatter service



1. create `YourFormatterService` extend `AbstractFormatterService`  



```java



public class YourFormatterService extends AbstractFormatterService {



    @Override

    public String format() {

        return "test";

    }



    @Override

    public Event applyString(Replicator replicator, RedisInputStream in, int version, byte[] key, int type, ContextKeyValuePair context) throws IOException {

        byte[] val = new DefaultRdbValueVisitor(replicator).applyString(in, version);

        getEscaper().encode(key, getOutputStream());

        getEscaper().encode(val, getOutputStream());

        getOutputStream().write('\n');

        return context;

    }

}



```



2. register this formatter using Java SPI  



```text

# create com.moilioncircle.redis.rdb.cli.api.format.FormatterService file in src/main/resources/META-INF/services/



|-src

|____main

| |____resources

| | |____META-INF

| | | |____services

| | | | |____com.moilioncircle.redis.rdb.cli.api.format.FormatterService



# add following content in com.moilioncircle.redis.rdb.cli.api.format.FormatterService



your.package.YourFormatterService



```



3. package and deploy



```shell



$ mvn clean install



$ cp ./target/your-service-1.0.0-jar-with-dependencies.jar /path/to/redis-rdb-cli/lib

```



4. run your formatter service



```shell



$ rct -f test -s redis://127.0.0.1:6379 -o ./out.csv -t string -d 0 -e json

```



## Contributors

  

* [Baoyi Chen](https://github.com/leonchen83)

* [Jintao Zhang](https://github.com/tao12345666333)

* [Maz Ahmadi](https://github.com/cmdshepard)

* [Anish Karandikar](https://github.com/anishkny)

* [Air](https://github.com/air3ijai)

* [Raghu Nandan B S](https://github.com/raghu-nandan-bs)

* Special thanks to [Kater Technologies](https://www.kater.com/)

  

## Consulting



Commercial support for `redis-rdb-cli` is available. The following services are currently available:

* Onsite consulting. $10,000 per day

* Onsite training. $10,000 per day



You may also contact `Baoyi Chen` directly, mail to [[email protected]](mailto:[email protected]).



## Supported by 宁文君



27 January 2023, A sad day that I lost my mother 宁文君, She was encouraging and supporting me in developing this tool. Every time a company uses this tool, she got excited like a child and encouraged me to keep going.

Without her I couldn't have maintained this tool for so many years. Even I didn't achieve much but she is still proud of me, R.I.P and hope God bless her.



## Supported by IntelliJ IDEA



[IntelliJ IDEA](https://www.jetbrains.com/?from=redis-rdb-cli) is a Java integrated development environment (IDE) for developing computer software.  

It is developed by JetBrains (formerly known as IntelliJ), and is available as an Apache 2 Licensed community edition,  

and in a proprietary commercial edition. Both can be used for commercial development.