{"id":13527958,"url":"https://github.com/sripathikrishnan/redis-rdb-tools","last_synced_at":"2025-05-13T21:09:57.266Z","repository":{"id":2686298,"uuid":"3678981","full_name":"sripathikrishnan/redis-rdb-tools","owner":"sripathikrishnan","description":"Parse Redis dump.rdb files, Analyze Memory, and Export Data to JSON","archived":false,"fork":false,"pushed_at":"2023-04-03T10:52:51.000Z","size":520,"stargazers_count":5152,"open_issues_count":97,"forks_count":747,"subscribers_count":195,"default_branch":"master","last_synced_at":"2025-04-28T14:57:02.398Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://rdbtools.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sripathikrishnan.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2012-03-10T11:09:12.000Z","updated_at":"2025-04-26T15:36:48.000Z","dependencies_parsed_at":"2023-07-05T20:16:52.206Z","dependency_job_id":null,"html_url":"https://github.com/sripathikrishnan/redis-rdb-tools","commit_stats":{"total_commits":173,"total_committers":24,"mean_commits":7.208333333333333,"dds":0.5722543352601156,"last_synced_commit":"548b11ec3c81a603f5b321228d07a61a0b940159"},"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sripathikrishnan%2Fredis-rdb-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sripathikrishnan%2Fredis-rdb-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sripathikrishnan%2Fredis-rdb-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sripathikrishnan%2Fredis-rdb-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sripathikrishnan","download_url":"https://codeload.github.com/sripathikrishnan/redis-rdb-tools/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254029002,"owners_count":22002283,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T06:02:08.318Z","updated_at":"2025-05-13T21:09:52.238Z","avatar_url":"https://github.com/sripathikrishnan.png","language":"Python","funding_links":[],"categories":["Development","Python","Web 后端","Tools/Repositories","Higher level libraries and tools"],"sub_categories":["Time Series"],"readme":"# Parse Redis dump.rdb files, Analyze Memory, and Export Data to JSON #\n\nRdbtools is a parser for Redis' dump.rdb files. The parser generates events similar to an xml sax parser, and is very efficient memory wise.\n\nIn addition, rdbtools provides utilities to :\n\n 1.  Generate a Memory Report of your data across all databases and keys\n 2.  Convert dump files to JSON\n 3.  Compare two dump files using standard diff tools\n\nRdbtools is written in Python, though there are similar projects in other languages. See [FAQs](https://github.com/sripathikrishnan/redis-rdb-tools/wiki/FAQs) for more information.\n\nSee https://rdbtools.com for a gui to administer redis, commercial support, and other enterprise features.\n\n## Installing rdbtools ##\n\nPre-Requisites : \n\n1. python-lzf is optional but highly recommended to speed up parsing.\n1. redis-py is optional and only needed to run test cases.\n\nTo install from PyPI (recommended) :\n\n    pip install rdbtools python-lzf\n    \nTo install from source : \n\n    git clone https://github.com/sripathikrishnan/redis-rdb-tools\n    cd redis-rdb-tools\n    sudo python setup.py install\n\n# Command line usage examples\n\nEvery run of RDB Tool requires to specify a command to indicate what should be done with the parsed RDB data.\nValid commands are: json, diff, justkeys, justkeyvals and protocol.\n\nJSON from a two database dump:\n\n    \u003e rdb --command json /var/redis/6379/dump.rdb\n\n    [{\n    \"user003\":{\"fname\":\"Ron\",\"sname\":\"Bumquist\"},\n    \"lizards\":[\"Bush anole\",\"Jackson's chameleon\",\"Komodo dragon\",\"Ground agama\",\"Bearded dragon\"],\n    \"user001\":{\"fname\":\"Raoul\",\"sname\":\"Duke\"},\n    \"user002\":{\"fname\":\"Gonzo\",\"sname\":\"Dr\"},\n    \"user_list\":[\"user003\",\"user002\",\"user001\"]},{\n    \"baloon\":{\"helium\":\"birthdays\",\"medical\":\"angioplasty\",\"weather\":\"meteorology\"},\n    \"armadillo\":[\"chacoan naked-tailed\",\"giant\",\"Andean hairy\",\"nine-banded\",\"pink fairy\"],\n    \"aroma\":{\"pungent\":\"vinegar\",\"putrid\":\"rotten eggs\",\"floral\":\"roses\"}}]\n\n## Filter parsed output\n\nOnly process keys that match the regex, and only print key and values:\n\n    \u003e rdb --command justkeyvals --key \"user.*\" /var/redis/6379/dump.rdb\n\n    user003 fname Ron,sname Bumquist,\n    user001 fname Raoul,sname Duke,\n    user002 fname Gonzo,sname Dr,\n    user_list user003,user002,user001\n    \nOnly process hashes starting with \"a\", in database 2:\n\n    \u003e rdb -c json --db 2 --type hash --key \"a.*\" /var/redis/6379/dump.rdb\n    \n    [{},{\n    \"aroma\":{\"pungent\":\"vinegar\",\"putrid\":\"rotten eggs\",\"floral\":\"roses\"}}]\n\n## Converting dump files to JSON ##\n\nThe `json` command output is UTF-8 encoded JSON.\nBy default, the callback try to parse RDB data using UTF-8 and escape non 'ASCII printable' characters with the `\\U` notation, or non UTF-8 parsable bytes with `\\x`.\nAttempting to decode RDB data can lead to binary data curroption, this can be avoided by using the `--escape raw` option.\nAnother option, is to use `-e base64` for Base64 encoding of binary data.\n\n\nParse the dump file and print the JSON on standard output:\n\n    \u003e rdb -c json /var/redis/6379/dump.rdb\n\n    [{\n    \"Citat\":[\"B\\u00e4ttre sent \\u00e4n aldrig\",\"Bra karl reder sig sj\\u00e4lv\",\"Man ska inte k\\u00f6pa grisen i s\\u00e4cken\"],\n    \"bin_data\":\"\\\\xFE\\u0000\\u00e2\\\\xF2\"}]\n\nParse the dump file to raw bytes and print the JSON on standard output:\n\n    \u003e rdb -c json /var/redis/6379/dump.rdb --escape raw\n\n    [{\n    \"Citat\":[\"B\\u00c3\\u00a4ttre sent \\u00c3\\u00a4n aldrig\",\"Bra karl reder sig sj\\u00c3\\u00a4lv\",\"Man ska inte k\\u00c3\\u00b6pa grisen i s\\u00c3\\u00a4cken\"],\n    \"bin_data\":\"\\u00fe\\u0000\\u00c3\\u00a2\\u00f2\"}]\n\n## Generate Memory Report ##\n\nRunning with the  `-c memory` generates a CSV report with the approximate memory used by that key. `--bytes C` and `'--largest N` can be used to limit output to keys larger than C bytes, or the N largest keys.\n\n    \u003e rdb -c memory /var/redis/6379/dump.rdb --bytes 128 -f memory.csv\n\t\u003e cat memory.csv\n\n\tdatabase,type,key,size_in_bytes,encoding,num_elements,len_largest_element\n\t0,list,lizards,241,quicklist,5,19\n\t0,list,user_list,190,quicklist,3,7\n\t2,hash,baloon,138,ziplist,3,11\n\t2,list,armadillo,231,quicklist,5,20\n\t2,hash,aroma,129,ziplist,3,11\n\n\nThe generated CSV has the following columns - Database Number, Data Type, Key, Memory Used in bytes and RDB Encoding type.\nMemory usage includes the key, the value and any other overheads.\n\nNote that the memory usage is approximate. In general, the actual memory used will be slightly higher than what is reported.\n\nYou can filter the report on keys or database number or data type.\n\nThe memory report should help you detect memory leaks caused by your application logic. It will also help you optimize Redis memory usage. \n\n## Find Memory used by a Single Key ##\n\nSometimes you just want to find the memory used by a particular key, and running the entire memory report on the dump file is time consuming.\n\nIn such cases, you can use the `redis-memory-for-key` command:\n\n    \u003e redis-memory-for-key person:1\n    \n    \u003e redis-memory-for-key -s localhost -p 6379 -a mypassword person:1\n\n    Key \t\t\tperson:1\n    Bytes\t\t\t\t111\n    Type\t\t\t\thash\n    Encoding\t\t\tziplist\n    Number of Elements\t\t2\n    Length of Largest Element\t8\n\nNOTE : \n\n1. This was added to redis-rdb-tools version 0.1.3\n2. This command depends [redis-py](https://github.com/andymccurdy/redis-py) package\n\n## Comparing RDB files ##\n\nFirst, use the --command diff option, and pipe the output to standard sort utility\n\n    \u003e rdb --command diff /var/redis/6379/dump1.rdb | sort \u003e dump1.txt\n    \u003e rdb --command diff /var/redis/6379/dump2.rdb | sort \u003e dump2.txt\n    \nThen, run your favourite diff program\n\n    \u003e kdiff3 dump1.txt dump2.txt\n\nTo limit the size of the files, you can filter on keys using the `--key` option\n\n## Emitting Redis Protocol ##\n\nYou can convert RDB file into a stream of [redis protocol](http://redis.io/topics/protocol) using the `protocol` command.\n\n    \u003e rdb -c protocol /var/redis/6379/dump.rdb\n    \n    *4\n    $4\n    HSET\n    $9\n    users:123\n    $9\n    firstname\n    $8\n    Sripathi\n\nYou can pipe the output to netcat and re-import a subset of the data.\nFor example, if you want to shard your data into two redis instances, you can use the --key flag to select a subset of data,\nand then pipe the output to a running redis instance to load that data.\nRead [Redis Mass Insert](http://redis.io/topics/mass-insert) for more information on this.\n\nWhen printing protocol output, the `--escape` option can be used with `printable` or `utf8` to avoid non printable/control characters.\n\nBy default, expire times are emitted verbatim if they are present in the rdb file, causing all keys that expire in the past to be removed.\nIf this behaviour is unwanted the `-x/--no-expire` option will ignore all key expiry commands.\n\nOtherwise you may want to set an expiry time in the future with `-a/--amend-expire` option which adds an integer number of seconds to the expiry time of each key which is already set to expire.\nThis will not change keys that do not already have an expiry set.\n\n# Using the Parser ##\n\n    from rdbtools import RdbParser, RdbCallback\n    from rdbtools.encodehelpers import bytes_to_unicode\n\n    class MyCallback(RdbCallback):\n        ''' Simple example to show how callback works.\n            See RdbCallback for all available callback methods.\n            See JsonCallback for a concrete example\n        '''\n\n        def __init__(self):\n            super(MyCallback, self).__init__(string_escape=None)\n\n        def encode_key(self, key):\n            return bytes_to_unicode(key, self._escape, skip_printable=True)\n\n        def encode_value(self, val):\n            return bytes_to_unicode(val, self._escape)\n\n        def set(self, key, value, expiry, info):\n            print('%s = %s' % (self.encode_key(key), self.encode_value(value)))\n\n        def hset(self, key, field, value):\n            print('%s.%s = %s' % (self.encode_key(key), self.encode_key(field), self.encode_value(value)))\n\n        def sadd(self, key, member):\n            print('%s has {%s}' % (self.encode_key(key), self.encode_value(member)))\n\n        def rpush(self, key, value):\n            print('%s has [%s]' % (self.encode_key(key), self.encode_value(value)))\n\n        def zadd(self, key, score, member):\n            print('%s has {%s : %s}' % (str(key), str(member), str(score)))\n\n\n    callback = MyCallback()\n    parser = RdbParser(callback)\n    parser.parse('/var/redis/6379/dump.rdb')\n\n## Other Pages\n\n 1. [Frequently Asked Questions](https://github.com/sripathikrishnan/redis-rdb-tools/wiki/FAQs)\n 2. [Redis Dump File Specification](https://github.com/sripathikrishnan/redis-rdb-tools/wiki/Redis-RDB-Dump-File-Format)\n 3. [Redis Dump File Version History](https://github.com/sripathikrishnan/redis-rdb-tools/blob/master/docs/RDB_Version_History.textile) - this also has notes on converting a dump file to an older version.\n\n## License\n\nrdbtools is licensed under the MIT License. See [LICENSE](https://github.com/sripathikrishnan/redis-rdb-tools/blob/master/LICENSE)\n\n## Maintained By \n\nSripathi Krishnan : @srithedabbler\n\n## Credits\n\n 1. [Didier Spézia](https://twitter.com/#!/didier_06)\n 2. [Yoav Steinberg](https://github.com/yoav-steinberg)\n 3. [Daniel Mezzatto](https://github.com/mezzatto)\n 4. [Carlo Cabanilla](https://github.com/clofresh)\n 5. [Josep M. Pujol](https://github.com/solso)\n 6. [Charles Gordon](https://github.com/cgordon)\n 7. [Justin Poliey](https://github.com/jdp)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsripathikrishnan%2Fredis-rdb-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsripathikrishnan%2Fredis-rdb-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsripathikrishnan%2Fredis-rdb-tools/lists"}