{"id":13492019,"url":"https://github.com/gamenet/redis-memory-analyzer","last_synced_at":"2025-05-15T16:02:17.964Z","repository":{"id":38355973,"uuid":"50424582","full_name":"gamenet/redis-memory-analyzer","owner":"gamenet","description":"Redis memory profiler to find the RAM bottlenecks throw scaning key space in real time and aggregate RAM usage statistic by patterns.","archived":false,"fork":false,"pushed_at":"2024-01-25T03:48:05.000Z","size":120,"stargazers_count":784,"open_issues_count":24,"forks_count":81,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-05-08T04:19:41.659Z","etag":null,"topics":["bottleneck-testing","memory-management","profiling","redis"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gamenet.png","metadata":{"files":{"readme":"README.rst","changelog":"CHANGES","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-01-26T11:32:04.000Z","updated_at":"2025-04-22T06:40:38.000Z","dependencies_parsed_at":"2022-08-09T03:01:54.408Z","dependency_job_id":"a2015711-41ab-4bf0-99d1-f8656f3dbb56","html_url":"https://github.com/gamenet/redis-memory-analyzer","commit_stats":{"total_commits":94,"total_committers":13,"mean_commits":7.230769230769231,"dds":"0.15957446808510634","last_synced_commit":"dd1e74994ae1d258b2bfca9cd226acb05b5516f5"},"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gamenet%2Fredis-memory-analyzer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gamenet%2Fredis-memory-analyzer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gamenet%2Fredis-memory-analyzer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gamenet%2Fredis-memory-analyzer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gamenet","download_url":"https://codeload.github.com/gamenet/redis-memory-analyzer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254374388,"owners_count":22060609,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bottleneck-testing","memory-management","profiling","redis"],"created_at":"2024-07-31T19:01:02.400Z","updated_at":"2025-05-15T16:02:17.910Z","avatar_url":"https://github.com/gamenet.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"|PyPI version| |Build Status| |Code Health| |GitHub license|\n\nRedis Memory Analyzer\n=====================\n\nRMA is a console tool to scan Redis key space in real time and aggregate\nmemory usage statistic by key patterns. You may use this tools without\nmaintenance on production servers. You can scanning by all or selected\nRedis types such as \"string\", \"hash\", \"list\", \"set\", \"zset\" and use\nmatching pattern as you like. RMA try to discern key names by patterns,\nfor example if you have keys like 'user:100' and 'user:101' application\nwould pick out common pattern 'user:\\*' in output so you can analyze\nmost memory distressed data in your instance.\n\nInstalling rma\n~~~~~~~~~~~~~~\n\nPre-Requisites :\n\n1. python \u003e= 3.5 and pip.\n2. redis-py.\n\nTo install from PyPI (recommended) :\n\n::\n\n    pip install rma\n\nTo install from source :\n\n::\n\n    pip install git+https://github.com/gamenet/redis-memory-analyzer@v0.2.0\n\nRunning\n-------\n\nAfter install used it from console:\n\n::\n\n    \u003erma --help\n    usage: rma [-h] [-s HOST] [-p PORT] [-a PASSWORD] [-d DB] [-m MATCH] [-l LIMIT]\n               [-b BEHAVIOUR] [-t TYPES]\n\n    RMA is used to scan Redis key space in and aggregate memory usage statistic by\n    key patterns.\n\n    optional arguments:\n      -h, --help                 show this help message and exit\n      -s, --server HOST          Redis Server hostname. Defaults to 127.0.0.1\n      -p, --port PORT            Redis Server port. Defaults to 6379\n      -a, --password PASSWORD    Password to use when connecting to the server\n      -d, --db DB                Database number, defaults to 0\n      -m, --match MATCH          Keys pattern to match\n      -l, --limit LIMIT          Get max key matched by pattern\n      -b, --behaviour BEHAVIOUR  Specify application working mode. Allowed values\n                                 are all, scanner, ram, global\n      -t, --type TYPES           Data types to include. Possible values are string,\n                                 hash, list, set. Multiple types can be provided. If\n                                 not specified, all data types will be returned.\n                                 Allowed values arestring, hash, list, set, zset\n      -f --format TYPE           Output type format: json or text (by default)\n      -x --separator SEPARATOR   Specify namespace separator. Default is ':'\n\nIf you have large database try running first with ``--limit`` option to\nrun first limited amount of keys. Also run with ``--types`` to limit\nonly specified Redis types in large database. Not this tool has\nperformance issues - call encoding for individual keys instead if batch\nqueue with LUA (like in scanner does). So this option may be very\nuseful. You can choose what kind of data would be aggregated from Redis\nnode using ``-b (--behaviour)`` option as console argument. Supported\nbehaviours are 'global', 'scanner', 'ram' and 'all'.\n\nInternals\n---------\n\nRMA shows statistics separated by types. All works in application\nseparated by few steps:\n\n1. Load type and encoding for each key matched by given pattern with Lua\n   scripting in batch mode. ``SCAN`` used to iterate keys from Redis key\n   db.\n2. Separate keys by types and match patterns.\n3. Run behaviours and rules for given data set.\n4. Output result with given reported (now only TextReported implemented)\n\nGlobal output ('global' behaviour)\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThe global data is some Redis server statistics which helps you to\nunderstand other data from this tools:\n\n::\n\n    | Stat                             | Value          |\n    |:---------------------------------|:---------------|\n    | Total keys in db                 | 28979          |\n    | RedisDB key space overhead       | 790528         |\n    | Used `set-max-intset-entries`    | 512            |\n    | ....                             | ...            |\n    | Info `total_system_memory`       | 3190095872     |\n    | ....                             | ...            |\n\nThe one of interesting things here is \"RedisDB key space overhead\". The\namount of memory used Redis to store key space data. If you have lots of\nkeys in your Redis instance this actually shows your overhead for this.\nKeep in mind that part of data such as total keys in db or key space overhead\nshows data for selected db. But statistics started with ``Info`` or ``Config``\nkeywords is server based.\n\nKey types ('scanner' behaviour)\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThis table helps then you do not know actually that kind of keys stored\nin your Redis database. For example then DevOps or system administrator\nwant to understand what kind of keys stored in Redis instance. Which\ndata structure is most used in system. This also helps if you are new to\nsome big project - this kind of ``SHOW ALL TABLES`` request :)\n\n::\n\n    | Match                 |   Count | Type   | %      |\n    |:----------------------|--------:|:-------|:-------|\n    | job:*                 |    5254 | hash   | 18.13% |\n    | game:privacy:*        |    2675 | hash   | 9.23%  |\n    | user:*                |    1890 | hash   | 6.52%  |\n    | group:*               |    1885 | set    | 6.50%  |\n\nData related output ('ram' behaviour)\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nAll output separated by keys and values statistics. This division is\nused because: 1. Keys of any type in Redis actually stored in RedisDB\ninternal data structure based on dict (more about this on\n`RedisPlanet \u003chttp://redisplanet.com/\u003e`__). 2. This type of data\nspecially important in Redis instances with lots of keys.\n\n::\n\n    | Match                         | Count | Useful |   Real | Ratio | Encoding                     | Min | Max |   Avg |\n    |:------------------------------|------:|-------:|-------:|------:|:-----------------------------|----:|----:|------:|\n    | event:data:*                  |  1198 |  17970 |  76672 |  4.27 | embstr [50.0%] / raw [50.0%] |  15 |  71 | 41.20 |\n    | mm:urllist:*                  |   524 |   7648 |  33536 |  4.38 | embstr [100.0%]              |  12 |  15 | 14.60 |\n    | Provider:ParallelForm:*:*:*:* |   459 |  43051 |  66096 |  1.54 | raw [100.0%]                 |  92 |  94 | 93.79 |\n    | user:spamblocked:dialy:post:* |    48 |   2208 |   4608 |  2.09 | raw [100.0%]                 |  46 |  46 | 46.00 |\n    | ...                           |   ... |    ... |    ... |   ... |                          ... | ... | ... |   ... |\n    | Total:                        |  2432 |  80493 | 200528 |  0.00 |                              |   0 |   0 |  0.00 |\n\nSo you can see count of keys matching given pattern, expected (by\ndeveloper) and real memory with taking into account the Redis data\nstructures and allocator overhead. Ratio and encoding distribution\nmin/max/avg len of key. For example in sample above keys some keys\nencoded as ``raw`` (sds string). Each sds encoded string:\n\n1. Has useful payload\n2. Has sds string header overhead\n3. Has ``redis object`` overhead\n4. The Redis implementation during memory allocation would be\n   align(redis object) + align(sds header + useful payload)\n\nIn x64 instance of Redis key ``event:data:f1wFFqgqqwgeg`` (24 byte len)\nactually would use 24 bytes payload bytes, 9 bytes sds header and 32\nbytes in r\\_obj (``redis object``). So we may think this would use 65\nbytes. But after jemalloc allocator align it this 24 byte (65 byte data\nwith Redis internals) would use 80 bytes - in ~3,3 more times as you\nexpect (\\`Ratio\\`\\` value in table).\n\nNot we can look at values. All values output individual by Redis type.\nEach type has they own limitations so here is some common data for each\ntype and some unique. The ``strings`` data type value same as keys\noutput above. The only one difference is ``Free`` field which shows\nunused but allocated memory by SDS strings in ``raw`` encoding.\n\nSo for example look at output for ``HASH`` values:\n\n::\n\n    | Match                 | Count | Avg field count | Key mem |   Real | Ratio | Value mem |   Real |    Ratio |   System | Encoding         | Total mem |  Total aligned |\n    |:----------------------|------:|----------------:|--------:|-------:|------:|----------:|-------:|---------:|---------:|:-----------------|----------:|---------------:|\n    | job:*                 |  5254 |            9.00 |  299485 | 619988 |  2.07 |    685451 | 942984 |     1.38 |  1345024 | ziplist [100.0%] |    984936 |        2907996 |\n    | LIKE:*                |  1890 |            1.02 |    5744 |  30262 |  5.27 |      1932 |  15432 |     7.99 |    91344 | ziplist [100.0%] |      7676 |         137038 |\n    | game:*:count:*        |  1231 |            1.00 |    7386 |  19696 |  2.67 |      1234 |   9848 |     7.98 |    59088 | ziplist [100.0%] |      8620 |          88632 |\n    | LIKE:game:like:*      |  1207 |            1.00 |    3621 |  19312 |  5.33 |      1210 |   9656 |     7.98 |    57936 | ziplist [100.0%] |      4831 |          86904 |\n    | integration:privacy:* |   530 |            3.00 |   20140 |  33920 |  1.68 |         0 |  25440 | 25440.00 |    42400 | ziplist [100.0%] |     20140 |         101760 |\n\nLook at ``job:*`` hashes. This instance contains 5254 such keys with 9\nfields each. Looks like this data has regular structure like python\ntuple. This means you can change data structure of this data from Redis\n``hash`` to ``list`` and use 2 times less memory then now. Why do this?\nNow you ``job:*`` hash uses ~3,2 times more memory as you developers\nexpect.\n\nWhy doesn't reported memory match actual memory used?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThe memory reported by this tool is approximate. In general, the\nreported memory should be within 10% of what is reported by\n`info \u003chttp://redis.io/commands/info\u003e`__.\n\nAlso note that the tool does not (and cannot) account for the following:\n- Memory used by allocator metadata (it is actually not possible without ``c``)\n- Memory used for pub/sub (no any commands in Redis for that)\n- Redis process internals (like shared objects)\n\nKnown issues\n~~~~~~~~~~~~\n\n1. ``Skiplist`` (``zset`` actually) encoding actually not realized.\n2. ``Quicklist`` now calculated as ``ziplist``.\n3. SDS strings from redis 3.2 (optimized headers) not implemented. Now\n   used fixed 9 bytes header.\n\nWhats next?\n~~~~~~~~~~~\n\nNow we use this tools as awesome helper. We most used data structures in\nour Redis instances is ``hash`` and ``list``. After upgradings our\nservers to Redis 3.2.x planning to fix known issues. Be glad to know\nthat are you think about this tool. In my dreams this tools should used\nas ``redis-lint`` tools which can say you\n``Hey, change this from this to this and save 30% of RAM``,\n``Hey, you are using PHP serializer for strings - change to msgpack and save 15% of RAM``\nand so on.\n\nLicense\n-------\n\nThis application was developed for using in\n`GameNet \u003chttps://gamenet.ru/\u003e`__ project as part of Redis memory\noptimizations and analise. RMA is licensed under the MIT License. See\n`LICENSE \u003chttps://github.com/gamenet/redis-memory-analyzer/blob/master/LICENSE\u003e`__\n\n.. |PyPI version| image:: https://badge.fury.io/py/rma.svg\n   :target: https://badge.fury.io/py/rma\n.. |Build Status| image:: https://travis-ci.org/gamenet/redis-memory-analyzer.svg?branch=master\n   :target: https://travis-ci.org/gamenet/redis-memory-analyzer\n.. |Code Health| image:: https://landscape.io/github/gamenet/redis-memory-analyzer/master/landscape.svg?style=flat-square\n   :target: https://landscape.io/github/gamenet/redis-memory-analyzer/master\n.. |GitHub license| image:: https://img.shields.io/badge/license-MIT-blue.svg\n   :target: https://raw.githubusercontent.com/gamenet/redis-memory-analyzer/master/LICENSE\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgamenet%2Fredis-memory-analyzer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgamenet%2Fredis-memory-analyzer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgamenet%2Fredis-memory-analyzer/lists"}