{"id":13730652,"url":"https://github.com/antirez/rax","last_synced_at":"2025-05-16T06:06:55.980Z","repository":{"id":19587660,"uuid":"83552012","full_name":"antirez/rax","owner":"antirez","description":"A radix tree implementation in ANSI C","archived":false,"fork":false,"pushed_at":"2023-11-26T12:39:33.000Z","size":273,"stargazers_count":1144,"open_issues_count":24,"forks_count":168,"subscribers_count":46,"default_branch":"master","last_synced_at":"2025-04-08T16:04:32.703Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/antirez.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-03-01T12:27:58.000Z","updated_at":"2025-04-07T11:37:03.000Z","dependencies_parsed_at":"2024-09-20T17:42:44.359Z","dependency_job_id":"abc97f79-9109-4688-8eda-0a47a989eda8","html_url":"https://github.com/antirez/rax","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antirez%2Frax","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antirez%2Frax/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antirez%2Frax/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antirez%2Frax/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/antirez","download_url":"https://codeload.github.com/antirez/rax/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254478190,"owners_count":22077676,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T02:01:17.666Z","updated_at":"2025-05-16T06:06:50.954Z","avatar_url":"https://github.com/antirez.png","language":"C","readme":"# Rax, an ANSI C radix tree implementation\n\nRax is a radix tree implementation initially written to be used in a specific\nplace of Redis in order to solve a performance problem, but immediately\nconverted into a stand alone project to make it reusable for Redis itself, outside the initial intended application, and for other projects as well.\n\nThe primary goal was to find a suitable balance between performances\nand memory usage, while providing a fully featured implementation of radix trees\nthat can cope with many different requirements.\n\nDuring the development of this library, while getting more and more excited\nabout how practical and applicable radix trees are, I was very surprised to\nsee how hard it is to write a robust implementation, especially of a fully\nfeatured radix tree with a flexible iterator. A lot of things can go wrong\nin node splitting, merging, and various edge cases. For this reason a major\ngoal of the project is to provide a stable and battle tested implementation\nfor people to use and in order to share bug fixes. The project relies a lot\non fuzz testing techniques in order to explore not just all the lines of code\nthe project is composed of, but a large amount of possible states.\n\nRax is an open source project, released under the BSD two clause license.\n\nMajor features:\n\n* Memory conscious:\n    + Packed nodes representation.\n    + Able to avoid storing a NULL pointer inside the node if the key is set to NULL (there is an `isnull` bit in the node header).\n    + Lack of parent node reference. A stack is used instead when needed.\n* Fast lookups:\n    + Edges are stored as arrays of bytes directly in the parent node, no need to access non useful children while trying to find a match. This translates into less cache misses compared to other implementations.\n    + Cache line friendly scanning of the correct child by storing edges as two separated arrays: an array of edge chars and one of edge pointers.\n* Complete implementation:\n    + Deletion with nodes re-compression as needed.\n    + Iterators (including a way to use iterators while the tree is modified).\n    + Random walk iteration.\n    + Ability to report and resist out of memory: if malloc() returns NULL the API can report an out of memory error and always leave the tree in a consistent state.\n* Readable and fixable implementation:\n    + All complex parts are commented with algorithms details.\n    + Debugging messages can be enabled to understand what the implementation is doing when calling a given function.\n    + Ability to print the radix tree nodes representation as ASCII art.\n* Portable implementation:\n    + Never does unaligned accesses to memory.\n    + Written in ANSI C99, no extensions used.\n* Extensive code and possible states test coverage using fuzz testing.\n    + Testing relies a lot on fuzzing in order to explore non trivial states.\n    + Implementation of the dictionary and iterator compared with behavior-equivalent implementations of simple hash tables and sorted arrays, generating random data and checking if the two implementations results match.\n    + Out of memory condition tests. The implementation is fuzzed with a special allocator returning `NULL` at random. The resulting radix tree is tested for consistency. Redis, the primary target of this implementation, does not use this feature, but the ability to handle OOM may make this implementation useful where the ability to survive OOMs is needed.\n    + Part of Redis: the implementation is stressed significantly in the real world.\n\nThe layout of a node is as follows. In the example, a node which represents\na key (so has a data pointer associated), has three children `x`, `y`, `z`.\nEvery space represents a byte in the diagram.\n\n    +----+---+--------+--------+--------+--------+\n    |HDR |xyz| x-ptr  | y-ptr  | z-ptr  |dataptr |\n    +----+---+--------+--------+--------+--------+\n\nThe header `HDR` is actually a bitfield with the following fields:\n\n    uint32_t iskey:1;     /* Does this node contain a key? */\n    uint32_t isnull:1;    /* Associated value is NULL (don't store it). */\n    uint32_t iscompr:1;   /* Node is compressed. */\n    uint32_t size:29;     /* Number of children, or compressed string len. */\n\nCompressed nodes represent chains of nodes that are not keys and have\nexactly a single child, so instead of storing:\n\n    A -\u003e B -\u003e C -\u003e [some other node]\n\nWe store a compressed node in the form:\n\n    \"ABC\" -\u003e [some other node]\n\nThe layout of a compressed node is:\n\n    +----+---+--------+\n    |HDR |ABC|chld-ptr|\n    +----+---+--------+\n\n# Basic API\n\nThe basic API is a trivial dictionary where you can add or remove elements.\nThe only notable difference is that the insert and remove APIs also accept\nan optional argument in order to return, by reference, the old value stored\nat a key when it is updated (on insert) or removed.\n\n## Creating a radix tree and adding a key\n\nA new radix tree is created with:\n\n    rax *rt = raxNew();\n\nIn order to insert a new key, the following function is used:\n\n    int raxInsert(rax *rax, unsigned char *s, size_t len, void *data,\n                  void **old);\n\nExample usage:\n\n    raxInsert(rt,(unsigned char*)\"mykey\",5,some_void_value,NULL);\n\nThe function returns 1 if the key was inserted correctly, or 0 if the key\nwas already in the radix tree: in this case, the value is updated. The\nvalue of 0 is also returned on out of memory, however in that case\n`errno` is set to `ENOMEM`.\n\nIf the associated value `data` is NULL, the node where the key\nis stored does not use additional memory to store the NULL value, so\ndictionaries composed of just keys are memory efficient if you use\nNULL as associated value.\n\nNote that keys are unsigned arrays of chars and you need to specify the\nlength: Rax is binary safe, so the key can be anything.\n\nThe insertion function is also available in a variant that will not\noverwrite the existing key value if any:\n\n    int raxTryInsert(rax *rax, unsigned char *s, size_t len, void *data,\n                     void **old);\n\nThe function is exactly the same as raxInsert(), however if the key\nexists the function returns 0 (like raxInsert) without touching the\nold value. The old value can be still returned via the 'old' pointer\nby reference.\n\n## Key lookup\n\nThe lookup function is the following:\n\n    void *raxFind(rax *rax, unsigned char *s, size_t len);\n\nThis function returns the special value `raxNotFound` if the key you\nare trying to access is not there, so an example usage is the following:\n\n    void *data = raxFind(rax,mykey,mykey_len);\n    if (data == raxNotFound) return;\n    printf(\"Key value is %p\\n\", data);\n\nraxFind() is a read only function so no out of memory conditions are\npossible, the function never fails.\n\n## Deleting keys\n\nDeleting the key is as you could imagine it, but with the ability to\nreturn by reference the value associated to the key we are about to\ndelete:\n\n    int raxRemove(rax *rax, unsigned char *s, size_t len, void **old);\n\nThe function returns 1 if the key gets deleted, or 0 if the key was not\nthere. This function also does not fail for out of memory, however if\nthere is an out of memory condition while a key is being deleted, the\nresulting tree nodes may not get re-compressed even if possible: the radix\ntree may be less efficiently encoded in this case.\n\nThe `old` argument is optional, if passed will be set to the key associated\nvalue if the function successfully finds and removes the key.\n\n# Iterators\n\nThe Rax key space is ordered lexicographically, using the value of the\nbytes the keys are composed of in order to decide which key is greater\nbetween two keys. If the prefix is the same, the longer key is considered\nto be greater.\n\nRax iterators allow to seek a given element based on different operators\nand then to navigate the key space calling `raxNext()` and `raxPrev()`.\n\n## Basic iterator usage\n\nIterators are normally declared as local variables allocated on the stack,\nand then initialized with the `raxStart` function:\n\n    raxIterator iter;\n    raxStart(\u0026iter, rt); // Note that 'rt' is the radix tree pointer.\n\nThe function `raxStart` never fails and returns no value.\nOnce an iterator is initialized, it can be sought (sought is the past tens\nof 'seek', which is not 'seeked', in case you wonder) in order to start\nthe iteration from the specified position. For this goal, the function\n`raxSeek` is used:\n\n    int raxSeek(raxIterator *it, unsigned char *ele, size_t len, const char *op);\n\nFor instance one may want to seek the first element greater or equal to the\nkey `\"foo\"`:\n\n    raxSeek(\u0026iter,\"\u003e=\",(unsigned char*)\"foo\",3);\n\nThe function raxSeek() returns 1 on success, or 0 on failure. Possible failures are:\n\n1. An invalid operator was passed as last argument.\n2. An out of memory condition happened while seeking the iterator.\n\nOnce the iterator is sought, it is possible to iterate using the function\n`raxNext` and `raxPrev` as in the following example:\n\n    while(raxNext(\u0026iter)) {\n        printf(\"Key: %.*s\\n\", (int)iter.key_len, (char*)iter.key);\n    }\n\nThe function `raxNext` returns elements starting from the element sought\nwith `raxSeek`, till the final element of the tree. When there are no more\nelements, 0 is returned, otherwise the function returns 1. However the function\nmay return 0 when an out of memory condition happens as well: while it attempts\nto always use the stack, if the tree depth is large or the keys are big the\niterator starts to use heap allocated memory.\n\nThe function `raxPrev` works exactly in the same way, but will move towards\nthe first element of the radix tree instead of moving towards the last\nelement.\n\n# Releasing iterators\n\nAn iterator can be used multiple times, and can be sought again and again\nusing `raxSeek` without any need to call `raxStart` again. However, when the\niterator is not going to be used again, its memory must be reclaimed\nwith the following call:\n\n    raxStop(\u0026iter);\n\nNote that even if you do not call `raxStop`, most of the times you'll not\ndetect any memory leak, but this is just a side effect of how the\nRax implementation works: most of the times it will try to use the stack\nallocated data structures. However for deep trees or large keys, heap memory\nwill be allocated, and failing to call `raxStop` will result into a memory\nleak.\n\n## Seek operators\n\nThe function `raxSeek` can seek different elements based on the operator.\nFor instance in the example above we used the following call:\n\n    raxSeek(\u0026iter,\"\u003e=\",(unsigned char*)\"foo\",3);\n\nIn order to seek the first element `\u003e=` to the string `\"foo\"`. However\nother operators are available. The first set are pretty obvious:\n\n* `==` seek the element exactly equal to the given one.\n* `\u003e` seek the element immediately greater than the given one.\n* `\u003e=` seek the element equal, or immediately greater than the given one.\n* `\u003c` seek the element immediately smaller than the given one.\n* `\u003c=` seek the element equal, or immediately smaller than the given one.\n* `^` seek the smallest element of the radix tree.\n* `$` seek the greatest element of the radix tree.\n\nWhen the last two operators, `^` or `$` are used, the key and key length\nargument passed are completely ignored since they are not relevant.\n\nNote how certain times the seek will be impossible, for example when the\nradix tree contains no elements or when we are asking for a seek that is\nnot possible, like in the following case:\n\n    raxSeek(\u0026iter,\"\u003e\",(unsigned char*)\"zzzzz\",5);\n\nWe may not have any element greater than `\"zzzzz\"`. In this case, what\nhappens is that the first call to `raxNext` or `raxPrev` will simply return\nzero, so no elements are iterated.\n\n## Iterator stop condition\n\nSometimes we want to iterate specific ranges, for example from AAA to BBB.\nIn order to do so, we could seek and get the next element. However we need\nto stop once the returned key is greater than BBB. The Rax library offers\nthe `raxCompare` function in order to avoid you need to code the same string\ncomparison function again and again based on the exact iteration you are\ndoing:\n\n    raxIterator iter;\n    raxStart(\u0026iter);\n    raxSeek(\u0026iter,\"\u003e=\",(unsigned char*)\"AAA\",3); // Seek the first element\n    while(raxNext(\u0026iter)) {\n        if (raxCompare(\u0026iter,\"\u003e\",(unsigned char*)\"BBB\",3)) break;\n        printf(\"Current key: %.*s\\n\", (int)iter.key_len,(char*)iter.key);\n    }\n    raxStop(\u0026iter);\n\nThe above code shows a complete range iterator just printing the keys\ntraversed by iterating.\n\nThe prototype of the `raxCompare` function is the following:\n\n    int raxCompare(raxIterator *iter, const char *op, unsigned char *key, size_t key_len);\n\nThe operators supported are `\u003e`, `\u003e=`, `\u003c`, `\u003c=`, `==`.\nThe function returns 1 if the current iterator key satisfies the operator\ncompared to the provided key, otherwise 0 is returned.\n\n## Checking for iterator EOF condition\n\nSometimes we want to know if the itereator is in EOF state before calling\nraxNext() or raxPrev(). The iterator EOF condition happens when there are\nno more elements to return via raxNext() or raxPrev() call, because either\nraxSeek() failed to seek the requested element, or because EOF was reached\nwhile navigating the tree with raxPrev() and raxNext() calls.\n\nThis condition can be tested with the following function that returns 1\nif EOF was reached:\n\n    int raxEOF(raxIterator *it);\n\n## Modifying the radix tree while iterating\n\nIn order to be efficient, the Rax iterator caches the exact node we are at,\nso that at the next iteration step, it can start from where it left.\nHowever an iterator has sufficient state in order to re-seek again\nin case the cached node pointers are no longer valid. This problem happens\nwhen we want to modify a radix tree during an iteration. A common pattern\nis, for instance, deleting all the elements that match a given condition.\n\nFortunately there is a very simple way to do this, and the efficiency cost\nis only paid as needed, that is, only when the tree is actually modified.\nThe solution consists of seeking the iterator again, with the current key,\nonce the tree is modified, like in the following example:\n\n    while(raxNext(\u0026iter,...)) {\n        if (raxRemove(rax,...)) {\n            raxSeek(\u0026iter,\"\u003e\",iter.key,iter.key_size);\n        }\n    }\n\nIn the above case we are iterating with `raxNext`, so we are going towards\nlexicographically greater elements. Every time we remove an element, what we\nneed to do is to seek it again using the current element and the `\u003e` seek\noperator: this way we'll move to the next element with a new state representing\nthe current radix tree (after the change).\n\nThe same idea can be used in different contexts, considering the following:\n\n* Iterators need to be sought again with `raxSeek` every time keys are added or removed while iterating.\n* The current iterator key is always valid to access via `iter.key_size` and `iter.key`, even after it was deleted from the radix tree.\n\n## Re-seeking iterators after EOF\n\nAfter iteration reaches an EOF condition since there are no more elements\nto return, because we reached one or the other end of the radix tree, the\nEOF condition is permanent, and even iterating in the reverse direction will\nnot produce any result.\n\nThe simplest way to continue the iteration, starting again from the last\nelement returned by the iterator, is simply to seek itself:\n\n    raxSeek(\u0026iter,iter.key,iter.key_len,\"==\");\n\nSo for example in order to write a command that prints all the elements\nof a radix tree from the first to the last, and later again from the last\nto the first, reusing the same iterator, it is possible to use the following\napproach:\n\n    raxSeek(\u0026iter,\"^\",NULL,0);\n    while(raxNext(\u0026iter,NULL,0,NULL))\n        printf(\"%.*s\\n\", (int)iter.key_len, (char*)iter.key);\n\n    raxSeek(\u0026iter,\"==\",iter.key,iter.key_len);\n    while(raxPrev(\u0026iter,NULL,0,NULL))\n        printf(\"%.*s\\n\", (int)iter.key_len, (char*)iter.key);\n\n## Random element selection\n\nTo extract a fair element from a radix tree so that every element is returned\nwith the same probability is not possible if we require that:\n\n1. The radix tree is not larger than expected (for example augmented with information that allows elements ranking).\n2. We want the operation to be fast, at worst logarithmic (so things like reservoir sampling are out since it's O(N)).\n\nHowever a random walk which is long enough, in trees that are more or less balanced, produces acceptable results, is fast, and eventually returns every possible element, even if not with the right probability.\n\nTo perform a random walk, just seek an iterator anywhere and call the\nfollowing function:\n\n    int raxRandomWalk(raxIterator *it, size_t steps);\n\nIf the number of steps is set to 0, the function will perform a number of\nrandom walk steps between 1 and two times the logarithm in base two of the\nnumber of elements inside the tree, which is often enough to get a decent\nresult. Otherwise, you may specify the exact number of steps to take.\n\n## Printing trees\n\nFor debugging purposes, or educational ones, it is possible to use the\nfollowing call in order to get an ASCII art representation of a radix tree\nand the nodes it is composed of:\n\n    raxShow(mytree);\n\nHowever note that this works well enough for trees with a few elements, but\nbecomes hard to read for very large trees.\n\nThe following is an example of the output raxShow() produces after adding\nthe specified keys and values:\n\n* alligator = (nil)\n* alien = 0x1\n* baloon = 0x2\n* chromodynamic = 0x3\n* romane = 0x4\n* romanus = 0x5\n* romulus = 0x6\n* rubens = 0x7\n* ruber = 0x8\n* rubicon = 0x9\n* rubicundus = 0xa\n* all = 0xb\n* rub = 0xc\n* ba = 0xd\n\n```\n[abcr]\n `-(a) [l] -\u003e [il]\n               `-(i) \"en\" -\u003e []=0x1\n               `-(l) \"igator\"=0xb -\u003e []=(nil)\n `-(b) [a] -\u003e \"loon\"=0xd -\u003e []=0x2\n `-(c) \"hromodynamic\" -\u003e []=0x3\n `-(r) [ou]\n        `-(o) [m] -\u003e [au]\n                      `-(a) [n] -\u003e [eu]\n                                    `-(e) []=0x4\n                                    `-(u) [s] -\u003e []=0x5\n                      `-(u) \"lus\" -\u003e []=0x6\n        `-(u) [b] -\u003e [ei]=0xc\n                      `-(e) [nr]\n                             `-(n) [s] -\u003e []=0x7\n                             `-(r) []=0x8\n                      `-(i) [c] -\u003e [ou]\n                                    `-(o) [n] -\u003e []=0x9\n                                    `-(u) \"ndus\" -\u003e []=0xa\n```\n\n# Running the Rax tests\n\nTo run the tests try:\n\n    $ make\n    $ ./rax-test\n\nTo run the benchmark:\n\n    $ make\n    $ ./rax-test --bench\n\nTo test Rax under OOM conditions:\n\n    $ make\n    $ ./rax-oom-test\n\nThe last one is very verbose currently.\n\nIn order to test with Valgrind, just run the tests using it, however\nif you want accurate leaks detection, let Valgrind run the *whole* test,\nsince if you stop it earlier it will detect a lot of false positive memory\nleaks. This is due to the fact that Rax put pointers at unaligned addresses\nwith `memcpy`, so it is not obvious where pointers are stored for Valgrind,\nthat will detect the leaks. However, at the end of the test, Valgrind will\ndetect that all the allocations were later freed, and will report that\nthere are no leaks.\n\n# Debugging Rax\n\nWhile investigating problems in Rax it is possible to turn debugging messages\non by compiling with the macro `RAX_DEBUG_MSG` enabled. Note that it's a lot\nof output, and may make running large tests too slow.\n\nIn order to active debugging selectively in a dynamic way, it is possible to\nuse the function raxSetDebugMsg(0) or raxSetDebugMsg(1) to disable/enable\ndebugging.\n\nA problem when debugging code doing complex memory operations like a radix\ntree implemented the way Rax is implemented, is to understand where the bug\nhappens (for instance a memory corruption). For that goal it is possible to\nuse the function raxTouch() that will basically recursively access every\nnode in the radix tree, itearting every sub child. In combination with\ntools like Valgrind, it is possible then to perform the following pattern\nin order to narrow down the state causing a give bug:\n\n1. The rax-test is executed using Valgrind, adding a printf() so that for\n   the fuzz tester we see what iteration in the loop we are in.\n2. After every modification of the radix tree made by the fuzz tester\n   in rax-test.c, we add a call to raxTouch().\n3. Now as soon as an operation will corrupt the tree, raxTouch() will\n   detect it (via Valgrind) immediately. We can add more calls to narrow\n   the state.\n4. At this point a good idea is to enable Rax debugging messages immediately\n   before the moment the tree is corrupted, to see what happens. This can\n   be achieved by adding a few \"if\" statements inside the code, since we\n   know the iteration that causes the corruption (because of step 1).\n\nThis method was used with success during rafactorings in order to debug the\nintroduced bugs.\n","funding_links":[],"categories":["C","C++"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fantirez%2Frax","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fantirez%2Frax","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fantirez%2Frax/lists"}