{"id":13550063,"url":"https://github.com/sebadob/redhac","last_synced_at":"2025-04-02T23:31:42.426Z","repository":{"id":177772271,"uuid":"657457800","full_name":"sebadob/redhac","owner":"sebadob","description":"Rust Embedded Distributed Highly Available Cache","archived":false,"fork":false,"pushed_at":"2024-08-01T10:25:40.000Z","size":143,"stargazers_count":19,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-05T03:19:14.018Z","etag":null,"topics":["async","cache","distributed","embedded","high-availability","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sebadob.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-23T05:30:37.000Z","updated_at":"2025-02-26T23:10:27.000Z","dependencies_parsed_at":null,"dependency_job_id":"d0a851d6-f44d-4966-99e5-633a4aa82a5b","html_url":"https://github.com/sebadob/redhac","commit_stats":null,"previous_names":["sebadob/redhac"],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sebadob%2Fredhac","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sebadob%2Fredhac/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sebadob%2Fredhac/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sebadob%2Fredhac/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sebadob","download_url":"https://codeload.github.com/sebadob/redhac/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246911057,"owners_count":20853652,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","cache","distributed","embedded","high-availability","rust"],"created_at":"2024-08-01T12:01:28.572Z","updated_at":"2025-04-02T23:31:38.106Z","avatar_url":"https://github.com/sebadob.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"# redhac\n\n`redhac` is derived from **Rust Embedded Distributed Highly Available Cache**\n\nThe keywords **embedded** and **distributed** are a bit strange at the same time.\u003cbr\u003e\nThe idea of `redhac` is to provide a caching library which can be embedded into any Rust\napplication, while still providing the ability to build a distributed HA caching layer.\u003cbr\u003e\nIt can be used as a cache for a single instance too, of course, but then it will not be the\nmost performant, since it needs clones of most values to be able to send them over the network\nconcurrently. If you need a distributed cache however, you might give `redhac` a try.\n\nThe underlying system works with quorum and will elect a leader dynamically on startup. Each\nnode will start a server and multiple client parts for bidirectional gRPC streaming. Each client will\nconnect to each of the other servers.\n\n## Release\n\nThis crate has been used and tested already before going open source.\u003cbr\u003e\nBy now, did thousands of HA cluster startups, where I produced leadership election conflicts on purpose to make\nsure they are handled and resolved properly.\n\nHowever, some benchmarks and maybe performance finetuning is still missing, which is the reason why this crate has not\nreached v1.0.0 yet. After the benchmarks the API may change, if another approach of handling everything is just faster.\n\n## Consistency and guarantees\n\n**Important:** `redhac` is used for caching, not for persistent data!\n\nDuring normal operation, all the instances will forward their cache modifications to the other cache members.\u003cbr\u003e\nIf a node goes down or has temporary network issues and therefore loses connection to the cluster, it will invalidate\nits own cache to make 100% sure, that it would never provide possibly outdated data from the cache. Once a node\nrejoins the cluster, it will start receiving and saving updates from the remotes again.\u003cbr\u003e\nThere is an option in the `cache_get!` macro to fetch cached values from remote instances after such an event. This can\nbe used if you maybe have some values that only live inside the cache and cannot be refreshed from the database, for\ninstance.\n\nThe other situation, when the cache will not save any values is when there is no quorum and cluster leader.\n\nIf you need more consistency / guarantees / syncing after a late join, you may take a look at\n[openraft](https://github.com/datafuselabs/openraft) or other projects like this.\n\n### Versions and dependencies\n\n- MSRV is Rust 1.70\n- Everything is checked and working with `-Zminimal-versions`\n- No issue reported by `cargo audit` (2024-04-09)\n\n## Single Instance Example\n\n```rust\n// These 3 are needed for the `cache_get` macro\nuse redhac::{cache_get, cache_get_from, cache_get_value};\nuse redhac::{cache_put, cache_recv, CacheConfig, SizedCache};\n\n#[tokio::main]\nasync fn main() {\n    let (_, mut cache_config) = CacheConfig::new();\n    \n    // The cache name is used to reference the cache later on\n    let cache_name = \"my_cache\";\n    // We need to spawn a global handler for each cache instance.\n    // Communication is done over channels.\n    cache_config.spawn_cache(\n        cache_name.to_string(), SizedCache::with_size(16), None\n    );\n    \n    // Cache keys can only be `String`s at the time of writing.\n    let key = \"myKey\";\n    // The value you want to cache must implement `serde::Serialize`.\n    // The serialization of the values is done with `bincode`.\n    let value = \"myCacheValue\".to_string();\n    \n    // At this point, we need cloned values to make everything work\n    // nicely with networked connections. If you only ever need a \n    // local cache, you might be better off with using the `cached`\n    // crate directly and use references whenever possible.\n    cache_put(\n        cache_name.to_string(), key.to_string(), \u0026cache_config, \u0026value\n    )\n        .await\n        .unwrap();\n    \n    let res = cache_get!(\n        // The type of the value we want to deserialize the value into\n        String,\n        // The cache name from above. We can start as many for our\n        // application as we like\n        cache_name.to_string(),\n        // For retrieving values, the same as above is true - we\n        // need real `String`s\n        key.to_string(),\n        // All our caches have the necessary information added to\n        // the cache config. Here a\n        // reference is totally fine.\n        \u0026cache_config,\n        // This does not really apply to this single instance \n        // example. If we would have started a HA cache layer we\n        // could do remote lookups for a value we cannot find locally,\n        // if this would be set to `true`\n        false\n    )\n        .await\n        .unwrap();\n    \n    assert!(res.is_some());\n    assert_eq!(res.unwrap(), value);\n}\n```\n\n## High Availability Setup\n\nThe High Availability (HA) works in a way, that each cache member connects to each other. When\neventually quorum is reached, a leader will be elected, which then is responsible for all cache\nmodifications to prevent collisions (if you do not decide against it with a direct `cache_put`).\nSince each node connects to each other, it means that you cannot just scale up the cache layer\ninfinitely. The ideal number of nodes is 3. You can scale this number up for instance to 5 or 7\nif you like, but this has not been tested in greater detail so far.\n\n**Write performance** will degrade the more nodes you add to the cluster, since you simply need\nto wait for more Ack's from the other members.  \n\n**Read performance** however should stay the same.\n\nEach node will keep a local copy of each value inside the cache (if it has not lost connection\nor joined the cluster at some later point), which means in most cases reads do not require any\nremote network access.\n\n## Configuration\n\nThe way to configure the `HA_MODE` is optimized for a Kubernetes deployment but may seem a bit\nodd at the same time, if you deploy somewhere else. You can either provide the `.env` file and\nuse it as a config file, or just set these variables for the environment directly. You need\nto set the following values:\n\n### `HA_MODE`\n\nThe first one is easy, just set `HA_MODE=true`\n\n### `HA_HOSTS`\n\n**NOTE:**\u003cbr\u003e\nIn a few examples down below, the name for deployments may be `rauthy`. The reason is, that this\ncrate was written originally to complement another project of mine, Rauthy (link will follow),\nwhich is an OIDC Provider and Single Sign-On Solution written in Rust.\n\nThe `HA_HOSTS` is working in a way, that it is really easy inside Kubernetes to configure it,\nas long as a `StatefulSet` is used for the deployment.\nThe way a cache node finds its members is by the `HA_HOSTS` and its own `HOSTNAME`.\nIn the `HA_HOSTS`, add every cache member. For instance, if you want to use 3 replicas in HA\nmode which are running and deployed as a `StatefulSet` with the name `rauthy` again:\n\n```text\nHA_HOSTS=\"http://rauthy-0:8000, http://rauthy-1:8000 ,http://rauthy-2:8000\"\n```\n\nThe way it works:\n\n1. **A node gets its own hostname from the OS**\u003cbr\u003e\nThis is the reason, why you use a StatefulSet for the deployment, even without any volumes\nattached. For a `StatefulSet` called `rauthy`, the replicas will always have the names `rauthy-0`,\n`rauthy-1`, ..., which are at the same time the hostnames inside the pod.\n2. **Find \"me\" inside the `HA_HOSTS` variable**\u003cbr\u003e\nIf the hostname cannot be found in the `HA_HOSTS`, the application will panic and exit because\nof a misconfiguration.\n3. **Use the port from the \"me\"-Entry that was found for the server part**\u003cbr\u003e\nThis means you do not need to specify the port in another variable which eliminates the risk of\nhaving inconsistencies\nor a bad config in that case.\n4. **Extract \"me\" from the `HA_HOSTS`**\u003cbr\u003e\nthen take the leftover nodes as all cache members and connect to them\n5. **Once a quorum has been reached, a leader will be elected**\u003cbr\u003e\nFrom that point on, the cache will start accepting requests\n6. **If the leader is lost - elect a new one - No values will be lost**\n7. **If quorum is lost, the cache will be invalidated**\u003cbr\u003e\nThis happens for security reasons to provide cache inconsistencies. Better invalidate the cache\nand fetch the values fresh from the DB or other cache members than working with possibly invalid\nvalues, which is especially true in an authn / authz situation.\n\n**NOTE:**\u003cbr\u003e\nIf you are in an environment where the described mechanism with extracting the hostname would\nnot work, you can set the `HOSTNAME_OVERWRITE` for each instance to match one of the `HA_HOSTS`\nentries, or you can overwrite the name when using the `redhac::start_cluster`.\n\n### `CACHE_AUTH_TOKEN`\n\nYou need to set a secret for the `CACHE_AUTH_TOKEN`, which is then used for authenticating\ncache members.\n\n### TLS\n\nFor the sake of this example, we will not dig into TLS and disable it in the example, which\ncan be done with `CACHE_TLS=false`.\u003cbr\u003e\nYou can add your TLS certificates in PEM format and an optional Root CA. This is true for the\nServer and the Client part separately. This means you can configure the cache layer to use mTLS\nconnections.\n\n#### Generating TLS certificates (optional)\n\nYou can of course provide your own set of certificates, if you already have some, or just use your preferred way of\ncreating some. However, I want to show a way of doing this in the most simple way possible with another tool of mine\ncalled [Nioca](https://github.com/sebadob/nioca).\u003cbr\u003e\nIf you have `docker` or similar available, this is the easiest option. If not, you can grab one of the binaries from\nthe [out]() folder, which are available for linux amd64 and arm64.\n\nI suggest to use `docker` for this task. Otherwise, you can use the `nioca` binary directly on any linux machine.\nIf you want a permanent way of generating certificates for yourself, take a look at Rauthy's `justfile` and copy\nand adjust the recipes `create-root-ca` and `create-end-entity-tls` to your liking.  \nIf you just want to get everything started quickly, follow these steps:\n\n##### Folder for your certificates\n\nLet's create a folder for our certificates:\n\n```\nmkdir ca\n```\n\n##### Create an alias for the `docker` command\n\nIf you use one of the binaries directly, you can skip this step.\n\n```\nalias nioca='docker run --rm -it -v ./ca:/ca -u $(id -u ${USER}):$(id -g ${USER}) ghcr.io/sebadob/nioca'\n```\n\nTo see the full feature set for more customization than mentioned below:\n```\nnioca x509 -h\n```\n\n##### Generate full certificate chain\n\nWe can create and generate a fully functioning, production ready Root Certificate Authority (CA) with just a single\ncommand. Make sure that at least one of your `--alt-name-dns` from here matches the `CACHE_TLS_CLIENT_VALIDATE_DOMAIN`\nfrom the redhac config later on.\u003cbr\u003e\nTo keep things simple, we will use the same certificate for the server and the client. You can of course create\nseparate ones, if you like:\n\n```\nnioca x509 \\\n    --cn 'redhac.local' \\\n    --alt-name-dns redhac.local \\\n    --usages-ext server-auth \\\n    --usages-ext client-auth \\\n    --stage full \\\n    --clean\n```\n\nYou will be asked 6 times (yes, 6) for an at least 16 character password:\n- The first 3 times, you need to provide the encryption password for your Root CA\n- The last 3 times, you should provide a different password for your Intermediate CA\n\nWhen everything was successful, you will have a new folder named `x509` with sub folders `root`, `intermediate`\nand `end_entity` in your current one.\n\nFrom these, you will need the following files:\n\n```\ncp ca/x509/intermediate/ca-chain.pem ./redhac.ca-chain.pem \u0026\u0026 \\\ncp ca/x509/end_entity/$(cat ca/x509/end_entity/serial)/cert-chain.pem ./redhac.cert-chain.pem \u0026\u0026 \\\ncp ca/x509/end_entity/$(cat ca/x509/end_entity/serial)/key.pem ./redhac.key.pem\n```\n\n- You should have 3 files in `ls -l`:\n```\nredhac.ca-chain.pem\nredhac.cert-chain.pem\nredhac.key.pem\n```\n\n**4. Create Kubernetes Secrets**\n```\nkubectl create secret tls redhac-tls-server --key=\"redhac.key.pem\" --cert=\"redhac.cert-chain.pem\" \u0026\u0026 \\\nkubectl create secret tls redhac-tls-client --key=\"redhac.key.pem\" --cert=\"redhac.cert-chain.pem\" \u0026\u0026 \\\nkubectl create secret generic redhac-server-ca --from-file redhac.ca-chain.pem \u0026\u0026 \\\nkubectl create secret generic redhac-client-ca --from-file redhac.ca-chain.pem\n``` \n\n### Reference Config\n\nThe following variables are the ones you can use to configure `redhac` via env vars.  \nAt the time of writing, the configuration can only be done via the env.\n\n```text\n# If the cache should start in HA mode or standalone\n# accepts 'true|false', defaults to 'false'\nHA_MODE=true\n\n# The connection strings (with hostnames) of the HA instances\n# as a CSV. Format: 'scheme://hostname:port'\nHA_HOSTS=\"http://redhac.redhac:8080, http://redhac.redhac:8180, http://redhac.redhac:8280\"\n\n# This can overwrite the hostname which is used to identify each\n# cache member. Useful in scenarios, where all members are on the\n# same host or for testing. You need to add the port, since `redhac`\n# will do an exact match to find \"me\".\n#HOSTNAME_OVERWRITE=\"127.0.0.1:8080\"\n\n# Secret token, which is used to authenticate the cache members\nCACHE_AUTH_TOKEN=SuperSafeSecretToken1337\n\n# Enable / disable TLS for the cache communication (default: true)\nCACHE_TLS=true\n\n# The path to the server TLS certificate PEM file\n# default: tls/redhac.cert-chain.pem\nCACHE_TLS_SERVER_CERT=tls/redhac.cert-chain.pem\n# The path to the server TLS key PEM file\n# default: tls/redhac.key.pem\nCACHE_TLS_SERVER_KEY=tls/redhac.key.pem\n\n# The path to the client mTLS certificate PEM file. This is optional.\nCACHE_TLS_CLIENT_CERT=tls/redhac.cert-chain.pem\n# The path to the client mTLS key PEM file. This is optional.\nCACHE_TLS_CLIENT_KEY=tls/redhac.key.pem\n\n# If not empty, the PEM file from the specified location will be\n# added as the CA certificate chain for validating\n# the servers TLS certificate. This is optional.\nCACHE_TLS_CA_SERVER=tls/redhac.ca-chain.pem\n# If not empty, the PEM file from the specified location will\n# be added as the CA certificate chain for validating\n# the clients mTLS certificate. This is optional.\nCACHE_TLS_CA_CLIENT=tls/redhac.ca-chain.pem\n\n# The domain / CN the client should validate the certificate\n# against. This domain MUST be inside the\n# 'X509v3 Subject Alternative Name' when you take a look at \n# the servers certificate with the openssl tool.\n# default: redhac.local\nCACHE_TLS_CLIENT_VALIDATE_DOMAIN=redhac.local\n\n# Can be used if you need to overwrite the SNI when the \n# client connects to the server, for instance if you are \n# behind a loadbalancer which combines multiple certificates. \n# default: \"\"\n#CACHE_TLS_SNI_OVERWRITE=\n\n# Define different buffer sizes for channels between the \n# components. Buffer for client request on the incoming \n# stream - server side (default: 128)\n# Makes sense to have the CACHE_BUF_SERVER roughly set to:\n# `(number of total HA cache hosts - 1) * CACHE_BUF_CLIENT`\nCACHE_BUF_SERVER=128\n# Buffer for client requests to remote servers for all cache \n# operations (default: 64)\nCACHE_BUF_CLIENT=64\n\n# Connections Timeouts\n# The Server sends out keepalive pings with configured timeouts\n# The keepalive ping interval in seconds (default: 5)\nCACHE_KEEPALIVE_INTERVAL=5\n# The keepalive ping timeout in seconds (default: 5)\nCACHE_KEEPALIVE_TIMEOUT=5\n\n# The timeout for the leader election. If a newly saved leader\n# request has not reached quorum after the timeout, the leader\n# will be reset and a new request will be sent out.\n# CAUTION: This should not be below \n# CACHE_RECONNECT_TIMEOUT_UPPER, since cold starts and\n# elections will be problematic in that case.\n# value in seconds, default: 2\nCACHE_ELECTION_TIMEOUT=2\n\n# These 2 values define the reconnect timeout for the HA Cache\n# Clients. The values are in ms and a random between these 2 \n# will be chosen each time to avoid conflicts\n# and race conditions (default: 500)\nCACHE_RECONNECT_TIMEOUT_LOWER=500\n# (default: 2000)\nCACHE_RECONNECT_TIMEOUT_UPPER=2000\n```\n\n## Example Code\n\nFor example code, please take a look at `./examples`.\n\nThe `conflict_resolution_tests` are used for producing conflicts on purpose by starting multiple nodes from the same\ncode at the exact same time, which is maybe not too helpful if you want to take a look at how it's done.\n\nThe better example then would be the `ha_setup`, which can be used to start 3 nodes in 3 different terminals to observe\nthe behavior.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsebadob%2Fredhac","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsebadob%2Fredhac","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsebadob%2Fredhac/lists"}