{"id":18542757,"url":"https://github.com/coinbase/redisbetween","last_synced_at":"2025-04-09T18:32:09.907Z","repository":{"id":43860694,"uuid":"337154083","full_name":"coinbase/redisbetween","owner":"coinbase","description":null,"archived":false,"fork":false,"pushed_at":"2024-01-29T07:03:54.000Z","size":350,"stargazers_count":34,"open_issues_count":6,"forks_count":11,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-03-24T10:38:47.354Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/coinbase.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-02-08T17:26:00.000Z","updated_at":"2025-03-18T02:08:05.000Z","dependencies_parsed_at":"2024-01-29T08:38:27.045Z","dependency_job_id":null,"html_url":"https://github.com/coinbase/redisbetween","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coinbase%2Fredisbetween","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coinbase%2Fredisbetween/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coinbase%2Fredisbetween/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coinbase%2Fredisbetween/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/coinbase","download_url":"https://codeload.github.com/coinbase/redisbetween/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248087855,"owners_count":21045602,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T20:10:24.903Z","updated_at":"2025-04-09T18:32:04.897Z","avatar_url":"https://github.com/coinbase.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# redisbetween\n\nThis is a connection pooling proxy for redis. It was originally built for an application that was hitting \nconnection limits against its redis clusters. Its purpose is to solve a specific problem: many application processes\nthat cannot otherwise share a connection pool need to connect to a single redis cluster.\n\nredisbetween supports both standalone and clustered redis deployments with the following caveats:\n\n- **Blocking Commands** that cause the client to hold a connection open are partially supported, with limitations.\nSpecifically, `BRPOPLPUSH` and `SUBSCRIBE`/`PSUBSCRIBE` are supported in non-clustered mode. In order to prevent\nexhausting the connection pool, these commands are implemented using reserved connections, configured via\n`maxsubscriptions` and `maxblockers` (both default to 1). The supported commands are specifically intended to support\nsidekiq servers with reliable fetch. Other blocking commands such as `BLPOP` or `WAIT` are not yet supported.\n\n- **Pipelines** are supported, but require a client patch. Normally, redis clients may send multiple commands\nback-to-back before reading a batch of responses all at once from the server. Since redisbetween shares upstream\nconnections among many clients, it relies on special \"signal\" messages to indicate the beginning and end of batched\ncommands. Clients using redisbetween must prepend a `GET 🔜` and append a `GET 🔚` to their batch of messages in order\nfor redisbetween to properly proxy the pipelined commands and responses.\n\n- **Transactions are only supported _within pipelines_.** This means that the commands `DISCARD`, `EXEC`, `MULTI`,\n`UNWATCH` and `WATCH` will return errors from the proxy before they reach an upstream host unless they occur between the\ntwo signal values described above in the **Pipelines** section. This is because redis stores state about open\ntransactions on the server side, attached to each client connection. In order to support transactions without\nconnection-pinning, we require that the full set of operations be sent in one batch so that the connection we check back\ninto the pool does not leak state to other clients.\n\n- The **SELECT** command, which is used by redis clients when connecting to a db other than the default `0`, is not\nallowed. However, redisbetween _does_ support multiple dbs by specifying the db number in the endpoint url path. With an\nexample URL of `redis://example.com/3`, the resulting connection pool would be mapped to the socket path\n`/var/tmp/redisbetween-example.com-3.sock` suffix, and all connections would issue a `SELECT 3` command before entering\nthe pool. Note that each db number gets its own connection pool, so adjust `maxpoolsize` accordingly when using this\nfeature.\n\n- The **AUTH** command is not supported. If this is needed in the future, we\ncould add support by pre-emptively sending the AUTH command on all new connections, like we do with `SELECT`.\n\n### How it works\n\nredisbetween creates a connection pool for each upstream redis server it discovers (either via configuration at start\ntime, snooping on `CLUSTER` commands or via `ASK`/`MOVED` errors) and maps a local unix socket to that pool.\nApplications running on the same host can connect to redis via this unix socket instead of connecting directly to the\nredis server, thus sharing a relatively smaller number of connections among the many processes on a machine.\n\nUpon startup, redisbetween creates a pool of connections to the redis endpoint provided and listens on a unix socket\nnamed after the endpoint. By default, it will be named `/var/tmp/redisbetween-${host}-${port}(-${db})(-ro).sock`. This can be\ncustomized using the `-localsocketprefix` and `-localsocketsuffix` options. For standalone redis deployments, this will\nbe the only socket created. However, redisbetween will inspect responses to `CLUSTER` commands, looking for references to\ncluster members that it hasn't yet seen. When it sees a new cluster member, it allocates a new connection pool and unix\nsocket for it before relaying the response to the client.\n\n### Redisbetween Gem\n\nThe [ruby](/ruby) directory contains a ruby gem that monkey patches the ruby redis client to support redisbetween. See\nthe [readme](/ruby/README.md) for more details.\n\nHere's an example of a patch to the go-redis client. Note that this one does not handle db number selection, as that is\nnot supported by redis cluster anyway.\n\n```go\nreadonly := true\nopt := \u0026redis.ClusterOptions{\n    Addrs: []string{address},\n    Dialer: func(ctx context.Context, network, addr string) (net.Conn, error) {\n        if strings.Contains(network, \"tcp\") {\n            host, port, err := net.SplitHostPort(addr)\n            if err != nil {\n                return nil, err\n            }\n            addr = \"/var/tmp/redisbetween-\" + host + \"-\" + port\n            if readonly {\n                addr += \"-ro\"\n            }\n            addr += \".sock\"\n            network = \"unix\"\n        }\n        return net.Dial(network, addr)\n    },\n}\nclient := redis.NewClusterClient(opt)\nres := client.Do(context.Background(), \"ping\")\n```\n\n### Installation\n```\ngo install github.com/coinbase/redisbetween\n```\n\n### Usage\n```\nUsage: bin/redisbetween [OPTIONS] uri1 [uri2] ...\n  -localsocketprefix string\n    \tprefix to use for unix socket filenames (default \"/var/tmp/redisbetween-\")\n  -localsocketsuffix string\n    \tsuffix to use for unix socket filenames (default \".sock\")\n  -loglevel string\n    \tone of: debug, info, warn, error, dpanic, panic, fatal (default \"info\")\n  -network string\n    \tone of: tcp, tcp4, tcp6, unix or unixpacket (default \"unix\")\n  -pretty\n    \tpretty print logging\n  -statsd string\n    \tstatsd address (default \"localhost:8125\")\n  -unlink\n    \tunlink existing unix sockets before listening\n  -healthcheck\n      start a background process to check the health of server connections\n  -healthcheckcycle\n      duration after which the healthcheck process should repeat itself (default 60s)\n  -healthcheckthreshold\n      count of consecutive healthcheck failures after which a server is declared unhealthy (default 3)\n  -idletimeout\n      how long can an inactive connection remain idle in the pool (default 0 meaning no timeout)\n```\n\nEach URI can specify the following settings as GET params:\n\n- `minpoolsize` sets the min connection pool size for this host. Defaults to 1\n- `maxpoolsize` sets the max connection pool size for this host. Defaults to 10\n- `label` optionally tags events and metrics for proxy activity on this host or cluster. Defaults to `\"\"` (disabled)\n- `readtimeout` timeout for reads to this upstream. Defaults to 5s\n- `writetimeout` timeout for writes to this upstream. Defaults to 5s\n- `readonly` every connection issues a [READONLY](https://redis.io/commands/readonly) command before entering the pool. Defaults to false\n- `maxsubscriptions` sets the max number of channels that can be subscribed to at one time. Defaults to 1.\n- `maxblockers` sets the max number of commands that can be blocking at one time. Defaults to 1. \n- `idletimeout` how long can an inactive connection remain idle in the pool. Default is the global level `idletimeout` or 0 (no timeout)\n\nExample: `./redisbetween -unlink -pretty -loglevel debug redis://localhost:7001?maxsubscriptions=2\u0026maxblockers=2`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoinbase%2Fredisbetween","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcoinbase%2Fredisbetween","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoinbase%2Fredisbetween/lists"}