{"id":13983787,"url":"https://github.com/coinbase/mongobetween","last_synced_at":"2025-04-07T11:11:04.298Z","repository":{"id":39583563,"uuid":"266219593","full_name":"coinbase/mongobetween","owner":"coinbase","description":null,"archived":false,"fork":false,"pushed_at":"2024-09-25T16:06:49.000Z","size":157,"stargazers_count":130,"open_issues_count":16,"forks_count":50,"subscribers_count":19,"default_branch":"master","last_synced_at":"2025-03-28T03:08:14.043Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/coinbase.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-22T22:29:41.000Z","updated_at":"2025-03-17T09:45:18.000Z","dependencies_parsed_at":"2024-06-17T19:39:28.457Z","dependency_job_id":"c10cf5d7-2728-471c-8025-82f64554ad76","html_url":"https://github.com/coinbase/mongobetween","commit_stats":null,"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coinbase%2Fmongobetween","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coinbase%2Fmongobetween/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coinbase%2Fmongobetween/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coinbase%2Fmongobetween/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/coinbase","download_url":"https://codeload.github.com/coinbase/mongobetween/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247640465,"owners_count":20971557,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-09T05:01:55.593Z","updated_at":"2025-04-07T11:11:04.275Z","avatar_url":"https://github.com/coinbase.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# mongobetween\n`mongobetween` is a lightweight MongoDB connection pooler written in Golang. It's primary function is to handle a large number of incoming connections, and multiplex them across a smaller connection pool to one or more MongoDB clusters.\n\n`mongobetween` is used in production at Coinbase. It is currently deployed as a Docker sidecar alongside a Rails application using the [Ruby Mongo driver](https://github.com/mongodb/mongo-ruby-driver), connecting to a number of sharded MongoDB clusters. It was designed to connect to `mongos` routers who are responsible for server selection for read/write preferences (connecting directly to a replica set's `mongod` instances hasn't been battle tested).\n\n### How it works\n`mongobetween` listens for incoming connections from an application, and proxies any queries to the [MongoDB Go driver](https://github.com/mongodb/mongo-go-driver) which is connected to a MongoDB cluster. It also intercepts any `ismaster` commands from the application, and responds with `\"I'm a shard router (mongos)\"`, without proxying. This means `mongobetween` appears to the application as an always-available MongoDB shard router, and any MongoDB connection issues or failovers are handled internally by the Go driver.\n\n### Installation\n```\ngo install github.com/coinbase/mongobetween\n```\n\n### Usage\n```\nUsage: mongobetween [OPTIONS] address1=uri1 [address2=uri2] ...\n  -loglevel string\n    \tOne of: debug, info, warn, error, dpanic, panic, fatal (default \"info\")\n  -network string\n    \tOne of: tcp, tcp4, tcp6, unix or unixpacket (default \"tcp4\")\n  -password string\n    \tMongoDB password\n  -ping\n    \tPing downstream MongoDB before listening\n  -pretty\n    \tPretty print logging\n  -statsd string\n    \tStatsd address (default \"localhost:8125\")\n  -unlink\n    \tUnlink existing unix sockets before listening\n  -username string\n    \tMongoDB username\n  -dynamic string\n    \tFile or URL to query for dynamic configuration\n  -enable-sdam-metrics\n        Enable SDAM(Server Discovery And Monitoring) metrics\n  -enable-sdam-logging\n        Enable SDAM(Server Discovery And Monitoring) logging\n```\n\nTCP socket example:\n```\nmongobetween \":27016=mongodb+srv://username:password@cluster.mongodb.net/database?maxpoolsize=10\u0026label=cluster0\"\n```\n\nUnix socket example:\n```\nmongobetween -network unix \"/tmp/mongo.sock=mongodb+srv://username:password@cluster.mongodb.net/database?maxpoolsize=10\u0026label=cluster0\"\n```\n\nProxying multiple clusters:\n```\nmongobetween -network unix \\\n  \"/tmp/mongo1.sock=mongodb+srv://username:password@cluster1.mongodb.net/database?maxpoolsize=10\u0026label=cluster1\" \\\n  \"/tmp/mongo2.sock=mongodb+srv://username:password@cluster2.mongodb.net/database?maxpoolsize=10\u0026label=cluster2\"\n```\n\nThe `label` query parameter in the connection URI is used to any tag statsd metrics or logs for that connection.\n\n### Dynamic configuration\n\nPassing a file or URL as the `-dynamic` argument will allow somewhat dynamic configuration of `mongobetween`. Example supported file format:\n```json\n{\n  \"Clusters\": {\n    \":12345\": {\n      \"DisableWrites\": true,\n      \"RedirectTo\": \"\"\n    },\n    \"/var/tmp/cluster1.sock\": {\n      \"DisableWrites\": false,\n      \"RedirectTo\": \"/var/tmp/cluster2.sock\"\n    }\n  }\n}\n```\n\nThis will disable writes to the proxy served from address `:12345`, and redirect any traffic sent to `/var/tmp/cluster1.sock` to the proxy running on `/var/tmp/cluster2.sock`. This is useful for minimal-downtime migrations between clusters.\n\n### TODO\n\nCurrent known missing features:\n - [X] Transaction server pinning\n - [X] Different cursors on separate servers with the same cursor ID value\n\n\n### Statsd\n`mongobetween` supports reporting health metrics to a local statsd sidecar, using the [Datadog Go library](github.com/DataDog/datadog-go). By default it reports to `localhost:8125`. The following metrics are reported:\n - `mongobetween.handle_message` (Timing) - end-to-end time handling an incoming message from the application\n - `mongobetween.round_trip` (Timing) - round trip time sending a request and receiving a response from MongoDB\n - `mongobetween.request_size` (Distribution) - request size to MongoDB\n - `mongobetween.response_size` (Distribution) - response size from MongoDB\n - `mongobetween.open_connections` (Gauge) - number of open connections between the proxy and the application\n - `mongobetween.connection_opened` (Counter) - connection opened with the application\n - `mongobetween.connection_closed` (Counter) - connection closed with the application\n - `mongobetween.cursors` (Gauge) - number of open cursors being tracked (for cursor -\u003e server mapping)\n - `mongobetween.transactions` (Gauge) - number of transactions being tracked (for client sessions -\u003e server mapping)****\n - `mongobetween.server_selection` (Timing) - Go driver server selection timing\n - `mongobetween.checkout_connection` (Timing) - Go driver connection checkout timing\n - `mongobetween.pool.checked_out_connections` (Gauge) - number of connections checked out from the Go driver connection pool\n - `mongobetween.pool.open_connections` (Gauge) - number of open connections from the Go driver to MongoDB\n - `mongobetween.pool_event.connection_closed` (Counter) - Go driver connection closed\n - `mongobetween.pool_event.connection_pool_created` (Counter) - Go driver connection pool created\n - `mongobetween.pool_event.connection_created` (Counter) - Go driver connection created\n - `mongobetween.pool_event.connection_check_out_failed` (Counter) - Go driver connection check out failed\n - `mongobetween.pool_event.connection_checked_out` (Counter) - Go driver connection checked out\n - `mongobetween.pool_event.connection_checked_in` (Counter) - Go driver connection checked in\n - `mongobetween.pool_event.connection_pool_cleared` (Counter) - Go driver connection pool cleared\n - `mongobetween.pool_event.connection_pool_closed` (Counter) - Go driver connection pool closed\n\n### Background\n`mongobetween` was built to address a connection storm issue between a high scale Rails app and MongoDB (see [blog post](https://blog.coinbase.com/scaling-connections-with-ruby-and-mongodb-99204dbf8857)). Due to Ruby MRI's global interpreter lock, multi-threaded web applications don't utilize multiple CPU cores. To achieve better CPU utilization, Puma is run with multiple workers (processes), each of which need a separate MongoDB connection pool. This leads to a large number of connections to MongoDB, sometimes exceeding MongoDB's upstream connection limit of 128k connections.\n\n`mongobetween` has reduced connection counts by an order of magnitude, spikes of up to 30k connections are now reduced to around 2k. It has also significantly reduced `ismaster` commands on the cluster, as there's only a single monitor goroutine per `mongobetween` process, instead of a monitor thread for each Ruby process.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoinbase%2Fmongobetween","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcoinbase%2Fmongobetween","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoinbase%2Fmongobetween/lists"}