{"id":15431001,"url":"https://github.com/cookpad/mixed_gauge","last_synced_at":"2025-10-08T17:13:34.978Z","repository":{"id":32313632,"uuid":"35888760","full_name":"cookpad/mixed_gauge","owner":"cookpad","description":"A simple and robust database sharding with ActiveRecord.","archived":false,"fork":false,"pushed_at":"2018-06-30T00:31:22.000Z","size":101,"stargazers_count":58,"open_issues_count":0,"forks_count":1,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-07-28T03:59:44.378Z","etag":null,"topics":["activerecord","database-sharding","hash-slots","rails","sharding"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cookpad.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-05-19T14:51:55.000Z","updated_at":"2025-06-19T11:34:36.000Z","dependencies_parsed_at":"2022-08-21T00:20:20.616Z","dependency_job_id":null,"html_url":"https://github.com/cookpad/mixed_gauge","commit_stats":null,"previous_names":["taiki45/mixed_gauge"],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/cookpad/mixed_gauge","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cookpad%2Fmixed_gauge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cookpad%2Fmixed_gauge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cookpad%2Fmixed_gauge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cookpad%2Fmixed_gauge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cookpad","download_url":"https://codeload.github.com/cookpad/mixed_gauge/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cookpad%2Fmixed_gauge/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278551097,"owners_count":26005385,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-06T02:00:05.630Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["activerecord","database-sharding","hash-slots","rails","sharding"],"created_at":"2024-10-01T18:19:53.964Z","updated_at":"2025-10-08T17:13:34.941Z","avatar_url":"https://github.com/cookpad.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mixed_gauge\n[![Build Status](https://travis-ci.org/cookpad/mixed_gauge.svg?branch=master)](https://travis-ci.org/cookpad/mixed_gauge) [![Coverage Status](https://coveralls.io/repos/taiki45/mixed_gauge/badge.svg?branch=master)](https://coveralls.io/r/taiki45/mixed_gauge?branch=master) [![Code Climate](https://codeclimate.com/github/taiki45/mixed_gauge/badges/gpa.svg)](https://codeclimate.com/github/taiki45/mixed_gauge) [![Gem Version](https://badge.fury.io/rb/mixed_gauge.svg)](http://badge.fury.io/rb/mixed_gauge)\n\nA simple and robust ActiveRecord extension for database sharding.\nmixed_gauge offers shards management with hash slots and re-sharding support.\nIt enable you to execute efficient queries to single node with KVS-like interface.\nAnd you can even execute limited RDB queries to all nodes with ActiveRecord interface in-parallel.\n\nmixed_gauge is already used in production. [(blog post in Japanese)](http://techlife.cookpad.com/entry/2015/06/22/134108)\n\n## Goal and concept\n- Simple\n- No downtime migrations\n- Rollback-able operations\n\nDatabase sharding tend to be over-complexed. There are cases which need these complex database sharding but in some cases database sharding can be more simple. The large data set which is enoght big to partition should be designed to be distributed, or should be re-design if it wasn't. Design to be distributed uses key based relation or reverse indexes to fits its limitation. In that case, the data set is almost design to be distributed, mixed_gauge strongly encourages your database sharding by its simplicity.\n\nWe, offer 24/7 services, must keep our services running. mixed_gauge supports online migrations: adding new nodes to cluster or removing some existing nodes from cluster. It comes with \"key distibution model with hash slots\" and  database replication and multi-master replication. In sharding we need re-sharding, move data from node to another node in cluster, when adding or removing new nodes from cluster. But by setting some rule to node management and using replication, we can finish moving data before adding or removing nodes. The detail operations are specified later chapter of this document.\n\nAll operaions should be rollback-able in case of any failures. mixed_gauge's node management can rollback adding and removing nodes operation. The detail operations are specified later chapter of this document.\n\n\n## Main components of sharding teqnique\n### Distribution model\nmixed_gauge's database sharding is based on keys distribution model with hash slots. The key space is split into arbitrary size of slots. `hash(v) mod N` determines which slot is used where `N` is size of configured hash slots. Hash slot is a virtual node and it is assigned to real node.\n\nThe default hash function is CRC32 which has better perfomance for this kind of cases. You can use other hash function.\n\n### Node management\nmixed_gauge's database sharding sets a rule to both adding nodes and removing nodes. The node size must be incresed by multiple of 2. At first, the node size is 1. Then the node size is incresed to 2, next is 4, and next of next is 8.\n\nBy setting this rule, we can move (copy) data from node to node before adding or removing nodes by \"database replication\". For example, when we have `cluster(A)`, which has single node A and node A is assigned (0..1023) hash slots, and plan to migrate to `cluster(A, B)`, which has 2 nodes A and B and node A is assigned (0..511) slots and node B is assigned (512..1023) slots, we can copy and replicate from A to B before migration then just balance hash slots to node B.\n\n```\n(1)              (2)                               (3)\n   ┌───────┐           ┌───────┐       ┌───────┐         ┌───────┐       ┌───────┐\n   │       │           │       │       │       │         │       │       │       │\n   │   A   │           │   A   │──────▶│   B   │         │   A   │       │   B   │\n   │       │           │       │       │       │         │       │       │       │\n   └───────┘           └───────┘       └───────┘         └───────┘       └───────┘\n\n    0..1023             0..1023                           0..511         512..1023\n```\n\n### Migration operations\n```\n(1) From 1 node cluster\n\n     0..1023\n    ┌───────┐\n    │       │\n    │       │\n    │   A   │\n    │       │\n    │       │\n    └───────┘\n\n(2) Copy data and start replication\n\n     0..1023\n    ┌───────┐       ┌───────┐\n    │       │       │       │\n    │       │       │       │\n    │   A   │──────▶│   B   │\n    │       │       │       │\n    │       │       │       │\n    └───────┘       └───────┘\n\n(3) Change auto_increment config\nnot to conflict id column\n\n     0..1023\n  1 ┌───────┐       ┌───────┐ 2\n  3 │       │       │       │ 4\n  5 │       │       │       │ 6\n  . │   A   │──────▶│   B   │ .\n  . │       │       │       │ .\n    │       │       │       │\n    └───────┘       └───────┘\n   increment=2    increment=2\n     offset=1       offset=2\n\n(4) Start Multi-master replication\n\n     0..1023\n    ┌───────┐       ┌───────┐\n    │       │──────▶│       │\n    │       │       │       │\n    │   A   │       │   B   │\n    │       │       │       │\n    │       │◀──────│       │\n    └───────┘       └───────┘\n\n(5) Deploy app and apply new cluster\nconfiguration\n\n    0..511         512..1023\n   ┌───────┐       ┌───────┐\n   │       │──────▶│       │\n   │       │       │       │\n   │   A   │       │   B   │\n   │       │       │       │\n   │       │◀──────│       │\n   └───────┘       └───────┘\n\n(6) Stop Multi-master replication\n\n    0..511         512..1023\n   ┌───────┐       ┌───────┐\n   │       │       │       │\n   │       │       │       │\n   │   A   │       │   B   │\n   │       │       │       │\n   │       │       │       │\n   └───────┘       └───────┘\n```\n\nIn step 3, we set enough big offset not to conflict auto increment value\non applying config.\n\n\n## Usage\nAdd additional database connection config to `database.yml`.\n\n```yaml\n# database.yml\nproduction_user_001:\n  adapter: mysql2\n  username: user_writable\n  host: db-user-001\nproduction_user_002:\n  adapter: mysql2\n  username: user_writable\n  host: db-user-002\nproduction_user_003:\n  adapter: mysql2\n  username: user_writable\n  host: db-user-003\nproduction_user_004:\n  adapter: mysql2\n  username: user_writable\n  host: db-user-004\n```\n\nConfigure slots (virtual node for cluster) then assign slots to real node.\n\n```ruby\nMixedGauge.configure do |config|\n  config.define_cluster(:user) do |cluster|\n    # When slots per node * max nodes per cluster = (2 ** 10) * (2 ** 10)\n    cluster.define_slot_size(1048576)\n    cluster.register(0..262143, :production_user_001)\n    cluster.register(262144..524287, :production_user_002)\n    cluster.register(524288..786431, :production_user_003)\n    cluster.register(786432..1048575, :production_user_004)\n  end\nend\n```\n\nInclude `MixedGauge::Model` to your model class, specify cluster name for the\nmodel, specify distkey which determines node to store.\n\n```ruby\nclass User \u003c ActiveRecord::Base\n  include MixedGauge::Model\n  use_cluster :user\n  def_distkey :email\nend\n```\n\nUse `.get` to retrive single record which is connected to proper\ndatabase node. Use `.put!` to create new record to proper database node.\n\n`.all_shards` returns each model class which is connected to proper\ndatabase node. You can query with these models and aggregate result.\n\n```ruby\nUser.put!(email: 'alice@example.com', name: 'alice')\n\nalice = User.get('alice@example.com')\nalice.age = 1\nalice.save!\n\nUser.all_shards.flat_map {|m| m.find_by(name: 'alice') }.compact\n```\n\nWhen you want to execute queries in all nodes in parallel, use `.all_shards_in_parallel`.\nIt returns `Mixedgauge::AllShardsInParallel` and it offers some collection\nactions which runs in parallel. It is aliased to `.parallel`.\n\n```ruby\nUser.all_shards_in_parallel.map(\u0026count) #=\u003e 1\nUser.parallel.flat_map {|m| m.where(age: 1) }.size #=\u003e 1\n```\n\nWhen you want find by non-distkey, not recomended though, you can define finder\nmethods to model class for convenience.\n\n```ruby\nclass User \u003c ActiveRecord::Base\n  include MixedGauge::Model\n  use_cluster :user\n  def_distkey :email\n\n  parent_methods do\n    def find_from_all_by_name(name)\n      all_shards_in_parallel.map {|m| m.find_by(name: name) }.compact.first\n    end\n  end\nend\n\nalice = User.find_from_all_by_name('Alice')\nalice.age = 0\nalice.save!\n```\n\nSometimes you want to generates distkey value before validation. Since mixed_gauge\ngenerates sub class of your models, AR's callback is usesless for this usecase,\nso mixed_gauge offers its own callback method.\n\n```ruby\nclass AccessToken \u003c ActiveRecord::Base\n  include MixedGauge::Model\n  use_cluster :access_token\n  def_distkey :token\n\n  validates :token, presence: true\n\n  def self.generate_token\n    SecureRandom.uuid\n  end\n\n  before_put do |attributes|\n    unless attributes[:token] || attributes['token']\n      attributes[:token] = generate_token\n    end\n  end\nend\n\naccess_token = AccessToken.put!\naccess_token.token #=\u003e a generated token\n```\n\n## Sharding with Replication\nmixed_gauge also supports replication.\n\nIn case you have 2 shards in cluster and each shard have read replica.\n\n- db-user-101 --replicated--\u003e db-user-102\n- db-user-201 --replicated--\u003e db-user-202\n\nYour database connection configuration might be like this:\n\n```yaml\n# database.yml\nproduction_user_001:\n  adapter: mysql2\n  username: user_writable\n  host: db-user-101\nproduction_user_002:\n  adapter: mysql2\n  username: user_writable\n  host: db-user-201\nproduction_user_readonly_001:\n  adapter: mysql2\n  username: user_readonly\n  host: db-user-102\nproduction_user_readonly_002:\n  adapter: mysql2\n  username: user_writable\n  host: db-user-202\n```\n\nYour initializer for mixed_gauge might be like this:\n\n```ruby\nMixedGauge.configure do |config|\n  config.define_cluster(:user) do |cluster|\n    cluster.define_slot_size(1048576)\n    cluster.register(0..524287, :production_user_001)\n    cluster.register(524288..1048575, :production_user_002)\n  end\n\n  config.define_cluster(:user_readonly) do |cluster|\n    cluster.define_slot_size(1048576)\n    cluster.register(0..524287, :production_user_readonly_001)\n    cluster.register(524288..1048575, :production_user_readonly_002)\n  end\nend\n```\n\nYou can split read/write by defining AR model class for each connection:\n\n```ruby\nclass User \u003c ActiveRecord::Base\n  include MixedGauge::Model\n  use_cluster :user\n  def_distkey :email\nend\n\nclass UserReadonly \u003c ActiveRecord::Base\n  self.table_name = 'users'\n\n  include MixedGauge::Model\n  use_cluster :user_readonly\n  def_distkey :email\nend\n\nUser.put!(name: 'Alice', email: 'alice@example.com')\nUserReadonly.get('alice@example.com')\n```\n\nIf you want to switch specific shard to another shard in another cluster, define mapping between each model:\n\n```ruby\nclass User \u003c ActiveRecord::Base\n  include MixedGauge::Model\n  use_cluster :user\n  def_distkey :email\n\n  replicates_with slave: :UserReadonly\nend\n\nclass UserReadonly \u003c ActiveRecord::Base\n  self.table_name = 'users'\n\n  include MixedGauge::Model\n  use_cluster :user_readonly\n  def_distkey :email\n\n  replicates_with master: :User\nend\n```\n\nYou can switch to another model which have connection to the shard by calling `.switch`:\n\n```ruby\nUserReadonly.all_shards do |readonly|\n  target_ids = readonly.where(age: 0).pluck(:id)\n  readonly.switch(:master) do |writable|\n    writable.where(id: target_ids).delete_all\n  end\nend\n```\n\n## Advanced configuration\n### Hash fucntion\nDefault hash fucntion is CRC32, which has better perfomance for this kind of\nusecase.\n\nBut you can use arbitrary hash function like:\n\n```ruby\n# gem install fnv\nrequire \"fnv\"\nMixedgauge.configure do |config|\n  config.register_hash_function do |key|\n    FNV.new.fnv1a_64(key)\n  end\nend\n```\n\nSuggested hash functions are:\n\n- MurmurHash\n- FNV Hash\n- SuperFastHash\n\n\n## Installation\nAdd this line to your application's Gemfile:\n\n```ruby\ngem 'mixed_gauge'\n```\n\nAnd then execute:\n\n    $ bundle\n\nOr install it yourself as:\n\n    $ gem install mixed_gauge\n\n\n## Contributing\nFeel free to pull request and issue :)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcookpad%2Fmixed_gauge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcookpad%2Fmixed_gauge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcookpad%2Fmixed_gauge/lists"}