{"id":13590661,"url":"https://github.com/go-gorm/sharding","last_synced_at":"2025-04-08T13:31:48.743Z","repository":{"id":37956902,"uuid":"449563934","full_name":"go-gorm/sharding","owner":"go-gorm","description":"High performance table sharding plugin for Gorm.","archived":false,"fork":true,"pushed_at":"2024-12-02T18:10:15.000Z","size":171,"stargazers_count":292,"open_issues_count":33,"forks_count":64,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-06T07:41:24.626Z","etag":null,"topics":["gorm","mysql","postgresql","sharding"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"longbridge/gorm-sharding","license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/go-gorm.png","metadata":{"funding":{"github":["jinzhu"],"patreon":"jinzhu","open_collective":"gorm"},"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-01-19T05:49:29.000Z","updated_at":"2025-04-04T02:15:38.000Z","dependencies_parsed_at":"2023-02-02T02:32:13.489Z","dependency_job_id":null,"html_url":"https://github.com/go-gorm/sharding","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/go-gorm%2Fsharding","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/go-gorm%2Fsharding/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/go-gorm%2Fsharding/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/go-gorm%2Fsharding/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/go-gorm","download_url":"https://codeload.github.com/go-gorm/sharding/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247851747,"owners_count":21006810,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gorm","mysql","postgresql","sharding"],"created_at":"2024-08-01T16:00:49.271Z","updated_at":"2025-04-08T13:31:43.732Z","avatar_url":"https://github.com/go-gorm.png","language":"Go","funding_links":["https://github.com/sponsors/jinzhu","https://patreon.com/jinzhu","https://opencollective.com/gorm"],"categories":["开源类库","Open source library","Go"],"sub_categories":["数据库","Database"],"readme":"# Gorm Sharding\n\n[![Go](https://github.com/go-gorm/sharding/actions/workflows/tests.yml/badge.svg)](https://github.com/go-gorm/sharding/actions/workflows/tests.yml)\n\nGorm Sharding plugin using SQL parser and replace for splits large tables into smaller ones, redirects Query into sharding tables. Give you a high performance database access.\n\nGorm Sharding 是一个高性能的数据库分表中间件。\n\n它基于 Conn 层做 SQL 拦截、AST 解析、分表路由、自增主键填充，带来的额外开销极小。对开发者友好、透明，使用上与普通 SQL、Gorm 查询无差别，只需要额外注意一下分表键条件。\n\n## Features\n\n- Non-intrusive design. Load the plugin, specify the config, and all done.\n- Lighting-fast. No network based middlewares, as fast as Go.\n- Multiple database (PostgreSQL, MySQL) support.\n- Integrated primary key generator (Snowflake, PostgreSQL Sequence, Custom, ...).\n\n## Install\n\n```bash\ngo get -u gorm.io/sharding\n```\n\n## Usage\n\nConfig the sharding middleware, register the tables which you want to shard.\n\n```go\nimport (\n  \"fmt\"\n\n  \"gorm.io/driver/postgres\"\n  \"gorm.io/gorm\"\n  \"gorm.io/sharding\"\n)\n\ndb, err := gorm.Open(postgres.New(postgres.Config{DSN: \"postgres://localhost:5432/sharding-db?sslmode=disable\"))\n\ndb.Use(sharding.Register(sharding.Config{\n    ShardingKey:         \"user_id\",\n    NumberOfShards:      64,\n    PrimaryKeyGenerator: sharding.PKSnowflake,\n}, \"orders\", Notification{}, AuditLog{}))\n// This case for show up give notifications, audit_logs table use same sharding rule.\n```\n\nUse the db session as usual. Just note that the query should have the `Sharding Key` when operate sharding tables.\n\n```go\n// Gorm create example, this will insert to orders_02\ndb.Create(\u0026Order{UserID: 2})\n// sql: INSERT INTO orders_2 ...\n\n// Show have use Raw SQL to insert, this will insert into orders_03\ndb.Exec(\"INSERT INTO orders(user_id) VALUES(?)\", int64(3))\n\n// This will throw ErrMissingShardingKey error, because there not have sharding key presented.\ndb.Create(\u0026Order{Amount: 10, ProductID: 100})\nfmt.Println(err)\n\n// Find, this will redirect query to orders_02\nvar orders []Order\ndb.Model(\u0026Order{}).Where(\"user_id\", int64(2)).Find(\u0026orders)\nfmt.Printf(\"%#v\\n\", orders)\n\n// Raw SQL also supported\ndb.Raw(\"SELECT * FROM orders WHERE user_id = ?\", int64(3)).Scan(\u0026orders)\nfmt.Printf(\"%#v\\n\", orders)\n\n// This will throw ErrMissingShardingKey error, because WHERE conditions not included sharding key\nerr = db.Model(\u0026Order{}).Where(\"product_id\", \"1\").Find(\u0026orders).Error\nfmt.Println(err)\n\n// Update and Delete are similar to create and query\ndb.Exec(\"UPDATE orders SET product_id = ? WHERE user_id = ?\", 2, int64(3))\nerr = db.Exec(\"DELETE FROM orders WHERE product_id = 3\").Error\nfmt.Println(err) // ErrMissingShardingKey\n```\n\nThe full example is [here](./examples/order.go).\n\n\u003e 🚨 NOTE: Gorm config `PrepareStmt: true` is not supported for now.\n\u003e\n\u003e 🚨 NOTE: Default snowflake generator in multiple nodes may result conflicted primary key, use your custom primary key generator, or regenerate a primary key when conflict occurs.\n\n## Primary Key\n\nWhen you sharding tables, you need consider how the primary key generate.\n\nRecommend options:\n\n- [Snowflake](https://github.com/bwmarrin/snowflake)\n- [Database sequence by manully](https://www.postgresql.org/docs/current/sql-createsequence.html)\n\n### Use Snowflake\n\nBuilt-in Snowflake primary key generator.\n\n```go\ndb.Use(sharding.Register(sharding.Config{\n    ShardingKey:         \"user_id\",\n    NumberOfShards:      64,\n    PrimaryKeyGenerator: sharding.PKSnowflake,\n}, \"orders\")\n```\n\n### Use PostgreSQL Sequence\n\nThere has built-in PostgreSQL sequence primary key implementation in Gorm Sharding, you just configure `PrimaryKeyGenerator: sharding.PKPGSequence` to use.\n\nYou don't need create sequence manually, Gorm Sharding check and create when the PostgreSQL sequence does not exists.\n\nThis sequence name followed `gorm_sharding_${table_name}_id_seq`, for example `orders` table, the sequence name is `gorm_sharding_orders_id_seq`.\n\n```go\ndb.Use(sharding.Register(sharding.Config{\n    ShardingKey:         \"user_id\",\n    NumberOfShards:      64,\n    PrimaryKeyGenerator: sharding.PKPGSequence,\n}, \"orders\")\n```\n\n### Use MySQL Sequence\n\nThere has built-in MySQL sequence primary key implementation in Gorm Sharding, you just configure `PrimaryKeyGenerator: sharding.PKMySQLSequence` to use.\n\nYou don't need create sequence manually, Gorm Sharding check and create when the MySQL sequence does not exists.\n\nThis sequence name followed `gorm_sharding_${table_name}_id_seq`, for example `orders` table, the sequence name is `gorm_sharding_orders_id_seq`.\n\n```go\ndb.Use(sharding.Register(sharding.Config{\n    ShardingKey:         \"user_id\",\n    NumberOfShards:      64,\n    PrimaryKeyGenerator: sharding.PKMySQLSequence,\n}, \"orders\")\n```\n\n### No primary key\n\nIf your table doesn't have a primary key, or has a primary key that isn't called `id`, anyway, you don't want to auto-fill the `id` field, then you can set `PrimaryKeyGenerator` to `PKCustom` and have `PrimaryKeyGeneratorFn` return `0`.\n\n## Combining with dbresolver\n\n\u003e 🚨 NOTE: Use dbresolver first.\n\n```go\ndsn := \"host=localhost user=gorm password=gorm dbname=gorm port=5432 sslmode=disable\"\ndsnRead := \"host=localhost user=gorm password=gorm dbname=gorm-slave port=5432 sslmode=disable\"\n\nconn := postgres.Open(dsn)\nconnRead := postgres.Open(dsnRead)\n\ndb, err := gorm.Open(conn, \u0026gorm.Config{})\ndbRead, err := gorm.Open(conn, \u0026gorm.Config{})\n\ndb.Use(dbresolver.Register(dbresolver.Config{\n  Replicas: []gorm.Dialector{dbRead.Dialector},\n}))\n\ndb.Use(sharding.Register(sharding.Config{\n  ShardingKey:         \"user_id\",\n  NumberOfShards:      64,\n  PrimaryKeyGenerator: sharding.PKSnowflake,\n}))\n```\n\n## Sharding process\n\nThis graph show up how Gorm Sharding works.\n\n```mermaid\ngraph TD\nfirst(\"SELECT * FROM orders WHERE user_id = ? AND status = ?\nargs = [100, 1]\")\n\nfirst---\u003egorm([\"Gorm Query\"])\n\nsubgraph \"Gorm\"\n  gorm---\u003egorm_query\n  gorm---\u003egorm_exec\n  gorm---\u003egorm_queryrow\n  gorm_query[\"connPool.QueryContext(sql, args)\"]\n  gorm_exec[/\"connPool.ExecContext\"/]\n  gorm_queryrow[/\"connPool.QueryRowContext\"/]\nend\n\nsubgraph \"database/sql\"\n  gorm_query--\u003econn([\"Conn\"])\n  gorm_exec--\u003econn([\"Conn\"])\n  gorm_queryrow--\u003econn([\"Conn\"])\n  ExecContext[/\"ExecContext\"/]\n  QueryContext[/\"QueryContext\"/]\n  QueryRowContext[/\"QueryRowContext\"/]\n\n\n  conn--\u003eExecContext\n  conn--\u003eQueryRowContext\n  conn--\u003eQueryContext\nend\n\nsubgraph sharding [\"Sharding\"]\n  QueryContext--\u003erouter--\u003e| Format to get full SQL string |format_sql--\u003e| Parser to AST |parse--\u003echeck_table\n  router[[\"router(sql, args)\u003cbr\u003e\"]]\n  format_sql\u003e\"sql = SELECT * FROM orders WHERE user_id = 100 AND status = 1\"]\n\n  check_table{\"Check sharding rules\u003cbr\u003eby table name\"}\n  check_table--\u003e| Exist |process_ast\n  check_table_1{{\"Return Raw SQL\"}}\n  not_match_error[/\"Return Error\u003cbr\u003eSQL query must has sharding key\"\\]\n\n  parse[[\"ast = sqlparser.Parse(sql)\"]]\n\n  check_table-.-\u003e| Not exist |check_table_1\n  process_ast((\"Sharding rules\"))\n  get_new_table_name[[\"Use value in WhereValue (100) for get sharding table index\u003cbr\u003eorders + (100 % 16)\u003cbr\u003eSharding Table = orders_4\"]]\n  new_sql{{\"SELECT * FROM orders_4 WHERE user_id = 100 AND status = 1\"}}\n\n  process_ast-.-\u003e| Not match ShardingKey |not_match_error\n  process_ast--\u003e| Match ShardingKey |match_sharding_key--\u003e| Get table name |get_new_table_name--\u003e| Replace TableName to get new SQL |new_sql\nend\n\n\nsubgraph database [Database]\n  orders_other[(\"orders_0, orders_1 ... orders_3\")]\n  orders_4[(orders_4)]\n  orders_last[(\"orders_5 ... orders_15\")]\n  other_tables[(Other non-sharding tables\u003cbr\u003eusers, stocks, topics ...)]\n\n  new_sql--\u003e| Sharding Query | orders_4\n  check_table_1-.-\u003e| None sharding Query |other_tables\nend\n\norders_4--\u003eresult\nother_tables-.-\u003eresult\nresult[/Query results\\]\n```\n\n## License\n\nMIT license.\n\nOriginal fork from [Longbridge](https://github.com/longbridgeapp/gorm-sharding).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgo-gorm%2Fsharding","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgo-gorm%2Fsharding","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgo-gorm%2Fsharding/lists"}