{"id":13591004,"url":"https://github.com/blastrain/octillery","last_synced_at":"2025-03-02T13:13:07.405Z","repository":{"id":49745433,"uuid":"159152840","full_name":"blastrain/octillery","owner":"blastrain","description":"Go package for sharding databases ( Supports every ORM or raw SQL )","archived":false,"fork":false,"pushed_at":"2023-11-05T03:57:08.000Z","size":196,"stargazers_count":190,"open_issues_count":6,"forks_count":30,"subscribers_count":20,"default_branch":"master","last_synced_at":"2024-05-29T21:22:00.117Z","etag":null,"topics":["database-sharding","go","golang","golang-library","mysql","sqlite3"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/blastrain.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-11-26T10:39:35.000Z","updated_at":"2024-06-19T05:19:33.545Z","dependencies_parsed_at":"2024-06-19T05:19:31.542Z","dependency_job_id":"17351e03-95ad-408e-a4cd-db270f977506","html_url":"https://github.com/blastrain/octillery","commit_stats":null,"previous_names":["knocknote/octillery"],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blastrain%2Foctillery","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blastrain%2Foctillery/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blastrain%2Foctillery/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blastrain%2Foctillery/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/blastrain","download_url":"https://codeload.github.com/blastrain/octillery/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241509654,"owners_count":19974071,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database-sharding","go","golang","golang-library","mysql","sqlite3"],"created_at":"2024-08-01T16:00:52.645Z","updated_at":"2025-03-02T13:13:07.382Z","avatar_url":"https://github.com/blastrain.png","language":"Go","funding_links":[],"categories":["Go","Generators","Uncategorized"],"sub_categories":["Database Tools"],"readme":"# Octillery [![GoDoc](https://godoc.org/go.knocknote.io/octillery?status.svg)](https://godoc.org/go.knocknote.io/octillery) [![CircleCI](https://circleci.com/gh/blastrain/octillery.svg?style=shield)](https://circleci.com/gh/blastrain/octillery)  [![codecov](https://codecov.io/gh/blastrain/octillery/branch/master/graph/badge.svg?token=hRKqugQMsg)](https://codecov.io/gh/blastrain/octillery) [![Go Report Card](https://goreportcard.com/badge/go.knocknote.io/octillery)](https://goreportcard.com/report/go.knocknote.io/octillery)\n\n\n\u003cimg width=\"300px\" height=\"238px\" src=\"https://user-images.githubusercontent.com/209884/29391665-d1d6e1d0-8333-11e7-9a33-1db3dc9d2f72.png\"\u003e\u003c/img\u003e\n\n`Octillery` is a Go package for sharding databases.\nIt can use with every OR Mapping library ( `xorm` , `gorp` , `gorm` , `dbr` ...) implementing `database/sql` interface, or raw SQL.\n\nCurrently supports `MySQL` (for product) and `SQLite3` (for testing) .\n\n# Motivation\n\nWe need database sharding library in Go. Of course, we know some libraries like ( https://github.com/evalphobia/wizard , https://github.com/go-pg/sharding ). But OR Mapping library they support are restricted and we want to write sharding configuration declaratively, also expect to pluggable for sharding algorithm or database adapter, and expect to configurable sharding key or whether use sequencer or not.\n\n# Features\n\n- Supports every OR Mapping library implementing `database/sql` interface ( `xorm` , `gorp` , `gorm` , `dbr` , ... )\n- Supports using `database/sql` ( raw SQL ) directly\n- Pluggable sharding algorithm ( preinstalled algorithms are `modulo` and `hashmap` )\n- Pluggable database adapter ( preinstalled adapters are `mysql` and `sqlite3` )\n- Declarative describing for sharding configuration in `YAML`\n- Configurable sharding algorithm, database adapter, sharding key, whether use sequencer or not.\n- Supports capture read/write queries just before passing to database driver\n- Supports database migration by CLI ( powered by `schemalex` )\n- Supports import seeds from CSV\n\n# Install\n\n## Install as a CLI tool\n\n```shell\ngo get go.knocknote.io/octillery/cmd/octillery\n```\n\n## Install as a library\n\n```shell\ngo get go.knocknote.io/octillery\n```\n\n# How It Works\n\n## 1. How database sharding works\n\nWe explain by using `posts` table.\n\n`posts` table schema is\n\n```sql\nCREATE TABLE `posts` (\n  `id` bigint unsigned NOT NULL AUTO_INCREMENT,\n  `user_id` bigint unsigned NOT NULL,\n  `created_at` datetime NOT NULL,\n  `updated_at` datetime NOT NULL,\n  PRIMARY KEY (`id`),\n  UNIQUE KEY `uq_posts_01` (`user_id`)\n) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;\n```\n\nAnd we want to shard this table to four databases for load distribution.\n\nIn this case, we can try to two approach according to requirements.\n\n### 1. Using Sequencer\n\nIf you want `id` to be unique in all databases, you should use this approach.  \nArchitecture of this approach would be the following.\n\n![architecture](https://user-images.githubusercontent.com/209884/48552381-c8612a80-e91b-11e8-8625-af0043b2536c.png)\n\nApplication create SQL ( like `insert into posts (id, user_id, ...) values (null, 1, ...)` ), in this point, id value is null because still not decide. In accordance with the above graph, insert this query to one of the databases.\n\n1. Application requests id value to sequencer\n2. Sequencer generates next unique id in all shards\n3. Sequencer returns id value to application ( ex. `id = 1` )\n4. Replace `id` value from `null` to `1` (ex. `insert into posts (id, user_id, ...) values (1, 1, ...)` )\n5. Decide target based of `id` value by sharding algorithm ( default `modulo` ) and insert record to selected database.\n\nBy using sequencer approach, you can get **unique** `id` value in all databases.\nTherefore, if you insert multiple records, database records should looks like the following.\n\n\u003ctable \u003e\n\u003ctr\u003e\n  \u003cth colspan=2\u003eposts_shard_1\u003c/th\u003e\n  \u003cth colspan=2\u003eposts_shard_2\u003c/th\u003e\n  \u003cth colspan=2\u003eposts_shard_3\u003c/th\u003e\n  \u003cth colspan=2\u003eposts_shard_4\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr align=\"center\"\u003e\n  \u003ctd\u003eid\u003c/td\u003e\n  \u003ctd\u003euser_id\u003c/td\u003e\n  \u003ctd\u003eid\u003c/td\u003e\n  \u003ctd\u003euser_id\u003c/td\u003e\n  \u003ctd\u003eid\u003c/td\u003e\n  \u003ctd\u003euser_id\u003c/td\u003e\n  \u003ctd\u003eid\u003c/td\u003e\n  \u003ctd\u003euser_id\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr align=\"center\"\u003e\n  \u003ctd\u003e1\u003c/td\u003e\n  \u003ctd\u003e1\u003c/td\u003e\n  \u003ctd\u003e2\u003c/td\u003e\n  \u003ctd\u003e2\u003c/td\u003e\n  \u003ctd\u003e3\u003c/td\u003e\n  \u003ctd\u003e3\u003c/td\u003e\n  \u003ctd\u003e4\u003c/td\u003e\n  \u003ctd\u003e4\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr align=\"center\"\u003e\n  \u003ctd\u003e5\u003c/td\u003e\n  \u003ctd\u003e5\u003c/td\u003e\n  \u003ctd\u003e6\u003c/td\u003e\n  \u003ctd\u003e6\u003c/td\u003e\n  \u003ctd\u003e7\u003c/td\u003e\n  \u003ctd\u003e7\u003c/td\u003e\n  \u003ctd\u003e8\u003c/td\u003e\n  \u003ctd\u003e8\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n### 2. Using Sharding Key ( without Sequencer )\nIf you don't care about uniqueness of `id`, you can use sharding key approach.  \nArchitecture of this appraoch would be the following.\n\n![architecture2](https://user-images.githubusercontent.com/209884/48595164-e455e200-e996-11e8-8207-f9a432812cc6.png)\n\n1. Decide target based of `user_id` value by sharding algorithm ( default `modulo` ) and insert record to selected database.\n\nBy using sharding key approach, same id value will appear in multiple databases.\nTherefore, if you insert multiple records, database record should looks like the following.\n\n\u003ctable \u003e\n\u003ctr\u003e\n  \u003cth colspan=2\u003eposts_shard_1\u003c/th\u003e\n  \u003cth colspan=2\u003eposts_shard_2\u003c/th\u003e\n  \u003cth colspan=2\u003eposts_shard_3\u003c/th\u003e\n  \u003cth colspan=2\u003eposts_shard_4\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr align=\"center\"\u003e\n  \u003ctd\u003eid\u003c/td\u003e\n  \u003ctd\u003euser_id\u003c/td\u003e\n  \u003ctd\u003eid\u003c/td\u003e\n  \u003ctd\u003euser_id\u003c/td\u003e\n  \u003ctd\u003eid\u003c/td\u003e\n  \u003ctd\u003euser_id\u003c/td\u003e\n  \u003ctd\u003eid\u003c/td\u003e\n  \u003ctd\u003euser_id\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr align=\"center\"\u003e\n  \u003ctd\u003e1\u003c/td\u003e\n  \u003ctd\u003e1\u003c/td\u003e\n  \u003ctd\u003e1\u003c/td\u003e\n  \u003ctd\u003e2\u003c/td\u003e\n  \u003ctd\u003e1\u003c/td\u003e\n  \u003ctd\u003e3\u003c/td\u003e\n  \u003ctd\u003e1\u003c/td\u003e\n  \u003ctd\u003e4\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr align=\"center\"\u003e\n  \u003ctd\u003e2\u003c/td\u003e\n  \u003ctd\u003e5\u003c/td\u003e\n  \u003ctd\u003e2\u003c/td\u003e\n  \u003ctd\u003e6\u003c/td\u003e\n  \u003ctd\u003e2\u003c/td\u003e\n  \u003ctd\u003e7\u003c/td\u003e\n  \u003ctd\u003e2\u003c/td\u003e\n  \u003ctd\u003e8\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n## 2. Requirements of database sharding library\n\nWe explained how to sharding database at section 1.\nFrom this we define requirements of database sharding library.\n\n- Know about database sharding configuration\n- Capture query just before passing to database driver\n- Parse query and find sharding key\n- If use sequencer, requests `id` value to sequencer and replace value of `id` column by it\n- Select sharding target based of sharding key by sharding algorithm\n\n## 3. How Octillery works\n\n### How To Capture Query\n\n`Octillery` CLI tool supports `transpose` command.\nIt replace import statement of `database/sql` to `go.knocknote.io/octillery/database/sql`.\n\n`go.knocknote.io/octillery/database/sql` package has compatible interface of `database/sql`.\n\nTherefore, OR Mapping library call `Octillery`'s interface. and it can capture all queries.\n\n### How To Parse SQL\n\n`Octillery` use [github.com/blastrain/vitess-sqlparser](https://github.com/blastrain/vitess-sqlparser) as SQL parser. It implements powered by `vitess` and `tidb` .\n\n### How To Use New Database Adapter\n\n`Octillery` supports `mysql` and `sqlite3` adapter by default.  \nIf you want to use new database adapter, need to the following two steps.\n\n1. Write `DBAdapter` interface. ( see https://godoc.org/go.knocknote.io/octillery/connection/adapter )\n2. Put new adapter file to `go.knocknote.io/octillery/plugin` directory\n\n### How To Use New Database Sharding Algorithm\n\n`Octillery` supports `modulo` and `hashmap` algorithm by default.  \nIf you want to use new algorithm, need to the following two steps.\n\n1. Write `ShardingAlgorithm` interface. ( see https://godoc.org/go.knocknote.io/octillery/algorithm )\n2. Put new algorithm file to `go.knocknote.io/octillery/algorithm` directory\n\n# Usage\n\n## 1. Install CLI tool\n\n```shell\n$ go get go.knocknote.io/octillery/cmd/octillery\n```\n\n## 2. Install library\n\n```shell\n$ go get go.knocknote.io/octillery\n```\n\n\n## 3. Replace already imported `database/sql` statement\n\n```shell\n$ octillery transpose\n```\n※ `--dry-run` option confirms without overwriting\n\n## 4. Install database adapter\n\n```shell\n$ octillery install --mysql\n```\n\n## 5. Describe database cofiguration in YAML\n\n`databases.yml`\n\n```yaml\ndefault: \u0026default\n  adapter: mysql\n  encoding: utf8mb4\n  username: root\n  master:\n    - localhost:3306\n\ntables:\n  posts:\n    shard: true\n    shard_key: user_id\n    shards:\n      - post_shard_1:\n          \u003c\u003c: *default\n          database: posts_shard_1\n      - post_shard_2:\n          \u003c\u003c: *default\n          database: posts_shard_2\n```\n\n## 6. Migrate database\n\n```shell\n$ octillery migrate --config databases.yml /path/to/schema\n```\n\n※ `--dry-run` option confirms migration plan\n\n## 7. Load configuration file\n\n```go\npackage main\n\nimport (\n\t\"go.knocknote.io/octillery\"\n\t\"go.knocknote.io/octillery/database/sql\"\n)\n\nfunc main() {\n\tif err := octillery.LoadConfig(\"databases.yml\"); err != nil {\n\t\tpanic(err)\n\t}\n\tdb, _ := sql.Open(\"mysql\", \"\")\n\tdb.QueryRow(\"...\")\n}\n```\n\n# Document\n\nSee [GoDoc](https://godoc.org/go.knocknote.io/octillery)\n\n# Development\n\n## Install dependencies\n\n```shell\n$ make deps\n```\n\nIf update dependencies, the following\n\n1. Modify `glide.yaml`\n2. Run `make update-deps`\n3. Commit `glide.yaml` and `glide.lock`\n\n## Run tests\n\n```shell\n$ make test\n```\n\n# See also\n\n- `sqlparser` : https://github.com/blastrain/vitess-sqlparser\n- `schemalex` : https://github.com/schemalex/schemalex\n\n# Committers\n\nMasaaki Goshima ([@goccy](https://github.com/goccy))\n\n# LICENSE\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblastrain%2Foctillery","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblastrain%2Foctillery","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblastrain%2Foctillery/lists"}