{"id":16301666,"url":"https://github.com/bbkr/exodus","last_synced_at":"2026-03-18T18:53:39.808Z","repository":{"id":25606104,"uuid":"29041161","full_name":"bbkr/exodus","owner":"bbkr","description":"Migrate chosen data between relational databases.","archived":false,"fork":false,"pushed_at":"2022-08-13T00:07:21.000Z","size":39,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-10T01:18:10.966Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"artistic-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bbkr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-01-10T00:08:22.000Z","updated_at":"2022-08-13T00:07:24.000Z","dependencies_parsed_at":"2022-07-07T22:53:18.076Z","dependency_job_id":null,"html_url":"https://github.com/bbkr/exodus","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/bbkr/exodus","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bbkr%2Fexodus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bbkr%2Fexodus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bbkr%2Fexodus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bbkr%2Fexodus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bbkr","download_url":"https://codeload.github.com/bbkr/exodus/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bbkr%2Fexodus/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28669711,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-22T19:36:09.361Z","status":"ssl_error","status_checked_at":"2026-01-22T19:36:05.567Z","response_time":144,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-10T20:55:17.626Z","updated_at":"2026-01-22T19:49:36.928Z","avatar_url":"https://github.com/bbkr.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# EXODUS\n\nComprehensive guide and tools to split monolithic database into shards.\n\nYou can find most recent version of this tutorial at https://github.com/bbkr/exodus .\n\n## Intro\n\nWhen you suddenly get this brilliant idea, the revolutionary game-changer, all you want to do is to immediately hack some proof of concept to start small project flame from spark of creativity. So I'll leave you alone for now, with your third mug of coffee and birds chirping first morning songs outside of the window...\n\n...Few years later we meet again. Your proof of concept has grown into a mature, recognizable product. Congratulations! But why the sad face? Your clients are complaining that your product is slow and unresponsive? They want more features? They generate more data? And you cannot do anything about it despite the fact that you bought most shiny, expensive database server that is available?\n\nWhen you were hacking your project on day 0 you were not thinking about long term scalability. All you wanted to do was to create working prototype as fast as possible. So single database design was easiest, fastest and most obvious to use. You didn't think back then, that single machine cannot be scaled up infinitely. And now it is already too late.\n\nLOOKS LIKE YOU'RE STUCK ON THE SHORES OF [monolithic design database] HELL.\nTHE ONLY WAY OUT IS THROUGH...\n(DOOM quote)\n\n## Sharding to the rescue!\n\nSharding is the process of distributing your clients data across multiple databases (called shards).\nBy doing so you will be able to:\n\n* Scale your whole system by adding more cheap database machines.\n* Add more complex features that use a lot of database CPU, I/O and RAM resources.\n* Load balance your environment by moving clients between shards.\n* Handle exceptionally large clients by dedicating resources to them.\n* Utilize data centers in different countries to reduce network lag for clients and be more compliant with local data processing laws.\n* Reduce risk of global system failures\n* Do faster crash recovery due to smaller size of databases.\n\nBut if you already have single (monolithic) database this process is like converting your motorcycle into a car... while riding.\n\n## Outline\n\nThis is step-by-step guide of a very tricky process. And the worst thing you can do is to panic because your product is collapsing under its own weight and you have a lots of pressure from clients. Whole process may take weeks, even months. Will use significant amount of human and time resources. And will pause new features development. Be prepared for that. And do not rush to the next stage until you are absolutely sure the current one is completed.\n\nSo what is the plan?\n\n* You will have to look at your data and fix a lot of schema design errors.\n* Then set up new hardware and software environment components.\n* Adapt your product to sharding logic.\n* Do actual data migration.\n* Adapt your internal tools and procedures.\n* Resume product development.\n\n## Understand your data\n\nIn monolithic database design data classification is irrelevant but it is the most crucial part of sharding design. Your tables can be divided into three groups: client, context and neutral.\n\nLet's assume your product is car rental software and do a quick exercise:\n\n```\n\n  +----------+      +------------+      +-----------+\n  | clients  |      | cities     |      | countries |\n  +----------+      +------------+      +-----------+\n+-| id       |   +--| id         |   +--| id        |\n| | city_id  |\u003e--+  | country_id |\u003e--+  | name      |\n| | login    |   |  | name       |      +-----------+\n| | password |   |  +------------+\n| +----------+   |\n|                +------+\n+--------------------+  |\n                     |  |     +-------+\n  +---------------+  |  |     | cars  |\n  | rentals       |  |  |     +-------+\n  +---------------+  |  |  +--| id    |\n+-| id            |  |  |  |  | vin   |\n| | client_id     |\u003e-+  |  |  | brand |\n| | start_city_id |\u003e----+  |  | model |\n| | start_date    |     |  |  +-------+\n| | end_city_id   |\u003e----+  |\n| | end_date      |        |\n| | car_id        |\u003e-------+   +--------------------+\n| | cost          |            | anti_fraud_systems |\n| +---------------+            +--------------------+\n|                              | id                 |--+\n|      +-----------+           | name               |  |\n|      | tracks    |           +--------------------+  |\n|      +-----------+                                   |\n+-----\u003c| rental_id |                                   |\n       | latitude  |     +--------------------------+  |\n       | longitude |     | blacklisted_credit_cards |  |\n       | timestamp |     +--------------------------+  |\n       +-----------+     | anti_fraud_system_id     |\u003e-+\n                         | number                   |\n                         +--------------------------+\n```\n\n### Client tables\n\nThey contain data owned by your clients. To find them you must start in some root table - `clients` in our example. Then follow every parent-to-child relation (only in this direction) as deep as you can. In this case we descend into `rentals` and then from `rentals` further to `tracks`. So our client tables are: `clients`, `rentals` and `tracks`.\n\nSingle client owns subset of rows from those tables, and those rows will always be moved together in a single transaction between shards.\n\n### Context tables\n\nThey put your clients data in context. To find them follow every child-to-parent relation (only in this direction) from every client table as shallow as you can. Skip if table is already classified. In this case we ascend from `clients` to `cities` and from `cities` further to `countries`. Then from `rentals` we can ascend to `clients` (already classified), `cities` (already classified) and `cars`. And from `tracks` we can ascend into `rentals` (already classified). So our context tables are: `cities`, `countries` and `cars`.\n\nContext tables should be synchronized across all shards.\n\n### Neutral tables\n\nEverything else. They must not be reachable from any client or context table through any relation. However, there may be relations between them. So our neutral tables are: `anti_fraud_systems` and `blacklisted_credit_cards`.\n\nNeutral tables should be moved outside of shards.\n\n### Checkpoint\n\nTake any tool that can visualize your database in the form of a diagram. Print it and pin it on the wall.\nThen take markers in 3 different colors - each for every table type -  and start marking tables in your schema.\n\nIf you have some tables not connected due to technical reasons (for example MySQL partitioned tables or TokuDB tables do not support foreign keys), draw this relation and assume it is there.\n\nIf you are not certain about specific table, leave it unmarked for now.\n\nDone? Good :)\n\n### Q\u0026A\n\n***Q:*** Is it a good idea to cut all relations between client and context tables, so that only two types - client and neutral - remain?\n\n***A:*** You will save a bit of work because no synchronization of context data across all shards will be required.\nBut at the same time any analytics will be nearly impossible. For example, even simple task to find which car was rented the most times will require software script to do the join.\nAlso there won't be any protection against software bugs, for example it will be possible to rent a car that does not even exist.\n\nThere are two cases when converting context table to neutral table is justified:\n\n* Context data is really huge or takes huge amount of transfer to synchronize. We're talking gigabytes here.\n* Reference is \"weak\". And that means it only exists for some debug purposes and is not used in business logic. For example if we present different version of website to user based on country he is from - that makes \"hard\" references between `clients`, `cities` and `countries`, so `cities` and `countries` should remain as context tables.\n\nIn every other case it is very bad idea to make neutral data out of context data.\n\n***Q:*** Is it a good idea to shard only big tables and leave all small tables together on monolithic database?\n\n***A:*** In our example you have one puffy table - `tracks`. It keeps GPS trail of every car rental and will grow really fast. So if you only shard this data you will save a lot of work because there will be only small application changes required. But in real world you will have 100 puffy tables and that means 100 places in application logic when you have to juggle database handles to locate all client data. That also means you won't be able to split your clients between many data centers. Also you won't be able to reduce downtime costs to 1/nth of the amount of shards if some data corruption in monolithic database occurs and recovery is required. And analytics argument mentioned above also applies here.\n\nIt is bad idea to do such sub-sharding. May seem easy and fast - but the sooner you do proper sharding that includes all of your clients data, the better.\n\n## Fix your monolithic database\n\nThere are few design patterns that are perfectly fine or acceptable in monolithic database design but are no-go in sharding.\n\n### Lack of foreign key\n\nAside from obvious risk of referencing nonexistent records, this issue can leave junk when you will migrate clients between shards later for load balancing.\nThe fix is simple - add foreign key if there should be one.\n\nThe only exception is when it cannot be added due to technical limitations, such as usage of TokuDB or partitioned MySQL tables that simply do not support foreign keys.\nSkip those, I'll tell you how to deal with them during data migration later.\n\n### Direct connection between clients\n\nBecause clients may be located on different shards their rows may not point at each other.\nTypical case where it happens is affiliation.\n\n```\n+-----------------------+\n| clients               |\n+-----------------------+\n| id                    |------+\n| login                 |      |\n| password              |      |\n| referred_by_client_id |\u003e-----+\n+-----------------------+\n```\n\nTo fix this issue you must remove foreign key and rely on software instead to match those records.\n\n### Nested connection between clients\n\nBecause clients may be located on different shards their rows may not reference another client (also indirectly).\nTypical case where it happens is post-and-comment discussion.\n\n```\n  +----------+        +------------+\n  | clients  |        | blog_posts |\n  +----------+        +------------+\n+-| id       |---+    | id         |---+\n| | login    |   +---\u003c| client_id  |   |\n| | password |        | text       |   |\n| +----------+        +------------+   |\n|                                      |\n|    +--------------+                  |\n|    | comments     |                  |\n|    +--------------+                  |\n|    | blog_post_id |\u003e-----------------+\n+---\u003c| client_id    |\n     | text         |\n     +--------------+\n```\n\nFirst client posted something and second client commented it. This comment references two clients at the same time - second one directly and first one indirectly through `blog_posts` table.\nThat means it will be impossible to satisfy both foreign keys in `comments` table if those clients are not in single database.\n\nTo fix this you must choose which relation from table that refers to multiple clients is more important, remove the other foreign keys and rely on software instead to match those records.\n\nSo in our example you may decide that relation between `comments` and `blog_posts` remains, relation between `comments` and `clients` is removed and you will use application logic to find which client wrote which comment.\n\n### Accidental connection between clients\n\nThis is the same issue as nested connection but caused by application errors instead of intentional design.\n\n\n```\n                    +----------+\n                    | clients  |\n                    +----------+\n+-------------------| id       |--------------------+\n|                   | login    |                    |\n|                   | password |                    |\n|                   +----------+                    |\n|                                                   |\n|  +-----------------+        +------------------+  |\n|  | blog_categories |        | blog_posts       |  |\n|  +-----------------+        +------------------+  |\n|  | id              |----+   | id               |  |\n+-\u003c| client_id       |    |   | client_id        |\u003e-+\n   | name            |    +--\u003c| blog_category_id |\n   +-----------------+        | text             |\n                              +------------------+\n```\n\nFor example first client defined his own blog categories for his own blog posts.\nBut lets say there was mess with www sessions or some caching mechanism and blog post of second client was accidentally assigned to category defined by first client.\n\nThose issues are very hard to find, because schema itself is perfectly fine and only data is damaged.\n\n### Not reachable clients data\n\nClient tables must be reached exclusively by descending from root table through parent-to-child relations.\n\n\n```\n              +----------+\n              | clients  |\n              +----------+\n+-------------| id       |\n|             | login    |\n|             | password |\n|             +----------+\n|\n|  +-----------+        +------------+\n|  | photos    |        | albums     |\n|  +-----------+        +------------+\n|  | id        |    +---| id         |\n+-\u003c| client_id |    |   | name       |\n   | album_id  |\u003e---+   | created_at |\n   | file      |        +------------+\n   +-----------+\n```\n\nSo we have photo management software this time and when client synchronizes photos from camera new album is created automatically for better import visualization.\nThis is obvious issue even in monolithic database - when all photos from album are removed then it becomes zombie row.\nWon't be deleted automatically by cascade and cannot be matched with `client` anymore.\nIn sharding this also causes misclassification of client table as context table.\n\nTo fix this issue foreign key should be added from `albums` to `clients`.\nThis may also fix classification for some tables below `albums`, if any.\n\n### Polymorphic data\n\nTable cannot be classified as two types at the same time.\n\n```\n              +----------+\n              | clients  |\n              +----------+\n+-------------| id       |-------------+\n|             | login    |             |\n|             | password |             |\n|             +----------+         (nullable)\n|                                      |\n|  +-----------+        +-----------+  |\n|  | blogs     |        | skins     |  |\n|  +-----------+        +-----------+  |\n|  | id        |    +---| id        |  |\n+-\u003c| client_id |    |   | client_id |\u003e-+\n   | skin_id   |\u003e---+   | color     |\n   | title     |        +-----------+\n   +-----------+\n```\n\nIn this product client can choose predefined skin for his blog.\nBut can also define his own skin color and use it as well.\n\nHere single interface of `skins` table is used to access data of both client and context type.\nA lot of \"let's allow client to customize that\" features end up implemented this way.\nWhile being a smart hack - with no table schema duplication and only simple `WHERE client_id IS NULL OR client_id = 123` added to query to present both public and private templates for client - this may cause a lot of trouble in sharding.\n\nThe fix is to go with dual foreign key design and separate tables. Create constraint (or trigger) that will protect against assigning blog to public and private skin at the same time. And write more complicated query to get blog skin color.\n\n```\n              +----------+\n              | clients  |\n              +----------+\n+-------------| id       |\n|             | login    |\n|             | password |\n|             +----------+\n|\n|   +---------------+        +--------------+\n|   | private_skins |        | public_skins |\n|   +---------------+        +--------------+\n|   | id            |--+  +--| id           |\n+--\u003c| client_id     |  |  |  | color        |\n|   | color         |  |  |  +--------------+\n|   +---------------+  |  |    \n|                      |  |\n|                   (nullable)\n|                      |  |\n|                      |  +------+\n|                      +-----+   |\n|                            |   |\n|       +-----------------+  |   |\n|       | blogs           |  |   |\n|       +-----------------+  |   |\n|       | id              |  |   | \n+------\u003c| client_id       |  |   |\n        | private_skin_id |\u003e-+   |\n        | public_skin_id  |\u003e-----+\n        | title           |\n        +-----------------+\n```\n\nHowever - this fix is optional. I'll show you how to deal with maintaining mixed data types in chapter about mutually exclusive IDs.\nIt will be up to you to decide if you want less refactoring but more complicated synchronization.\n\n***Beware!*** Such fix may also accidentally cause another issue described below.\n\n### Opaque uniqueness (a.k.a. horse riddle)\n\nEvery client table without unique constraint must be reachable by not nullable path of parent-to-child relations or at most single nullable path of parent-to-child relations.\nThis is very tricky issue which may cause data loss or duplication during client migration to database shard.\n\n```\n              +----------+\n              | clients  |\n              +----------+\n+-------------| id       |-------------+\n|             | login    |             |\n|             | password |             |\n|             +----------+             |\n|                                      |\n|  +-----------+        +-----------+  |\n|  | time      |        | distance  |  |\n|  +-----------+        +-----------+  |\n|  | id        |--+  +--| id        |  |\n+-\u003c| client_id |  |  |  | client_id |\u003e-+\n   | amount    |  |  |  | amount    |\n   +-----------+  |  |  +-----------+\n                  |  |\n               (nullable)\n                  |  |\n                  |  |\n         +--------+  +---------+\n         |                     |\n         |   +-------------+   |\n         |   | parts       |   |\n         |   +-------------+   |\n         +--\u003c| time_id     |   |\n             | distance_id |\u003e--+\n             | name        |\n             +-------------+\n```\n\nThis time our product is application that helps you with car maintenance schedule.\nOur clients car has 4 tires that must be replaced after 10 years or 100000km\nand 4 spark plugs that must be replaced after 100000km.\nSo 4 indistinguishable rows for tires are added to `parts` table (they reference both `time` and `distance`)\nand 4 indistinguishable rows are added for spark plugs (they reference only `distance`).\n\nNow to migrate client to shard we have to find which rows from `parts` table does he own.\nBy following relations through `time` table we will get 4 tires. But because this path is nullable at some point\nwe are not sure if we found all records. And indeed, by following relations through `distance` table we found 4 tires and 4 spark plugs.\nSince this path is also nullable at some point we are not sure if we found all records.\nSo we must combine result from time and distance paths, which gives us... 8 tires and 4 spark plugs?\nWell, that looks wrong. Maybe let's group it by time and distance pair, which gives us... 1 tire and 1 spark plug?\nSo depending how you combine indistinguishable rows from many nullable paths to get final row set, you may suffer either data duplication or data loss.\n\nYou may say: Hey, that's easy - just select all rows through time path, then all rows from distance path that do not have `time_id`, then union both results.\nUnfortunately paths may be nullable somewhere earlier and several nullable paths may lead to single table,\nwhich will produce bizarre logic to get indistinguishable rows set properly.\n\nTo solve this issue make sure there is at least one not nullable path that leads to every client table (does not matter how many tables it goes through).\nExtra foreign key should be added between `clients` and `part` in our example.\n\nTL;DR\n\n***Q:*** How many legs does the horse have?\n\n***A1:*** Eight. Two front, two rear, two left and two right.\n\n***A2:*** Four. Those attached to it.\n\n### Foreign key to not unique rows\n\nMySQL specific issue.\n\n```\nCREATE TABLE `foo` (\n  `id` int(10) unsigned DEFAULT NULL,\n  KEY `id` (`id`)\n) ENGINE=InnoDB DEFAULT CHARSET=utf8;\n\nCREATE TABLE `bar` (\n  `foo_id` int(10) unsigned NOT NULL,\n  KEY `foo_id` (`foo_id`),\n  CONSTRAINT `bar_ibfk_1` FOREIGN KEY (`foo_id`) REFERENCES `foo` (`id`)\n) ENGINE=InnoDB DEFAULT CHARSET=utf8;\n\nmysql\u003e INSERT INTO `foo` (`id`) VALUES (1);\nQuery OK, 1 row affected (0.01 sec)\n\nmysql\u003e INSERT INTO `foo` (`id`) VALUES (1);\nQuery OK, 1 row affected (0.00 sec)\n\nmysql\u003e INSERT INTO `bar` (`foo_id`) VALUES (1);\nQuery OK, 1 row affected (0.01 sec)\n```\n\nWhich row from `foo` table is referenced by row in `bar` table?\n\nYou don't know because behavior of foreign key constraint is defined as \"it there any parent I can refer to?\" instead of \"do I have exactly one parent?\".\nThere are no direct row-to-row references as in other databases. And it's not a bug, it's a feature.\n\nOf course this causes a lot of weird bugs when trying to locate all rows that belong to given client, because results can be duplicated on JOINs.\n\nTo fix this issue just make sure every referenced column (or set of columns) is unique.\nThey ***must not*** be nullable and ***must*** all be used as primary or unique key.\n\n### Self loops\n\nRows in the same client table cannot be in direct relation.\nTypical case is expressing all kinds of tree or graph structures.\n\n```\n+----------+\n| clients  |\n+----------+\n| id       |----------------------+\n| login    |                      |\n| password |                      |\n+----------+                      |\n                                  |\n          +--------------------+  |\n          | albums             |  |\n          +--------------------+  |\n      +---| id                 |  |\n      |   | client_id          |\u003e-+\n      +--\u003c| parent_album_id    |\n          | name               |\n          +--------------------+\n```\n\nThis causes issues when client data is inserted into target shard.\n\nFor example in our photo management software client has album with `id` = 2 as subcategory of album with `id` = 1.\nThen he flips this configuration, so that the album with `id` = 2 is on top.\nIn such scenario if database returned client rows in default, primary key order\nthen it won't be possible to insert album with `id` = 1 because it requires presence of album with `id` = 2.\n\nYes, you can disable foreign key constraints to be able to insert self-referenced data in any order.\nBut by doing so you may mask many errors - for example broken references to context data.\n\nFor good sharding experience all relations between rows of the same table should be stored in separate table.\n\n```\n+----------+\n| clients  |\n+----------+\n| id       |----------------------+\n| login    |                      |\n| password |                      |\n+----------+                      |\n                                  |\n          +--------------------+  |\n          | albums             |  |\n          +--------------------+  |\n  +-+=====| id                 |  |\n  | |     | client_id          |\u003e-+\n  | |     | name               |\n  | |     +--------------------+\n  | |\n  | |    +------------------+\n  | |    | album_hierarchy  |\n  | |    +------------------+\n  | +---\u003c| parent_album_id  |\n  +-----\u003c| child_album_id   |\n         +------------------+ \n```\n\n### Triggers\n\nTriggers cannot modify rows.\nOr roar too loud :)\n\n```\n            +----------+\n            | clients  |\n            +----------+\n +----------| id       |------------------+\n |          | login    |                  |\n |          | password |                  |\n |          +----------+                  |\n |                                        |\n |  +------------+        +------------+  |\n |  | blog_posts |        | activities |  |\n |  +------------+        +------------+  |\n |  | id         |        | id         |  |\n +-\u003c| client_id  |        | client_id  |\u003e-+\n    | content    |        | counter    |\n    +------------+        +------------+\n          :                      :\n          :                      :\n (on insert post create or increase activity)\n```\n\nThis is common usage of a trigger to automatically aggregate some statistics.\nVery useful and safe - doesn't matter which part of application adds new blog post,\nactivities counter will always go up.\n\nHowever, when sharding this causes a lot of trouble when inserting client data.\nLet's say he has 4 blog posts and 4 activities.\nIf posts are inserted first they bump activity counter through trigger and we have collision in `activties` table due to unexpected row.\nWhen activities are inserted first they are unexpectedly increased by posts inserts later, ending with invalid 8 activities total.\n\nIn sharding triggers can only be used if they do not modify data.\nFor example it is OK to do sophisticated constraints using them.\nTriggers that modify data must be removed and their logic ported to application.\n\n### Checkpoint\n\nCheck if there are any issues described above in your printed schema and fix them.\n\nAnd this is probably the most annoying part of sharding process\nas you will have to dig through a lot of code.\nSometimes old, untested undocumented and unmaintained.\n\nWhen you are done your printed schema on the wall should not contain any unclassified tables.\n\nReady for next step?\n\n## Prepare schema\n\nIt is time to dump monolithic database complete schema (tables, triggers, views and functions/procedures) to the `shard_schema.sql` file and prepare for sharding environment initialization.\n\n### Separate neutral tables\n\nMove all tables that are marked as neutral from `shard_schema.sql` file to separate `neutral_schema.sql` file.\nDo not forget to also move triggers, views or procedures associated with them.\n\n### Bigints\n\nEvery primary key on shard should be of `unsigned bigint` type.\nYou do not have to modify your existing schema installed on monolithic database.\nJust edit `shard_schema.sql` file and massively replace all numeric primary and foreign keys to unsigned big integers.\nI'll explain later why this is needed.\n\n### Create schema for dispatcher\n\nDispatcher tells on which shard specific client is located.\nAbsolute minimum is to have table where you will keep client id and shard number.\nSave it to `dispatch_schema.sql` file.\n\nMore complex dispatchers will be described later.\n\n### Dump common data\n\nFrom monolithic database dump data for neutral tables to `neutral_data.sql` file\nand for context tables to `context_data.sql` file.\nWatch out for tables order to avoid breaking foreign keys constraints.\n\n### Checkpoint\n\nYou should have `shard_schema.sql`, `neutral_schema.sql`, `dispatch_schema.sql`, `neutral_data.sql` and `context_data.sql` files.\n\nAt this point you should also freeze all schema and common data changes in your application until sharding is completed.\n\n## Set up environment\n\nFinally you can put all those new, shiny machines to good use.\n\nTypical sharding environment contains of:\n\n* Database for neutral data.\n* Database for dispatch.\n* Databases for shards.\n\nEach database should of course be replicated.\n\n### Database for neutral data\n\nNothing fancy, just regular database.\nInstall `neutral_schema.sql` and feed `neutral_data.sql` to it.\n\nMake separate user for application with read-only grants to read neutral data\nand separate user with read-write grants for managing data.\n\n### Database for dispatch\n\nEvery time client logs in to your product you will have to find which shard he is located on.\nMake sure all data fits into RAM, have a huge connection pool available.\nAnd install `dispatch_schema.sql` to it.\n\nThis is a weak point of all sharding designs.\nShould be off-loaded by various caches as much as possible.\n\n### Databases for shards\n\nThey should all have the same power (CPU/RAM/IO) -\nthis will speed things up because you can just randomly select shard for your new or migrated client without bothering with different hardware capabilities.\n\nConfiguration of shard databases is pretty straightforward.\nFor every shard just install `shard_schema.sql`, feed `context_data.sql` file and follow two more steps.\n\n### Databases for shards - users\n\nRemember that context tables should be identical on all shards.\nTherefore it is a good idea to have separate user with read-write grants for managing context data.\nApplication user should have read-only access to context tables to prevent accidental context data change.\n\nThis ideal design may be too difficult to maintain - every new table will require setting up separate grants.\nIf you decide to go with single user make sure you will add some mechanism that monitors context data consistency across all shards.\n\n### Databases for shards - mutually exclusive primary keys\n\nPrimary keys in client tables ***must be globally unique across whole product***.\n\nFirst of all - data split is a long process. Just pushing data between databases may take days or even weeks!\nAnd because of that it should be performed without any global downtime.\nSo during monolithic to shard migration phase new rows will still be created in monolithic database\nand already migrated users will create rows on shards. Those rows must never collide.\n\nSecond of all - sharding does not end there.\nLater on you will have to load balance shards, move client between different physical locations, backup and restore them if needed.\nSo rows must never collide at any time of your product life.\n\nHow to achieve that? Use offset and increment while generating your primary keys.\n\nMySQL has ready to use mechanism: \n\n* auto_increment_increment - Set this global variable to 100 on all of your shards. That is also the maximum amount of shards you can have. Be generous here, as it will not be possible to change it later! You must have spare slots even if you don't have such amount of shards right now!\n* auto_increment_offset - Set this global value differently on all of your shards. First shard should get 1, second shard should get 2, and so on. Of course you cannot exceed value of auto_increment_increment.\n\nNow your first shard for any table will generate 1, 101, 201, 301, ... , 901, 1001, 1101 auto increments and second shard will generate 2, 102, 202, 302, ... , 902, 1002, 1102 auto increments.\nAnd that's all! Your new rows will never collide, doesn't matter which shard they were generated on and without any communication between shards needed.\n\nTODO: add recipes for another database types\n\nNow you should understand why I've told you to convert all numerical primary and foreign keys to unsigned big integers. The sequences will grow really fast, in our example 100x faster than on monolithic database.\n\n***Remember to set the same increment and corresponding offsets on replicas.*** Forgetting to do so will be lethal to whole sharding design.\n\n### Checkpoint\n\nYour database servers should be set up. Check routings from application, check user grants.\nAnd again - remember to have correct configurations (in puppet for example) for shards and their replicas offsets.\nDo some failures simulations.\n\nAnd move to the next step :)\n\n### Q\u0026A\n\n***Q:*** Can I do sharding without dispatch database? When client wants to log in I can just ask all shards for his data and use the one shard that will respond.\n\n***A:*** No. This may work when you start with three shards, but when you have 64 shards in 8 different data centers such fishing queries become impossible. Not to mention you will be very vulnerable to brute-force attacks - every password guess attempt will be multiplied by your application causing instant overload of the whole infrastructure.\n\n\n***Q:*** Can I use any no-SQL technology for dispatch and neutral databases?\n\n***A:*** Sure. You can use it instead of traditional SQL or as a supplementary cache.\n\n## Adapt code\n\n### Product\n\nThere will be additional step in your product. When user logs in then dispatch database must be asked for shard number first. Then you connect to this shard and... it works!\nYour code will also have to use separate database connection to access neutral data.\nAnd it will have to roll shard when new client registers and note this selection in dispatch database.\n\nThat is the beauty of whole clients sharding - majority of your code is not aware of it.\n\n### Synchronization of common data\n\nIf you modify neutral data this change should be propagated to every neutral database (you may have more of those in different physical locations).\n\nSame thing applies to context data on shard, but ***all auto increment columns must be forced***. This is because every shard will generate different sequence. When you execute ```INSERT INTO skins (color) VALUES ('#AA55BC')``` then shard 1 will assign different IDs for them than shard 2. And all client data that reference this language will be impossible to migrate between shards.\n\n### Dispatch\n\nDispatch serves two purposes. It allows you to find client on shard by some unique attribute (login, email, and so on)\nand it also helps to guarantee such uniqueness. So for example when new client is created then dispatch database must be asked if chosen login is available. Take an extra care of dispatch database. Off-load as much as you can by caching and schedule regular consistency checks between it and the shards.\n\nThings get complicated if you have shards in many data centers. Unfortunately I cannot give you universal algorithm of how to keep them in sync, this is too much application specific.\n\n### Analytic tools\n\nBecause your clients data will be scattered across multiple shard databases you will have to fix a lot of global queries used in analytical and statistical tools. What was trivial previously - for example `SELECT city.name, COUNT(*) AS amount FROM clients JOIN cities ON clients.city_id = cities.id GROUP BY city.name ORDER BY amount DESC LIMIT 8` - will now require gathering all data needed from shards and performing intermediate materialization for grouping, ordering and limiting.\n\nThere are tools that helps you with that. I've tried several solutions, but none was versatile, configurable and stable enough that I could recommend it.\n\n### Make a wish-ard \n\nWe got to the point when you have to switch your application to sharding flow.\nTo avoid having two versions of code - old one for still running monolithic design and a new one for sharding, we will just connect monolithic database as another \"fake\" shard.\n\nFirst you have to deal with auto increments. Set up in the configuration the same increment on your monolithic database as on shards and set up any free offset. Then check what is the current auto increment value for every client or context table and set the same value for this table located on every shard. Now your primary keys won't collide between \"fake\" and real shards during the migration. But beware: this can easily overflow tiny or small ints in your monolithic database, for example just adding three rows can overflow tiny int unsigned capacity of 255.\n\nAfter that synchronize data on dispatch database - every client you have should point to this \"fake\" shard. Deploy your code changes to use dispatch logic.\n\n### Checkpoint\n\nYou should be running code with complete sharding logic but on reduced environment - with only one \"fake\" shard made out of your monolithic database. You may also already enable creating new client accounts on your real shards.\n\nTons of small issues to fix will pop up at this point. Forgotten pieces of code, broken analytics tools, broken support panels, need of neutral or dispatch databases tune up.\n\nAnd when you squash all the bugs it is time for grande finale: clients migration.\n\n## Migrate clients to shards\n\n### Downtime semaphore\n\nYou do not need any global downtime to perform clients data migration. Disabling your whole product for a few weeks would be unacceptable and would cause huge financial loses. But you need some mechanism to disable access for individual clients while they are migrated. Single flag in dispatch databases should do, but your code should be aware of it and present nice information screen for client when he tries to log in. And of course do not modify clients data.\n\n### Timing\n\nIf you have some history of your client habits - use it. For example if client is usually logging in at 10:00 and logging out at 11:00 schedule his migration to another hour. You may also figure out which timezones clients are in and schedule migration for the night for each of them. The migration process should be as transparent to client as possible. One day he should just log in and bam - fast and responsive product all of a sudden.\n\n### UpRooted tool\n\nReady for some heavy lifting?\n[UpRooted](https://github.com/bbkr/UpRooted) tool for [Raku](https://www.raku.org) language allows to read tree of data from relational database and write it directly to another database. Here is example for MySQL:\n\n\n```\n    use DBIish;\n    use UpRooted::Schema::MySQL;\n    use UpRooted::Tree;\n    use UpRooted::Reader::MySQL;\n    use UpRooted::Writer::MySQL;\n    \n    my $monolithic-connection = DBIish.connect( 'mysql', host =\u003e ..., port =\u003e ..., ... );\n    my $shard-connection = DBIish.connect( 'mysql', host =\u003e ..., port =\u003e ..., ... );\n    \n    # discover schema\n    my $schema = UpRooted::Schema::MySQL.new( connection =\u003e $monolithic-connection );\n    \n    # define which table is root of data tree\n    my $tree = UpRooted::Tree.new( root-table =\u003e $schema.table( 'clients' ) );\n    \n    # monolithic database is the source of data\n    my $reader = UpRooted::Reader::MySQL.new( connection =\u003e $monolithic-connection, :$tree );\n\n    # shard database is destination for data\n    my $writer = UpRooted::Writer::MySQL.new( connection =\u003e $shard-connection );\n    \n    # start cloning client of id = 1\n    $writer.write( :$reader, id =\u003e 1 );\n```\n\nUpdate dispatch shard for this client, check that product works for him and remove his rows from monolithic database.\n\nRepeat for every client.\n\n### MySQL issues\n\nDo not use user with SUPER grant to perform migration. Not because it is unsafe, but because they do not have locales loaded by default. You may end up with messed character encodings if you do so.\n\nMySQL is quite dumb when it comes to cascading DELETE operations. If you have such schema\n\n```\n            +----------+\n            | clients  |\n            +----------+\n +----------| id       |----------------+\n |          | login    |                |\n |          | password |                |\n |          +----------+                |\n |                                      |\n |  +-----------+        +-----------+  |\n |  | foo       |        | bar       |  |\n |  +-----------+        +-----------+  |\n |  | id        |----+   | id        |  |\n +-\u003c| client_id |    |   | client_id |\u003e-+\n    +------------+   +--\u003c| foo_id    |\n                         +----------+\n```\n\nand all relations are ON DELETE CASCADE then sometimes it cannot resolve proper order and may try to delete data from `foo` table before data from `bar` table, causing constraint error. In such cases you must help it a bit and manually delete clients data from `bar` table before you will be able to delete row from `clients` table.\n\n\n## Cleanup\n\nWhen all of your clients are migrated simply remove your \"fake\" shard from infrastructure.\n\n## Contact\n\nIf you have any questions about database sharding or want to contribute to this guide or Exodus tool contact me in person or on IRC as \"bbkr\".\n\n# Congratulations\n\nYOU'VE PROVEN TOO TOUGH FOR [monolithic database design] HELL TO CONTAIN\n(DOOM quote)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbbkr%2Fexodus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbbkr%2Fexodus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbbkr%2Fexodus/lists"}