{"id":25996908,"url":"https://github.com/clickup/pg-microsharding","last_synced_at":"2025-03-05T16:55:28.703Z","repository":{"id":248452196,"uuid":"741396430","full_name":"clickup/pg-microsharding","owner":"clickup","description":"Support for microshards on top of PostgreSQL schemas with cross-nodes rebalancing","archived":false,"fork":false,"pushed_at":"2025-02-20T16:28:42.000Z","size":242,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-20T17:36:14.753Z","etag":null,"topics":["microshards","postgresql","sharding"],"latest_commit_sha":null,"homepage":"https://www.npmjs.com/package/@clickup/pg-microsharding","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/clickup.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-10T10:03:40.000Z","updated_at":"2025-02-20T16:28:41.000Z","dependencies_parsed_at":"2024-07-15T03:22:16.513Z","dependency_job_id":"a8fb4c74-cb13-44f7-b247-6c115606e2a3","html_url":"https://github.com/clickup/pg-microsharding","commit_stats":null,"previous_names":["clickup/pg-sharding","clickup/pg-microsharding"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clickup%2Fpg-microsharding","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clickup%2Fpg-microsharding/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clickup%2Fpg-microsharding/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clickup%2Fpg-microsharding/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/clickup","download_url":"https://codeload.github.com/clickup/pg-microsharding/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242067705,"owners_count":20066750,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["microshards","postgresql","sharding"],"created_at":"2025-03-05T16:55:26.869Z","updated_at":"2025-03-05T16:55:28.696Z","avatar_url":"https://github.com/clickup.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pg-microsharding: Microshards Support for PostgreSQL\n\nSee also [TypeScript API documentation](https://github.com/clickup/pg-microsharding/blob/master/docs/globals.md).\n\n[![CI run](https://github.com/clickup/pg-microsharding/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/clickup/pg-microsharding/actions/workflows/ci.yml/badge.svg?branch=main)\n\nThe [pg-microsharding](https://www.npmjs.com/package/@clickup/pg-microsharding) CLI tool enables microshard schemas management across multiple PostgreSQL servers. You can do the following:\n\n* Add and remove microshard schemas.\n* Activate and deactivate schemas.\n* Enumerate active microshard schemas.\n* View the entire cluster layout: what microshard schemas are where, of what size, and how many reads/writes do the experience.\n* Move a microshard from one PostgreSQL server to another with no downtime.\n* Automatically rebalance microshards among multiple servers, so that each server will become of approximately the same size.\n* Weighted rebalancing: when one server looks overloaded, you can \"dissolve out\" some shards from it to other servers to achieve equal load.\n\nEach microshard is a PostgreSQL schema with numeric suffix. Microshard schemas have the same set of tables with same names; it's up to the higher-level tools to keep the schemas of all those tables in sync (e.g. see [pg-mig](https://www.npmjs.com/package/@clickup/pg-mig) tool).\n\n## Usage\n\n```\npg-microsharding list | ls\n  [--weight-sql='SELECT returning weight with optional unit']\n  [--verbose]\n  [--dsn=DSN | --dsns=DNS1,DSN2,...]\n\npg-microsharding allocate\n  --shard=N | --shards=N-M\n  --migrate-cmd='shell command to run migrations'\n  --activate={yes | no}\n  [--dsn=DSN | --dsns=DNS1,DSN2,...]\n\npg-microsharding factor\n  --shard=N | --shards=N,M,... | --shards=DSN-PREFIX\n  --factor=P|+P.Q|-P.Q|\"*P.Q\"\n  [--dsn=DSN | --dsns=DNS1,DSN2,...]\n\npg-microsharding move\n  --shard=N\n  --from=DSN\n  --to=DSN\n  --activate-on-destination={yes | no}\n  [--deactivate-sql='SQL $1 SQL']\n\npg-microsharding rebalance\n  --activate-on-destination={yes | no}\n  [--deactivate-sql='SQL $1 SQL']\n  [--weight-sql='SELECT returning weight with optional unit']\n  [--decommission=DSN1,DSN2,...]\n  [--parallelism=N]\n  [--dsn=DSN | --dsns=DNS1,DSN2,...]\n\npg-microsharding cleanup\n  [--dsn=DSN | --dsns=DNS1,DSN2,...]\n```\n\n## Environment Variables\n\nThe tool receives parameters from command line option, but allows to set defaults for most of them using environment variables.\n\nSome variables are standard for psql command:\n\n* `PGUSER`: default database user\n* `PGPASSWORD`: default database password\n* `PGHOST`: default database host (or multiple hosts, comma-separated)\n* `PGPORT`: default database port\n* `PGDATABASE`: default database name\n* `PGSSLMODE`: default SSL mode (e.g. \"prefer\")\n\nCustom variables of the tool itself:\n\n* `DSNS` or `PGDSNS` or `PGHOST`: default value for `--dsns` option, comma-separated list of DSNs (see below)\n* `MIGRATE_CMD`: default value for `--migrate-cmd` option\n* `WEIGHT_SQL`: default value for `--weight-sql` option\n* `DEACTIVATE_SQL`: default value for `--deactivate-sql` option\n\n## Configuration File: pg-microsharding.config.ts\n\nInstead of setting the environment variables, you can export the same exact values in `pg-microsharding.config.ts` file by e.g. deriving them directly from the [Ent Framework](https://ent-framework.org/) cluster configuration:\n\n```javascript\nimport { cluster } from \"ents/cluster\";\n\nexport default async function(action: \"apply\" | \"undo\" | string) {\n  const islands = cluster.options.islands();\n  return {\n    PGDSNS: islands\n      .map((island) =\u003e island.nodes.map(({ host }) =\u003e host)\n      .flat()\n      .join(\",\"),\n    PGPORT: 5432, // we don't want to use pgbouncer port here\n    PGUSER: firstNode.user,\n    PGPASSWORD: firstNode.password,\n    PGDATABASE: firstNode.database,\n    PGSSLMODE: firstNode.ssl ? \"prefer\" : undefined,\n    MIGRATE_CMD: \"yarn -s pg-mig\",\n  };\n}\n```\n\nThe file `pg-microsharding.config.ts` is searched in all parent folders starting from the current working directory when `pg-microsharding` is run (typically you want to have it in the root of your project, near the other configuration files).\n\nYou can export-default a regular function, an async function, or even a plain constant object.\n\n## DSN and Addressing Databases\n\nOption `--dsns`, if required, should be a comma separated list of DSNs.\n\nAlso, you may pass duplicated DSNs and even DSNs of replicas: the tool will filter them out and remain only master DSNs in the list.\n\nDSN format examples (parts defaults are from environment variables):\n\n* `postgresql://user:pass@hostname/db?options` (all parts are optional)\n* `hostname:port/db` (all parts except the hostname are optional\n\n## Command Line Tool\n\nThe `pg-microsharding` library consists of 2 parts:\n\n1. A CLI tool allowing you to manipulate with microshards.\n2. A set of PostgreSQL stored functions to call them from your app.\n\n### Show Cluster Layout: pg-microsharding list\n\n```bash\npg-microsharding list\n```\n\nThis action prints the list of all PostgreSQL islands (pointed by DNSn), microshards and some statistics.\n\nIn `--verbose` mode, also prints detailed statistics about insert/update/delete, index scans and seqscans.\n\n\u003cdiv align=\"center\"\u003e\u003cfigure\u003e\u003cimg src=\"https://raw.githubusercontent.com/dimikot/ent-framework/refs/heads/main/gitbook/.gitbook/assets/pg-microsharding-list.png\" alt=\"\" width=\"563\"\u003e\u003cfigcaption\u003e\u003c/figcaption\u003e\u003c/figure\u003e\u003c/div\u003e\n\n### Allocate New Microshards: pg-microsharding allocate\n\n```typescript\npg-microsharding allocate --shards=301-309 --activate=yes\n```\n\nThis action allows you to create more microshard schemas in the cluster. The microshards are created on PostgreSQL the host pointed by the 1st DSN, so after it's done, run `pg-microsharding rebalance` to spread that new schemas across other nodes.\n\nEach microshard can either be \"active\" or \"inactive\". When you create them, you tell the tool, should the microshards become active immediately (and thus, visible to `microsharding_list_active_shards()` API) or not. You can always activate the schemas later using the same exact command (it is idempotent).\n\nThe tool runs `--migrate-cmd` command right after creating the inactive microshards, assuming that your migration tool will initialize them properly.\n\n\u003cfigure\u003e\u003cimg src=\"https://raw.githubusercontent.com/dimikot/ent-framework/refs/heads/main/gitbook/.gitbook/assets/pg-microsharding-allocate.png\" alt=\"\"\u003e\u003cfigcaption\u003e\u003c/figcaption\u003e\u003c/figure\u003e\n\n\u003cfigure\u003e\u003cimg src=\"https://raw.githubusercontent.com/dimikot/ent-framework/refs/heads/main/gitbook/.gitbook/assets/pg-microsharding-allocate-list.png\" alt=\"\" width=\"375\"\u003e\u003cfigcaption\u003e\u003c/figcaption\u003e\u003c/figure\u003e\n\n### Move One Microshard: pg-microsharding move\n\n```bash\npg-microsharding move \\\n  --shard=42 --from=host1 --to=host2 \\\n  --activate-on-destination=yes\n```\n\nMicroshards can be moved from one PostgreSQL node to another. There is no need to stop writes while moving microshards: the tool uses PostgreSQL logical replication to stream each microshard table's data, and in the very end, acquires a quick write lock to finalize the move.\n\nThere are many aspects and corner cases addressed in the move action, here are some of them:\n\n* The move is fast even for large microshards. The tool internally uses the same approach for data copying as `pg_dump`. First recreates the tables structure on the destination, except most of the indexes and foreign key constraints (only the primary key indexes or REPLICA IDENTITY indexes are created at this stage, since they are required for the logical replication to work). Then, it copies the data, utilizing the built-in PostgreSQL tablesync worker; this process is fast, since it inserts the data in bulk and doesn't update indexes. In the end, the tool creates the remaining indexes and foreign key constraints (this is where you may want to increase [maintenance\\_work\\_mem](https://www.postgresql.org/docs/current/runtime-config-resource.html) for the role you pass to pg-microsharding, since it directly affects the indexes creation time). Overall, this approach speeds up the copying by \\~10x comparing to the naive way of using logical subscriptions.\n* At each long running step, the tool shows a descriptive progress information: how many tuples are copied so far, what is the elapsed %, how much time is left, what are the SQL queries it executes (dynamically updatable block in console) etc.\n* It also shows replication lag statistics for all physical replicas of the source and the destination, plus the logical replication lag of the temporary subscription.\n* In the end, the tool activates the microshard on the destination and deactivates on the source, but it does it only when the replication lag in seconds dropped below some reasonable threshold (defaults to 20 seconds, but you can pass a lower value to be on a safe side). So the write lock is guaranteed to be acquired for only a brief moment.\n* The tool runs it all in an automatically created tmux session. If you accidentally disconnect, then just connect back and rerun the same command line: instead of running another move action, if will jump you back in the existing session.\n\nIf you're unsure, you can practice with the move without activating the microshard on the destination (and without deactivating it on the source) by passing `--activate-on-destination=no` option. This is like a \"dry-run\" mode, where the tool does all the work, except the very last step. The moved schema on the destination won't be activated, and it will also be renamed using some long descriptive prefix (including the move date).\n\nAt any moment, you can abort the move with ^C. It is safe: half-moved data will remain on the destination, but the microshard schema will remaim invisible there for e.g. `microsharding_list_active_shards()` API (see below). If you then rerun the `move` action, it will start from scratch.\n\n\u003cfigure\u003e\u003cimg src=\"https://raw.githubusercontent.com/dimikot/ent-framework/refs/heads/main/gitbook/.gitbook/assets/pg-microsharding-move.png\" alt=\"\"\u003e\u003cfigcaption\u003e\u003c/figcaption\u003e\u003c/figure\u003e\n\n### Clean Old Moved Copies: pg-microsharding cleanup\n\n```bash\npg-microsharding cleanup\n```\n\nWhen you move one or more microshards, pg-microsharding doesn't delete the old copy from the source host. Instead, it renames the schema (using some long descriptive prefix with date) and deactivates it.\n\nLater, when you are sure that everything went well, you can remove such \"backup\" copies by running the cleanup action. It is interactive: it will tell you, what it wants to delete, and ask for an explicit confirmation.\n\n### Rebalance All Islands: pg-microsharding rebalance\n\n```bash\npg-microsharding rebalance --activate-on-destination=yes\n```\n\nThis action runs multiple \"move\" sub-actions in parallel, utilizing [tmux](https://github.com/tmux/tmux/wiki) panes. Notice that tmux is required: it allows to resume the rebalancing if your SSH console on the server gets disconnected (in this case, just run the `rebalance` action again, and you'll \"jump into\" the existing session).\n\nBefore running the moves, the action calculates weights of each shard (by default, the weight is the microshard tables size in bytes, mutuplied by per-shard \"weight factor\"; see below). Then, it estimates, which microshards need to be moved to what islands, to achieve a more or less uniform distribution. The algorithm is complicated: among other heuristics, it tries to make sure that each island gets approximately the same number of microshards with comparable sizes (e.g. if you allocate 100 new empty microshards, then rebalancing will spread them across islands uniformly).\n\n\u003cfigure\u003e\u003cimg src=\"https://raw.githubusercontent.com/dimikot/ent-framework/refs/heads/main/gitbook/.gitbook/assets/pg-microsharding-rebalance-plan.png\" alt=\"\" width=\"563\"\u003e\u003cfigcaption\u003e\u003c/figcaption\u003e\u003c/figure\u003e\n\nOnce the rebalancing plan is ready, the tool will print it to you and ask for your confirmation. You can always run `pg-microsharding rebalance` and then press ^C to just see, what _would_ happen if you rebalance.\n\nAfter rebalancing, the result may look like:\n\n\u003cfigure\u003e\u003cimg src=\"https://raw.githubusercontent.com/dimikot/ent-framework/refs/heads/main/gitbook/.gitbook/assets/pg-microsharding-rebalance-after.png\" alt=\"\" width=\"375\"\u003e\u003cfigcaption\u003e\u003c/figcaption\u003e\u003c/figure\u003e\n\nAt any time, you can abort the rebalancing with ^C in any of the tmux panes. It is as safe as aborting the `move` action.\n\n### Evacuate All Microshards from an Island (Decommissioning)\n\n```bash\npg-microsharding rebalance \\\n  --decommission=host2 --activate-on-destination=yes\n```\n\nThis mode of \"rebalance\" action allows you to remove a PostgreSQL host from the cluster, or even upgrade PostgreSQL to the next major version with no downtime. It moves all the microshards from the provided DSN, so the host becomes \"empty\".\u0026#x20;\n\nE.g. after decommissioning, the result may look like (notice that one node became empty):\n\n\u003cfigure\u003e\u003cimg src=\"https://raw.githubusercontent.com/dimikot/ent-framework/refs/heads/main/gitbook/.gitbook/assets/pg-microsharding-decommission.png\" alt=\"\" width=\"375\"\u003e\u003cfigcaption\u003e\u003c/figcaption\u003e\u003c/figure\u003e\n\nNow, you can remove the host from the cluster or upgrade PostgreSQL, then rebalance the microshards back (rebalancing works fine across different major PostgreSQL versions).\n\n### Tweak Island Weights: pg-microsharding factor\n\n```bash\npg-microsharding factor --shards=host1 --factor=\"*1.2\"\n```\n\nImagine you have 10 PostgreSQL islands with rebalanced microshards, and you see on your monitoring charts that some island is loaded more than all other islands. E.g. it may experience higher CPU load, higher disk throughput etc.\n\nSuch situation typically happens, because one microshard became too large, or there is a customer data in some microshard that causes more load than the data of an average customer. In case you don't want to investigate this case too much, you can \"duct tape\" it by artificially \"dissolving\" a fraction of microshards from that overloaded island to other islands.\n\nWhen you run `pg-microsharding factor --factor=\"*1.2\"`, the tool artificially increases the \"weight\" of each microshard on the provided host (in this example, the increase is by 1.2, i.e. by 20%). This information is then remembered in the microshards themselves (and is displayed in `list` action), so you can run rebalancing and \"dissolve\" some of the microshards among other hosts. As a result, your target island will become less loaded (on average), and by repeating this step several times, you can achieve a more fair load distribution.\n\nThe \"weight increase factor\" is technically stored as a SQL comment on the microshard schema, and it travels along with the microshard when you move it.\n\n## PostgreSQL Stored Functions API\n\nThis is the second part of pg-microsharding tool: a set of stored functions you add to your database.\n\n### Installing into the Database\n\nRun the following SQL files in your up- and down-migrations to install (or upgrade) and uninstall the tool:\n\n* sql/pg-microsharding-up.sql: to install/upgrade the library\n* sql/pg-microsharding-down.sql: to uninstall\n\nE.g.:\n\n```sql\n-- mig/20250628100000.add-pg-microsharding.public.up.sql\nCREATE SCHEMA microsharding;\nSET search_path TO microsharding;\n\\ir ../pg-microsharding/sql/pg-microsharding-up.sql\n```\n\n```sql\n-- mig/20250628100000.add-pg-microsharding.public.dn.sql\nSET search_path TO microsharding;\n\\ir ../pg-microsharding/sql/pg-microsharding-down.sql\nDROP SCHEMA microsharding;\n```\n\nIn the above example, we create a separtate schema for the library, but it is not mandatory: you can also install it into schema `public` (all of the API functions have `microsharding_` prefix).\n\n### List Active Shards: microsharding.list\\_active\\_shards()\n\nThis function returns the list of active microshard schemas in the current PostgreSQL database. When using the tool with [Ent Framework](https://ent-framework.org/), mention it in your `Cluster` object:\n\n```typescript\nexport const cluster = new Cluster({\n  shards: {\n    nameFormat: \"sh%04d\",\n    discoverQuery:\n      \"SELECT unnest FROM unnest(microsharding.microsharding_list_active_shards())\",\n  },\n  ...\n});\n```\n\n### Microsharding Debug Views\n\nThe `microsharding_migration_after()` function creates so-called \"debug views\" for each sharded table in your cluster. For instance, it you have `sh0001.users`, `sh0002.users` etc. tables. then it will create a debug view `public.users` with the definition like:\n\n```sql\n-- This is what pg-microsharding creates automatically.\nCREATE VIEW public.users AS\n  SELECT * FROM sh0001.users\n  UNION ALL\n  SELECT * FROM sh0002.users\n  UNION ALL\n  ...;\n```\n\nEven more, if you pass the list of all PostgreSQL hosts, and those hosts can access each other without a password (e.g. they have  `/var/lib/postgresql/N/.pgpass` files), then those debug views will work **across all shards on all nodes, including the remote ones** (using [foreign-data wrapper](https://www.postgresql.org/docs/current/postgres-fdw.html) functionality).\n\nSo **for debugging purposes**, you'll be able to run queries across all microshards in your `psql` sessions. This is typically very convenient.\n\nOf course those **debug views are not suitable for production traffic**: cross-node communication in PostgreSQL, as well as query planning, work not enough inefficiently. Do not even try, use application-level microshards routing, like e.g. [Ent Framework](https://ent-framework.org/) provides.\n\n```\n$ psql\npostgres=# SELECT shard, email FROM users\n  WHERE created_at \u003e now() - '1 hour'::interval;\n-- Prints all recent users from all microshards, including\n-- the microshards on other PosgreSQL nodes! Use for\n-- debugging purposes only.\n```\n\nAs of `microsharding_migration_before()`, you must call it before any changes are applied to your microsharded tables. The function drops all of the debug views mentioned above. E.g. if you remove a column from a table, PostgreSQL would not allow you to do it it this column is mentioned in any of the views, so it's important to drop the views and re-create them afterwards.\n\nTypically, you just call `microsharding_migration_before()` in your pre-migration sequence and then call `microsharding_migration_after()` in your post-migration steps.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclickup%2Fpg-microsharding","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fclickup%2Fpg-microsharding","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclickup%2Fpg-microsharding/lists"}