{"id":22382480,"url":"https://github.com/jcoreio/postgres-shard-coordinator","last_synced_at":"2026-05-10T16:37:59.080Z","repository":{"id":78315489,"uuid":"164362496","full_name":"jcoreio/postgres-shard-coordinator","owner":"jcoreio","description":null,"archived":false,"fork":false,"pushed_at":"2024-08-06T20:53:19.000Z","size":1405,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-26T19:45:42.032Z","etag":null,"topics":["postgres","sharding"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jcoreio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-01-07T01:31:28.000Z","updated_at":"2024-08-06T20:52:13.000Z","dependencies_parsed_at":"2024-03-01T02:30:27.394Z","dependency_job_id":"8a711cc4-85d6-4440-b74f-69f7051d4d36","html_url":"https://github.com/jcoreio/postgres-shard-coordinator","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"purl":"pkg:github/jcoreio/postgres-shard-coordinator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jcoreio%2Fpostgres-shard-coordinator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jcoreio%2Fpostgres-shard-coordinator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jcoreio%2Fpostgres-shard-coordinator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jcoreio%2Fpostgres-shard-coordinator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jcoreio","download_url":"https://codeload.github.com/jcoreio/postgres-shard-coordinator/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jcoreio%2Fpostgres-shard-coordinator/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266580322,"owners_count":23951198,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["postgres","sharding"],"created_at":"2024-12-05T00:13:12.243Z","updated_at":"2026-05-10T16:37:54.034Z","avatar_url":"https://github.com/jcoreio.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# @jcoreio/postgres-shard-coordinator\n\n[![CircleCI](https://circleci.com/gh/jcoreio/postgres-shard-coordinator.svg?style=svg\u0026circle-token=b008aa0f99297a9d8fb7ac7d0d1329b1a1e16f95)](https://circleci.com/gh/jcoreio/postgres-shard-coordinator)\n[![Coverage Status](https://codecov.io/gh/jcoreio/postgres-shard-coordinator/branch/master/graph/badge.svg)](https://codecov.io/gh/jcoreio/postgres-shard-coordinator)\n[![semantic-release](https://img.shields.io/badge/%20%20%F0%9F%93%A6%F0%9F%9A%80-semantic--release-e10079.svg)](https://github.com/semantic-release/semantic-release)\n[![Commitizen friendly](https://img.shields.io/badge/commitizen-friendly-brightgreen.svg)](http://commitizen.github.io/cz-cli/)\n[![npm version](https://badge.fury.io/js/%40jcoreio%2Fpostgres-shard-coordinator.svg)](https://badge.fury.io/js/%40jcoreio%2Fpostgres-shard-coordinator)\n\nHelps processes pick a unique shard index and determine the number of shards,\nusing Postgres to coordinate registration.\n\n# Introduction\n\nThis is designed for any situation where batch processing needs to be divided\nbetween multiple processes using hash-based sharding. For example, Clarity uses\nmultiple processes to handle the notification queue; each process restricts\nitself to events where\n\n```\n  knuth_hash(userId) % numShards \u003e= (shard * MAX_USER_ID) / numShards \u0026\u0026\n  knuth_hash(userId) % numShards \u003c ((shard + 1) * MAX_USER_ID) / numShards\n```\n\nEach of these processes can use `@jcoreio/postgres-shard-coordinator` to pick a unique\n`shard` index and determine the total `numShards` (number of processes) in a\ndecentralized fashion that automatically adapts as processes are spawned or die.\n\n# Usage\n\n```\nnpm i --save @jcoreio/postgres-shard-coordinator\n```\n\n## Database migration\n\nYou will need to perform provided migrations to create the tables and functions\nfor coordination:\n\n```js\nimport { Client } from 'pg'\nimport Umzug from 'umzug'\nimport { umzugMigrationOptions } from '@jcoreio/postgres-shard-coordinator'\n\nexport default async function migrate({ database }) {\n  const umzug = new Umzug({\n    storage: 'umzug-postgres-storage',\n    storageOptions: {\n      database,\n      relation: '\"SequelizeMeta\"',\n      column: 'name',\n    },\n    migrations: {\n      ...umzugMigrationOptions(),\n      params: [{ query }],\n    },\n  })\n\n  await umzug.up()\n}\n\nasync function query(sql) {\n  const client = new Client(database)\n  try {\n    await client.connect()\n    migrationDebug(sql)\n    return await client.query(sql)\n  } finally {\n    await client.end()\n  }\n}\n```\n\n## Shard registration\n\n```js\nimport { ShardRegistrar } from '@jcoreio/postgres-shard-coordinator'\nimport { Pool } from 'pg'\nimport PgIpc from '@jcoreio/pg-ipc'\nimport requireEnv from '@jcoreio/require-env'\nimport migrate from './migrate'\n\nconst database = {\n  user: requireEnv('DB_USER'),\n  host: requireEnv('DB_HOST'),\n  database: requireEnv('DB_NAME'),\n  password: requireEnv('DB_PASSWORD'),\n  port: parseInt(requireEnv('DB_PORT')),\n}\n\nconst ipc = new PgIpc({\n  newClient: () =\u003e new Client(database),\n})\nipc.on('error', (err) =\u003e console.error(err.stack))\n\nconst pool = new Pool(database)\n\nconst registrar = new ShardRegistrar({\n  pool,\n  ipc,\n  cluster: 'clarity_notifications',\n  heartbeatInterval: 60, // seconds\n  gracePeriod: 30, // seconds\n  reshardInterval: 60, // seconds\n})\n\nregistrar.on('shardChanged', ({ shard, numShards }) =\u003e {\n  // reconfigure the notification queue processor\n})\nregistrar.on('error', (err) =\u003e console.error(err.stack))\n\nasync function go() {\n  await migrate({ database })\n  await registrar.start()\n}\n\ngo()\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjcoreio%2Fpostgres-shard-coordinator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjcoreio%2Fpostgres-shard-coordinator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjcoreio%2Fpostgres-shard-coordinator/lists"}