{"id":14974170,"url":"https://github.com/jrsaunders/shardmatrix","last_synced_at":"2025-10-27T06:31:56.550Z","repository":{"id":49074342,"uuid":"256291351","full_name":"JRSaunders/ShardMatrix","owner":"JRSaunders","description":"Database Sharding system for MYSQL and Postgres","archived":false,"fork":false,"pushed_at":"2023-02-01T21:09:00.000Z","size":325,"stargazers_count":24,"open_issues_count":4,"forks_count":4,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-01T03:22:12.918Z","etag":null,"topics":["async-database-driver","database-sharding","laravel","mysql","postgres"],"latest_commit_sha":null,"homepage":"","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JRSaunders.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-04-16T18:01:39.000Z","updated_at":"2024-09-04T10:33:49.000Z","dependencies_parsed_at":"2023-02-17T11:15:45.256Z","dependency_job_id":null,"html_url":"https://github.com/JRSaunders/ShardMatrix","commit_stats":null,"previous_names":[],"tags_count":27,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JRSaunders%2FShardMatrix","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JRSaunders%2FShardMatrix/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JRSaunders%2FShardMatrix/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JRSaunders%2FShardMatrix/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JRSaunders","download_url":"https://codeload.github.com/JRSaunders/ShardMatrix/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238455088,"owners_count":19475376,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async-database-driver","database-sharding","laravel","mysql","postgres"],"created_at":"2024-09-24T13:50:05.297Z","updated_at":"2025-10-27T06:31:56.115Z","avatar_url":"https://github.com/JRSaunders.png","language":"PHP","readme":"\n[![Latest Stable Version](https://poser.pugx.org/jrsaunders/shard-matrix/v/stable)](https://packagist.org/packages/jrsaunders/shard-matrix)\n[![Total Downloads](https://poser.pugx.org/jrsaunders/shard-matrix/downloads)](https://packagist.org/packages/jrsaunders/shard-matrix)\n[![Latest Unstable Version](https://poser.pugx.org/jrsaunders/shard-matrix/v/unstable)](https://packagist.org/packages/jrsaunders/shard-matrix)\n[![License](https://poser.pugx.org/jrsaunders/shard-matrix/license)](https://packagist.org/packages/jrsaunders/shard-matrix)\n# ShardMatrix for PHP\n\n### Database Sharding system for MYSQL and Postgres\n\n* Requirements\n    * PHP 7.4^\n    \n* Supports:\n    * **A single Yaml configuration file**\n    * **Multiple Nodes** (DB servers)\n    * **Mysql**\n    * **Postgres**\n    * **Mysql \u0026 Postgres can be used together** and hot swapped\n    * **Multiple Geo Locations**\n    * **UUIDs** bakes in all relevant data for tables and to which node it belongs\n    * **Docker**\n    * **Kubernetes**\n    * **Fast Asynchronous DB queries** (using a purpose built GoThreaded service https://github.com/jrsaunders/go-threaded | https://hub.docker.com/r/jrsaunders/gothreaded or PHP Forking for crons or dev work)\n    * Caching to File or to **Redis** or **MemcacheD**\n    * Unique table columns across nodes\n    * Table Grouping to ensure data is kept in the right shards so joins can be done\n    * Using popular ORM from Laravel ( though your project does **not** need be in Laravel ) https://laravel.com/docs/7.x\n    * Query building being database agnostic\n    * Efficient pagination system across Nodes using caching\n    * Raw SQL Queries\n    \n## Quick Usage\n\nOnce you have initiated it as outlined in the [INSTALLATION](#installation) section below - here are some quick examples of usage.\n\n_If you are familiar with the ORM in Laravel - this is just an extension of that._\n\n### Create Table\n* Creates Table across all appropriate Nodes (Mysql and Postgres simultaneously).  This follows the guidance you have given in your Yaml Config file as to what tables belong on what nodes\n```php\nuse ShardMatrix\\DB\\Builder\\Schema;\n\n# Creates Table across all appropriate Nodes (Mysql and Postgres simultaneously).\n# This follows the guidance you have given in your Yaml Config file as to what tables\n# belong on what nodes\n\nSchema::create( 'users',\n    function ( \\Illuminate\\Database\\Schema\\Blueprint $table ) {\n          \n        $table-\u003estring( 'uuid', 50 )-\u003eprimary();\n        $table-\u003estring('username',255)-\u003eunique();\n        $table-\u003estring('email',255)-\u003eunique();\n        $table-\u003einteger('something');\n        $table-\u003edateTime( 'created' );\n\n    } \n);\n```\n\n\n### Insert Record\n* Insert Data - the system will choose an appropriate shard node and create a UUID for it that will be attributed to an appropriate node\n```php\nuse ShardMatrix\\DB\\Builder\\DB;\n\n# Insert Data - the system will choose an appropriate shard node and create a UUID for it that will be attributed to an appropriate node\n\n$uuid = DB::table( 'users' )-\u003einsert( \n    [\n\t'username' =\u003e 'jack-malone',\n\t'password' =\u003e 'poootpooty',\n\t'created'   =\u003e (new \\DateTime())-\u003eformat('Y-m-d H:i:s'),\n\t'something' =\u003e 5,\n\t'email'    =\u003e 'jack.malone@yatti.com',\n    ]\n);\n\necho $uuid-\u003etoString();\n# outputs 06a00233-1ea8af83-9b6f-6104-b465-444230303037\n\necho $uuid-\u003egetNode()-\u003egetName();\n# outputs DB0007\n\necho $uuid-\u003egetTable()-\u003egetName();\n# outputs users\n\n```\n**Inserted Data**\n```\nuuid        06a00233-1ea8af83-9b6f-6104-b465-444230303037\nusername    jack-malone\npassword    poootpooty\nemail       jack.malone@yatti.com\ncreated     2020-04-30 15:35:31.000000\nsomething   5\n```\n\n* Any further inserts done in this php process will be inserted into the same shard, if in the correct table group\n\n### Get Record By UUID and Update Record\n* Get the record directly from the correct node (shard)\n* Manipulate the record\n* Update the record\n\n```php\n    use ShardMatrix\\DB\\Builder\\DB;\n    use ShardMatrix\\DB\\Interfaces\\DBDataRowTransactionsInterface;\n\n    # Get the record directly from the correct node (shard)\n    $record = DB::getByUuid( '06a00233-1ea8af83-9b6f-6104-b465-444230303037' );\n\n    # Manipulate the record\n    if ( $record \u0026\u0026 $record instanceof DBDataRowTransactionsInterface) {\n\n        # As above you could run an additional check for the instance of the record returned, but it should always follow this interface through the query builder\n        \n    \techo $record-\u003eusername;\n    \t# outputs jack-malone\n    \t\n    \techo $record-\u003eemail;\n    \t# outputs jack.malone@yatti.com\n    \t\n    \t# overwrite the email attribute\n    \t$record-\u003eemail = 'anotheremail@yatti.com';\n    \n    \t# Update the record\n    \t$record-\u003esave();\n    }\n\n```\n\n### Query Data and Conditionally Delete a Record\n* Query all relevant nodes for the data\n* Data returns as a Collection that can be iterated through\n* Use data conditionally\n* Manipulate the record and commit changes\n\n```php\nuse ShardMatrix\\DB\\Builder\\DB;\nuse ShardMatrix\\DB\\Interfaces\\DBDataRowTransactionsInterface;\n\n# Query all relevant nodes for the data\n$collection = DB::allNodesTable( 'users')-\u003ewhere('email','like','%yatti%')-\u003elimit(50)-\u003eget();\n\n# Data returns as a Collection that can be iterated through\n$collection-\u003eeach( function(DBDataRowTransactionsInterface $record){\n\n    # Use data conditionally\n\tif($record-\u003eusername == 'a-bad-user'){\n        \n        # Manipulate the record and commit changes\n        $record-\u003edelete();\n\t}\n\n});\n\n```\n\n## Pagination\n\n### Pagination of Data from all shards\n\n\n```php\nuse ShardMatrix\\DB\\Builder\\DB;\nuse ShardMatrix\\DB\\Interfaces\\DBDataRowTransactionsInterface;\n\n$pagination = DB::allNodesTable( 'users' )\n              -\u003eorderBy( 'created', 'desc' )\n              -\u003epaginate();\n\n$pagination-\u003eeach( function ( DBDataRowTransactionsInterface $record) {\n\n\techo $record-\u003eusername;\n\n\techo $record-\u003egetUuid();\n});\n\necho $pagination-\u003etotal();\n\necho $pagination-\u003eperPage();\n\necho $pagination-\u003enextPageUrl();\n\necho $pagination-\u003epreviousPageUrl();\n```\n### Pagination of Data from one shard defined by the UUID location\n\n```php\nuse ShardMatrix\\DB\\Builder\\DB;\nuse ShardMatrix\\DB\\Interfaces\\DBDataRowTransactionsInterface;\n\n$uuidFromCurrentUser = \"06a00233-1ea8af83-d514-6a76-83ae-444230303037\";\n\n$pagination = DB::table( 'users' )\n              -\u003euuidAsNodeReference($uuidFromCurrentUser)\n              -\u003eorderBy( 'created', 'desc' )\n              -\u003epaginate();\n\n$pagination-\u003eeach( function ( DBDataRowTransactionsInterface $record) {\n\n\techo $record-\u003eusername;\n\n\techo $record-\u003egetUuid();\n});\n\necho $pagination-\u003etotal();\n\necho $pagination-\u003eperPage();\n\necho $pagination-\u003enextPageUrl();\n\necho $pagination-\u003epreviousPageUrl();\n```\n\n    \n# \u003ca id=\"installation\"\u003e\u003c/a\u003e Installation\n\n## Installing ShardMatrix for PHP\n\nUse Composer to install ShardMatrix, or pull the repo from github.\n\n```\ncomposer require jrsaunders/shard-matrix\n```\n\n## Preparing the YAML config file\n\nShardMatrix needs to know how your tables and columns and databases interact, so this config file will define this in a simple yaml file.\n* You will need your credentials for your databases, and access privileges setup.\n[Reference Yaml file](shard_matrix.yaml)\n\n### Example\nThis is a full example of how a configuration file should look.\n```yaml\nversion: 1\n\ntable_groups:\n  user:\n    - users\n    - payments\n    - offers\n  tracking:\n    - visitors\n    - sign_ups\n  published:\n    - published_offers\n\nunique_columns:\n  users:\n    - email\n    - username\n\nnodes:\n  DB0001:\n    dsn: mysql:dbname=shard;host=localhost:3301;user=root;password=password\n    docker_network: DB0001:3306\n    geo: UK\n    table_groups:\n      - user\n      - published\n  DB0002:\n    dsn: mysql:dbname=shard;host=localhost:3302;user=root;password=password\n    docker_network: DB0002:3306\n    geo: UK\n    table_groups:\n      - user\n      - published\n  DB0003:\n    dsn: mysql:dbname=shard;host=localhost:3303;user=root;password=password\n    docker_network: DB0003:3306\n    geo: UK\n    table_groups:\n      - user\n      - published\n  DB0004:\n    dsn: mysql:dbname=shard;host=localhost:3304;user=root;password=password\n    docker_network: DB0004:3306\n    geo: UK\n    table_groups:\n      - published\n  DB0005:\n    dsn: mysql:dbname=shard;host=localhost:3305;user=root;password=password\n    docker_network: DB0005:3306\n    table_groups:\n      - tracking\n  DB0006:\n    dsn: mysql:dbname=shard;host=localhost:3306;user=root;password=password\n    docker_network: DB0006:3306\n    geo: UK\n    insert_data: false\n    table_groups:\n      - tracking\n  DB0007:\n    dsn: pgsql:dbname=shard;host=localhost:5407;user=postgres;password=password\n    docker_network: DB0007:5432\n    geo: UK\n    table_groups:\n      - user\n      - tracking\n\n```\n## Anatomy of the Configuration File\n\n### Version\nDefine the version.  The most recent version is 1.\n```yaml\nversion: 1\n```\n### Table Groups\nDefine the table groups.  As you add tables to your Application you will need to explicitly add them here to.\n\nThe group name is only used in ShardMatrix.\n\nThe table names are attributed to the groups.  A table can only be in one group at a time and once you have written to the Databases, it is best not to change any table assigned to a group.\n* Denotes the table groups section on config\n* Denotes the name of a group of tables\n* Denotes the table name\n\n```yaml\n# Denotes the table groups section on config\n\ntable_groups:\n\n  # Denotes the name of a group of tables\n\n  user:\n\n    # Denotes the table name\n\n    - users\n```\nThis section as it may appear.\n```yaml\ntable_groups:\n  user:\n    - users\n    - payments\n    - offers\n  tracking:\n    - visitors\n    - sign_ups\n  published:\n    - published_offers\n```\n### Unique Columns in Tables\nUnique Columns can be defined here.  So in the `users` table `email` and `username` must be unique across all Nodes (shard databases).\n```yaml\nunique_columns:\n  users:\n    - email\n    - username\n  facebook_users:\n    - fb_id\n```\n### Nodes\n\nThis is where you define your database connections, credentials, and what table groups and geos the node maybe using.\n\nNodes can be extended and added to as you go.\n\nNode names must remain the same though as must the table groups they correspond to.\n\nThe anatomy of the node section.\n* Denotes the where the nodes are defined\n* Node Name\n* DSN for connection to DB\n* *optional Docker service name and port number\n* *optional Geo - if a geo is stated the application inserting data will use this to choose this node to write new inserts to it\n* *optional Stop new data being written here, unless connected to an existing UUID from this node\n* Table groups that use this node must be defined here\n* Table group user (that consists of the users, offers, payments tables)\n```yaml\n# Denotes the where the nodes are defined\n\nnodes:\n\n  # Node Name\n\n  DBUK01:\n\n    # DSN for connection to DB\n\n    dsn: mysql:dbname=shard;host=localhost:3301;user=root;password=password\n\n    # *optional docker service name and port number\n\n    docker_network: DBUK:3306\n    \n    # *optional Geo - if a geo is stated the application inserting data will use this to choose this node to write new inserts to it\n\n    geo: UK\n\n    # *optional Stop new data being written here, unless connected to an existing UUID from this node\n\n    insert_data: false\n\n    # Table groups that use this node must be defined here\n\n    table_groups:\n\n      # Table group user (that consists of the users, offers, payments tables)\n\n      - user\n      \n      - published\n```\nThe Node Section as it may appear in the config yaml.\n```yaml\nnodes:\n  DBUK01:\n    dsn: mysql:dbname=shard;host=localhost:3301;user=root;password=password\n    docker_network: DBUK:3306\n    geo: UK\n    table_groups:\n      - user\n      - published\n  postg1:\n    dsn: pgsql:dbname=shard;host=localhost:5407;user=postgres;password=password\n    docker_network: postg1_db:5432\n    table_groups:\n      - tracking\n  DB0001:\n    dsn: mysql:dbname=shard;host=localhost:3304;user=root;password=password\n    docker_network: DB0001:3306\n    insert_data: false\n    table_groups:\n      - user\n      - published\n```\n\n### Once Yaml Config File is Complete\n\nSave the **file** to where the application is, in either a protected directory or externally inaccessible directory.\n\nAlternatively it can be made into a **Kubernetes Secret** and given to your application that way.\n\n\n# Initiate in PHP\n\nIn these examples we have saved our Config file as `shard_matrix.yaml` and placed it in the same directory as our applications index php.\n\n#### Basic Setup Using Only PHP and Webserver Resources\n* Our config file\n* Specifying a local directory to write db data to when it needs to\n```php\n\nuse ShardMatrix\\ShardMatrix;\n\n\n# Our config file\n\nShardMatrix::initFromYaml( __DIR__ . '/shard_matrix.yaml' );  \n\n\n# Specifying a local directory to write db data to when it needs to\n\nShardMatrix::setPdoCachePath( __DIR__ . '/shard_matrix_cache' );  \n\n```\n#### Setup Using Only GoThreaded and Redis\n* Our config file\n* Changes the service from PHP forking for asynchronous queries to GoThreaded\n* Uses GoThreaded for asynchronous DB calls when we have to query all relevant shards\n* This overwrites the PdoCache Service that was using writing to file, and now instead uses Redis caching\n```php\n\nuse ShardMatrix\\ShardMatrix;\n\n\n# Our config file\n\nShardMatrix::initFromYaml( __DIR__ . '/shard_matrix.yaml' );  \n\n\n# Changes the service from PHP forking for asynchronous queries to GoThreaded\n\nShardMatrix::useGoThreadedForAsyncQueries();\n\n\n# Uses GoThreaded for asynchronous DB calls when we have to query all relevant shards\n\nShardMatrix::setGoThreadedService( function () {\n\treturn new \\ShardMatrix\\GoThreaded\\Client( '127.0.0.1', 1534, 'gothreaded', 'password' );\n} );\n\n# This overwrites the PdoCache Service that was used to write to file, and now instead uses Redis caching\n\nShardMatrix::setPdoCacheService( function () {\n\treturn new \\ShardMatrix\\PdoCacheRedis( new \\Predis\\Client( 'tcp://127.0.0.1:6379' ) );\n} );  \n\n```\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjrsaunders%2Fshardmatrix","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjrsaunders%2Fshardmatrix","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjrsaunders%2Fshardmatrix/lists"}