{"id":24289285,"url":"https://github.com/rla/sql-sync","last_synced_at":"2025-09-25T12:30:49.621Z","repository":{"id":8833414,"uuid":"10536064","full_name":"rla/sql-sync","owner":"rla","description":"Offline replication between SQLite (clients) and MySQL (master).","archived":false,"fork":false,"pushed_at":"2013-06-06T21:02:24.000Z","size":134,"stargazers_count":58,"open_issues_count":1,"forks_count":15,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-04-11T20:56:27.671Z","etag":null,"topics":["mysql","replication","sql","sqlite","sync"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rla.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-06-06T20:41:41.000Z","updated_at":"2023-10-05T03:47:59.000Z","dependencies_parsed_at":"2022-09-19T08:11:03.981Z","dependency_job_id":null,"html_url":"https://github.com/rla/sql-sync","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rla%2Fsql-sync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rla%2Fsql-sync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rla%2Fsql-sync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rla%2Fsql-sync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rla","download_url":"https://codeload.github.com/rla/sql-sync/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234188314,"owners_count":18793215,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mysql","replication","sql","sqlite","sync"],"created_at":"2025-01-16T10:51:48.607Z","updated_at":"2025-09-25T12:30:49.245Z","avatar_url":"https://github.com/rla.png","language":"JavaScript","readme":"sql-sync\n========\n\nOffline replication between SQLite (clients) and MySQL (master). This\nproject is not a library, rather it is example code. Some parts of it\n(code in `lib`) could be reused in other projects. The code mainly\njust tests that the approach described here works. The code is built\nfor Node.JS.\n\nAssumes the following:\n\n* Primary keys are [UUIDs](http://en.wikipedia.org/wiki/Uuid) (here 36-character strings);\n* No foreign key constraints.\n\nIt is possible to use foreign key constraints but then table\nupdates must be reordered correctly (complicates the code a lot!).\n\nIt is possible to use [natural keys](http://en.wikipedia.org/wiki/Natural_key)\ninstead of UUIDs but there do not always exist good natural keys.\nIn most cases large composite keys would have to be used.\nIt is not possible to use autoincremented keys because of key value conflicts.\n\nMetainfo\n--------\n\nOn the client side table actions (INSERT/UPDATE/DELETE) are recorded\ninto the metadata table `sync` with the following structure:\n\n    CREATE TABLE sync (\n        action INTEGER NOT NULL,\n        keyval CHARACTER(36) NOT NULL,\n        tid INTEGER NOT NULL,\n        PRIMARY KEY (keyval) ON CONFLICT REPLACE\n    );\n\nIn the table:\n\n* action - 0 marks insert/update, 1 marks delete;\n* keyval - primary key value in the row;\n* tid - table id (used by triggers below).\n\nSynced tables are kept in the table `sync_table`:\n\n    CREATE TABLE sync_table (\n        tid INTEGER NOT NULL,\n        name VARCHAR(255) NOT NULL,\n        keycol VARCHAR(255) NOT NULL,\n        PRIMARY KEY (tid),\n        UNIQUE (name)\n    );\n\nIn the table:\n\n* tid - table id.\n* name - table name.\n* keycol - name of primary key column in the table.\n\nFor each table in `sync_table` the following triggers have\nto be created:\n\n    CREATE TRIGGER \u003ctable\u003e_insert\n    AFTER INSERT ON \u003ctable\u003e FOR EACH ROW\n    BEGIN\n        INSERT INTO sync (action, keyval, tid)\n        VALUES (0, NEW.\u003ckeycol\u003e, \u003ctableid\u003e);\n    END;\n    \n    CREATE TRIGGER \u003ctable\u003e_update\n    AFTER UPDATE ON \u003ctable\u003e FOR EACH ROW\n    BEGIN\n        INSERT INTO sync(action, keyval, tid)\n        VALUES (1, OLD.\u003ckeycol\u003e, \u003ctableid\u003e);\n        INSERT INTO sync(action, keyval, tid)\n        VALUES (0, NEW.\u003ckeycol\u003e, \u003ctableid\u003e);\n    END;\n    \n    CREATE TRIGGER \u003ctable\u003e_delete\n    AFTER DELETE ON \u003ctable\u003e FOR EACH ROW\n    BEGIN\n        INSERT INTO sync(action, keyval, tid)\n        VALUES (1, OLD.\u003ckeycol\u003e, \u003ctableid\u003e);\n    END;\n\nA special table is used for storing the last revision number (given by the\nserver at the end of sync). This is sent with each sync request (but is not updated on\neach data table action on the client):\n\n    CREATE TABLE revision (\n        rev UNSIGNED BIG INT NOT NULL\n    );\n\nMetainfo tables on the server are similar. The main difference is\nin the `sync` table:\n\n    CREATE TABLE sync (\n        action TINYINT UNSIGNED NOT NULL,\n        keyval CHAR(36) NOT NULL,\n        tid TINYINT UNSIGNED NOT NULL,\n        rev BIGINT UNSIGNED NOT NULL,\n        PRIMARY KEY (keyval)\n    );\n\nThe `rev` field is updated with each data table action. This is done with\nthe help of the following stored procedure:\n\n    CREATE PROCEDURE sync_mark (\n        in_keyval CHAR(36),\n        in_tid TINYINT UNSIGNED,\n        in_action VARCHAR(10)\n    )\n    BEGIN\n        INSERT INTO sync (action, keyval, tid, rev)\n        VALUES (\n            in_action,\n            in_keyval,\n            in_tid,\n            (SELECT rev + 1 FROM revision)\n        ) ON DUPLICATE KEY UPDATE action = VALUES(action), rev = VALUES(rev);\n        UPDATE revision SET rev = rev + 1;\n    END\n\nThe procedure is called by triggers. They have to be created for\neach data table:\n\n    CREATE TRIGGER \u003ctable\u003e_insert\n    AFTER INSERT ON \u003ctable\u003e FOR EACH ROW\n    BEGIN CALL sync_mark(NEW.\u003ckeycol\u003e, \u003ctableid\u003e, 0); END\n    \n    CREATE TRIGGER \u003ctable\u003e_update\n    AFTER UPDATE ON \u003ctable\u003e FOR EACH ROW\n    BEGIN\n        CALL sync_mark(OLD.\u003ckeycol\u003e, \u003ctableid\u003e, 1);\n        CALL sync_mark(NEW.\u003ckeycol\u003e, \u003ctableid\u003e, 0);\n    END\n    \n    CREATE TRIGGER \u003ctable\u003e_delete\n    AFTER DELETE ON \u003ctable\u003e FOR EACH ROW\n    BEGIN CALL sync_mark(OLD.\u003ckeycol\u003e, \u003ctableid\u003e, 1); END\n\nGeneral algorithm\n-----------------\n\n1. Client finds all changes.\n2. Client finds all deletes.\n3. Client finds **clrev**, the last revision the client synced with the server.\n4. Client sends changes, deletes and **clrev** to server.\n5. Server locks meta and data tables.\n6. Server finds current revision **crrev**.\n7. Server applies client changes and deletes.\n8. Server finds current revison again, **nwrev**.\n9. Server finds all changes between **clrev** and **crrev**.\n10. Server finds all deletes between **clrev** and **crrev**.\n11. Server unlocks tables.\n12. Server sends changes, deletes and **nwrev** back to the client.\n13. Client stores **nwrev**.\n\nFinding changes on the client (this and others have to be\nexecuted per data table):\n\n    SELECT \u003ctable\u003e.* FROM \u003ctable\u003e\n    JOIN sync ON (\u003ctable\u003e.\u003ckeycol\u003e = sync.keyval)\n    WHERE sync.action = 0 AND sync.tid = \u003ctableid\u003e\n    ORDER BY \u003ckeycol\u003e;\n\nFinding deletes on the client:\n\n    SELECT keyval FROM sync\n    WHERE action = 1 AND tid = \u003ctableid\u003e ORDER BY keyval;\n\nApplying changes on the server (per row):\n\n    INSERT INTO \u003ctable\u003e (col1, col2, ...)\n    VALUES (val1, val2, ...)\n    ON DUPLICATE KEY UPDATE col1 = VALUES(col1),\n    col2 = VALUES(col2), ...;\n\nApplying deletes on the server (per deleted row):\n\n    DELETE FROM \u003ctable\u003e WHERE \u003ckeycol\u003e = value;\n\nFinding changes on the server:\n\n    SELECT \u003ctable\u003e.* FROM \u003ctable\u003e\n    JOIN sync ON (\u003ctable\u003e.\u003ckeycol\u003e = sync.keyval\n    WHERE sync.action = 0 AND sync.rev \u003e \u003cclrev\u003e\n    AND sync.rev \u003c= \u003ccrrev\u003e\n    AND sync.tid = \u003ctableid\u003e ORDER BY \u003ckeycol\u003e;\n\nFinding deletes on the server:\n\n    SELECT keyval FROM sync\n    WHERE action = 1 AND rev \u003e \u003cclrev\u003e\n    AND rev \u003c= \u003ccrrev\u003e AND tid = \u003ctableid\u003e\n    ORDER BY keyval;\n\nApplying changes on the client (per row, both\nqueries are needed):\n\n    INSERT OR IGNORE INTO \u003ctable\u003e (col1, col2, ...)\n    VALUES (val1, val2, ...);\n    UPDATE \u003ctable\u003e SET col1 = val1, col2 = val2, ...\n    WHERE \u003ckeycol\u003e = value;\n\nApplying deletes on the server (per deleted row):\n\n    DELETE FROM \u003ctable\u003e\n    WHERE \u003ckeycol\u003e = value;\n\nMultiuser case\n--------------\n\nWhen data is kept by user (data tables contain some sort of user id)\nthen the `revision` table on the server side must also contain user id.\nQueries on the server must take it into account (add to `WHERE` clauses\nor `SET` user id when inserting). The procedure `sync_mark` has to be rewritten\nto update by-user `rev` value.\n\nSync data over HTTP\n-------------------\n\nBoth the server and the client send JSON object in the form:\n\n    {\n        \"\u003ctable\u003e\": {\n            \"deletes\": [ \"keyval1\", ... ],\n            \"changes\": [\n                {\n                    \"prop1\": \"value1\",\n                    \"prop2\": 23\n                }\n            ]\n        },\n        \"revision\": 120\n    }\n\nRunning tests\n-------------\n\nInstall dependencies:\n\n    npm install\n\nAnd install Mocha:\n\n    npm install mocha -g\n\nModify MySQL connection details in `Makefile` and `tests/helpers/server.js`.\nRun `make test`. Tests will create test schema in two clients and the server\nand run various random operations in each. Then synchronize and check for\ndata consistency.\n\nLicense\n-------\n\nThe MIT License.\n\n```\nCopyright (c) 2013 Raivo Laanemets\n\nPermission is hereby granted, free of charge, to any person\nobtaining a copy of this software and associated documentation\nfiles (the \"Software\"), to deal in the Software without restriction,\nincluding without limitation the rights to use, copy, modify, merge,\npublish, distribute, sublicense, and/or sell copies of the Software,\nand to permit persons to whom the Software is furnished to do so,\nsubject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included\nin all copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS\nOR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL\nTHE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING\nFROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS\nIN THE SOFTWARE.\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frla%2Fsql-sync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frla%2Fsql-sync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frla%2Fsql-sync/lists"}