{"id":16660209,"url":"https://github.com/metasoarous/datsync","last_synced_at":"2025-04-04T14:07:17.379Z","repository":{"id":37270279,"uuid":"51035659","full_name":"metasoarous/datsync","owner":"metasoarous","description":"Datomic \u003c-\u003e DataScript syncing/replication utilities","archived":false,"fork":false,"pushed_at":"2021-12-22T23:00:45.000Z","size":3576,"stargazers_count":324,"open_issues_count":6,"forks_count":18,"subscribers_count":30,"default_branch":"dev","last_synced_at":"2024-10-13T10:28:28.776Z","etag":null,"topics":["clojure","clojurescript","datascript","datomic","frp"],"latest_commit_sha":null,"homepage":null,"language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"epl-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/metasoarous.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-02-03T22:34:46.000Z","updated_at":"2024-05-31T07:40:56.000Z","dependencies_parsed_at":"2022-09-04T16:01:32.393Z","dependency_job_id":null,"html_url":"https://github.com/metasoarous/datsync","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metasoarous%2Fdatsync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metasoarous%2Fdatsync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metasoarous%2Fdatsync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metasoarous%2Fdatsync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/metasoarous","download_url":"https://codeload.github.com/metasoarous/datsync/tar.gz/refs/heads/dev","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247190250,"owners_count":20898702,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clojure","clojurescript","datascript","datomic","frp"],"created_at":"2024-10-12T10:28:21.451Z","updated_at":"2025-04-04T14:07:17.355Z","avatar_url":"https://github.com/metasoarous.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n![Datsync](datsync.jpg)\n\nDatomic \u0026lt;-\u003e DataScript syncing/replication utilities\n\n[![Gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/metasoarous/datsync?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge)\n\n## Status\n\nYou may be able to get some of this stuff to work for basic prototyping pruposes, but had to more or less drop this project due to other demands (family, [occupational](https://github.com/compdemocracy) and other [open source projects](https://github.com/metasoarous) (in particular, [Oz](https://github.com/metasoarous/oz)).\n\nI do hope to get back to it in the near future, once things settle down with the latest version of Oz, but don't expect a ton out of this project at the moment unless you're willing to get your hands dirty.\n\nThanks\n\n\n\u003cbr/\u003e\n\n\n\n## Introduction\n\nThis library offers tools for building DataScript databases as materialized views (very much in the re-frame/samsa sense) of some master/central Datomic database.\nEventually we hope to also provide helpers for handling optimistic updates, offline availability, scoped syncing, and security filters.\nThe current set of features:\n\n* Translate ids between Datomic and DataScript in transaction flow\n* Update DataScript schema based on transaction data from Datomic (shared schema, consequently)\n\nThis library is also part of the [Datsys architecture](https://github.com/metasoarous/datspec).\nAs such, it offers a set of ready-made system components (a la Stuart Sierra) for plugging into a componentized system (client only right now; server coming soon).\nFor a look at how Datsync hooks up in this fashion see the Datsys README.\n\nFor more general information about Datsys and Datsync:\n\n* Clojure/West 2016 talk: [Datalog all the way down](https://www.youtube.com/watch?v=aI0zVzzoK_E)\n* If you'd like to chat, see the [Datsys chatroom](https://gitter.im/metasoarous/datsys) and the [Datsync chatroom](https://gitter.im/metasoarous/datsync)\n* For more in depth documentation (including the [big picture system vision](http://github.com/metasoarous/datsync/wiki/The-Vision) and this library's [current limitations and future directions](http://github.com/metasoarous/datsync/wiki/Current-limitations-and-future-directions)), see the [datsync GitHub wiki](https://github.com/metasoarous/datsync/wiki).\n* For an API walkthrough, read on.\n* The source is also heavily documented (though there's need for some cleanup).\n\n\n## Quickstart\n\nThe easiest way to get Datsync running is to clone [Datsys](https://github.com/metasoarous/datsys).\nIt's a pretty minimal template, so it shouldn't be difficult to adapt it to your needs.\n\nBut for the sake of thoroughness, we'll cover here how you'd set things up.\nFirst, add the following to your `project.clj`:\n\n```\n[datsync \"0.0.1-alpha3\"]\n```\n\n\n### On the client (with system components)\n\nDatsync includes some optional system components for setting things up on the client side.\nYou don't have to use these, if you'd rather just piece together the utility functions yourself.\nIf you do, you can use the implementation and design of these components as guidance in your own setup/architecture.\n(There's also a small example below of how you'd set things up using Mount below).\n\nAssuming you do want to use the component setup, Datsync provides:\n\n* `dat.sync.client/Datsync`: Runs all sync processes, given a `:dispatcher` and a `:remote`\n* `dat.remote.impl.sente/SenteRemote`: A valid `:remote` dependency of the `Datsync` component above.\n\nAdditionally, you'll need to pass the `:remote` and the `:dispatcher` components along to a `:reactor` component that actually reduces over the DataScript conn.\nA `:reactor` implementation is available in [datreactor](https://github.com/metasoarous/datreactor) as `dat.reactor/SimpleReactor`.\nThe Datreactor project also has a couple of `:dispatcher` implementations, in particular `dat.reactor.dispatcher/StrictlyOrderedDispatcher`.\n\nHere's what all this looks like put together with the standard component constructors:\n\n```clj\n(ns your-app\n  (:require [dat.sync.client]\n            [dat.remote]\n            [dat.remote.impl.sente :as sente-remote]\n            [datascript.core :as d]\n            [dat.reactor.dispatcher :as dispatcher]))\n\n(def conn (d/create-conn dat.sync.client/base-schema))\n\n(defn new-system []\n  (-\u003e (component/system-map\n        ;;          ;; A sente remote implementation handles client/server comms\n        :remote     (sente-remote/new-sente-remote)\n        ;;          :: Dispatcher accepts messages from wherever (pluggable event streams?) and presents them to the reactor\n        :dispatcher (dispatcher/new-strictly-ordered-dispatcher)\n        ;;          ;; The reactor is what orchestrates the event processing and handles side effects\n        :reactor    (component/using\n                      ;; the reactor needs an :app attribute which points to the DataScript :conn\n                      (reactor/new-simple-reactor {:app {:conn conn}})\n                      [:remote :dispatcher])\n        ;;          ;; The Datsync component pipes data from the remote in to the reactor for processing, and registers\n        ;;          ;; default handlers on the reactor for this data\n        :datsync    (component/using\n                      (dat.sync.client/new-datsync)\n                      [:remote :dispatcher]))))\n```\n\nYou may also specify `:app` in this system as a system component.\nThis is how [Datview](https://github.com/metasoarous/datview) works.\nIf you're using Datview, we recommend you look at the Datsys project for how that's set up.\nAnd even if you're not, you may want to take a look at how it handles routes, history, etc.\n\nBecause all of these pieces are designed and build around protocols and specs, the semantics of how you'd\nswap-out/customize system components are described in Datspec's [`dat.spec.protocol` namespace](https://github.com/metasoarous/datspec/blob/master/src/dat/spec/protocols.cljc).\nThat's also the best place to go for understanding the Datsys architectural vision.\n\n#### What's going on here?\n\nBehind the scenes, Datsync hooks things up so that messages coming from the server with event id `:dat.sync.client/recv-remote-tx` get transacted into our local database subject to the following conditions:\n\n* Nested maps are expanded, so inner maps aren't just treated as single values in the DataScript db (thinking of\n  revising this; should maybe just require that nested maps satisfy `:db/isComponent true`).\n* Every entity gets a `:datsync.remote.db/id` mapping to the Datomic id\n* Local ids try to match remote ids when possible\n* A `:dat.sync.client/bootstrap` message is sent to the server to initiate the initial data payload from the server\n* Any entity specified in the transaction which has a `:db/ident` attribute will be treated as part of the\n  schema for the local `conn`'s db, and `assoc`'d into the db's `:schema` in a DataScript compatible manner\n    * Any `:db/valueType` other than `:db.type/ref` will be removed, since DataScript errors on anything other\n      than `:db.type/ref`\n    * This operation updates the db indices by creating a new database with the new schema, and moving all the\n      datoms over into it.\n    * Schema entities are included as datoms in the db, not just as `:schema`\n        * Keep in mind that idents aren't supported in DataScript, so to look up attribute entities, use attribute references\n        * This can facilitate really powerful UI patterns based on schema metadata which direct the composition\n          of UI components (in an overrideable fashion); have some WIP on this I might put in another lib eventually\n    * We can also apply schema changes to an existing database using the `dat.sync.client/update-schema!` function, or by\n      dispatching a `:dat.sync.client/apply-schema-changes` event.\n\nAdditionally, there is a `:dat.sync.client/send-remote-tx` event handler that takes transactions from the client and submits them to the server.\nThis function:\n\n* Translates eids via `dat.sync.client/datomic-tx`\n* Send a message to server over the remote as `[:dat.sync.remote/tx translated-tx]`\n\nHandlers for the following messages are also set up:\n\n* `[:dat.sync.client/apply-schema-changes schema-tx]` - Applies schema transaction to DataScript db conn\n* `[:dat.sync.client/merge-schema schema-map]` - Merges the schema-map (DataScript style) into the DataScript db conn\n* `[:dat.sync.client/bootstrap tx-data]` - Takes a bootstrap response and more or less delegates to `:dat.sync.client/recv-remote-tx`\n* `[:dat.sync.client/request-bootstrap _]` - Initiates a remote request for the boostrap message (this is triggered automatically when the Datsync component fires up; you'll have to handle this on server)\n\n### On the client (without system component)\n\nIf you're not into component, you can also set things manually using the helper functions defined in Datsync.\n\nFirst create your database connection, and load it with some datsync specific schema:\n\n```clj\n(def conn (d/create-conn dat.sync.remote/base-schema))\n```\n\nWhile there are no restrictions presently as to what methods you may use for sending messages between client\nand server, we'll show you roughly how you'd set things up manually using [Sente](https://github.com/ptaoussanis/sente).\n\n#### Getting data into the client db\n\nYou'll need the client to receive data from the server as transactions.\n\nUsing `event-msg-handler` multimethod dispatching on message id (first element of the message vector, by default), we can intercept\nmessages with an id of (e.g.) `:dat.sync.client/recv-remote-tx`, and handle it as follows:\n\n```clj\n(def sente (sente/make-channel-socket ...))\n\n(defmethod event-msg-handler first)\n\n(defmethod event-msg-handler :dat.sync.client/recv-remote-tx\n  [[_ tx-data]]\n  (dat.sync/apply-remote-tx! conn tx-data))\n  \n;; Add whatever other methods to the handler you like\n  \n(sente/start-chsk-router (:ch-recv sente) event-msg-handler)\n```\n\nThe `dat.sync.client/apply-remote-tx!` function takes your DataScript `conn` and a collection of transaction forms\n(should be compatible with any Datomic transaction form), and applies that to the `conn`.\nThis function takes care of translating eids between Datomic and DataScript, and updates the DataScript schema when transactions are recieved for anything with a `:db/ident` attribute\n(should maybe make this based on the install attribute, like in Datomic, but for now...).\n\nYou'll probably also want to kick off your own bootstrap process with a message sent to the server to trigger the initial data payload.\nThis might look something like:\n\n```clj\n(let [{:keys [send-fn]} sente]\n  (send-fn [:dat.sync.client/request-bootstrap true]))\n```\n\nThe server side handler then only has to return the corresponding `:dat.sync.client/recv-remote-tx` message in order to complete the cycle.\n\n#### Sending transactions to the server\n\nWhen we send transactions to the server, we need to translate their entity ids to the corresponding `:dat.sync.remote.db/id` values.\nThe `dat.sync.client/datomic-tx` utility function does this for us.\nIn sente you could write a little function that wraps this as follows.\n\n```clj\n(defn send-tx! [tx]\n  (let [{:keys [send-fn]} sente]\n    (send-fn [:dat.sync.client/tx (dat.sync.client/datomic-tx conn tx)])))\n```\n\n#### Schema API\n\nThere are additionally two functions presented for directly manipulating the database schema\n\n* `(dat.sync.client/apply-schema-tx! conn schema-tx)` - Apply a schema transaction in Datomic's tx-form to the DataScript database\n* `(dat.sync.client/update-schema! conn schema-spec)` - Merge a DataScript style schema map with the existing schema map\n\nNote that at least the latter of these will probably be deprecated once a sufficiently clean solution is [added to DataScript](https://github.com/tonsky/datascript/issues/174)\nThe solution as presented does not yet do any validation, so be careful not to abuse them.\nHowever, note that since messages coming from the remote are assumed to have been passed through Datomic, in the case of our `:dat.sync.client/recv-remote-tx` handler, we don't have to worry about this quite as much.\n\n#### Using mount\n\nYou prefer mount to Stuart Sierra's vision of system components?\nNo problem!\nYou can simply call the `component/Lifecycle` `start` and `stop` method inside of your mount `:start` and `:stop` specs, as follows:\n\n```clj\n(defstate datsys\n  :start (let [datsys (new-sytem)]\n          (component/start datsys))\n  :stop (component/stop @datsys))\n```\n\nOr if you'd rather, you can break up each of the individual Stuart Sierra components into their own mounts:\n\n```clj\n(defstate datremote\n  :start (component/start (sente-remote/new-sente-remote))\n  :stop (component/stop @datremote))\n  \n(defstate dispatcher\n  ;; ...\n  )\n  \n;; ...\n\n(defstate datsync\n  :start (component/start (dat.sync.client/new-datsync {:remote @datremote :dispatcher @dispatcher))\n  :stop (component/stop @datsync))\n```\n\nThis is a nice thing about the data-driven nature of Stuart Sierra's vision.\nWhile it is less convenient in practice to have to pass around your system components everywhere,\ndesigning around data makes it easy to plug these things into a mount setup.\nBy contrast, it would be much more cumbersome to try and wrap mount components into a Sierra-style system, hence, we argue the latter is a better target for libs.\n\nGo ahead, eat your cake.\n\n\n### On the server\n\nHere things look pretty similar; we need to send and receive transactions.\nHowever, as far as implementation goes, things are much simpler, since a lot of the translational work between DataScript and Datomic is set up to happen on the client side.\n\nEventually we'll add server-side system component protocols, specs and default implementations, so that this is all much more modular (as on the client).\nBut for now, we'll leave you with a quick demonstration of how you'd set things up with Sente.\n(Though again, we'd recommend just cloning [Datsys](https://github.com/metasoarous/datsys) and tweaking from there, even if you don't want the rest of Datsys)\n\nTo start, let's require the `dat.sync.server` namespace and get sente set up:\n\n```clj\n(require '[dat.sync.server])\n\n;; Set up sente\n(def sente (sente/make-channel-socket-server! ...))\n\n(def send! (:send-fn sente))\n\n(defn broadcast! [event]\n  (doseq [uids (:any @(:connected-uids @ws-connection))]\n    (send! uid event))))\n    \n```\n\n#### Receiving transactions\n\nNext we need to handle `:dat.sync.client/tx` messages sent from the client.\nIf you're using regular http requests, you can just call this in your handler functions as you might normally handle a form submission and send a response indicating whether the transaction went through.\nIn sente, you might set this up as follows:\n\n```clj\n(defmethod event-msg-handler :id)\n\n(defmethod event-msg-handler :dat.sync.remote/tx\n  [{:as ev-msg :keys [?data]}]\n  (dat.sync.server/apply-remote-tx! datomic-conn ?data))\n\n(defn get-bootstrap []\n  (concat\n    ;; At least one of these should probably be for your schema, and we'll probably give you a helper for that\n    (d/q ...)\n    (d/q ...)))\n\n(defmethod event-msg-handler :dat.sync.client/bootstrap\n  [{:keys [uid]}]\n  (send! uid [:dat.sync.client/bootstrap (get-bootstrap)]))\n\n(sente/start-chsk-router (:ch-recv sente) event-msg-handler)\n```\n\n\n#### Sending transactions to clients\n\nEvery time we get a transaction, we want to send the results of that transaction to any client that needs to be notified.\nEventually we can get fancy with installing subscription middleware, so for each client we have a specification of what they have \"checked out\", but this is just a starting point.\n\nAssuming we just send all changes to all clients using sente, you might write a function like this as a handler:\n\n```clj\n(defn handle-transaction-report! [tx-deltas]\n  (broadcast! [:dat.sync.client/recv-remote-tx tx-deltas]))\n```\n\nThis handler function should take a collection `tx-deltas` of `:db/add` and `:db/retract` tx forms which will automatically get computed from the datoms created in the transaction.\nThis handler function is also where you could implement your own scope restriction functionality and read authorization security protocols if needed.\n\nWe apply this handler function using the `dat.sync/start-transaction-listener!` function:\n\n```clj\n(dat.sync/start-transaction-listener! (d/tx-report-queue datomic-conn) handle-transaction-report!)\n```\n\nThis function currently takes the Java blocking queue returned by `d/tx-report-queue` and consumes all changes placed on that queue.\nWe'll eventually make it possible to pass in a `core.async` channel as well, so you can pull messages off Datomic's single `tx-report-queue` and mult them out to various processes that needed these notifications as well.\n\n\n## Contributing\n\nThis project is developed in a \"gitflow\" methodology.\nSubmit PRs to the `develop` branch.\nReleases will be merged to master and tagged with version numbers.\n\n\n## Alpha Disclaimer\n\nThis API is not yet stable.\nIn particular, anything not explicitly mentioned in this document (or one of the other datsys documents) should be considered an implementation detail, and is not to be relied upon.\nAdditionally, as we deal with issues around scoping, security filters, offline availability and decentralized sync, the API may need to adjust to meet the demands.\nSo until the path is clear there, we may yet change some of what has been described here, if it's in the best interest of the overall direction of the library and system.\nHowever, we'll do the best we can to avoid where possible.\n\n## License\n\nCopyright © 2016 Christopher Small\n\nDistributed under the Eclipse Public License either version 1.0 or (at your option) any later version.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmetasoarous%2Fdatsync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmetasoarous%2Fdatsync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmetasoarous%2Fdatsync/lists"}