{"id":25875621,"url":"https://github.com/ivarref/double-trouble","last_synced_at":"2025-03-02T10:18:41.578Z","repository":{"id":61281883,"uuid":"488586762","full_name":"ivarref/double-trouble","owner":"ivarref","description":"Handle duplicate Datomic transactions with ease (on-prem).","archived":false,"fork":false,"pushed_at":"2022-12-16T18:06:09.000Z","size":1296,"stargazers_count":9,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-02T05:56:17.261Z","etag":null,"topics":["cas","clojure","compare-and-swap","datomic"],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"epl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ivarref.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-05-04T12:50:49.000Z","updated_at":"2023-07-06T11:43:21.000Z","dependencies_parsed_at":"2023-01-29T15:45:37.929Z","dependency_job_id":null,"html_url":"https://github.com/ivarref/double-trouble","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivarref%2Fdouble-trouble","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivarref%2Fdouble-trouble/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivarref%2Fdouble-trouble/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivarref%2Fdouble-trouble/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ivarref","download_url":"https://codeload.github.com/ivarref/double-trouble/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241465092,"owners_count":19967243,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cas","clojure","compare-and-swap","datomic"],"created_at":"2025-03-02T10:18:41.016Z","updated_at":"2025-03-02T10:18:41.569Z","avatar_url":"https://github.com/ivarref.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# \u003cimg align=\"right\" src=\"the_clash.png\" width=\"187\" height=\"200\"\u003e [No more] double trouble\n\nHandle duplicate Datomic transactions with ease. This library contains:\n\n* A modified compare-and-swap function, `:dt/cas`, that gracefully handles duplicates.\n* A set-and-change function, `:dt/sac`, that cancels a transaction if a value does not change.\n* A just-increment-it function, `:dt/jii`, that increments the value of an attribute.\n* A counter function, `:dt/counter` that returns increasing numbers starting from 1 for a given counter name.\n* A counter function, `:dt/counter-str` that returns increasing numbers starting from 1 for a given counter name as a string.\n\nOn-prem only.\n\n## Installation\n\n[![Clojars Project](https://img.shields.io/clojars/v/com.github.ivarref/double-trouble.svg)](https://clojars.org/com.github.ivarref/double-trouble)\n\n## 2-minute `:dt/cas` example\n\n```clojure\n(require '[com.github.ivarref.double-trouble :as dt])\n(require '[datomic.api :as d])\n\n(def conn (let [uri (str \"datomic:mem://README-example\")]\n            (d/delete-database uri)\n            (d/create-database uri)\n            (d/connect uri)))\n\n; Setup:\n@(d/transact conn dt/schema)\n\n(def example-schema\n  [#:db{:ident :e/id, :cardinality :db.cardinality/one, :valueType :db.type/string :unique :db.unique/identity}\n   #:db{:ident :e/status, :cardinality :db.cardinality/one, :valueType :db.type/keyword}\n   #:db{:ident :e/version, :cardinality :db.cardinality/one, :valueType :db.type/long}\n   #:db{:ident :e/info, :cardinality :db.cardinality/one, :valueType :db.type/string}])\n@(d/transact conn example-schema)\n\n; Add sample data:\n@(d/transact conn [{:e/id \"my-id\" :e/version 1 :e/info \"Initial version\"}])\n\n; Sample payload:\n(def payload {:e/id \"my-id\"\n              :e/info \"Second version\"\n              :e/version 1})\n\n; Initial commit using :dt/cas is fine:\n@(dt/transact conn [(dissoc payload :e/version)\n                    [:dt/cas [:e/id \"my-id\"] :e/version 1 2 (dt/sha payload)]])\n; =\u003e {:db-before datomic.db.Db@a108c315\n;     :db-after datomic.db.Db@260c18eb\n;     :transacted? true}\n; Notice the key :transacted? with value true\n\n; Transacting one more time is also fine:\n@(dt/transact conn [(dissoc payload :e/version)\n                    [:dt/cas [:e/id \"my-id\"] :e/version 1 2 (dt/sha payload)]])\n; =\u003e {:db-before datomic.db.Db@d06fccb2\n;     :db-after datomic.db.Db@d06fccb1\n;     :transacted? false}\n; Notice the key :transacted? with value false\n\n; Why did the above succeed?\n; :dt/cas detected that\n; :e/version 1 -\u003e 2 had already been asserted in a previous transaction and\n; that the sha asserted in that previous transaction matched the sha\n; in the current transaction.\n; Thus this is a duplicate transaction that should be allowed.\n; :dt/cas throws an exception indicating this.\n\n; dt/transact, yes, that is com.github.ivarref.double-trouble/transact,\n; catches this particular exception, and returns a result as if the\n; transaction had succeeded: a map with the keys :db-before, :db-after and\n; :transacted? (false).\n```\n\n## Motivation\n\n\u003e The network is reliable\n\n*The first assertion of [fallacies of distributed computing](https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing).*\n\n`:db/cas`, compare-and-swap, can be used as a lock/protection\nif you want to disallow concurrent edits on some entity.\nImagine the following scenario where a network failure\noccurs after a transaction has successfully completed:\n\n```\n1. The client sends HTTP request to Clojure backend.\n2. Backend issues @(d/transact ... [[:db/cas [...] :e/version 1 2]]) to the Datomic transactor.\n3. The transactor writes successfully to the storage, e.g. PostgreSQL.\n; Everything fine so far.\n\n4. The network between the Clojure backend and the Datomic transactor goes down.\n5. The invocation of d/transact throws an exception, and the backend returns HTTP status code 500.\n6. The network between the Clojure backend and the Datomic transactor comes back up.\n\n; Now what?\n\n7. The client retries the HTTP request.\n8. Backend issues an identical @(d/transact ...).\n9. Now d/transact throws a cas exception.\n10. Backend returns what?\n\n; The client is now stuck.\n; What should it do?\n```\n\nThe aim of this library is to handle such scenarios.\n\nHad `:dt/cas`, and not `:db/cas`, been used, the retry would have\nbeen successful.\n\n## Usage\n\n`:dt/cas` takes 5 arguments, one more than `:db/cas`:\n1. The entity id or lookup ref.\n2. The attribute to change.\n3. The old value.\n4. The new value.\n5. The sha representing the essence of the transaction.\n\nThe first four arguments are identical to Datomic's cas.\n\nThe last argument, the sha, should represent the essence of the transaction. \nIt will be used to check if the transaction has already been committed in\nthe event of a cas failure, i.e. if this transaction has a duplicate.\n\nYou may use `(dt/sha my-data)` to generate a sha. `my-data` must obviously\nbe identical for transactions that should be considered identical.\n`dt/sha` converts maps and sets into their sorted forms before\ncalculating the sha.\n\nYou may use `com.github.ivarref.double-trouble/transact` to transact data\ncontaining `:dt/cas`. It will catch exceptions, return success in the\ncase of a duplicate transaction, or re-throw other exceptions.\nIt will also resolve tempids on the input tx-data:\n\n```clojure\n(require '[com.github.ivarref.double-trouble :as dt])\n\n(let [{:keys [transacted? db-after]} @(dt/transact conn [[:dt/cas e attr old-val new-val sha]])]\n  (if transacted?\n    (log/info \"transacted data\")\n    (log/info \"duplicate transaction is fine\"))\n  (d/pull db-after [:*] lookup-ref))\n```\n\nIf you prefer handling the exception yourself, you may do it as the following:\n```clojure\n(require '[com.github.ivarref.double-trouble :as dt])\n(require '[datomic.api :as d])\n\n(try\n  @(d/transact conn ...)\n  (catch Exception e\n    (if (dt/already-transacted? e)\n      :ok\n      '...handle-exception...)))\n```\n\n## Example usage in the case of a HTTP backend\n\n```clojure\n(try\n  (let [{:keys [transacted? db-after]} @(dt/transact conn ...)]\n    {:status (if transacted? 201 200) :body (d/pull db-after [:*] lookup-ref)})\n  (catch Exception e\n    (if (dt/cas-failure? e :e/version)\n      {:status 409 :body {:message \"Conflict\" :expected (dt/expected-val e) :given (dt/given-val e)}}\n      {:status 500 :body {:message \"Internal server error\"}})))\n```\nThe code is simplified, but illustrates the gist of a typical endpoint.\nYou'll want to handle:\n* Actual executed transaction: response code 201.\n* Duplicate transaction: response code 200.\n* Cas error: response code 409.\n* Other errors: response code 500.\n\n## Using `:dt/sac` to cancel a transaction if a value does not change\n\nIf your transaction is primarily concerned with changing some kind of status, you may\nwant to cancel the transaction if there is no actual status change.\nYou may use `:dt/sac` for this scenario. `sac` stands for `set-and-change`.\n\n```clojure\n@(dt/transact conn [{:e/id \"sac-demo\" :e/status :INIT}])\n\n(:transacted? @(dt/transact conn [[:dt/sac [:e/id \"sac-demo\"] :e/status :PROCESSING]]))\n; =\u003e true\n\n(:transacted? @(dt/transact conn [[:dt/sac [:e/id \"sac-demo\"] :e/status :PROCESSING]]))\n; =\u003e false\n```\n\nWhen you are using `dt/transact`, `:dt/sac` will take precedence over `:dt/cas`.\n\n`:dt/sac` handles the combination where attributes are references and values are keywords.\n\n\n## Incrementing an attribute's value\n\nYou may use `dt/jii`, where `jii` stands for `just-increment-it`:\n\n```clojure\n@(dt/transact conn [{:e/id \"jii-demo\" :e/version 1}])\n\n@(dt/transact conn [[:dt/jii [:e/id \"jii-demo\"] :e/version]])\n\n(d/pull (d/db conn) [:e/version] [:e/id \"jii-demo\"])\n; =\u003e #:e{:version 2}\n```\n\n## Counters\n\nYou may use `:dt/counter`:\n\n```clojure\n(let [tx [[:dt/counter \"my-counter\" \"my-tempid\" :e/id-long]\n          {:db/id \"my-tempid\" :e/info \"info\"}]\n      {:keys [db-after tempids]} @(d/transact conn tx)]\n  (d/pull db-after [:*] (get tempids \"my-tempid\")))\n; =\u003e {:db/id ... :e/id-long 1 :e/info \"info\"}\n\n; evaluate the same again, and you will get e.g.:\n(let [tx [[:dt/counter \"my-counter\" \"my-tempid\" :e/id-long]\n          {:db/id \"my-tempid\" :e/info \"info\"}]\n      {:keys [db-after tempids]} @(d/transact conn tx)]\n  (d/pull db-after [:*] (get tempids \"my-tempid\")))\n; =\u003e {:db/id ... :e/id-long 2 :e/info \"info\"}\n```\n\nNotice here that it is `datomic.api/transact` that is being used.\n\nUse `:dt/counter-str` if the added attribute should be a string.\n\n## Error handling and health checking\n\nIf there is a regular cas mismatch and thus an actual conflict, `:dt/cas`\nwill throw an exception identical to the one thrown by `:db/cas`.\nThus if you are already catching this kind of exception, you may keep\nusing that code.\n\nIf there is an older duplicate transaction, e.g. cas from version 1 to 2\nwith a previously asserted and correct sha,\nbut the current version now being at 3 (and not 2), `double-trouble` considers\nthis an error. In this case `double-trouble` will also throw a regular `:db/cas` exception.\nThis most likely means that the client/issuer of the transaction is out of sync, and\nthus you will want it to fail.\n\nIf `com.github.ivarref.double-trouble/transact` detects a duplicate sha\nfor a non-duplicate transaction, it will reset the atom\n`com.github.ivarref.double-trouble/healthy?` to `false`.\nYou may monitor this atom in a healthcheck. It defaults to `true`.\n\nIf a duplicate sha occurs, your code is most likely broken (or a sha collision\nhas occurred). It could also be that `double trouble`'s logic failed in some\nunforeseen way, and thus did in fact introduce additional trouble ¯\\\\\\_(ツ)\\_/¯.\n\n## FAQ\n\n\u003e Can I use shas for versioning?\n\nYes.\n\n\u003e Where is the sha asserted? \n\nThe sha is asserted on the transaction metadata attribute `:com.github.ivarref.double-trouble/sha-1`.\n\n## Limitations\n\n`com.github.ivarref.double-trouble/transact` does not return\n`tempids` nor `tx-data` for duplicate transactions. For regular\ntransactions it removes those keys so that you do not mistakenly\nrely on them when using this library.\n\n## Alternatives and related software\n\nI did not find any similar libraries for Datomic.\n\n[toxiproxy](https://github.com/Shopify/toxiproxy), a TCP proxy to simulate network and system conditions for chaos\nand resiliency testing, as well as [clj-test-containers](https://github.com/javahippie/clj-test-containers)\nand/or [testcontainers-java](https://github.com/testcontainers/testcontainers-java) is worth checking out.\n\nI've also written [mikkmokk-proxy](https://github.com/ivarref/mikkmokk-proxy), an HTTP reverse proxy\nfor doing fault injection on the HTTP layer, as well as [yoltq](https://github.com/ivarref/yoltq), a\npersistent Datomic queue for building (more) reliable systems.\n\n## Changelog\n\n#### 2022-12-16 v0.1.105\nBugfix `resolve-lookup-ref` for remote transactor.\n\n#### 2022-11-21 v0.1.103\nSupport tempids in counter functions.\n\n#### 2022-11-18 v0.1.102\nAdded `ensure-partition!` function.\n\n#### 2022-10-12 v0.1.101\nAdded `:dt/counter-str` function.\n\n#### 2022-10-12 v0.1.100\nAdded `:dt/counter` function.\n\n#### 2022-07-03 v0.1.96\nAdded `:dt/sac` and `:dt/jii` functions.\n\n#### 2022-06-20 v0.1.92\nAllow `cas-failure?` to work on a single exception and no attribute.\n\n#### 2022-06-17 v0.1.89\nFirst publicly announced release.\n\n## License\n\nCopyright © 2022 Ivar Refsdal\n\nThis program and the accompanying materials are made available under the terms of the Eclipse Public License 2.0 which is available at http://www.eclipse.org/legal/epl-2.0.\n\nThis Source Code may also be made available under the following Secondary Licenses when the conditions for such availability set forth in the Eclipse Public License, v. 2.0 are satisfied: GNU General Public License as published by the Free Software Foundation, either version 2 of the License, or (at your option) any later version, with the GNU Classpath Exception which is available at https://www.gnu.org/software/classpath/license.html.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fivarref%2Fdouble-trouble","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fivarref%2Fdouble-trouble","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fivarref%2Fdouble-trouble/lists"}