{"id":13509858,"url":"https://github.com/minio/dsync","last_synced_at":"2025-03-30T14:31:49.854Z","repository":{"id":53674911,"uuid":"65050615","full_name":"minio/dsync","owner":"minio","description":"A distributed sync package.","archived":true,"fork":false,"pushed_at":"2019-11-12T10:46:21.000Z","size":3064,"stargazers_count":400,"open_issues_count":3,"forks_count":41,"subscribers_count":24,"default_branch":"master","last_synced_at":"2024-11-01T10:35:02.490Z","etag":null,"topics":["cloud-storage","distributed-locks","distributed-systems","locking","object-storage","quorum"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/minio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-08-05T21:36:55.000Z","updated_at":"2024-10-25T15:51:42.000Z","dependencies_parsed_at":"2022-09-26T17:41:30.692Z","dependency_job_id":null,"html_url":"https://github.com/minio/dsync","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minio%2Fdsync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minio%2Fdsync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minio%2Fdsync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minio%2Fdsync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/minio","download_url":"https://codeload.github.com/minio/dsync/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246332234,"owners_count":20760444,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud-storage","distributed-locks","distributed-systems","locking","object-storage","quorum"],"created_at":"2024-08-01T02:01:14.958Z","updated_at":"2025-03-30T14:31:48.601Z","avatar_url":"https://github.com/minio.png","language":"Go","readme":"\u003e This project has moved to https://github.com/minio/minio/tree/master/pkg/dsync\n\ndsync [![Slack](https://slack.minio.io/slack?type=svg)](https://slack.minio.io) [![Go Report Card](https://goreportcard.com/badge/minio/minio)](https://goreportcard.com/report/minio/minio) [![codecov](https://codecov.io/gh/minio/dsync/branch/master/graph/badge.svg)](https://codecov.io/gh/minio/dsync)\n=====\n\nA distributed locking and syncing package for Go.\n\nIntroduction\n------------\n\n`dsync` is a package for doing distributed locks over a network of `n` nodes. It is designed with simplicity in mind and hence offers limited scalability (`n \u003c= 32`). Each node will be connected to all other nodes and lock requests from any node will be broadcast to all connected nodes. A node will succeed in getting the lock if `n/2 + 1` nodes (whether or not including itself) respond positively. If the lock is acquired it can be held for as long as the client desires and needs to be released afterwards. This will cause the release to be broadcast to all nodes after which the lock becomes available again.\n\nMotivation\n----------\n\nThis package was developed for the distributed server version of [Minio Object Storage](https://minio.io/). For this we needed a distributed locking mechanism for up to 32 servers that each would be running `minio server`. The locking mechanism itself should be a reader/writer mutual exclusion lock meaning that it can be held by a single writer or an arbitrary number of readers.\n\nFor [minio](https://minio.io/) the distributed version is started as follows (for a 6-server system):\n\n```\n$ minio server http://server1/disk http://server2/disk http://server3/disk http://server4/disk http://server5/disk http://server6/disk\n```\n\n_(note that the same identical command should be run on servers `server1` through to `server6`)_\n\nDesign goals\n------------\n\n* **Simple design**: by keeping the design simple, many tricky edge cases can be avoided.\n* **No master node**: there is no concept of a master node which, if this would be used and the master would be down, causes locking to come to a complete stop. (Unless you have a design with a slave node but this adds yet more complexity.)\n* **Resilient**: if one or more nodes go down, the other nodes should not be affected and can continue to acquire locks (provided not more than `n/2 - 1` nodes are down).\n* Drop-in replacement for `sync.RWMutex` and supports [`sync.Locker`](https://github.com/golang/go/blob/master/src/sync/mutex.go#L30) interface.\n* Automatically reconnect to (restarted) nodes.\n\nRestrictions\n------------\n\n* Limited scalability: up to 32 nodes.\n* Fixed configuration: changes in the number and/or network names/IP addresses need a restart of all nodes in order to take effect.\n* If a down node comes up, it will not try to (re)acquire any locks that it may have held.\n* Not designed for high performance applications such as key/value stores.\n\nPerformance\n-----------\n\n* Support up to a total of 7500 locks/second for a size of 16 nodes (consuming 10% CPU usage per server) on moderately powerful server hardware.\n* Lock requests (successful) should not take longer than 1ms (provided decent network connection of 1 Gbit or more between the nodes).\n\nThe tables below show detailed performance numbers.\n\n### Performance with varying number of nodes\n\nThis table shows test performance on the same (EC2) instance type but with a varying number of nodes:\n\n| EC2 Instance Type    | Nodes |     Locks/server/sec | Total Locks/sec | CPU Usage |\n| -------------------- | -----:| --------------------:| ---------------:| ---------:|\n| c3.2xlarge           |     4 | (min=3110, max=3376) |           12972 |       25% |\n| c3.2xlarge           |     8 | (min=1884, max=2096) |           15920 |       25% |\n| c3.2xlarge           |    12 | (min=1239, max=1558) |           16782 |       25% |\n| c3.2xlarge           |    16 |  (min=996, max=1391) |           19096 |       25% |\n\nThe min and max locks/server/sec gradually declines but due to the larger number of nodes the overall total number of locks rises steadily (at the same CPU usage level).\n\n### Performance with difference instance types\n\nThis table shows test performance for a fixed number of 8 nodes on different EC2 instance types:\n\n| EC2 Instance Type    | Nodes |     Locks/server/sec | Total Locks/sec | CPU Usage |\n| -------------------- | -----:| --------------------:| ---------------:| ---------:|\n| c3.large (2 vCPU)    |     8 |  (min=823,  max=896) |            6876 |       75% |\n| c3.2xlarge (8 vCPU)  |     8 | (min=1884, max=2096) |           15920 |       25% |\n| c3.8xlarge (32 vCPU) |     8 | (min=2601, max=2898) |           21996 |       10% |\n\nWith the rise in the number of cores the CPU load decreases and overall performance increases.\n\n### Stress test\n\nStress test on a c3.8xlarge (32 vCPU) instance type:\n\n| EC2 Instance Type    | Nodes |     Locks/server/sec | Total Locks/sec | CPU Usage |\n| -------------------- | -----:| --------------------:| ---------------:| ---------:|\n| c3.8xlarge           |     8 | (min=2601, max=2898) |           21996 |       10% |\n| c3.8xlarge           |     8 | (min=4756, max=5227) |           39932 |       20% |\n| c3.8xlarge           |     8 | (min=7979, max=8517) |           65984 |       40% |\n| c3.8xlarge           |     8 | (min=9267, max=9469) |           74944 |       50% |\n\nThe system can be pushed to 75K locks/sec at 50% CPU load.\n\nUsage\n-----\n\n\u003e NOTE: Previously if you were using `dsync.Init([]NetLocker, nodeIndex)` to initialize dsync has\nbeen changed to `dsync.New([]NetLocker, nodeIndex)` which returns a `*Dsync` object to be used in\nevery instance of `NewDRWMutex(\"test\", *Dsync)`\n\n### Exclusive lock\n\nHere is a simple example showing how to protect a single resource (drop-in replacement for `sync.Mutex`):\n\n```go\nimport (\n\t\"github.com/minio/dsync/v3\"\n)\n\nfunc lockSameResource() {\n\n\t// Create distributed mutex to protect resource 'test'\n\tdm := dsync.NewDRWMutex(context.Background(), \"test\", ds)\n\n\tdm.Lock(\"lock-1\", \"example.go:505:lockSameResource()\")\n\tlog.Println(\"first lock granted\")\n\n\t// Release 1st lock after 5 seconds\n\tgo func() {\n\t\ttime.Sleep(5 * time.Second)\n\t\tlog.Println(\"first lock unlocked\")\n\t\tdm.Unlock()\n\t}()\n\n\t// Try to acquire lock again, will block until initial lock is released\n\tlog.Println(\"about to lock same resource again...\")\n\tdm.Lock(\"lock-1\", \"example.go:515:lockSameResource()\")\n\tlog.Println(\"second lock granted\")\n\n\ttime.Sleep(2 * time.Second)\n\tdm.Unlock()\n}\n```\n\nwhich gives the following output:\n\n```\n2016/09/02 14:50:00 first lock granted\n2016/09/02 14:50:00 about to lock same resource again...\n2016/09/02 14:50:05 first lock unlocked\n2016/09/02 14:50:05 second lock granted\n```\n\n### Read locks\n\nDRWMutex also supports multiple simultaneous read locks as shown below (analogous to `sync.RWMutex`)\n\n```go\nfunc twoReadLocksAndSingleWriteLock() {\n\n\tdrwm := dsync.NewDRWMutex(context.Background(), \"resource\", ds)\n\n\tdrwm.RLock(\"RLock-1\", \"example.go:416:twoReadLocksAndSingleWriteLock()\")\n\tlog.Println(\"1st read lock acquired, waiting...\")\n\n\tdrwm.RLock(\"RLock-2\", \"example.go:420:twoReadLocksAndSingleWriteLock()\")\n\tlog.Println(\"2nd read lock acquired, waiting...\")\n\n\tgo func() {\n\t\ttime.Sleep(1 * time.Second)\n\t\tdrwm.RUnlock()\n\t\tlog.Println(\"1st read lock released, waiting...\")\n\t}()\n\n\tgo func() {\n\t\ttime.Sleep(2 * time.Second)\n\t\tdrwm.RUnlock()\n\t\tlog.Println(\"2nd read lock released, waiting...\")\n\t}()\n\n\tlog.Println(\"Trying to acquire write lock, waiting...\")\n\tdrwm.Lock(\"Lock-1\", \"example.go:445:twoReadLocksAndSingleWriteLock()\")\n\tlog.Println(\"Write lock acquired, waiting...\")\n\n\ttime.Sleep(3 * time.Second)\n\n\tdrwm.Unlock()\n}\n```\n\nwhich gives the following output:\n\n```\n2016/09/02 15:05:20 1st read lock acquired, waiting...\n2016/09/02 15:05:20 2nd read lock acquired, waiting...\n2016/09/02 15:05:20 Trying to acquire write lock, waiting...\n2016/09/02 15:05:22 1st read lock released, waiting...\n2016/09/02 15:05:24 2nd read lock released, waiting...\n2016/09/02 15:05:24 Write lock acquired, waiting...\n```\n\nBasic architecture\n------------------\n\n### Lock process\n\nThe basic steps in the lock process are as follows:\n- broadcast lock message to all `n` nodes\n- collect all responses within certain time-out window\n  - if quorum met (minimally `n/2 + 1` responded positively) then grant lock\n  - otherwise release all underlying locks and try again after a (semi-)random delay\n- release any locks that (still) came in after time time-out window\n\n### Unlock process\n\nThe unlock process is really simple:\n- broadcast unlock message to all nodes that granted lock\n- if a destination is not available, retry with gradually longer back-off window to still deliver\n- ignore the 'result' (cover for cases where destination node has gone down and came back up)\n\nDealing with Stale Locks\n------------------------\n\nA 'stale' lock is a lock that is left at a node while the client that originally acquired the client either:\n- never released the lock (due to eg a crash) or\n- is disconnected from the network and henceforth not able to deliver the unlock message.\n\nToo many stale locks can prevent a new lock on a resource from being acquired, that is, if the sum of the stale locks and the number of down nodes is greater than `n/2 - 1`. In `dsync` a recovery mechanism is implemented to remove stale locks (see [here](https://github.com/minio/dsync/pull/22#issue-176751755) for the details).\n\nKnown deficiencies\n------------------\n\nKnown deficiencies can be divided into two categories, namely a) more than one write lock granted and b) lock not becoming available anymore.\n\n### More than one write lock\n\nSo far we have identified one case during which this can happen (example for 8 node system):\n- 3 nodes are down (say 6, 7, and 8)\n- node 1 acquires a lock on \"test\" (nodes 1 through to 5 giving quorum)\n- node 4 and 5 crash (dropping the lock)\n- nodes 4 through to 8 restart\n- node 4 acquires a lock on \"test\" (nodes 4 through to 8 giving quorum)\n\nNow we have two concurrent locks on the same resource name which violates the core requirement. Note that if just a single server out of 4 or 5 crashes that we are still fine because the second lock cannot acquire quorum.\n\nThis table summarizes the conditions for different configurations during which this can happen:\n\n| Nodes | Down nodes | Crashed nodes | Total nodes |\n| -----:| ----------:| -------------:| -----------:|\n|     4 |          1 |             2 |           3 |\n|     8 |          3 |             2 |           5 |\n|    12 |          5 |             2 |           7 |\n|    16 |          7 |             2 |           9 |\n\n(for more info see `testMultipleServersOverQuorumDownDuringLockKnownError` in [chaos.go](https://github.com/minio/dsync/blob/master/chaos/chaos.go))\n\n### Lock not available anymore\n\nThis would be due to too many stale locks and/or too many servers down (total over `n/2 - 1`). The following table shows the maximum toterable number for different node sizes:\n\n| Nodes |  Max tolerable |\n| -----:|  -------------:|\n|     4 |              1 |\n|     8 |              3 |\n|    12 |              5 |\n|    16 |              7 |\n\nIf you see any other short comings, we would be interested in hearing about them.\n\nTackled issues\n--------------\n\n* When two nodes want to acquire the same lock at precisely the same time, it is possible for both to just acquire `n/2` locks and there is no majority winner. Both will fail back to their clients and will retry later after a semi-randomized delay.\n\nServer side logic\n-----------------\n\nOn the server side just the following logic needs to be added (barring some extra error checking):\n\n```go\nconst WriteLock = -1\n\ntype lockServer struct {\n\tmutex   sync.Mutex\n\tlockMap map[string]int64 // Map of locks, with negative value indicating (exclusive) write lock\n\t                         // and positive values indicating number of read locks\n}\n\nfunc (l *lockServer) Lock(args *LockArgs, reply *bool) error {\n\tl.mutex.Lock()\n\tdefer l.mutex.Unlock()\n\tif _, *reply = l.lockMap[args.Name]; !*reply {\n\t\tl.lockMap[args.Name] = WriteLock // No locks held on the given name, so claim write lock\n\t}\n\t*reply = !*reply // Negate *reply to return true when lock is granted or false otherwise\n\treturn nil\n}\n\nfunc (l *lockServer) Unlock(args *LockArgs, reply *bool) error {\n\tl.mutex.Lock()\n\tdefer l.mutex.Unlock()\n\tvar locksHeld int64\n\tif locksHeld, *reply = l.lockMap[args.Name]; !*reply { // No lock is held on the given name\n\t\treturn fmt.Errorf(\"Unlock attempted on an unlocked entity: %s\", args.Name)\n\t}\n\tif *reply = locksHeld == WriteLock; !*reply { // Unless it is a write lock\n\t\treturn fmt.Errorf(\"Unlock attempted on a read locked entity: %s (%d read locks active)\", args.Name, locksHeld)\n\t}\n\tdelete(l.lockMap, args.Name) // Remove the write lock\n\treturn nil\n}\n```\n\nIf you also want RLock()/RUnlock() functionality, then add this as well:\n\n```go\nconst ReadLock = 1\n\nfunc (l *lockServer) RLock(args *LockArgs, reply *bool) error {\n\tl.mutex.Lock()\n\tdefer l.mutex.Unlock()\n\tvar locksHeld int64\n\tif locksHeld, *reply = l.lockMap[args.Name]; !*reply {\n\t\tl.lockMap[args.Name] = ReadLock // No locks held on the given name, so claim (first) read lock\n\t\t*reply = true\n\t} else {\n\t\tif *reply = locksHeld != WriteLock; *reply { // Unless there is a write lock\n\t\t\tl.lockMap[args.Name] = locksHeld + ReadLock // Grant another read lock\n\t\t}\n\t}\n\treturn nil\n}\n\nfunc (l *lockServer) RUnlock(args *LockArgs, reply *bool) error {\n\tl.mutex.Lock()\n\tdefer l.mutex.Unlock()\n\tvar locksHeld int64\n\tif locksHeld, *reply = l.lockMap[args.Name]; !*reply { // No lock is held on the given name\n\t\treturn fmt.Errorf(\"RUnlock attempted on an unlocked entity: %s\", args.Name)\n\t}\n\tif *reply = locksHeld != WriteLock; !*reply { // A write-lock is held, cannot release a read lock\n\t\treturn fmt.Errorf(\"RUnlock attempted on a write locked entity: %s\", args.Name)\n\t}\n\tif locksHeld \u003e ReadLock {\n\t\tl.lockMap[args.Name] = locksHeld - ReadLock // Remove one of the read locks held\n\t} else {\n\t\tdelete(l.lockMap, args.Name) // Remove the (last) read lock\n\t}\n\treturn nil\n}\n```\n\nSee [dsync-server_test.go](https://github.com/fwessels/dsync/blob/master/dsync-server_test.go) for a full implementation.\n\nSub projects\n------------\n\n* See [performance](https://github.com/minio/dsync/tree/master/performance) directory for performance measurements\n* See [chaos](https://github.com/minio/dsync/tree/master/chaos) directory for some edge cases\n\nTesting\n-------\n\nThe full test code (including benchmarks) from `sync/rwmutex_test.go` is used for testing purposes.\n\nExtensions / Other use cases\n----------------------------\n\n### Robustness vs Performance\n\nIt is possible to trade some level of robustness with overall performance by not contacting each node for every Lock()/Unlock() cycle. In the normal case (example for `n = 16` nodes) a total of 32 RPC messages is sent and the lock is granted if at least a quorum of `n/2 + 1` nodes respond positively. When all nodes are functioning normally this would mean `n = 16` positive responses and, in fact, `n/2 - 1 = 7` responses over the (minimum) quorum of `n/2 + 1 = 9`. So you could say that this is some overkill, meaning that even if 6 nodes are down you still have an extra node over the quorum.\n\nFor this case it is possible to reduce the number of nodes to be contacted to for example `12`. Instead of 32 RPC messages now 24 message will be sent which is 25% less. As the performance is mostly depending on the number of RPC messages sent, the total locks/second handled by all nodes would increase by 33% (given the same CPU load).\n\nYou do however want to make sure that you have some sort of 'random' selection of which 12 out of the 16 nodes will participate in every lock. See [here](https://gist.github.com/fwessels/dbbafd537c13ec8f88b360b3a0091ac0) for some sample code that could help with this.\n\n### Scale beyond 32 nodes?\n\nBuilding on the previous example and depending on how resilient you want to be for outages of nodes, you can also go the other way, namely to increase the total number of nodes while keeping the number of nodes contacted per lock the same.\n\nFor instance you could imagine a system of 64 nodes where only a quorum majority of `17` would be needed out of `28` nodes. Again this requires some sort of pseudo-random 'deterministic' selection of 28 nodes out of the total of 64 servers (same [example](https://gist.github.com/harshavardhana/44614a69650c9111defe3470941cdd16) as above).\n\nOther techniques\n----------------\n\nWe are well aware that there are more sophisticated systems such as zookeeper, raft, etc. However we found that for our limited use case this was adding too much complexity. So if `dsync` does not meet your requirements than you are probably better off using one of those systems.\n\nOther links that you may find interesting:\n- [Distributed locks with Redis](http://redis.io/topics/distlock)\n- Based on the above: [Redis-based distributed mutual exclusion lock implementation for Go](https://github.com/hjr265/redsync.go)\n\nPerformance of `net/rpc` vs `grpc`\n----------------------------------\n\nWe did an analysis of the performance of `net/rpc` vs `grpc`, see [here](https://github.com/golang/go/issues/16844#issuecomment-245261755), so we'll stick with `net/rpc` for now.\n\nLicense\n-------\n\nReleased under the Apache License v2.0. You can find the complete text in the file LICENSE.\n\nContributing\n------------\n\nContributions are welcome, please send PRs for any enhancements.\n","funding_links":[],"categories":["Misc","Go","distributed-systems"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fminio%2Fdsync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fminio%2Fdsync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fminio%2Fdsync/lists"}