Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/darach/dq
Distributed Fault Tolerant Queue library
https://github.com/darach/dq
Last synced: 2 months ago
JSON representation
Distributed Fault Tolerant Queue library
- Host: GitHub
- URL: https://github.com/darach/dq
- Owner: darach
- License: mit
- Created: 2015-01-16T17:55:16.000Z (almost 10 years ago)
- Default Branch: master
- Last Pushed: 2015-02-14T00:40:19.000Z (almost 10 years ago)
- Last Synced: 2024-10-13T13:35:52.179Z (3 months ago)
- Language: Erlang
- Size: 199 KB
- Stars: 35
- Watchers: 6
- Forks: 2
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-erlang - dq - Distributed Fault Tolerant Queue library. (Queue)
README
# **dq** [](https://travis-ci.org/darach/dq)
> Distributed Fault Tolerant Queue
## Status
Experimental.
## Overview
**dq** is a distributed fault tolerant queue based on Ulf Wiger's and Thomas Arts deadlock-resolving [locks](http://github.com/uwiger/locks) library.
The queue exposes the core, extended and Okasaki queue APIs but the queue is redundant and uses distributed erlang to maintain consistent state on many nodes. The queue can detect new nodes and recover from lost nodes and netsplits.
## Sample usage
A walkthrough of using the queue library.
**1. Start one or many distributed erlang nodes**
```
$ ERL_LIBS=deps erl -pa ebin -sname bill@darach -setcookie ted
Erlang R16B02 (erts-5.10.3) [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]
Eshell V5.10.3 (abort with ^G)
(bill@darach)1>
```
**2. Ensure that the locks application is available and start it**
```
(bill@darach)1> application:ensure_all_started(locks).
{ok,[locks]}
(bill@darach)2>
```
**3. Create a distributed queue**
```
(bill@darach)2> {ok,Q} = dq:new(myq).
{ok,<0.46.0>}
(bill@darach)3>
```
**4. Start using the queue**
```
bill@darach)3> [ dq:in(X,Q) || X <- lists:seq(1,3) ].
[{[1],[]},{[2],[1]},{[3,2],[1]}]
(bill@darach)4>
```
**5. We can start other nodes at any time ...**
```
$ ERL_LIBS=deps erl -pa ebin -sname jill@darach -setcookie ted
Erlang R16B02 (erts-5.10.3) [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]
Eshell V5.10.3 (abort with ^G)
(jill@darach)1> application:ensure_all_started(locks).
{ok,[locks]}
(jill@darach)2> {ok,Q} = dq:new(myq).
{ok,<0.46.0>}
(jill@darach)3> [ dq:in(X,Q) || X <- lists:seq(7,9) ].
[{[7],[]},{"\b",[7]},{"\t\b",[7]}]
(jill@darach)4>
```
Notice that the distributed queue on each node are distinct?
The nodes haven't been joined into a cluster and do not know about each other yet.
**6. Join nodes into a cluster**
```
(jill@darach)4> net_adm:ping(bill@darach).
pong
(jill@darach)5>
```
The distinct queues have now been merged into a single distributed queue.
This queue can be operated on from any of the nodes in the cluster.
```
(bill@darach)4> dq:in_r(777,Q).
{[9,8,7,3,2],[777,1]}
(bill@darach)5>
```
**7. There can be more than one distributed queue active in a cluster**
Create and populate a new queue on one node:
```
(bill@darach)5> {ok,P} = dq:new(otherq).
{ok,<0.54.0>}
(bill@darach)6> dq:in(1,P).
{[1],[]}
(bill@darach)7>
```
As the cluster is already active, we just need a reference to it on
the other cluster nodes to operate on it:
```
(jill@darach)5> {ok,P} = dq:new(otherq).
{ok,<0.56.0>}
(jill@darach)6> dq:in(2,P).
{[2],[1]}
(jill@darach)7>
```
## TODO
1. Make merge function replaceable
2. Resolve travis common test issues
3. Priority queue option
## NOTES
* It is not enough to run distributed erlang nodes. They must also be connected.
* The locks application must be running on at least 2 nodes...
* Calling erlang:disconnect_node/1 on a 2 node cluster is not supported by locks
* Calling dq:<fun>/<arity> on a >1 node cluster with locks running on a single node will fail
* Calling dq:<fun>/<arity> on a 1 node cluster with locks running on a single node will succeed
* Any queue state in a local queue before being adjoined to a cluster will be lost (not merged)
## Enjoy!