{"id":13800975,"url":"https://github.com/darach/dq","last_synced_at":"2025-03-23T14:31:35.211Z","repository":{"id":25919465,"uuid":"29360476","full_name":"darach/dq","owner":"darach","description":"Distributed Fault Tolerant Queue library","archived":false,"fork":false,"pushed_at":"2015-02-14T00:40:19.000Z","size":204,"stargazers_count":35,"open_issues_count":2,"forks_count":2,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-03-18T21:16:34.043Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Erlang","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/darach.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-01-16T17:55:16.000Z","updated_at":"2024-02-21T20:48:39.000Z","dependencies_parsed_at":"2022-08-24T07:40:23.382Z","dependency_job_id":null,"html_url":"https://github.com/darach/dq","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/darach%2Fdq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/darach%2Fdq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/darach%2Fdq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/darach%2Fdq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/darach","download_url":"https://codeload.github.com/darach/dq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245115915,"owners_count":20563254,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-04T00:01:18.155Z","updated_at":"2025-03-23T14:31:34.920Z","avatar_url":"https://github.com/darach.png","language":"Erlang","readme":"# **dq** [![Build Status](https://travis-ci.org/darach/dq.svg?branch=master)](https://travis-ci.org/darach/dq)\n\n\u003e Distributed Fault Tolerant Queue\n\n## Status\n\nExperimental.\n\n## Overview\n\n**dq** is a distributed fault tolerant queue based on Ulf Wiger's and Thomas Arts deadlock-resolving [locks](http://github.com/uwiger/locks) library.\n\nThe queue exposes the core, extended and Okasaki queue APIs but the queue is redundant and uses distributed erlang to maintain consistent state on many nodes. The queue can detect new nodes and recover from lost nodes and netsplits. \n\n## Sample usage\n\nA walkthrough of using the queue library.\n\n**1. Start one or many distributed erlang nodes**\n\n```\n$ ERL_LIBS=deps erl -pa ebin -sname bill@darach -setcookie ted\nErlang R16B02 (erts-5.10.3) [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]\n\nEshell V5.10.3  (abort with ^G)\n(bill@darach)1\u003e\n```\n\n**2. Ensure that the locks application is available and start it**\n\n```\n(bill@darach)1\u003e application:ensure_all_started(locks).\n{ok,[locks]}\n(bill@darach)2\u003e\n```\n\n**3. Create a distributed queue**\n\n```\n(bill@darach)2\u003e {ok,Q} = dq:new(myq).\n{ok,\u003c0.46.0\u003e}\n(bill@darach)3\u003e\n```\n\n**4. Start using the queue**\n\n```\nbill@darach)3\u003e [ dq:in(X,Q) || X \u003c- lists:seq(1,3) ].\n[{[1],[]},{[2],[1]},{[3,2],[1]}]\n(bill@darach)4\u003e\n```\n\n**5. We can start other nodes at any time ...**\n\n```\n$ ERL_LIBS=deps erl -pa ebin -sname jill@darach -setcookie ted\nErlang R16B02 (erts-5.10.3) [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]\n\nEshell V5.10.3  (abort with ^G)\n(jill@darach)1\u003e application:ensure_all_started(locks).\n{ok,[locks]}\n(jill@darach)2\u003e {ok,Q} = dq:new(myq).\n{ok,\u003c0.46.0\u003e}\n(jill@darach)3\u003e [ dq:in(X,Q) || X \u003c- lists:seq(7,9) ].\n[{[7],[]},{\"\\b\",[7]},{\"\\t\\b\",[7]}]\n(jill@darach)4\u003e\n```\n\nNotice that the distributed queue on each node are distinct?\nThe nodes haven't been joined into a cluster and do not know about each other yet.\n\n**6. Join nodes into a cluster**\n\n```\n(jill@darach)4\u003e net_adm:ping(bill@darach).\npong\n(jill@darach)5\u003e\n```\nThe distinct queues have now been merged into a single distributed queue.\nThis queue can be operated on from any of the nodes in the cluster.\n\n```\n(bill@darach)4\u003e dq:in_r(777,Q).\n{[9,8,7,3,2],[777,1]}\n(bill@darach)5\u003e\n```\n\n**7. There can be more than one distributed queue active in a cluster**\n\nCreate and populate a new queue on one node:\n\n```\n(bill@darach)5\u003e {ok,P} = dq:new(otherq).\n{ok,\u003c0.54.0\u003e}\n(bill@darach)6\u003e dq:in(1,P).\n{[1],[]}\n(bill@darach)7\u003e\n```\n\nAs the cluster is already active, we just need a reference to it on\nthe other cluster nodes to operate on it:\n\n```\n(jill@darach)5\u003e {ok,P} = dq:new(otherq).\n{ok,\u003c0.56.0\u003e}\n(jill@darach)6\u003e dq:in(2,P).\n{[2],[1]}\n(jill@darach)7\u003e\n```\n\n## TODO\n\n1. Make merge function replaceable\n2. Resolve travis common test issues\n3. Priority queue option\n\n## NOTES\n\n* It is not enough to run distributed erlang nodes. They must also be connected.\n* The locks application must be running on at least 2 nodes...\n  * Calling erlang:disconnect_node/1 on a 2 node cluster is not supported by locks \n  * Calling dq:\u0026lt;fun\u0026gt;/\u0026lt;arity\u0026gt; on a \u003e1 node cluster with locks running on a single node will fail\n  * Calling dq:\u0026lt;fun\u0026gt;/\u0026lt;arity\u0026gt; on a 1 node cluster with locks running on a single node will succeed\n* Any queue state in a local queue before being adjoined to a cluster will be lost (not merged)\n\n## Enjoy!\n","funding_links":[],"categories":["Queue"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdarach%2Fdq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdarach%2Fdq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdarach%2Fdq/lists"}