{"id":22664650,"url":"https://github.com/emqx/replayq","last_synced_at":"2025-04-04T07:05:40.245Z","repository":{"id":43135894,"uuid":"144913985","full_name":"emqx/replayq","owner":"emqx","description":"Generic on-disk persistent queue implementation for Erlang","archived":false,"fork":false,"pushed_at":"2025-03-04T15:45:44.000Z","size":132,"stargazers_count":38,"open_issues_count":1,"forks_count":20,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-03-28T06:05:44.153Z","etag":null,"topics":["disk-queue","elixir","erlang","persistent-queue","replay-queue"],"latest_commit_sha":null,"homepage":"https://www.emqx.com","language":"Erlang","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/emqx.png","metadata":{"files":{"readme":"README.md","changelog":"changelog.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-08-15T23:44:01.000Z","updated_at":"2025-03-04T15:44:35.000Z","dependencies_parsed_at":"2024-11-07T11:27:02.068Z","dependency_job_id":"b8036489-d9d6-4f5b-88a3-c69c358e8475","html_url":"https://github.com/emqx/replayq","commit_stats":{"total_commits":53,"total_committers":7,"mean_commits":7.571428571428571,"dds":0.5094339622641509,"last_synced_commit":"c5945583fafee534a337b52d8b65e9ee925cd80d"},"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emqx%2Freplayq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emqx%2Freplayq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emqx%2Freplayq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emqx%2Freplayq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/emqx","download_url":"https://codeload.github.com/emqx/replayq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247135142,"owners_count":20889420,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["disk-queue","elixir","erlang","persistent-queue","replay-queue"],"created_at":"2024-12-09T13:15:25.530Z","updated_at":"2025-04-04T07:05:40.222Z","avatar_url":"https://github.com/emqx.png","language":"Erlang","readme":"# ReplayQ\n\nA Disk Queue for Log Replay in Erlang\n\n## Features\n\n* Queue items are written to segment files on disk to survive restart.\n* Batch popping items out of queue with size/count limit.\n* An `ack/2` API is provided to record the reader position within a segment.\n* Add config option `max_total_bytes` to limit the total size of replayq logs\n\n## Usage Example\n\n### Mem Only\n\n```erlang\nQ0 = replayq:open(#{mem_only =\u003e true}),\nQ1 = replayq:append(Q0, [Binary1, Binary2]),\n{Q2, AckRef, [Binary1]} = replayq:pop(Q1, #{bytes_limt =\u003e 1}),\nok = replayq:ack(Q2, AckRef).\n```\n\n### Binary Queue Items\n\n```erlang\nQ0 = replayq:open(#{dir =\u003e \"/tmp/replayq-test\", seg_bytes =\u003e 10000000}),\nQ1 = replayq:append(Q0, [Binary1, Binary2]),\n{Q2, AckRef, [Binary1]} = replayq:pop(Q1, #{count_limit =\u003e 1}),\nok = replayq:ack(Q2, AckRef).\n```\n\n### User Defined Queue Items\n\n```erlang\nQ0 = replayq:open(#{dir =\u003e \"/tmp/replayq-test\",\n                    seg_bytes =\u003e 10000000,\n                    sizer =\u003e fun({K, V}) -\u003e size(K) + size(V) end,\n                    marshaller =\u003e fun({K, V}) -\u003e term_to_binary({K, V});\n                                     (Bin)    -\u003e binary_to_term(Bin)\n                                  end\n                   }),\nQ1 = replayq:append(Q0, [{\u003c\u003c\"k1\"\u003e\u003e, \u003c\u003c\"v1\"\u003e\u003e}, {\u003c\u003c\"k2\"\u003e\u003e, \u003c\u003c\"v2\"\u003e\u003e}]),\n{Q2, AckRef, [{\u003c\u003c\"k1\"\u003e\u003e, \u003c\u003c\"v1\"\u003e\u003e}]} = replayq:pop(Q1, #{count_limit =\u003e 1}),\nok = replayq:ack(Q2, AckRef).\n```\n\n### User Defined Stop Before Function\n\nIn this example, we want to create a batch of items such that all items in the\nbatch are the same. We stop adding items to the batch as soon as we encounter\nan item that differs from the first item in the batch.\n\n```erlang\nQ0 = replayq:open(#{mem_only =\u003e true}),\nQ1 = replayq:append(Q0,\n                    [\n                     \u003c\u003c\"type1\"\u003e\u003e,\n                     \u003c\u003c\"type1\"\u003e\u003e,\n                     \u003c\u003c\"type2\"\u003e\u003e,\n                     \u003c\u003c\"type2\"\u003e\u003e,\n                     \u003c\u003c\"type2\"\u003e\u003e,\n                     \u003c\u003c\"type3\"\u003e\u003e\n                    ]),\nStopBeforeFunc =\nfun(Item, #{current_type := none}) -\u003e\n        #{current_type =\u003e Item};\n   (Item, #{current_type := Item}) -\u003e\n        %% Item and current_type are the same\n        #{current_type =\u003e Item};\n   (Item, #{current_type := _OtherType}) -\u003e\n        %% Return true to stop collecting items before the current item\n        true\nend,\nStopBeforeInitialState = #{current_type =\u003e none},\nStopBefore = {StopBeforeFunc, StopBeforeInitialState},\n%% We will stop because the stop_before function returns true\n{Q2, AckRef1, [\u003c\u003c\"type1\"\u003e\u003e, \u003c\u003c\"type1\"\u003e\u003e]} =\n    replayq:pop(Q1, #{stop_before =\u003e StopBefore, count_limit =\u003e 10}),\nok = replayq:ack(Q2, AckRef1),\n%% We will stop because of the count_limit\n{Q3, AckRef2, [\u003c\u003c\"type2\"\u003e\u003e]} =\n    replayq:pop(Q2, #{stop_before =\u003e StopBefore, count_limit =\u003e 1}),\nok = replayq:ack(Q3, AckRef2),\n%% We will stop because the stop_before function returns true\n{Q4, AckRef3, [\u003c\u003c\"type2\"\u003e\u003e, \u003c\u003c\"type2\"\u003e\u003e]} =\n    replayq:pop(Q3, #{stop_before =\u003e StopBefore, count_limit =\u003e 10}),\nok = replayq:ack(Q4, AckRef3),\n%% We will stop because the queue gets empty\n{Q5, AckRef4, [\u003c\u003c\"type3\"\u003e\u003e]} =\n    replayq:pop(Q4, #{stop_before =\u003e StopBefore, count_limit =\u003e 10}),\nok = replayq:ack(Q5, AckRef4).\n```\n\n### Offload mode\n\nIn offload mode, the disk queue is only used to offload queue tail segments.\nAdd `offload =\u003e true` to `Config` for `replayq:open/1`.\n\n#### Volatile mode\n\nUsing `offload =\u003e {true, volatile}` in the `Config` for\n`replayq:open/1` will make it unconditionally clear any previous\nsegments and commit file in the given work directory.  Also, it will\nnot dump in-memory data back to the disk when closing.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Femqx%2Freplayq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Femqx%2Freplayq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Femqx%2Freplayq/lists"}