{"id":21526322,"url":"https://github.com/linxgnu/pqueue","last_synced_at":"2025-04-09T23:33:22.165Z","repository":{"id":43451714,"uuid":"388552900","full_name":"linxGnu/pqueue","owner":"linxGnu","description":"A fast durable queue for Go","archived":false,"fork":false,"pushed_at":"2024-11-25T17:49:49.000Z","size":94,"stargazers_count":25,"open_issues_count":1,"forks_count":2,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-01T17:12:43.134Z","etag":null,"topics":["durability","durable","go","golang","persistent-queue","queue"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/linxGnu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-07-22T17:58:32.000Z","updated_at":"2025-02-12T09:47:46.000Z","dependencies_parsed_at":"2023-11-06T11:24:24.138Z","dependency_job_id":"28102d64-2c82-4919-a6ec-db266e5d4984","html_url":"https://github.com/linxGnu/pqueue","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linxGnu%2Fpqueue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linxGnu%2Fpqueue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linxGnu%2Fpqueue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linxGnu%2Fpqueue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/linxGnu","download_url":"https://codeload.github.com/linxGnu/pqueue/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248130048,"owners_count":21052689,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["durability","durable","go","golang","persistent-queue","queue"],"created_at":"2024-11-24T01:44:36.574Z","updated_at":"2025-04-09T23:33:22.141Z","avatar_url":"https://github.com/linxGnu.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pqueue - a fast durable queue for Go\n\n[![](https://github.com/linxGnu/pqueue/workflows/Build/badge.svg)]()\n[![Go Report Card](https://goreportcard.com/badge/github.com/linxGnu/pqueue)](https://goreportcard.com/report/github.com/linxGnu/pqueue)\n[![Coverage Status](https://coveralls.io/repos/github/linxGnu/pqueue/badge.svg?branch=main)](https://coveralls.io/github/linxGnu/pqueue?branch=main)\n\n`pqueue` is thread-safety, serves environments where more durability is required (e.g., outages last longer than memory queues can sustain)\n\n`pqueue` only consumes a bit of your memory. Most of the time, you are only bound to disk-size.\n\n## Installation\n\n```\ngo get -u github.com/linxGnu/pqueue \n```\n\n## Usage\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"log\"\n\n\t\"github.com/linxGnu/pqueue\"\n\t\"github.com/linxGnu/pqueue/entry\"\n)\n\nfunc main() {\n\t// 1000 entries per segment\n\tq, err := pqueue.New(\"/tmp\", 1000) // anywhere you want, instead of /tmp\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tdefer q.Close() // it's important to close the queue before exit\n\n\t// enqueue\n\tif err = q.Enqueue([]byte{1, 2, 3, 4}); err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tif err = q.Enqueue([]byte{5, 6, 7, 8}); err != nil {\n\t\tlog.Fatal(err)\n\t}\n\n\t// peek\n\tvar v entry.Entry\n\tif hasItem := q.Peek(\u0026v); hasItem {\n\t\tfmt.Println(v) // print: [1 2 3 4]\n\t}\n\n\t// dequeue\n\tif hasItem := q.Dequeue(\u0026v); hasItem {\n\t\tfmt.Println(v) // print: [1 2 3 4]\n\t}\n\tif hasItem := q.Dequeue(\u0026v); hasItem {\n\t\tfmt.Println(v) // print: [5 6 7 8]\n\t}\n}\n```\n\n## Limitation\n- Entry size must not be larger than 1GB\n\n## Benchmark\n\n### Comparing\n\n- [github.com/grandecola/bigqueue](https://github.com/grandecola/bigqueue) - embedded, fast and persistent queue written in pure Go using memory mapped (`mmap`) files\n  - bigqueue.SetPeriodicFlushOps(5): for data safety\n  - bigqueue.SetMaxInMemArenas(256MB)\n  - bigqueue.SetArenaSize(512MB)\n\n### Result\n\n### HDD - Hitachi HTS725050A7\n\n```\nDisk model: HGST HTS725050A7\nUnits: sectors of 1 * 512 = 512 bytes\nSector size (logical/physical): 512 bytes / 4096 bytes\nI/O size (minimum/optimal): 4096 bytes / 4096 bytes\n```\n\n```\ngoos: linux\ngoarch: amd64\npkg: github.com/linxGnu/pqueue\ncpu: AMD Ryzen 9 3950X 16-Core Processor\nPASS\nbenchmark                          iter        time/iter      bytes alloc            allocs\n---------                          ----        ---------      -----------            ------\nBenchmarkPQueueWriting_16-32        124       9.51 ms/op      102634 B/op   10146 allocs/op\nBenchmarkBigQueueWriting_16-32        1    1115.03 ms/op        5096 B/op      29 allocs/op\nBenchmarkPQueueWriting_64-32        133      10.01 ms/op      102433 B/op   10146 allocs/op\nBenchmarkBigQueueWriting_64-32        2     958.87 ms/op       11308 B/op      30 allocs/op\nBenchmarkPQueueWriting_256-32       120      10.51 ms/op      102616 B/op   10146 allocs/op\nBenchmarkBigQueueWriting_256-32       1    1170.48 ms/op        5336 B/op      29 allocs/op\nBenchmarkPQueueWriting_2048-32       60      20.25 ms/op      104512 B/op   10146 allocs/op\nBenchmarkBigQueueWriting_2048-32      1    2046.51 ms/op        7128 B/op      29 allocs/op\nBenchmarkPQueueWriting_16K-32        10     100.61 ms/op      120151 B/op   10146 allocs/op\nBenchmarkBigQueueWriting_16K-32       1    9634.72 ms/op       29776 B/op      32 allocs/op\nBenchmarkPQueueWriting_64K-32         4     349.45 ms/op      171772 B/op   10147 allocs/op\nBenchmarkBigQueueWriting_64K-32       1   29679.39 ms/op       83848 B/op      87 allocs/op\nBenchmarkPQueueRW_16-32              24      49.06 ms/op      273011 B/op   20238 allocs/op\nBenchmarkBigQueueRW_16-32             1   18573.37 ms/op      176096 B/op   10061 allocs/op\nBenchmarkPQueueRW_64-32              22      52.32 ms/op      272786 B/op   20236 allocs/op\nBenchmarkBigQueueRW_64-32             1   20714.97 ms/op      648248 B/op   10059 allocs/op\nBenchmarkPQueueRW_256-32             20      54.85 ms/op      273814 B/op   20245 allocs/op\nBenchmarkBigQueueRW_256-32            1   18967.45 ms/op     2573648 B/op   10033 allocs/op\nBenchmarkPQueueRW_2048-32            13      85.23 ms/op      279630 B/op   20240 allocs/op\nBenchmarkBigQueueRW_2048-32           1   23003.57 ms/op    20511568 B/op   10201 allocs/op\nBenchmarkPQueueRW_16K-32              4     286.53 ms/op      319328 B/op   20247 allocs/op\nBenchmarkBigQueueRW_16K-32            1   43056.72 ms/op   163882064 B/op   10161 allocs/op\nBenchmarkPQueueRW_64K-32              2     674.56 ms/op      473848 B/op   20236 allocs/op\nBenchmarkBigQueueRW_64K-32            1   71556.97 ms/op   655469672 B/op   10357 allocs/op\n```\n\n### SSD - Samsung SSD 850 Pro\n\n```\nDisk model: Samsung SSD 850 Pro\nUnits: sectors of 1 * 512 = 512 bytes\nSector size (logical/physical): 512 bytes / 512 bytes\nI/O size (minimum/optimal): 512 bytes / 512 bytes\n```\n\n```\ngoos: linux\ngoarch: amd64\npkg: github.com/linxGnu/pqueue\ncpu: AMD Ryzen 9 3950X 16-Core Processor            \nPASS\nbenchmark                          iter        time/iter      bytes alloc            allocs\n---------                          ----        ---------      -----------            ------\nBenchmarkPQueueWriting_16-32        122       9.91 ms/op      102507 B/op   10146 allocs/op\nBenchmarkBigQueueWriting_16-32        9     117.01 ms/op        4186 B/op      27 allocs/op\nBenchmarkPQueueWriting_64-32        120       9.62 ms/op      102424 B/op   10146 allocs/op\nBenchmarkBigQueueWriting_64-32        8     138.96 ms/op        4634 B/op      28 allocs/op\nBenchmarkPQueueWriting_256-32       106      10.79 ms/op      102647 B/op   10146 allocs/op\nBenchmarkBigQueueWriting_256-32       5     242.03 ms/op        2052 B/op      27 allocs/op\nBenchmarkPQueueWriting_2048-32       57      21.44 ms/op      104817 B/op   10146 allocs/op\nBenchmarkBigQueueWriting_2048-32      3     354.33 ms/op       10029 B/op      30 allocs/op\nBenchmarkPQueueWriting_16K-32        10     104.84 ms/op      120153 B/op   10146 allocs/op\nBenchmarkBigQueueWriting_16K-32       1    1963.32 ms/op       30640 B/op      41 allocs/op\nBenchmarkPQueueWriting_64K-32         3     349.73 ms/op      170405 B/op   10146 allocs/op\nBenchmarkBigQueueWriting_64K-32       1    6077.02 ms/op       86728 B/op     117 allocs/op\nBenchmarkPQueueRW_16-32              26      51.32 ms/op      271447 B/op   20240 allocs/op\nBenchmarkBigQueueRW_16-32             1    5947.32 ms/op      173408 B/op   10033 allocs/op\nBenchmarkPQueueRW_64-32              22      51.97 ms/op      272084 B/op   20240 allocs/op\nBenchmarkBigQueueRW_64-32             1    5586.13 ms/op      653456 B/op   10033 allocs/op\nBenchmarkPQueueRW_256-32             20      55.29 ms/op      273002 B/op   20241 allocs/op\nBenchmarkBigQueueRW_256-32            1    6269.66 ms/op     2568792 B/op   10066 allocs/op\nBenchmarkPQueueRW_2048-32            13      81.82 ms/op      278406 B/op   20240 allocs/op\nBenchmarkBigQueueRW_2048-32           1    6099.57 ms/op    20520656 B/op   10289 allocs/op\nBenchmarkPQueueRW_16K-32              4     287.24 ms/op      327796 B/op   20252 allocs/op\nBenchmarkBigQueueRW_16K-32            1    9192.42 ms/op   163882128 B/op   10155 allocs/op\nBenchmarkPQueueRW_64K-32              2     707.96 ms/op      473884 B/op   20277 allocs/op\nBenchmarkBigQueueRW_64K-32            1   14538.79 ms/op   655459096 B/op   10252 allocs/op\n```\n\n#### NVMe - Corsair Force MP600\n\n```\nDisk model: Corsair Force Series Gen.4 PCIe MP600 500GB NVMe M.2 SSD\nUnits: sectors of 1 * 512 = 512 bytes\nSector size (logical/physical): 512 bytes / 512 bytes\nI/O size (minimum/optimal): 512 bytes / 512 bytes\n```\n\n```\ngoos: linux\ngoarch: amd64\npkg: github.com/linxGnu/pqueue\ncpu: AMD Ryzen 9 3950X 16-Core Processor\nPASS\nbenchmark                          iter      time/iter      bytes alloc            allocs\n---------                          ----      ---------      -----------            ------\nBenchmarkPQueueWriting_16-32        225     5.23 ms/op      101927 B/op   10142 allocs/op\nBenchmarkBigQueueWriting_16-32     1089     1.03 ms/op         988 B/op      27 allocs/op\nBenchmarkPQueueWriting_64-32        232     5.15 ms/op      102055 B/op   10142 allocs/op\nBenchmarkBigQueueWriting_64-32      828     1.37 ms/op         984 B/op      27 allocs/op\nBenchmarkPQueueWriting_256-32       190     5.98 ms/op      102279 B/op   10142 allocs/op\nBenchmarkBigQueueWriting_256-32     482     2.50 ms/op        1176 B/op      27 allocs/op\nBenchmarkPQueueWriting_2048-32       97    11.93 ms/op      104170 B/op   10142 allocs/op\nBenchmarkBigQueueWriting_2048-32     80    13.67 ms/op        3508 B/op      27 allocs/op\nBenchmarkPQueueWriting_16K-32        18    64.92 ms/op      118953 B/op   10142 allocs/op\nBenchmarkBigQueueWriting_16K-32      10   101.22 ms/op       20366 B/op      30 allocs/op\nBenchmarkPQueueWriting_64K-32         5   209.72 ms/op      167192 B/op   10142 allocs/op\nBenchmarkBigQueueWriting_64K-32       3   364.77 ms/op       70128 B/op      56 allocs/op\nBenchmarkPQueueRW_16-32              36    31.39 ms/op      271413 B/op   20237 allocs/op\nBenchmarkBigQueueRW_16-32           100    11.76 ms/op      163011 B/op   10030 allocs/op\nBenchmarkPQueueRW_64-32              36    31.89 ms/op      270448 B/op   20241 allocs/op\nBenchmarkBigQueueRW_64-32            99    12.34 ms/op      647321 B/op   10033 allocs/op\nBenchmarkPQueueRW_256-32             33    33.75 ms/op      272010 B/op   20243 allocs/op\nBenchmarkBigQueueRW_256-32           81    14.75 ms/op     2571970 B/op   10035 allocs/op\nBenchmarkPQueueRW_2048-32            20    52.16 ms/op      278024 B/op   20237 allocs/op\nBenchmarkBigQueueRW_2048-32          33    35.82 ms/op    20497430 B/op   10054 allocs/op\nBenchmarkPQueueRW_16K-32              5   200.83 ms/op      323212 B/op   20244 allocs/op\nBenchmarkBigQueueRW_16K-32            6   175.49 ms/op   163876804 B/op   10105 allocs/op\nBenchmarkPQueueRW_64K-32              3   471.45 ms/op      469130 B/op   20253 allocs/op\nBenchmarkBigQueueRW_64K-32            2   570.38 ms/op   655472440 B/op   10391 allocs/op\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinxgnu%2Fpqueue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flinxgnu%2Fpqueue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinxgnu%2Fpqueue/lists"}