{"id":21658419,"url":"https://github.com/grandecola/bigqueue","last_synced_at":"2025-07-17T21:31:03.974Z","repository":{"id":33976857,"uuid":"138180298","full_name":"grandecola/bigqueue","owner":"grandecola","description":"Embedded, Fast and Persistent bigqueue implementation","archived":false,"fork":false,"pushed_at":"2021-09-12T11:28:08.000Z","size":118,"stargazers_count":441,"open_issues_count":17,"forks_count":32,"subscribers_count":10,"default_branch":"main","last_synced_at":"2024-06-18T17:01:51.347Z","etag":null,"topics":["bigqueue","embedded","golang","persistence","queue"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/grandecola.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-06-21T14:21:30.000Z","updated_at":"2024-05-08T13:35:06.000Z","dependencies_parsed_at":"2022-08-07T23:31:10.106Z","dependency_job_id":null,"html_url":"https://github.com/grandecola/bigqueue","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grandecola%2Fbigqueue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grandecola%2Fbigqueue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grandecola%2Fbigqueue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grandecola%2Fbigqueue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/grandecola","download_url":"https://codeload.github.com/grandecola/bigqueue/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226304673,"owners_count":17603640,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigqueue","embedded","golang","persistence","queue"],"created_at":"2024-11-25T09:29:14.708Z","updated_at":"2024-11-25T09:29:23.088Z","avatar_url":"https://github.com/grandecola.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"[![PkgGoDev](https://pkg.go.dev/badge/github.com/grandecola/bigqueue)](https://pkg.go.dev/github.com/grandecola/bigqueue) [![MIT license](http://img.shields.io/badge/license-MIT-brightgreen.svg)](http://opensource.org/licenses/MIT) [![Build Status](https://travis-ci.com/grandecola/bigqueue.svg?branch=master)](https://travis-ci.com/grandecola/bigqueue) [![codecov](https://codecov.io/gh/grandecola/bigqueue/branch/master/graph/badge.svg)](https://codecov.io/gh/grandecola/bigqueue)\n\n [![Go Report Card](https://goreportcard.com/badge/github.com/grandecola/bigqueue)](https://goreportcard.com/report/github.com/grandecola/bigqueue) [![Codacy Badge](https://app.codacy.com/project/badge/Grade/f7a080f9ab2b4f7e9543b4eb8e404e2b)](https://www.codacy.com/gh/grandecola/bigqueue/dashboard?utm_source=github.com\u0026amp;utm_medium=referral\u0026amp;utm_content=grandecola/bigqueue\u0026amp;utm_campaign=Badge_Grade) [![Maintainability](https://api.codeclimate.com/v1/badges/b3e1b2f184edd8150ddd/maintainability)](https://codeclimate.com/github/grandecola/bigqueue/maintainability) [![CodeFactor](https://www.codefactor.io/repository/github/grandecola/bigqueue/badge)](https://www.codefactor.io/repository/github/grandecola/bigqueue)\n\n# bigqueue\n\n`bigqueue` provides embedded, fast and persistent queue written in pure Go using\nmemory mapped (`mmap`) files. `bigqueue` is now *thread safe* as well.\n\n## Installation\n```\ngo get github.com/grandecola/bigqueue\n```\n\n## Requirements\n* Only works for `linux` and `darwin` OS\n* Only works on Little Endian architecture\n\n## Usage\n\n### Standard API\nCreate or open a bigqueue:\n```go\nbq, err := bigqueue.NewMmapQueue(\"path/to/queue\")\ndefer bq.Close()\n```\n\nbigqueue persists the data of the queue in multiple Arenas.\nEach Arena is a file on disk that is mapped into memory (RAM)\nusing mmap syscall. Default size of each Arena is set to 128MB.\nIt is possible to create a bigqueue with custom Arena size:\n```go\nbq, err := bigqueue.NewMmapQueue(\"path/to/queue\", bigqueue.SetArenaSize(4*1024))\ndefer bq.Close()\n```\n\nBigqueue also allows setting up the maximum possible memory that it\ncan use. By default, the maximum memory is set to [3 x Arena Size].\n```go\nbq, err := bigqueue.NewQueue(\"path/to/queue\", bigqueue.SetArenaSize(4*1024),\n\t    bigqueue.SetMaxInMemArenas(10))\ndefer bq.Close()\n```\nIn this case, bigqueue will never allocate more memory than `4KB*10=40KB`. This\nmemory is above and beyond the memory used in buffers for copying data.\n\nBigqueue allows to set periodic flush based on either elapsed time or number\nof mutate (enqueue/dequeue) operations. Flush syncs the in memory changes of all\nmemory mapped files with disk. *This is a best effort flush*.\n\nThis is how we can set these options:\n```go\nbq, err := bigqueue.NewQueue(\"path/to/queue\", bigqueue.SetPeriodicFlushOps(2))\n```\nIn this case, a flush is done after every two mutate operations.\n\n```go\nbq, err := bigqueue.NewQueue(\"path/to/queue\", bigqueue.SetPeriodicFlushDuration(time.Minute))\n```\nIn this case, a flush is done after one minute elapses and an Enqueue/Dequeue is called.\n\nWrite to bigqueue:\n```go\nerr := bq.Enqueue([]byte(\"elem\"))\n```\n\nbigqueue allows writing string data directly, avoiding conversion to `[]byte`:\n```go\nerr := bq.EnqueueString(\"elem\")\n```\n\nRead from bigqueue:\n```go\nelem, err := bq.Dequeue()\n```\n\nwe can also read string data from bigqueue:\n```go\nelem, err := bq.DequeueString()\n```\n\nCheck whether bigqueue has non zero elements:\n```go\nisEmpty := bq.IsEmpty()\n```\n\n### Advanced API\nbigqueue allows reading data from bigqueue using consumers similar to Kafka. This allows\nmultiple consumers from reading data at different offsets (not in thread safe manner yet).\nThe offsets of each consumer are persisted on disk and can be retrieved by creating a\nconsumer with the same name. Data will be read from the same offset where it was left off.\n\nWe can create a new consumer as follows. The offsets of a new consumer are set at the\nstart of the queue wherever the first non-deleted element is.\n```go\nconsumer, err := bq.NewConsumer(\"consumer\")\n```\n\nWe can also copy an existing consumer. This will create a consumer that will have the\nsame offsets into the queue as that of the existing consumer.\n```go\ncopyConsumer, err := bq.FromConsumer(\"copyConsumer\", consumer)\n```\n\nNow, read operations can be performed on the consumer:\n```go\nisEmpty := consumer.IsEmpty()\nelem, err := consumer.Dequeue()\nelem, err := consumer.DequeueString()\n```\n\n## Benchmarks\n\n### Setup\n\n```\ngoos: linux\ngoarch: amd64\ncpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz\nGo version: 1.16\n```\n\n### NewMmapQueue\n```go\nBenchmarkNewMmapQueue/ArenaSize-4KB-12         \t     312\t   3762844 ns/op\t    2506 B/op\t      36 allocs/op\nBenchmarkNewMmapQueue/ArenaSize-128KB-12       \t     310\t   3897333 ns/op\t    2506 B/op\t      36 allocs/op\nBenchmarkNewMmapQueue/ArenaSize-4MB-12         \t     301\t   4033893 ns/op\t    2504 B/op\t      36 allocs/op\nBenchmarkNewMmapQueue/ArenaSize-128MB-12       \t     309\t   3954329 ns/op\t    2504 B/op\t      36 allocs/op\n```\n\n### Enqueue\n```go\nBenchmarkEnqueue/ArenaSize-4KB/MessageSize-128B/MaxMem-12KB-12         \t 1021855\t      1172 ns/op\t      15 B/op\t       0 allocs/op\nBenchmarkEnqueue/ArenaSize-4KB/MessageSize-128B/MaxMem-40KB-12         \t  999122\t      1178 ns/op\t      15 B/op\t       0 allocs/op\nBenchmarkEnqueue/ArenaSize-4KB/MessageSize-128B/MaxMem-NoLimit-12      \t 1000000\t      1027 ns/op\t      20 B/op\t       0 allocs/op\nBenchmarkEnqueue/ArenaSize-128KB/MessageSize-4KB/MaxMem-384KB-12       \t  258444\t      4602 ns/op\t      14 B/op\t       0 allocs/op\nBenchmarkEnqueue/ArenaSize-128KB/MessageSize-4KB/MaxMem-1.25MB-12      \t  246780\t      4610 ns/op\t      14 B/op\t       0 allocs/op\nBenchmarkEnqueue/ArenaSize-128KB/MessageSize-4KB/MaxMem-NoLimit-12     \t  271261\t      4118 ns/op\t      14 B/op\t       0 allocs/op\nBenchmarkEnqueue/ArenaSize-4MB/MessageSize-128KB/MaxMem-12MB-12        \t   10000\t    108440 ns/op\t      14 B/op\t       0 allocs/op\nBenchmarkEnqueue/ArenaSize-4MB/MessageSize-128KB/MaxMem-40MB-12        \t   10000\t    108159 ns/op\t      14 B/op\t       0 allocs/op\nBenchmarkEnqueue/ArenaSize-4MB/MessageSize-128KB/MaxMem-NoLimit-12     \t   10000\t    104991 ns/op\t      14 B/op\t       0 allocs/op\nBenchmarkEnqueue/ArenaSize-128MB/MessageSize-4MB/MaxMem-256MB-12       \t     330\t   3619772 ns/op\t      13 B/op\t       0 allocs/op\nBenchmarkEnqueue/ArenaSize-128MB/MessageSize-4MB/MaxMem-1.25GB-12      \t     339\t   3502254 ns/op\t      13 B/op\t       0 allocs/op\nBenchmarkEnqueue/ArenaSize-128MB/MessageSize-4MB/MaxMem-NoLimit-12     \t     336\t   3478795 ns/op\t      13 B/op\t       0 allocs/op\n```\n\n### EnqueueString\n```go\nBenchmarkEnqueueString/ArenaSize-4KB/MessageSize-128B/MaxMem-12KB-12   \t  843966\t      1186 ns/op\t      15 B/op\t       0 allocs/op\nBenchmarkEnqueueString/ArenaSize-4KB/MessageSize-128B/MaxMem-40KB-12   \t 1000000\t      1180 ns/op\t      15 B/op\t       0 allocs/op\nBenchmarkEnqueueString/ArenaSize-4KB/MessageSize-128B/MaxMem-NoLimit-12         \t 1000000\t      1026 ns/op\t      15 B/op\t       0 allocs/op\nBenchmarkEnqueueString/ArenaSize-128KB/MessageSize-4KB/MaxMem-384KB-12          \t  257824\t      4642 ns/op\t      14 B/op\t       0 allocs/op\nBenchmarkEnqueueString/ArenaSize-128KB/MessageSize-4KB/MaxMem-1.25MB-12         \t  256230\t      4621 ns/op\t      14 B/op\t       0 allocs/op\nBenchmarkEnqueueString/ArenaSize-128KB/MessageSize-4KB/MaxMem-NoLimit-12        \t  266560\t      4101 ns/op\t      14 B/op\t       0 allocs/op\nBenchmarkEnqueueString/ArenaSize-4MB/MessageSize-128KB/MaxMem-12MB-12           \t   10000\t    107929 ns/op\t      14 B/op\t       0 allocs/op\nBenchmarkEnqueueString/ArenaSize-4MB/MessageSize-128KB/MaxMem-40MB-12           \t   10000\t    107948 ns/op\t      14 B/op\t       0 allocs/op\nBenchmarkEnqueueString/ArenaSize-4MB/MessageSize-128KB/MaxMem-NoLimit-12        \t   11434\t    103482 ns/op\t      13 B/op\t       0 allocs/op\nBenchmarkEnqueueString/ArenaSize-128MB/MessageSize-4MB/MaxMem-256MB-12          \t     333\t   3650641 ns/op\t      13 B/op\t       0 allocs/op\nBenchmarkEnqueueString/ArenaSize-128MB/MessageSize-4MB/MaxMem-1.25GB-12         \t     339\t   3559835 ns/op\t      13 B/op\t       0 allocs/op\nBenchmarkEnqueueString/ArenaSize-128MB/MessageSize-4MB/MaxMem-NoLimit-12        \t     334\t   3546090 ns/op\t      13 B/op\t       0 allocs/op\n```\n\n### Dequeue\n```go\nBenchmarkDequeue/ArenaSize-4KB/MessageSize-128B/MaxMem-12KB-12                  \t 1000000\t      3201 ns/op\t     142 B/op\t       1 allocs/op\nBenchmarkDequeue/ArenaSize-4KB/MessageSize-128B/MaxMem-40KB-12                  \t 1000000\t      3187 ns/op\t     142 B/op\t       1 allocs/op\nBenchmarkDequeue/ArenaSize-4KB/MessageSize-128B/MaxMem-NoLimit-12               \t 6737412\t       174.0 ns/op\t     128 B/op\t       1 allocs/op\nBenchmarkDequeue/ArenaSize-128KB/MessageSize-4KB/MaxMem-384KB-12                \t  502522\t      3478 ns/op\t    4109 B/op\t       1 allocs/op\nBenchmarkDequeue/ArenaSize-128KB/MessageSize-4KB/MaxMem-1.25MB-12               \t  516555\t      3509 ns/op\t    4109 B/op\t       1 allocs/op\nBenchmarkDequeue/ArenaSize-128KB/MessageSize-4KB/MaxMem-NoLimit-12              \t 1000000\t      1156 ns/op\t    4096 B/op\t       1 allocs/op\nBenchmarkDequeue/ArenaSize-4MB/MessageSize-128KB/MaxMem-12MB-12                 \t   29844\t     39677 ns/op\t  131085 B/op\t       1 allocs/op\nBenchmarkDequeue/ArenaSize-4MB/MessageSize-128KB/MaxMem-40MB-12                 \t   30626\t     39388 ns/op\t  131085 B/op\t       1 allocs/op\nBenchmarkDequeue/ArenaSize-4MB/MessageSize-128KB/MaxMem-NoLimit-12              \t   45805\t     26247 ns/op\t  131072 B/op\t       1 allocs/op\nBenchmarkDequeue/ArenaSize-128MB/MessageSize-4MB/MaxMem-256MB-12                \t    1005\t   1241554 ns/op\t 4194316 B/op\t       1 allocs/op\nBenchmarkDequeue/ArenaSize-128MB/MessageSize-4MB/MaxMem-1.25GB-12               \t    1257\t   1164477 ns/op\t 4194314 B/op\t       1 allocs/op\nBenchmarkDequeue/ArenaSize-128MB/MessageSize-4MB/MaxMem-NoLimit-12              \t    1260\t    884842 ns/op\t 4194304 B/op\t       1 allocs/op\n```\n\n### DequeueString\n```go\nBenchmarkDequeueString/ArenaSize-4KB/MessageSize-128B/MaxMem-12KB-12            \t 1000000\t      3200 ns/op\t     142 B/op\t       1 allocs/op\nBenchmarkDequeueString/ArenaSize-4KB/MessageSize-128B/MaxMem-40KB-12            \t 1000000\t      3206 ns/op\t     142 B/op\t       1 allocs/op\nBenchmarkDequeueString/ArenaSize-4KB/MessageSize-128B/MaxMem-NoLimit-12         \t 6239718\t       188.8 ns/op\t     128 B/op\t       1 allocs/op\nBenchmarkDequeueString/ArenaSize-128KB/MessageSize-4KB/MaxMem-384KB-12          \t  501561\t      3511 ns/op\t    4109 B/op\t       1 allocs/op\nBenchmarkDequeueString/ArenaSize-128KB/MessageSize-4KB/MaxMem-1.25MB-12         \t  507860\t      3535 ns/op\t    4109 B/op\t       1 allocs/op\nBenchmarkDequeueString/ArenaSize-128KB/MessageSize-4KB/MaxMem-NoLimit-12        \t 1000000\t      1236 ns/op\t    4096 B/op\t       1 allocs/op\nBenchmarkDequeueString/ArenaSize-4MB/MessageSize-128KB/MaxMem-12MB-12           \t   29692\t     39532 ns/op\t  131085 B/op\t       1 allocs/op\nBenchmarkDequeueString/ArenaSize-4MB/MessageSize-128KB/MaxMem-40MB-12           \t   30268\t     39709 ns/op\t  131085 B/op\t       1 allocs/op\nBenchmarkDequeueString/ArenaSize-4MB/MessageSize-128KB/MaxMem-NoLimit-12        \t   46911\t     25956 ns/op\t  131072 B/op\t       1 allocs/op\nBenchmarkDequeueString/ArenaSize-128MB/MessageSize-4MB/MaxMem-256MB-12          \t     968\t   1254574 ns/op\t 4194316 B/op\t       1 allocs/op\nBenchmarkDequeueString/ArenaSize-128MB/MessageSize-4MB/MaxMem-1.25GB-12         \t    1429\t   1175763 ns/op\t 4194314 B/op\t       1 allocs/op\nBenchmarkDequeueString/ArenaSize-128MB/MessageSize-4MB/MaxMem-NoLimit-12        \t    1364\t    865977 ns/op\t 4194304 B/op\t       1 allocs/op$$\n```\n\n**Note:** Before running benchmarks `ulimit` and `vm.max_map_count` parameters should be adjusted using below commands:\n```\nulimit -n 50000\necho 262144 \u003e /proc/sys/vm/max_map_count\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrandecola%2Fbigqueue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgrandecola%2Fbigqueue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrandecola%2Fbigqueue/lists"}