{"id":21203812,"url":"https://github.com/phf/go-queue","last_synced_at":"2025-04-05T20:07:56.596Z","repository":{"id":11497957,"uuid":"13974333","full_name":"phf/go-queue","owner":"phf","description":"Queue data structure for Go; SAY NO TO GITHUB","archived":false,"fork":false,"pushed_at":"2017-05-04T03:16:29.000Z","size":65,"stargazers_count":129,"open_issues_count":2,"forks_count":22,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-29T19:06:16.161Z","etag":null,"topics":["datastructure","deque","fast","golang","queue"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/phf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-10-30T01:48:42.000Z","updated_at":"2024-10-31T06:06:37.000Z","dependencies_parsed_at":"2022-08-07T06:16:21.968Z","dependency_job_id":null,"html_url":"https://github.com/phf/go-queue","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phf%2Fgo-queue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phf%2Fgo-queue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phf%2Fgo-queue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phf%2Fgo-queue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/phf","download_url":"https://codeload.github.com/phf/go-queue/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247393570,"owners_count":20931812,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["datastructure","deque","fast","golang","queue"],"created_at":"2024-11-20T20:26:56.723Z","updated_at":"2025-04-05T20:07:56.578Z","avatar_url":"https://github.com/phf.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Queue data structure for Go\n\n[![GoDoc](https://godoc.org/github.com/phf/go-queue/queue?status.png)](http://godoc.org/github.com/phf/go-queue/queue)\n[![Go Report Card](https://goreportcard.com/badge/github.com/phf/go-queue)](https://goreportcard.com/report/github.com/phf/go-queue)\n\nPackage `queue` implements a double-ended queue (aka \"deque\") data structure\non top of a slice.\nAll operations run in (amortized) constant time.\nBenchmarks compare favorably to\n[container/list](https://golang.org/pkg/container/list/) as well as to\nGo's channels.\nThese queues are not safe for concurrent use.\n\nI tried to stick to the conventions established by\n[container/list](https://golang.org/pkg/container/list/)\neven though I disagree with them (see\n[`RANT.md`](https://github.com/phf/go-queue/blob/master/RANT.md)\nfor details).\nIn other words, this data structure is ready for the standard\nlibrary (hah!).\n\n## Background\n\nIn 2013 I was hacking a breadth-first search in Go and needed a\nqueue, but all I could find in the standard library was\n[container/list](https://golang.org/pkg/container/list/).\n\nNow in *principle* there's nothing wrong with\n[container/list](https://golang.org/pkg/container/list/), but I\nhad just admonished my students to *always* think carefully about\nthe number of memory allocations their programs make.\nIn other words, it felt wrong for me to use a data structure that\nallocates memory for *every* single vertex we visit during a\nbreadth-first search.\n\nAfter I got done with my project, I decided to clean up the queue\ncode a little and to push it here to give everybody else what I\nreally wanted to find in the standard library:\nA queue abstraction that doesn't allocate memory on every single\ninsertion.\n\n## Performance\n\n**Please read\n[`BENCH.md`](https://github.com/phf/go-queue/blob/master/BENCH.md)\nfor some perspective.\nSome numbers below are most likely \"contaminated\" in a way that makes\nour queues appear *worse* than they are.**\n\nHere are the numbers for my (ancient) home machine:\n\n```\n$ go test -bench=. -benchmem -count=10 \u003ebench.txt\n$ benchstat bench.txt\nname               time/op\nPushFrontQueue-2   82.8µs ± 1%\nPushFrontList-2     162µs ± 1%\nPushBackQueue-2    83.4µs ± 1%\nPushBackList-2      158µs ± 3%\nPushBackChannel-2   110µs ± 2%\nRandomQueue-2       161µs ± 2%\nRandomList-2        281µs ± 4%\nGrowShrinkQueue-2   110µs ± 1%\nGrowShrinkList-2    170µs ± 5%\n\nname               alloc/op\nPushFrontQueue-2   40.9kB ± 0%\nPushFrontList-2    57.4kB ± 0%\nPushBackQueue-2    40.9kB ± 0%\nPushBackList-2     57.4kB ± 0%\nPushBackChannel-2  24.7kB ± 0%\nRandomQueue-2      45.7kB ± 0%\nRandomList-2       90.8kB ± 0%\nGrowShrinkQueue-2  57.2kB ± 0%\nGrowShrinkList-2   57.4kB ± 0%\n\nname               allocs/op\nPushFrontQueue-2    1.03k ± 0%\nPushFrontList-2     2.05k ± 0%\nPushBackQueue-2     1.03k ± 0%\nPushBackList-2      2.05k ± 0%\nPushBackChannel-2   1.03k ± 0%\nRandomQueue-2       1.63k ± 0%\nRandomList-2        3.24k ± 0%\nGrowShrinkQueue-2   1.04k ± 0%\nGrowShrinkList-2    2.05k ± 0%\n$ go version\ngo version go1.8.1 linux/amd64\n$ cat /proc/cpuinfo | grep \"model name\" | uniq\nmodel name\t: AMD Athlon(tm) 64 X2 Dual Core Processor 6000+\n```\n\nThat's a [speedup](https://en.wikipedia.org/wiki/Speedup) of\n1.55-1.96\nover [container/list](https://golang.org/pkg/container/list/) and a speedup of\n1.31\nover Go's channels.\nWe also consistently allocate less memory in fewer allocations than\n[container/list](https://golang.org/pkg/container/list/).\n(Note that the number of allocations seems off: since we grow by *doubling*\nwe should only allocate memory *O(log n)* times.)\n\nThe same benchmarks on one of our department's servers:\n\n```\n$ go test -bench=. -benchmem -count=10 \u003ebench.txt\n$ benchstat bench.txt\nname               time/op\nPushFrontQueue-8   88.8µs ± 3%\nPushFrontList-8     156µs ± 5%\nPushBackQueue-8    88.3µs ± 1%\nPushBackList-8      159µs ± 2%\nPushBackChannel-8   132µs ± 2%\nRandomQueue-8       156µs ± 7%\nRandomList-8        279µs ±10%\nGrowShrinkQueue-8   117µs ± 0%\nGrowShrinkList-8    164µs ± 4%\n\nname               alloc/op\nPushFrontQueue-8   40.9kB ± 0%\nPushFrontList-8    57.4kB ± 0%\nPushBackQueue-8    40.9kB ± 0%\nPushBackList-8     57.4kB ± 0%\nPushBackChannel-8  24.7kB ± 0%\nRandomQueue-8      45.7kB ± 0%\nRandomList-8       90.8kB ± 0%\nGrowShrinkQueue-8  57.2kB ± 0%\nGrowShrinkList-8   57.4kB ± 0%\n\nname               allocs/op\nPushFrontQueue-8    1.03k ± 0%\nPushFrontList-8     2.05k ± 0%\nPushBackQueue-8     1.03k ± 0%\nPushBackList-8      2.05k ± 0%\nPushBackChannel-8   1.03k ± 0%\nRandomQueue-8       1.63k ± 0%\nRandomList-8        3.24k ± 0%\nGrowShrinkQueue-8   1.04k ± 0%\nGrowShrinkList-8    2.05k ± 0%\n$ go version\ngo version go1.7.5 linux/amd64\n$ cat /proc/cpuinfo | grep \"model name\" |uniq\nmodel name\t: Intel(R) Xeon(R) CPU           E5440  @ 2.83GHz\n```\n\nThat's a [speedup](https://en.wikipedia.org/wiki/Speedup) of\n1.76-1.80\nover [container/list](https://golang.org/pkg/container/list/) and a speedup of\n1.49\nover Go's channels.\n\nThe same benchmarks on a *different* department server:\n\n```\n$ go test -bench=. -benchmem -count=10 \u003ebench.txt\n$ benchstat bench.txt\nname                time/op\nPushFrontQueue-24   89.1µs ± 8%\nPushFrontList-24     176µs ± 8%\nPushBackQueue-24    86.8µs ± 5%\nPushBackList-24      178µs ± 6%\nPushBackChannel-24   151µs ±12%\nRandomQueue-24       180µs ±24%\nRandomList-24        334µs ± 7%\nGrowShrinkQueue-24   117µs ± 3%\nGrowShrinkList-24    187µs ± 6%\n\nname                alloc/op\nPushFrontQueue-24   40.9kB ± 0%\nPushFrontList-24    57.4kB ± 0%\nPushBackQueue-24    40.9kB ± 0%\nPushBackList-24     57.4kB ± 0%\nPushBackChannel-24  24.7kB ± 0%\nRandomQueue-24      45.7kB ± 0%\nRandomList-24       90.8kB ± 0%\nGrowShrinkQueue-24  57.2kB ± 0%\nGrowShrinkList-24   57.4kB ± 0%\n\nname                allocs/op\nPushFrontQueue-24    1.03k ± 0%\nPushFrontList-24     2.05k ± 0%\nPushBackQueue-24     1.03k ± 0%\nPushBackList-24      2.05k ± 0%\nPushBackChannel-24   1.03k ± 0%\nRandomQueue-24       1.63k ± 0%\nRandomList-24        3.24k ± 0%\nGrowShrinkQueue-24   1.04k ± 0%\nGrowShrinkList-24    2.05k ± 0%\n$ go version\ngo version go1.7.4 linux/amd64\n$ cat /proc/cpuinfo | grep \"model name\" |uniq\nmodel name\t: Intel(R) Xeon(R) CPU E5-2420 0 @ 1.90GHz\n```\n\nThat's a [speedup](https://en.wikipedia.org/wiki/Speedup) of\n1.86-**2.05**\nover [container/list](https://golang.org/pkg/container/list/) and a speedup of\n1.74\nover Go's channels.\n\nThe same benchmarks on an old\n[Raspberry Pi Model B Rev 1](https://en.wikipedia.org/wiki/Raspberry_Pi):\n\n```\n$ benchstat bench.txt\nname             time/op\nPushFrontQueue    788µs ±24%\nPushFrontList    2.74ms ±14%\nPushBackQueue    1.11ms ± 3%\nPushBackList     2.73ms ±14%\nPushBackChannel  1.25ms ± 3%\nRandomQueue      1.50ms ± 1%\nRandomList       4.92ms ± 6%\nGrowShrinkQueue  1.26ms ± 0%\nGrowShrinkList   2.88ms ± 2%\n\nname             alloc/op\nPushFrontQueue   16.5kB ± 0%\nPushFrontList    33.9kB ± 0%\nPushBackQueue    16.5kB ± 0%\nPushBackList     33.9kB ± 0%\nPushBackChannel  8.45kB ± 0%\nRandomQueue      16.5kB ± 0%\nRandomList       53.4kB ± 0%\nGrowShrinkQueue  24.6kB ± 0%\nGrowShrinkList   33.9kB ± 0%\n\nname             allocs/op\nPushFrontQueue     12.0 ± 0%\nPushFrontList     1.03k ± 0%\nPushBackQueue      12.0 ± 0%\nPushBackList      1.03k ± 0%\nPushBackChannel    1.00 ± 0%\nRandomQueue        12.0 ± 0%\nRandomList        1.63k ± 0%\nGrowShrinkQueue    20.0 ± 0%\nGrowShrinkList    1.03k ± 0%\n$ go version\ngo version go1.3.3 linux/arm\n$ cat /proc/cpuinfo |grep \"model name\"\nmodel name\t: ARMv6-compatible processor rev 7 (v6l)\n```\n\nThat's a [speedup](https://en.wikipedia.org/wiki/Speedup) of\n**2.46-3.48**\nover [container/list](https://golang.org/pkg/container/list/)\nbut only a speedup of\n1.13\nover Go's channels.\n(Note that I had to manually repeat the benchmarks and then run `benchtest`\nelsewhere since those features/tools are not available for Go 1.3;\nhowever, the number of allocations seems to be correct here for the first\ntime, maybe there's some breakage in the more recent benchmarking\nframework?)\n\n### Go's channels as queues\n\nGo's channels *used* to beat our queue implementation by about 22%\nfor `PushBack`.\nThat seemed sensible considering that channels are built into the\nlanguage and offer a lot less functionality:\nWe have to size them correctly if we want to use them as a simple\nqueue in an otherwise non-concurrent setting, they are not\ndouble-ended, and they don't support \"peeking\" at the next element\nwithout removing it.\n\nThat all changed with\n[two](https://github.com/phf/go-queue/commit/5652cbe39198516d853918fe64a4e70948b42f1a)\n[commits](https://github.com/phf/go-queue/commit/aa6086b89f98eb5cfd8df918e57612271ae1c137)\nin which I replaced the \"manual\" loop when a queue has to grow with\n[copy](https://golang.org/ref/spec#Appending_and_copying_slices)\nand the `%` operations to wrap indices around the slice with\nequivalent `\u0026` operations.\n(The code was originally written without these \"hacks\" because I wanted to\nshow it to my \"innocent\" Java students.)\nThose two changes *really* paid off.\n\n(I used to call channels \"*ridiculously* fast\" before and recommended their\nuse in situations where nothing but performance matters.\nAlas that may no longer be good advice.\nEither that, or I am just benchmarking incorrectly.)\n\n## Kudos\n\nHacking queue data structures in Go seems to be a popular way to spend\nan evening. Kudos to...\n\n- [Rodrigo Moraes](https://github.com/moraes) for posting\n  [this gist](https://gist.github.com/moraes/2141121) which reminded\n  me of Go's [copy](https://golang.org/ref/spec#Appending_and_copying_slices)\n  builtin and a similar trick I had previously used in Java.\n- [Evan Huus](https://github.com/eapache) for sharing\n  [his queue](https://github.com/eapache/queue) which reminded me of\n  the old \"replace % by \u0026\" trick I had used many times before.\n- [Dariusz Górecki](https://github.com/canni) for his\n  [commit](https://github.com/eapache/queue/commit/334cc1b02398be651373851653017e6cbf588f9e)\n  to [Evan](https://github.com/eapache)'s queue that simplified\n  [Rodrigo](https://github.com/moraes)'s snippet and hence mine.\n\nIf you find something in my code that helps you improve yours, feel\nfree to run with it!\n\n## Why use this queue?\n\nLooking around at other people's queues shows some \"common problems\" that\nthis implementation tries to avoid:\n\n- Most queues out there are just that, they are not double-ended.\n  Deques are more general but their implementation is *not* significantly\n  more complex. Why go for anything less as a general-purpose data structure?\n- Some queues panic. Personally I approve of this if it's done right\n  (see [`RANT.md`](https://github.com/phf/go-queue/blob/master/RANT.md)), but\n  *not* using panic is a better fit with the rest of the library/language.\n- Many queues use linked lists with one element per node, leading to the same\n  performance problems [container/list](https://golang.org/pkg/container/list/)\n  has.\n- Some queues offer \"strange\" operations that don't fit the queue/deque\n  abstraction. Call me a purist, but I prefer my interfaces complete yet\n  minimal.\n- Some queues are safe for concurrent use, but Go's philosophy seems to be to\n  leave this up to the programmer, not the library designer. The overhead of\n  locking doesn't \"disappear\" when you don't need it.\n- Some queues based on slices use `%` instead of `\u0026` for wrapping indices.\n  That *shouldn't* matter for performance but in fact it still does (as of\n  Go 1.7.5 anyway).\n- Some queues based on slices never shrink. In specific applications that\n  may be fine, for a general-purpose data structure it's not.\n\nWith that in mind, here's a list of queue/deque implementations that I\nwouldn't recommend:\n\n- https://github.com/eapache/queue not double-ended, panics, \"strange\" `Get`\n  operation\n- https://github.com/emnl/goods/tree/master/queue not double-ended, linked\n  list\n- https://github.com/oleiade/lane linked list, safe for concurrent use\n- https://github.com/Workiva/go-datastructures/tree/master/queue full\n  of semi-arcane concurrency stuff\n- https://github.com/pbberlin/tools/blob/master/util/util-fifo-queue.go not\n  double-ended, uses `%` instead of `\u0026`, requires starting size not less\n  than 5???\n- https://gist.github.com/moraes/2141121 not double-ended, doesn't shrink,\n  extra `Node` type\n- https://github.com/ErikDubbelboer/ringqueue/ not double-ended, uses\n  `%` instead of `\u0026`, seems to shrink too early (if I am reading the code\n  correctly)\n- https://github.com/iNamik/go_container/tree/master/queue not double-ended,\n  panics, \"strange\" `Peek` and `AtCapacity` operations, doesn't wrap around\n  in slice\n\nOf course you could always roll your own.\nI spent a reasonable amount of time on this one, making sure that it works\nwell as a general-purpose queue/deque data structure.\nBut go ahead, you can probably do better.\n\n## Where's the actual competition?\n\n- https://github.com/juju/utils/tree/master/deque seems pretty good. It uses\n  a viable alternative representation: a list of blocks. That should \"waste\"\n  less memory in some scenarios, but of course the code is more complicated\n  than ours. Sadly it doesn't have `Front` or `Back` operations, so the\n  interface isn't complete. But that would be an easy fix...\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphf%2Fgo-queue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fphf%2Fgo-queue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphf%2Fgo-queue/lists"}