{"id":15646548,"url":"https://github.com/justintimperio/gpq","last_synced_at":"2025-08-20T10:32:44.363Z","repository":{"id":224286768,"uuid":"762052698","full_name":"JustinTimperio/gpq","owner":"JustinTimperio","description":"GPQ is a high performance embeddable double priority queue with complex priority ordering guarantees ","archived":false,"fork":false,"pushed_at":"2024-09-09T03:26:28.000Z","size":716,"stargazers_count":51,"open_issues_count":0,"forks_count":4,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-12-14T00:44:23.114Z","etag":null,"topics":["double-priority-queue","golang","high-performance","message-bus","message-queue","priority-queue"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JustinTimperio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-23T01:44:54.000Z","updated_at":"2024-10-29T12:22:15.000Z","dependencies_parsed_at":"2024-02-25T04:22:30.844Z","dependency_job_id":"94524aaa-e89c-4259-8bb7-97e60ca7c58e","html_url":"https://github.com/JustinTimperio/gpq","commit_stats":null,"previous_names":["justintimperio/gpq"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JustinTimperio%2Fgpq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JustinTimperio%2Fgpq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JustinTimperio%2Fgpq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JustinTimperio%2Fgpq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JustinTimperio","download_url":"https://codeload.github.com/JustinTimperio/gpq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230415318,"owners_count":18222158,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["double-priority-queue","golang","high-performance","message-bus","message-queue","priority-queue"],"created_at":"2024-10-03T12:13:14.584Z","updated_at":"2025-08-20T10:32:44.349Z","avatar_url":"https://github.com/JustinTimperio.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./docs/gpq.png\"\u003e\n\u003c/p\u003e\n\n\u003ch4 align=\"center\"\u003e\n\tGPQ is an extremely fast and flexible priority queue, capable of millions transactions a second. GPQ supports a complex \"Double Priority Queue\" which allows for priorities to be distributed across N buckets, with each bucket holding a second priority queue which allows for internal escalation and timeouts of items based on parameters the user can specify during submission combined with how frequently you ask GPQ to prioritize the queue.\n\u003c/h4\u003e\n\n\u003cp align=\"center\"\u003e\n\t\u003ca href=\"https://go.dev/dl/\"\u003e\u003cimg alt=\"Go version\" src=\"https://img.shields.io/github/go-mod/go-version/JustinTimperio/gpq\"\u003e\u003c/a\u003e\n\t\u003ca href=\"https://pkg.go.dev/github.com/JustinTimperio/gpq\"\u003e\u003cimg alt=\"Go Reference\" src=\"https://pkg.go.dev/badge/github.com/JustinTimperio/gpq.svg\"\u003e\u003c/a\u003e\n\t\u003ca href=\"https://github.com/JustinTimperio/gpq/blob/master/LICENSE\"\u003e\u003cimg alt=\"GitHub License\" src=\"https://img.shields.io/github/license/JustinTimperio/gpq\"\u003e\u003c/a\u003e\n\t\u003ca href=\"https://github.com/JustinTimperio/gpq/releases\"\u003e\u003cimg alt=\"GitHub Release\" src=\"https://img.shields.io/github/v/release/JustinTimperio/gpq\"\u003e\u003c/a\u003e\n\t\u003ca href=\"https://github.com/JustinTimperio/gpq/issues\"\u003e\u003cimg alt=\"GitHub Issues\" src=\"https://img.shields.io/github/issues/JustinTimperio/gpq\"\u003e\u003c/a\u003e\n\t\u003ca href=\"https://github.com/JustinTimperio/gpq/actions\"\u003e\u003cimg alt=\"GitHub Branch Status\" src=\"https://img.shields.io/github/check-runs/JustinTimperio/gpq/master\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n## Notice\nWhile GPQ is largely stable, bugs are more than likely present at this early stage, and you should carefully consider if your application can tolerate any down time or lost messages that may result from adopting this project into a production workflow. If you run into any bugs please submit an issue or better a PR! Check out the guide to contributing below.\n\n\n## Table of Contents\n- [Notice](#notice)\n- [Table of Contents](#table-of-contents)\n- [Background](#background)\n\t- [Should I Use GPQ?](#should-i-use-gpq)\n\t- [Sister Projects](#sister-projects)\n- [Benchmarks](#benchmarks)\n- [Usage](#usage)\n\t- [Prerequisites](#prerequisites)\n\t- [API Reference](#api-reference)\n\t- [Submitting Items to the Queue](#submitting-items-to-the-queue)\n- [Contributing](#contributing)\n\t- [We Develop with Github](#we-develop-with-github)\n\t- [All Code Changes Happen Through Pull Requests](#all-code-changes-happen-through-pull-requests)\n\t- [Any contributions you make will be under the MIT Software License](#any-contributions-you-make-will-be-under-the-mit-software-license)\n\t- [Report bugs using Github's Issues](#report-bugs-using-githubs-issues)\n\t- [Write bug reports with detail, background, and sample code](#write-bug-reports-with-detail-background-and-sample-code)\n- [License](#license)\n\n## Background\nGPQ was written as an experiment when I was playing with [Fibonacci Heaps](https://en.wikipedia.org/wiki/Fibonacci_heap) and wanted to find something faster. I was disappointed by the state of research and libraries being used by most common applications, so GPQ is meant to be a highly flexible framework that can support a multitude of workloads.\n\n### Should I Use GPQ?\nGPQ is a concurrency safe, embeddable priority queue that can be used in a variety of applications. GPQ might be the right choice if:\n- Your data requires strict ordering guarantees\n- You need to prioritize items that are in the queue too long \n- You need to timeout items\n- You have multiple writers and readers that need to access the queue concurrently\n- You run critical workloads and need to store the queue on disk in case of a crash\n\n### Sister Projects\n- [fibheap (Fibonacci Heaps)](https://github.com/JustinTimperio/fibheap)\n- [rpq (Rust Priority Queue)](https://github.com/JustinTimperio/rpq)\n- [gpq-server (GPQ Server)](https://github.com/JustinTimperio/gpq-server)\n- [pq-bench (Priority Queue Benchmarks)](https://github.com/JustinTimperio/pq-bench)\n\n\n## Benchmarks\nDue to the fact that most operations are done in constant time `O(1)` or logarithmic time `O(log n)`, with the exception of the prioritize function which happens in linear time `O(n)`, all GPQ operations are extremely fast. A single GPQ can handle a few million transactions a second and can be tuned depending on your work load. I have included some basic benchmarks using C++, Rust, Zig, Python, and Go to measure GPQ's performance against the standard implementations of other languages that can be found here at: [pq-bench](https://github.com/JustinTimperio/pq-bench) \n\n|                                                                                                             |                                                                                                          |\n|-------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|\n| ![Time-Spent](https://github.com/JustinTimperio/pq-bench/blob/master/docs/Time-Spent-vs-Implementation.png) | ![Queue-Speed-WITHOUT-Reprioritize](./docs/Queue-Speed-Without-Prioritize.png)                           |\n| ![Queue-Speed-WITH-Reprioritize](./docs/Queue-Speed-With-Prioritize.png)                                    | ![Average Time to Send and Receive VS Bucket Count](./docs/Time-to-Send-and-Receive-VS-Bucket-Count.png) |\n\n## Usage\nGPQ at the core is a embeddable priority queue meant to be used at the core of critical workloads that require complex queueing and delivery order guarantees. The best way to use it is just to import it.\n\n```go\nimport \"github.com/JustinTimperio/gpq\"\n```\n\n### Prerequisites \nFor this you will need Go \u003e= `1.22` and gpq itself uses [hashmap](https://github.com/cornelk/hashmap), [btree](https://github.com/tidwall/btree) and [BadgerDB](https://github.com/dgraph-io/badger). \n\n### API Reference\n- `NewGPQ[d any](options schema.GPQOptions) (uint, *GPQ[d], error)` - Creates a new GPQ with the specified options and returns the number of restored items, the GPQ, and an error if one occurred. \n  - `ItemsInQueue() uint` - Returns the number of items in the queue.\n  - `ItemsInDB() uint` - Returns the number of items in the database.\n  - `ActiveBuckets() uint` - Returns the number of active buckets.\n  - `Enqueue(item schema.Item[d]) error` - Enqueues an item into the queue.\n  - `EnqueueBatch(items []schema.Item[d]) error` - Enqueues a batch of items into the queue.\n  - `Dequeue() (*schema.Item[d], error)` - Dequeues an item from the queue.\n  - `DequeueBatch(batchSize uint) ([]*schema.Item[d], error)` - Dequeues a batch of items from the queue.\n  - `Prioritize() error` - Prioritizes the queue based on the values in each item.\n  - `Close()` - Closes the queue and saves the queue to disk.\n\n### Submitting Items to the Queue\nOnce you have an initialized queue you can easily submit items like the following:\n```go\npackage main\n\nimport (\n  \"log\"\n  \"time\"\n\n  \"github.com/JustinTimperio/gpq\"\n)\n\nfunc main() {\n\tdefaultMessageOptions := schema.EnqueueOptions{\n\t\tShouldEscalate: true,\n\t\tEscalationRate: time.Duration(time.Second),\n\t\tCanTimeout:     true,\n\t\tTimeout:        time.Duration(time.Second * 5),\n\t}\n\n\topts := schema.GPQOptions{\n\t\tMaxPriority: maxBuckets,\n\n\t\tDiskCacheEnabled:      true,\n\t\tDiskCachePath:         \"/tmp/gpq\",\n\t\tDiskCacheCompression:  true,\n\t\tDiskEncryptionEnabled: true,\n\t\tDiskEncryptionKey:     []byte(\"12345678901234567890123456789012\"),\n\t\tDiskWriteDelay:       time.Duration(time.Second * 5),\n\n\t\tLazyDiskCacheEnabled: true,\n\t\tLazyDiskCacheChannelSize:  1_000_000,\n\t\tLazyDiskBatchSize:    10_000,\n\t}\n\n\t_, queue, err := gpq.NewGPQ[uint](opts)\n\tif err != nil {\n\t\tlog.Fatalln(err)\n\t}\n\n\tfor i := uint(0); i \u003c total; i++ {\n\t\tp := i % maxBuckets\n\t\titem := schema.NewItem(p, i, defaultMessageOptions)\n\n\t\terr := queue.Enqueue(item)\n\t\tif err != nil {\n\t\t\tlog.Fatalln(err)\n\t\t}\n\t}\n\n\tfor i := uint(0); i \u003c total; i++ {\n\t\titem, err := queue.Dequeue()\n\t\tif err != nil {\n\t\t\tlog.Fatalln(err)\n\t\t}\n\t}\n\n\tqueue.Close()\n}\n```\n\nYou have a few options when you submit a job such as if the item should escalate over time if not sent, or inversely can timeout if it has been enqueued to long to be relevant anymore.\n\n## Contributing\nGPQ is actively looking for maintainers so feel free to help out when:\n\n- Reporting a bug\n- Discussing the current state of the code\n- Submitting a fix\n- Proposing new features\n\n### We Develop with Github\nWe use github to host code, to track issues and feature requests, as well as accept pull requests.\n\n### All Code Changes Happen Through Pull Requests\n1. Fork the repo and create your branch from `master`.\n2. If you've added code that should be tested, add tests.\n3. If you've changed APIs, update the documentation.\n4. Ensure the test suite passes.\n5. Make sure your code lints.\n6. Issue that pull request!\n\n### Any contributions you make will be under the MIT Software License\nIn short, when you submit code changes, your submissions are understood to be under the same [MIT License](http://choosealicense.com/licenses/mit/) that covers the project. Feel free to contact the maintainers if that's a concern.\n\n### Report bugs using Github's [Issues](https://github.com/JustinTimperio/gpq/issues)\nWe use GitHub issues to track public bugs. Report a bug by opening a new issue; it's that easy!\n\n### Write bug reports with detail, background, and sample code\n**Great Bug Reports** tend to have:\n\n- A quick summary and/or background\n- Steps to reproduce\n  - Be specific!\n  - Give sample code if you can.\n- What you expected would happen\n- What actually happens\n- Notes (possibly including why you think this might be happening, or stuff you tried that didn't work)\n\n## License\nAll code here was originally written by me, Justin Timperio, under an MIT license with the exception of some code directly forked under a BSD license from the Go maintainers.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjustintimperio%2Fgpq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjustintimperio%2Fgpq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjustintimperio%2Fgpq/lists"}