{"id":15631037,"url":"https://github.com/tomarrell/miniqueue","last_synced_at":"2025-08-20T10:31:24.632Z","repository":{"id":37503179,"uuid":"327874458","full_name":"tomarrell/miniqueue","owner":"tomarrell","description":"A simple, single binary, message queue. Supports HTTP/2 and Redis Protocol.","archived":false,"fork":false,"pushed_at":"2023-02-03T16:05:38.000Z","size":12543,"stargazers_count":209,"open_issues_count":3,"forks_count":10,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-12-10T00:42:03.975Z","etag":null,"topics":["broker","golang","hacktoberfest","message","queue"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tomarrell.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-01-08T10:44:32.000Z","updated_at":"2024-11-27T09:24:13.000Z","dependencies_parsed_at":"2023-02-12T12:00:57.844Z","dependency_job_id":null,"html_url":"https://github.com/tomarrell/miniqueue","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomarrell%2Fminiqueue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomarrell%2Fminiqueue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomarrell%2Fminiqueue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomarrell%2Fminiqueue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tomarrell","download_url":"https://codeload.github.com/tomarrell/miniqueue/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230415317,"owners_count":18222158,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["broker","golang","hacktoberfest","message","queue"],"created_at":"2024-10-03T10:38:34.275Z","updated_at":"2024-12-19T10:08:48.928Z","avatar_url":"https://github.com/tomarrell.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# miniqueue\n\n![Tests](https://github.com/tomarrell/miniqueue/workflows/Tests/badge.svg)\n[![Go Report Card](https://goreportcard.com/badge/github.com/tomarrell/miniqueue)](https://goreportcard.com/report/github.com/tomarrell/miniqueue)\n![GitHub release (latest by date)](https://img.shields.io/github/v/release/tomarrell/miniqueue)\n\nA stupid simple, single binary message queue using HTTP/2 or Redis Protocol.\n\nMost messaging workloads don't require enormous amounts of data, endless\nfeatures or infinite scaling. Instead, they'd probably be better off with\nsomething dead simple.\n\nminiqueue is just that. A ridiculously simple, high performance queue. You can\npublish bytes to topics and be sure that your consumers will receive what you\npublished, nothing more.\n\n## Features\n\n- Redis Protocol Support\n- Simple to run\n- Very fast, see [benchmarks](#benchmarks)\n- Not infinitely scalable\n- Multiple topics\n- HTTP/2\n- Publish\n- Subscribe\n- Acknowledgements\n- Persistence\n- Prometheus metrics [WIP]\n\n## API\n\n### Redis\n\nYou can communicate with miniqueue using any major Redis library which supports\ncustom commands. The command set is identical to the HTTP/2 implementation and\nlisted under the [commands](#commands) heading.\n\nExamples of using the Redis interface can be found in the\n[redis_test.go](./redis_test.go) file.\n\n### HTTP/2\n\n- POST `/publish/:topic`, where the body contains the bytes to publish to the topic.\n\n  ```bash\n  curl -X POST https://localhost:8080/publish/foo --data \"helloworld\"\n  ```\n\n- POST `/subscribe/:topic` - streams messages separated by `\\n`\n\n  - `client → server: \"INIT\"`\n  - `server → client: { \"msg\": [base64], \"error\": \"...\", dackCount: 1 }`\n  - `client → server: \"ACK\"`\n\n- DELETE `/:topic` - deletes the given topic, removing all messages. Note, this\n    is an expensive operation for large topics.\n\nYou can also find examples in the [`./examples/`](./examples/) directory.\n\n## Usage\n\nminiqueue runs as a single binary, persisting the messages to the filesystem in\na directory specified by the `-db` flag and exposes an HTTP/2 server on the port\nspecified by the `-port` flag.\n\n**Note:** As the server uses HTTP/2, TLS is required. For testing, you can\ngenerate a certificate using [mkcert](https://github.com/FiloSottile/mkcert) and\nreplace the ones in `./testdata` as these will not be trusted by your client, or\nspecify your own certificate using the `-cert` and `-key` flags.\n\n```bash\nUsage of ./miniqueue:\n  -cert string\n        path to TLS certificate (default \"./testdata/localhost.pem\")\n  -db string\n        path to the db file (default \"./miniqueue\")\n  -human\n        human readable logging output\n  -key string\n        path to TLS key (default \"./testdata/localhost-key.pem\")\n  -level string\n        (disabled|debug|info) (default \"debug\")\n  -period duration\n        period between runs to check and restore delayed messages (default 1s)\n  -port int\n        port used to run the server (default 8080)\n```\n\nOnce running, miniqueue will expose an HTTP/2 server capable of bidirectional\nstreaming between client and server. Subscribers will be delivered incoming\nmessages and can send commands `ACK`, `NACK`, `BACK` [etc](#commands). Upon a\nsubscriber disconnecting, any outstanding messages are automatically `NACK`'ed\nand returned to the front of the queue.\n\nMessages sent to subscribers are JSON encoded, containing additional information\nin some cases to enable certain features. The consumer payload looks like: \n\n```js\n{\n  \"msg\": \"dGVzdA==\", // base64 encoded msg\n  \"dackCount\": 2,    // number of times the msg has been DACK'ed\n}\n```\n\nIn case of an error, the payload will be:\n```js\n{\n  \"error\": \"uh oh, something went wrong\"\n}\n```\n\nTo get you started, here are some common ways to get up and running with `miniqueue`.\n\n##### Start miniqueue with human readable logs\n\n```bash\nλ ./miniqueue -human\n```\n\n##### Start miniqueue with custom TLS certificate\n\n```bash\nλ ./miniqueue -cert ./localhost.pem -key ./localhost-key.pem\n```\n\n##### Start miniqueue on custom port\n\n```bash\nλ ./miniqueue -port 8081\n```\n\n## Docker \n\nAs of `v0.7.0` there are published miniqueue docker images available in the\nDocker hub repository\n[`tomarrell/miniqueue`](https://hub.docker.com/repository/docker/tomarrell/miniqueue).\n\nIt is recommended to use a tagged release build. The tag `latest` tracks the\n`master` branch.\n\nWith the TLS certificate and key in a relative directory `./certs` (can be\ngenerated using [mkcert](https://github.com/FiloSottile/mkcert)).\n\n```bash\n./certs\n├── localhost-key.pem\n└── localhost.pem\n```\n\nYou can execute the following Docker command to run the image.\n\n```bash\n$ docker run \\\n  -v $(pwd)/certs:/etc/miniqueue/certs \\\n  -p 8080:8080 \\\n  tomarrell/miniqueue:v0.7.0 \\\n  -cert /etc/miniqueue/certs/localhost.pem \\\n  -key /etc/miniqueue/certs/localhost-key.pem \\\n  -db /var/lib/miniqueue \\\n  -human\n```\n\n## Examples\n\nTo take a look at some common usage, we have compiled some examples for\nreference in the [`./examples/`](./examples/) directory. Here you will find\ncommon patterns such as:\n\n- [Exponential backoff](./examples/exponential_backoff), `1s → 2s → 4s` etc\n- Failure resistant [workers](./examples/workers)\n- Simple [echo](./examples/echo)\n\n## Commands\n\nA client may send commands to the server over a duplex connection. Commands are\nin the form of a **JSON string** to allow for simple encoding/decoding.\n\nAvailable commands are:\n\n- `\"INIT\"`: Establishes a new consumer on the topic. If you are consuming for\n    the first time, this should be sent along with the request.\n\n- `\"ACK\"`: Acknowledges the current message, popping it from the topic and\n    removing it.\n\n- `\"NACK\"`: Negatively acknowledges the current message, causing it to be\n    returned to the *front* of the queue. If there is a ready consumer waiting\n    for a message, it will immediately be delivered to this consumer. Otherwise\n    it will be delivered as as one becomes available.\n\n- `\"BACK\"`: Negatively acknowledges the current message, causing it to be\n    returned to the *back* of the queue. This will cause it to be processed\n    again after the currently waiting messages.\n\n- `\"DACK [seconds]\"`: Negatively acknowledges the current message, placing it on\n    a delay for a certain number of `seconds`. Once the delay expires, on the\n    next tick given by the `-period` flag, the message will be returned to the\n    front of the queue to be processed as soon as possible.\n\n    DACK'ed messages will contain a `dackCount` key when consumed. This allows\n    for doing exponential backoff for the same message if multiple failures\n    occur.\n\n## Benchmarks\n\nAs miniqueue is still under development, take these benchmarks with a grain of\nsalt. However, for those curious:\n\n**Publish**\n```bash\nλ go-wrk -c 12 -d 10 -M POST -body \"helloworld\" https://localhost:8080/publish/test\nRunning 10s test @ https://localhost:8080/publish/test\n  12 goroutine(s) running concurrently\n142665 requests in 9.919498387s, 7.89MB read\nRequests/sec:           14382.28\nTransfer/sec:           814.62KB\nAvg Req Time:           834.36µs\nFastest Request:        190µs\nSlowest Request:        141.091118ms\nNumber of Errors:       0\n```\n\n**Consume + Ack**\n```bash\nλ ./bench_consume -duration=10s\nconsumed 42982 times in 10s\n4298 (consume+ack)/second\n```\n\nRunning on my MacBook Pro (15-inch, 2019), with a 2.6 GHz 6-Core Intel Core i7\nusing Go `v1.15`.\n\n## Contributing\n\nContributors are more than welcome. Please feel free to open a PR to improve anything you don't like, or would like to add. No PR is too small!\n\n## License\n\nThis project is licensed under the MIT license.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftomarrell%2Fminiqueue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftomarrell%2Fminiqueue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftomarrell%2Fminiqueue/lists"}