{"id":36491519,"url":"https://github.com/giant-stone/gmq","last_synced_at":"2026-01-12T01:56:19.632Z","repository":{"id":40521041,"uuid":"507500626","full_name":"giant-stone/gmq","owner":"giant-stone","description":"一个支持自定义消费速率的简单消息队列 Simple, reliable, lightweight and efficient task queue in Go","archived":false,"fork":false,"pushed_at":"2025-08-04T04:15:20.000Z","size":161,"stargazers_count":4,"open_issues_count":7,"forks_count":1,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-08-04T06:40:40.156Z","etag":null,"topics":["crawler","message-queue","redis","task-manager"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/giant-stone.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-06-26T06:56:50.000Z","updated_at":"2025-08-04T03:48:38.000Z","dependencies_parsed_at":"2024-03-16T18:16:11.141Z","dependency_job_id":"08a6a176-bad4-4874-927e-811336c0003d","html_url":"https://github.com/giant-stone/gmq","commit_stats":null,"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"purl":"pkg:github/giant-stone/gmq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/giant-stone%2Fgmq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/giant-stone%2Fgmq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/giant-stone%2Fgmq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/giant-stone%2Fgmq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/giant-stone","download_url":"https://codeload.github.com/giant-stone/gmq/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/giant-stone%2Fgmq/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28331344,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-12T00:36:25.062Z","status":"ssl_error","status_checked_at":"2026-01-12T00:36:15.229Z","response_time":60,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","message-queue","redis","task-manager"],"created_at":"2026-01-12T01:56:19.093Z","updated_at":"2026-01-12T01:56:19.620Z","avatar_url":"https://github.com/giant-stone.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# About\n\ngmq 一个支持自定义消费速率的简单消息队列。\n\n[![Go](https://github.com/giant-stone/gmq/actions/workflows/go.yml/badge.svg)](https://github.com/giant-stone/gmq/actions/workflows/go.yml)\n[![CodeQL](https://github.com/giant-stone/gmq/actions/workflows/codeql-analysis.yml/badge.svg)](https://github.com/giant-stone/gmq/actions/workflows/codeql-analysis.yml)\n[![Go Report Card](https://goreportcard.com/badge/github.com/giant-stone/gmq)](https://goreportcard.com/report/github.com/giant-stone/gmq)\n[![GoDoc](https://godoc.org/github.com/giant-stone/gmq?status.svg)](https://godoc.org/github.com/giant-stone/gmq)\n\n特性\n\n- [x] 处理消息失败默认自动存档\n- [x] 支持暂停队列消费\n- [x] 支持类似 cron 定时任务\n- [x] 命令行队列管理工具\n- [x] 自定义消费速率\n- [x] 自定义时限内消息去重\n- [x] 中间件\n- [x] 测试覆盖核心逻辑\n- [x] 支持业务层通过 [gomock](https://github.com/golang/mock) 实现单元测试\n\n功能和接口设计都参考了 [hibiken/asynq](https://github.com/hibiken/asynq) 实现，差异：\n\n- 默认支持自定义消费速率（自定义从队列取消息的时间间隔）\n  - 所有的消息队列都希望消费节点尽可能快，在某些场景下，我们希望实现自定义间隔消费消息——注意：不是重试，不是预定将来某个准确时刻!\n- 消息自动保证入队某个时间段内唯一，`client.Enqueue(msg, gmq.UniqueIn(time.Hour*12))` 即可——无论消费结果成功还是失败，约束在 UniqueIn 内都有效\n  - asynq 需要同时指定生成 TaskId、Unique、Retention 三个参数，但默认消费成功后 Unique 限制自动删除\n\n和其他消息队列差异：\n\n- 性能和可靠性不是首要考虑，**易用性、实用性、满足真正多样化场景需求放在第一优先考虑**\n\n## Quickstart\n\n安装 Go stable 版本（当前应为 1.17 或以上）后，创建项目目录，通过以下命令安装 gmq 库\n\n```sh\ngo get -u github.com/giant-stone/gmq\n```\n\n实现一个最简单的消息队列生产-消费 demo：\n\n```go\npackage main\n\nimport (\n\t\"context\"\n\t\"flag\"\n\t\"time\"\n\n\t\"github.com/giant-stone/go/glogging\"\n\t\"github.com/giant-stone/go/gutil\"\n\n\t\"github.com/giant-stone/gmq/gmq\"\n)\n\nfunc main() {\n\tconst defaultDsn = \"redis://127.0.0.1:6379/0\"\n\tvar dsn string\n\tflag.StringVar(\u0026dsn, \"d\", defaultDsn, \"data source name of redis\")\n\tflag.Parse()\n\n\t// 初始化 glogging 打印日志，[glogging](https://github.com/giant-stone/go#custom-logging) 将集成格式化、自动切割、日志分级，满足大部分服务 99% 以上场景\n\tglogging.Init([]string{\"stdout\"}, \"debug\")\n\n\tbroker, err := gmq.NewBrokerRedis(dsn, \"\")\n\tgutil.ExitOnErr(err)\n\n\tctx := context.Background()\n\tsrv := gmq.NewServer(ctx, broker, \u0026gmq.Config{Logger: glogging.Sugared})\n\tmux := gmq.NewMux()\n\n\t// 用一个子协程模拟实现消息队列生产者\n\tcli, err := gmq.NewClientRedis(dsn)\n\tgutil.ExitOnErr(err)\n\tgo func() {\n\t\tfor {\n\t\t\tselect {\n\t\t\tcase \u003c-ctx.Done():\n\t\t\t\treturn\n\t\t\tdefault:\n\t\t\t\t{\n\t\t\t\t\t// 如果不指定队列名，gmq 默认使用 gmq.DefaultQueueName\n\t\t\t\t\tcli.Enqueue(ctx, \u0026gmq.Msg{Payload: []byte(`{\"data\":\"hello world\"}`)})\n\t\t\t\t\ttime.Sleep(time.Millisecond * 500)\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}()\n\n\t// 设置消息消费者，mux 类似于 web 框架中常用的多路复用路由处理，\n\t// 消费消息以队列名为 pattern，handler 为 gmq.HandlerFunc 类型函数\n\tmux.Handle(gmq.DefaultQueueName, gmq.HandlerFunc(func(ctx context.Context, msg gmq.IMsg) (err error) {\n\t\tglogging.Sugared.Debugf(\"consume id=%s queue=%s payload=%s\", msg.GetId(), msg.GetQueue(), string(msg.GetPayload()))\n\t\treturn nil\n\t}))\n\n\tif err := srv.Run(mux); err != nil {\n\t\tglogging.Sugared.Fatal(\"srv.Run \", err)\n\t}\n\tglogging.Sugared.Debug(\"gmq server started\")\n\tselect {}\n}\n\n```\n\n完整代码见 examples/pushmsg/main.go .\n\n如果不加队列配置，默认全速消费，每个队列会生成和机器 CPU (`runtime.NumCPU()`)数量一致 workers\n\n    gmq - queue foo - worker a\n                    - worker b\n                    - ...\n        - queue bar - worker aa\n                    - worker bb\n                    - ...\n\n在某些场合下，我们希望只有一个 worker 并且按一定间隔从队列中消费消息，可以通过队列配置控制：\n\n```go\n\t// ...\n\tfunc workIntervalFunc() time.Duration {\n\t\treturn time.Second * time.Duration(3)\n\t}\n\n\t// ...\n\n\tslowQueueName := \"seckill\"\n\tsrv := gmq.NewServer(ctx, broker, \u0026gmq.Config{Logger: glogging.Sugared,\n\t\tQueueCfgs: map[string]*gmq.QueueCfg{\n\t\t\t// 队列名 - 队列配置\n\t\t\tslowQueueName: gmq.NewQueueCfg(\n\t\t\t\tgmq.OptQueueWorkerNum(1),                    // 配置限制队列只有一个 worker\n\t\t\t\tgmq.OptWorkerWorkInterval(workIntervalFunc), // 配置限制队列消费间隔为每 3 秒从队列取一条消息\n\t\t\t),\n\t\t},\n\t})\n\n\t// ...\n```\n\n完整代码见 examples/queuecfg/main.go .\n\n消费效果输出\n\n```\n2022-07-05T10:56:00.779+0800    DEBUG   workInterval/main.go:65 consume id=a8f39ae3-de80-40a2-bdf4-a6cbbcf7cf59 queue=default payload={\"data\":\"hello world\"}\n2022-07-05T10:56:01.794+0800    DEBUG   workInterval/main.go:65 consume id=077d0595-9e8c-4211-b450-86fd4215fe40 queue=default payload={\"data\":\"hello world\"}\n2022-07-05T10:56:01.797+0800    DEBUG   workInterval/main.go:65 consume id=4c361217-59bc-4058-88f2-6aff96f51373 queue=default payload={\"data\":\"hello world\"}\n2022-07-05T10:56:02.701+0800    DEBUG   workInterval/main.go:70 consume id=116b8e73-9a8c-4832-b14a-9efba742159f queue=seckill payload={\"fulluri\":\"worldgold.xxoo/123\"}\n2022-07-05T10:56:02.813+0800    DEBUG   workInterval/main.go:65 consume id=6cc3331a-f054-4155-8b04-d57922ecb4c9 queue=default payload={\"data\":\"hello world\"}\n2022-07-05T10:56:02.815+0800    DEBUG   workInterval/main.go:65 consume id=0cb698e1-00ed-4353-9e00-a5102fa1f0cc queue=default payload={\"data\":\"hello world\"}\n2022-07-05T10:56:03.828+0800    DEBUG   workInterval/main.go:65 consume id=4e1db153-3177-4ffc-82f1-8f31c0a4dddf queue=default payload={\"data\":\"hello world\"}\n2022-07-05T10:56:03.833+0800    DEBUG   workInterval/main.go:65 consume id=d6cd0836-29df-448b-943a-0082d7bfe7fe queue=default payload={\"data\":\"hello world\"}\n2022-07-05T10:56:04.842+0800    DEBUG   workInterval/main.go:65 consume id=69569019-8448-47ef-9277-83ef85b4ef70 queue=default payload={\"data\":\"hello world\"}\n2022-07-05T10:56:04.844+0800    DEBUG   workInterval/main.go:65 consume id=2bd5db36-c13c-49de-abf7-26e2aad681df queue=default payload={\"data\":\"hello world\"}\n2022-07-05T10:56:05.712+0800    DEBUG   workInterval/main.go:70 consume id=9c1909f0-3f83-438e-8c98-c0dd7d7d1150 queue=seckill payload={\"fulluri\":\"worldgold.xxoo/123\"}\n2022-07-05T10:56:05.852+0800    DEBUG   workInterval/main.go:65 consume id=9f5d7b9f-fbf1-48cf-8ccc-e626f198e72d queue=default payload={\"data\":\"hello world\"}\n2022-07-05T10:56:05.857+0800    DEBUG   workInterval/main.go:65 consume id=ccd2f9ae-4b74-4b92-9321-d65872ae7b36 queue=default payload={\"data\":\"hello world\"}\n// ...\n```\n\ndefault 队列无限制 worker 数量，每 500 毫秒 生产 1 条，消费无间隔，所以每秒消费 2 条 ~= 500 毫秒/条；\nseckill 队列限制 worker 一个，每 500 毫秒 生产 1 条，消费间隔 3 秒一条，所以每秒消费 0.33/条 ~= 3 秒/条，和期望一致。\n\n## Advanced Topics\n\n更多功能见 [giant-stone/gmq/wiki](https://github.com/giant-stone/gmq/wiki)\n\n## Run tests\n\nrun tests on Windows\n\n```\nset GMQ_RDS=\"redis://localhost:6379/14?dial_timeout=1s\u0026read_timeout=1s\u0026max_retries=1\"\ngo test -v ./...\n```\n\nrun tests on macOS/Linux\n\n```\nexport GMQ_RDS=\"redis://localhost:6379/14?dial_timeout=1s\u0026read_timeout=1s\u0026max_retries=1\"\ngo test -v ./...\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgiant-stone%2Fgmq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgiant-stone%2Fgmq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgiant-stone%2Fgmq/lists"}