{"id":18655446,"url":"https://github.com/kevwan/mapreduce","last_synced_at":"2025-07-06T12:38:14.061Z","repository":{"id":37741888,"uuid":"443314716","full_name":"kevwan/mapreduce","owner":"kevwan","description":"A in-process MapReduce library to help you optimizing service response time or concurrent task processing.","archived":false,"fork":false,"pushed_at":"2024-07-02T01:48:28.000Z","size":46,"stargazers_count":174,"open_issues_count":2,"forks_count":24,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-07-04T16:14:40.245Z","etag":null,"topics":["concurrent","concurrent-programming","go","golang","mapreduce","mapreduce-go"],"latest_commit_sha":null,"homepage":"https://go-zero.dev","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kevwan.png","metadata":{"files":{"readme":"readme-cn.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-12-31T10:03:06.000Z","updated_at":"2025-07-02T02:29:21.000Z","dependencies_parsed_at":"2024-12-13T19:11:57.917Z","dependency_job_id":"1e978a71-fd48-44b5-beef-e09ce6eeeb94","html_url":"https://github.com/kevwan/mapreduce","commit_stats":{"total_commits":28,"total_committers":2,"mean_commits":14.0,"dds":0.0714285714285714,"last_synced_commit":"a6e43ced86582ee96c4df768ad7479956daa40bc"},"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/kevwan/mapreduce","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevwan%2Fmapreduce","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevwan%2Fmapreduce/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevwan%2Fmapreduce/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevwan%2Fmapreduce/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kevwan","download_url":"https://codeload.github.com/kevwan/mapreduce/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevwan%2Fmapreduce/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263901593,"owners_count":23527438,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concurrent","concurrent-programming","go","golang","mapreduce","mapreduce-go"],"created_at":"2024-11-07T07:18:58.610Z","updated_at":"2025-07-06T12:38:14.018Z","avatar_url":"https://github.com/kevwan.png","language":"Go","readme":"# mapreduce\n\n[English](readme.md) | 简体中文\n\n[![Go](https://github.com/kevwan/mapreduce/workflows/Go/badge.svg?branch=main)](https://github.com/kevwan/mapreduce/actions)\n[![codecov](https://codecov.io/gh/kevwan/mapreduce/branch/main/graph/badge.svg)](https://codecov.io/gh/kevwan/mapreduce)\n[![Go Report Card](https://goreportcard.com/badge/github.com/kevwan/mapreduce)](https://goreportcard.com/report/github.com/kevwan/mapreduce)\n[![Release](https://img.shields.io/github/v/release/kevwan/mapreduce.svg?style=flat-square)](https://github.com/kevwan/mapreduce)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n## 为什么会有这个项目\n\n`mapreduce` 其实是 [go-zero](https://github.com/zeromicro/go-zero) 的一部分，但是一些用户问我是不是可以单独使用 `mapreduce` 而不用引入 `go-zero` 的依赖，所以我考虑再三，还是单独提供一个吧。但是，我强烈推荐你使用 `go-zero`，因为 `go-zero` 真的提供了很多很好的功能。\n\n## 为什么需要 MapReduce\n\n在实际的业务场景中我们常常需要从不同的 rpc 服务中获取相应属性来组装成复杂对象。\n\n比如要查询商品详情：\n\n1. 商品服务-查询商品属性\n2. 库存服务-查询库存属性\n3. 价格服务-查询价格属性\n4. 营销服务-查询营销属性\n\n如果是串行调用的话响应时间会随着 rpc 调用次数呈线性增长，所以我们要优化性能一般会将串行改并行。\n\n简单的场景下使用 waitGroup 也能够满足需求，但是如果我们需要对 rpc 调用返回的数据进行校验、数据加工转换、数据汇总呢？继续使用 waitGroup 就有点力不从心了，go 的官方库中并没有这种工具（java 中提供了 CompleteFuture），go-zero 作者依据 mapReduce 架构思想实现了进程内的数据批处理 mapReduce 并发工具类。\n\n## 设计思路\n\n我们尝试梳理一下并发工具可能的示例业务场景：\n\n1. 查询商品详情：支持并发调用多个服务来组合产品属性，支持调用错误可以立即结束。\n2. 商品详情页自动推荐用户卡券：支持并发校验卡券，校验失败自动剔除，返回全部卡券。\n\n以上实际都是在进行对输入数据进行处理最后输出清洗后的数据，针对数据处理有个非常经典的异步模式：生产者消费者模式。于是我们可以抽象一下数据批处理的生命周期，大致可以分为三个阶段：\n\n\u003cimg src=\"https://raw.githubusercontent.com/zeromicro/zero-doc/main/doc/images/mapreduce-serial-cn.png\" width=\"500\"\u003e\n\n1. 数据生产 generate\n2. 数据加工 mapper\n3. 数据聚合 reducer\n\n其中数据生产是不可或缺的阶段，数据加工、数据聚合是可选阶段，数据生产与加工支持并发调用，数据聚合基本属于纯内存操作单协程即可。\n\n再来思考一下不同阶段之间数据应该如何流转，既然不同阶段的数据处理都是由不同 goroutine 执行的，那么很自然的可以考虑采用 channel 来实现 goroutine 之间的通信。\n\n\u003cimg src=\"https://raw.githubusercontent.com/zeromicro/zero-doc/main/doc/images/mapreduce-cn.png\" width=\"500\"\u003e\n\n\n如何实现随时终止流程呢？\n\n`goroutine` 中监听一个全局的结束 `channel` 和调用方提供的 `ctx` 就行。\n\n## 版本选择\n\n- `v1`（默认）- 非泛型版本\n- `v2`（泛型版）- 泛型版本，需要 Go 版本 \u003e= 1.18\n\n## 简单示例\n\n并行求平方和（不要嫌弃示例简单，只是模拟并发）\n\n```go\npackage main\n\nimport (\n    \"fmt\"\n    \"log\"\n\n    \"github.com/kevwan/mapreduce/v2\"\n)\n\nfunc main() {\n    val, err := mapreduce.MapReduce(func(source chan\u003c- int) {\n        // generator\n        for i := 0; i \u003c 10; i++ {\n            source \u003c- i\n        }\n    }, func(i int, writer mapreduce.Writer[int], cancel func(error)) {\n        // mapper\n        writer.Write(i * i)\n    }, func(pipe \u003c-chan int, writer mapreduce.Writer[int], cancel func(error)) {\n        // reducer\n        var sum int\n        for i := range pipe {\n            sum += i\n        }\n        writer.Write(sum)\n    })\n    if err != nil {\n        log.Fatal(err)\n    }\n    fmt.Println(\"result:\", val)\n}\n```\n\n更多示例：[https://github.com/zeromicro/zero-examples/tree/main/mapreduce](https://github.com/zeromicro/zero-examples/tree/main/mapreduce)\n\n## 强烈推荐！\n\ngo-zero: [https://github.com/zeromicro/go-zero](https://github.com/zeromicro/go-zero)\n\n## 欢迎 star！⭐\n\n如果你正在使用或者觉得这个项目对你有帮助，请 **star** 支持，感谢！\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkevwan%2Fmapreduce","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkevwan%2Fmapreduce","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkevwan%2Fmapreduce/lists"}