{"id":13459886,"url":"https://github.com/didi/sharingan","last_synced_at":"2025-05-15T07:07:32.497Z","repository":{"id":40239540,"uuid":"255014912","full_name":"didi/sharingan","owner":"didi","description":"Sharingan（写轮眼）是一个基于golang的流量录制回放工具，适合项目重构、回归测试等。","archived":false,"fork":false,"pushed_at":"2023-09-29T04:22:23.000Z","size":34581,"stargazers_count":2344,"open_issues_count":10,"forks_count":348,"subscribers_count":43,"default_branch":"master","last_synced_at":"2025-04-23T09:41:55.614Z","etag":null,"topics":["golang","mock-server","recorder","replayer","smoke-test","testing-tools"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/didi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-04-12T05:24:55.000Z","updated_at":"2025-04-14T08:32:26.000Z","dependencies_parsed_at":"2024-01-29T10:10:12.908Z","dependency_job_id":"2f4ede21-5c3f-499e-8450-81aaa7eeeb93","html_url":"https://github.com/didi/sharingan","commit_stats":{"total_commits":110,"total_committers":11,"mean_commits":10.0,"dds":0.6636363636363636,"last_synced_commit":"3abe16b4513398a86500e2ebd5551437a628636f"},"previous_names":["didichuxing/sharingan"],"tags_count":23,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/didi%2Fsharingan","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/didi%2Fsharingan/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/didi%2Fsharingan/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/didi%2Fsharingan/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/didi","download_url":"https://codeload.github.com/didi/sharingan/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254292043,"owners_count":22046426,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","mock-server","recorder","replayer","smoke-test","testing-tools"],"created_at":"2024-07-31T10:00:31.916Z","updated_at":"2025-05-15T07:07:27.487Z","avatar_url":"https://github.com/didi.png","language":"Go","readme":"# Sharingan\n\n[![Build Status](https://www.travis-ci.org/didi/sharingan.svg?branch=master)](https://www.travis-ci.org/didi/sharingan)\n[![Gitter](https://badges.gitter.im/didi-sharingan/community.svg)](https://gitter.im/didi-sharingan/community?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge)\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/didi/sharingan/blob/master/LICENSE)\n[![GoDoc](https://godoc.org/github.com/didi/sharingan?status.svg)](https://godoc.org/github.com/didi/sharingan)\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg alt=\"Sharingan\" src=\"https://dpubstatic.udache.com/static/dpubimg/614bd23f-b9dc-4225-9e39-76373edc35cf.png\" width=\"150\"\u003e\n\u003c/div\u003e\n\nSharingan，中文名：写轮眼，是漫画《火影忍者》中的一种瞳术，具有复制、幻术等能力，在幻术世界里，一切因素包括时间，地点，质量等，都由施术者掌控。\n\n## 一、简介\n\nSharingan是一个基于golang的流量录制回放工具，录制线上真实请求流量进行回放测试，适合项目重构、回归测试等。\n\n### 1.1、背景\n\n随着微服务架构的兴起，服务之间的依赖关系变的越来越复杂，软件测试也面临新的挑战：系统升级频繁、服务依赖众多等等。\n\n* 常见的测试方案（如：单元测试、系统测试等）构造和维护测试用例成本高，特别是业务复杂的场景。「构造测试数据**难**」\n* 依赖第三方服务众多，线下测试环境不太稳定，经常出现下游服务不可用导致测试失败的情况发生。「维护测试环境成本**高**」\n\n为此，我们需要开发一套工具来缓解上述问题。\n\n### 1.2、方案\n\n* 录制线上服务真实请求流量（包括下游调用流量），在线下进行回放，解决构造测试数据难的问题。「复制能力」\n* 回放的时候匹配Mock下游调用，不再依赖具体的下游服务，解决维护测试环境成本高的问题。「幻术能力」\n\n### 1.3、特性\n\n* 支持下游流量录制。相比[tcpcopy](https://github.com/session-replay-tools/tcpcopy)、[goreplay](https://github.com/buger/goreplay)等方案，回放不依赖下游环境。\n* 支持并发流量录制和回放。录制对服务影响小，回放速度更快。\n* 支持时间重置、噪音去除、批量回放、覆盖率报告、常见协议解析等等。\n* 支持写流量回放，不会污染应用数据。\n* 不依赖业务框架，低应用浸入。\n\n## 二、快速开始\n\n### 2.1、使用示例\n\n```shell\n# Step1: 下载sharingan项目\n$ git clone https://github.com/didi/sharingan.git\n$ cd sharingan\n\n# Step2: 使用定制版golang，以go1.13为例「慢？科学上网试试」\n$ sudo sh install.sh go1.13 # 支持go1.10 ~ go1.15，限mac、linux、amd64「执行出错？请查看底部常见问题」\n$ export GOROOT=/tmp/recorder-go1.13\n$ export PATH=$GOROOT/bin:$PATH\n\n# Step3: 编译、后台启动replayer-agent「默认会占用3515、8998端口，可修改」\n# [回放接入文档]内有直接下载bin文件的链接，无需build\n$ cd replayer-agent\n$ go build\n$ nohup ./replayer-agent \u003e\u003e run.log 2\u003e\u00261 \u0026\n\n# Step4: 编译、后台启动example示例「默认会占用9999端口，可修改」\n$ cd ../example\n$ go build -tags=\"replayer\" -gcflags=\"all=-N -l\"\n$ nohup ./example \u003e\u003e run.log 2\u003e\u00261 \u0026\n\n# Step5: 打开回放页面\n$ 浏览器打开，http://127.0.0.1:8998 # 非本机替换IP即可\n$ 页面选择要回放的流量点执行          # 内置提前录制好的3条example示例流量\n```\n\n### 2.2、接入文档\n\n* [录制接入文档](./doc/recorder/README.md) 「先录制」\n* [回放接入文档](./doc/replayer/README.md) 「后回放」\n\n## 三、技术方案\n\n### 3.1、模块划分\n\n* recorder: 流量录制模块，录制流量本地文件存储、发送流量到录制agent等。\n* recorder-agent：流量录制agent，单独进程启动，控制录制比例、流量存储等。\n* replayer: 流量回放模块，重定向连接到Mock Server、Mock时间、添加流量标识等。\n* replayer-agent：流量回放agent，单独进程启动，查询流量、查询/上报噪音、流量diff、批量回放、生成覆盖率报告等。\n\n### 3.2、整体架构图\n\n![架构图](http://img-hxy021.didistatic.com/static/sharingan/arch_v2.png)\n\n### 3.3、录制方案\n\n* 修改golang源码，对外暴露Hook接口。「所有改动通过官方测试用例」\n* 提供API串联不同goroutine之间的网络调用。「常见的http、mysql、redis流量都不需要特别设置」\n* 提供单独的agent筛选流量、控制比例。\n* 更多参考：[流量录制实现原理](https://github.com/didi/sharingan/wiki/%E6%B5%81%E9%87%8F%E5%BD%95%E5%88%B6%E5%AE%9E%E7%8E%B0%E5%8E%9F%E7%90%86)。\n\n### 3.4、回放方案\n\n* 连接重定向：将服务所有Connect网络调用重定向到Mock Server。「安装replayer-agent时候自带」\n* 流量匹配：Mock Server会根据服务真实的下游请求匹配一个返回。「mock下游调用」\n* 时间重置：将程序执行时间回退到录制执行时刻，尽量避免时间因素带来的干扰。\n* 噪音去除：提供API可以将已知的噪音流量去掉，如：traceID，每次请求本来就不一样。\n* 常见协议解析：会解析http、mysql、redis、thrift等协议，方便diff对比。\n* 更多参考：[流量回放实现原理](https://github.com/didi/sharingan/wiki/%E6%B5%81%E9%87%8F%E5%9B%9E%E6%94%BE%E5%AE%9E%E7%8E%B0%E5%8E%9F%E7%90%86)。\n\n## 四、演进之路\n\n关于流量录制和回放，在内部进行过多次探索，主要经历下面三个阶段：\n\n### 4.1、月光宝盒（串行录制、串行回放）\n\n录制：利用tcpdump录制流量，改造router层将请求串行化，利用时间来分割请求。\n\n回放：利用iptables转发流量到mock服务，匹配请求并mock返回。支持时间重置、流量对比等。\n\n不足：录制流量覆盖率低，一次只能录制一个请求。iptables转发，噪音干扰严重。\n\n### 4.2、Fastdev（并行录制、串行回放）\n\n录制：改造golang源码，利用goroutine+工作委托技术串联区分请求。[链路追踪原理](https://github.com/didi/sharingan/wiki/%E9%93%BE%E8%B7%AF%E8%BF%BD%E8%B8%AA)\n\n回放：利用mock库重定向connect系统调用，转发流量到mock服务。支持Dashboard、噪音去除等。[monkey mock原理](https://bou.ke/blog/monkey-patching-in-go/)\n\n不足：录制接口和实现混合，golang多版本支持困难。不支持并发回放，启动阶段流量无法代理会失败，定时任务流量干扰严重。\n\n### 4.3、Sharingan（并行录制、并行回放）\n\n录制：接口和实现分离，golang源码改造部分只暴露接口，具体录制实现单独提供包支持。确保golang源码改动通过官方测试，支持1.10~1.14所有版本；优化录制服务性能。\n\n回放：添加流量标识，支持并发回放；支持启动阶段流量代理；利用定制版golang，消除定时任务流量干扰；时间重置不再依赖本地文件，支持replayer-agent单独部署；支持常见协议解析。\n\n## 五、效果展示\n\n### 5.1、流量回放\n\n#### 5.1.1、单个回放\n\n![单个回放](http://img-hxy021.didistatic.com/static/sharingan/replayer_s_v2.png)\n\n对于下游请求很多且复杂的情况，支持对下游协议进行筛选\n![单个回放-协议刷选](http://img-hxy021.didistatic.com/static/sharingan/protocol_v2.png)\n\n#### 5.1.2、批量回放\n\n批量回放的并发度默认是10，可通过增加[-parallel](https://github.com/didi/sharingan/blob/master/replayer-agent/control.sh#L160)参数修改。\n![批量回放](http://img-hxy021.didistatic.com/static/sharingan/replayer_p_v2.png)\n\n### 5.2、覆盖率报告\n\n#### 5.2.1、整体报告\n\n覆盖率报告支持覆盖率结果累计，即支持 多次 单个回放和批量回放后，统一生成覆盖率结果。\n![整体报告](http://img-hxy021.didistatic.com/static/sharingan/codeCover_v2.png)\n\n#### 5.2.1、覆盖详情\n\n![覆盖详情](http://img-hxy021.didistatic.com/static/sharingan/codeCover_detail_v2.png)\n\n## 六、更多\n\n### 6.1、如何贡献\n\n欢迎大家参与进来，更多参考[Contribute](./CONTRIBUTING.md)。\n\n### 6.2、许可\n\n基于Apache-2.0协议进行分发和使用，更多参考[LICENSE](./LICENSE)。\n\n### 6.3、成员\n\n[hueng](https://github.com/hueng)、[yj20060714](https://github.com/yj20060714)、[qiaodandedidi](https://github.com/qiaodandedidi)、[bikong0411](https://github.com/bikong0411)、[plpan](https://github.com/plpan)、[fzl-yty](https://github.com/fzl-yty)。\n\n### 6.4、感谢\n\n特别感谢[TaoWen](https://github.com/taowen) ，流量录制和回放初版设计者，为后续开源奠定了很好的基础。\n\n### 6.5、联系我们\n\n* 微信交流群 【加管理员微信，拉进交流群】\n\n![WEIXIN](http://img-ys011.didistatic.com/static/dhack_wakeup/weixin_sharingan_v3.jpg)\n\n## 七、常见问题\n\n- 执行 `$ sudo sh install.sh go1.13` 提示 `wget: command not found...`\n\n  - 使用 [brew](http://brew.sh/)\n  ```shell\n  # 首先安装 brew\n  $ ruby -e \"$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)\"\n  # 然后用 `brew` 安装 `wget`：\n  $ brew install wget\n  ```\n  - 使用 [MacPorts](https://www.macports.org/install.php)\n  ```shell\n  # 下载并运行上述连接中安装包后\n  $ sudo port install wget\n  ```","funding_links":[],"categories":["Go","开源类库","Open source library","语言资源库"],"sub_categories":["网络","The Internet","go"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdidi%2Fsharingan","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdidi%2Fsharingan","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdidi%2Fsharingan/lists"}