{"id":21095109,"url":"https://github.com/dan-and-dna/espresso","last_synced_at":"2026-04-20T03:32:48.977Z","repository":{"id":189500094,"uuid":"680376228","full_name":"DAN-AND-DNA/espresso","owner":"DAN-AND-DNA","description":"简单模块化服务框架 simple modular service framework","archived":false,"fork":false,"pushed_at":"2023-08-20T12:35:50.000Z","size":4489,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-01T03:48:08.540Z","etag":null,"topics":["gin","grpc","http","microservice","modular","mvc"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DAN-AND-DNA.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-08-19T03:44:08.000Z","updated_at":"2023-08-20T12:37:20.000Z","dependencies_parsed_at":"2023-08-20T14:09:57.378Z","dependency_job_id":null,"html_url":"https://github.com/DAN-AND-DNA/espresso","commit_stats":null,"previous_names":["dan-and-dna/espresso"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/DAN-AND-DNA/espresso","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DAN-AND-DNA%2Fespresso","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DAN-AND-DNA%2Fespresso/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DAN-AND-DNA%2Fespresso/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DAN-AND-DNA%2Fespresso/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DAN-AND-DNA","download_url":"https://codeload.github.com/DAN-AND-DNA/espresso/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DAN-AND-DNA%2Fespresso/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32031572,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-20T00:18:06.643Z","status":"online","status_checked_at":"2026-04-20T02:00:06.527Z","response_time":94,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gin","grpc","http","microservice","modular","mvc"],"created_at":"2024-11-19T22:22:42.153Z","updated_at":"2026-04-20T03:32:48.955Z","avatar_url":"https://github.com/DAN-AND-DNA.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 简单模块化服务框架\n\n## 协议生成\n1. 下载 协议，放置pb目录\n2. 创建pb_gen目录\n```bash\nprotoc --go_out=./pb_gen  --go-grpc_out=./pb_gen  pb/*.proto\n```\n\n## 发布\n### windows编linux可执行文件\n1. cd .\\build\\\n2. .\\build-linux.bat\n\n## 教程\n### 创建新模块\n1. .\\build\\build.bat 生成windows版本的工具 \n2. 在modules目录创建一个模块目录，比如example\n3. 执行 .\\code-generator.exe来自动生成example模块代码\n\n### 配置文件\n1. 模块相关的配置： [modules.yaml](__output%2Fconfigs%2Fmodules.yaml)\n2. 词库相关配置：[__output](__output)\n\n### 例子\n1. 简单例子 [echo](examples%2Fecho)\n2. 接口形式的模块: 参考[nlp](modules%2Fnlp)模块（只暴露接口，不对外公开实现细节） \n3. mvc形式模块: 参考[cache](modules%2Fcache)模块（只在model里写业务逻辑，controller作为网络消息处理） \n4. 事件处理模块: 参考[metric](modules%2Fmetric)模块 （controller作为事件处理） \n5. 压力测试：参考[e2etest](cmd%2Fe2etest)\n6. 定时器: [时间格式](https://pkg.go.dev/github.com/robfig/cron/v3)跟crontab一样，例子参考[model.go](modules%2Fanalyzer%2Finternal%2Fmodel.go)\n\n## 简单测试\n1. 启动 [python文本分类服务](http://192.168.4.210/dmm-backend/easy-text-classifier)\n2. 启动本服务(./v2)\n3. 启动 ./e2etest.exe，以并发100，qps5000+的速度请求服务器\n4. 打开 [statsviz](http://127.0.0.1:8079/debug/statsviz/) 查看统计\n![dd](./imgs/screenshot-20230727-201022.png)\n![dd](./imgs/screenshot-20230727-200425.png)\n\n## metric\nTODO\n1. 往prometheus grafana里送数据，可视化\n2. 更多的数据\n打开 http://127.0.0.1:8080/metric\n\n二次开发参考[metric](modules%2Fmetric)\n\n![dd](./imgs/screenshot-20230813-024212.png)\n\n## 测试\nTODO\n## 压力测试\n未优化linux，并发150，qps 25000，发送8655777次请求：  \n![dd](./imgs/202385-224935.jpg)\n\ncpu和内存：  \n![dd](./imgs/202385-224950.jpg)\n\n延迟分布：99.5%的请求延迟在5ms以下：  \n![dd](./imgs/202385-225751.jpg)\n## 单元测试\nTODO\n## 限流、熔断和降级\nTODO  \nqps限制：python服务，qps太高就会出先cpu过高，redis也一样得qps控制  \nredis和python：服务异常得熔断，直接返回，避免本侧的请求堆积，内存涨起来  \n如果本机服务的负载很大：可以考虑降级，直接返回true  \n\n## profile、trace和stat\n性能分析，内存分析好过cpu分析，建议在发布前压力测试时分析，可以在生产环境的机器上进行分析和追踪，如果内存\n普遍都不复用，那么GC的频率和延迟可能都会很高，阻塞要看情况分析，添加pprof的代码参考[network](pkg%2Fnetwork)\n### profile\n打开 http://127.0.0.1:8080/debug/pprof/ 分类代表啥意思，完整的可以看官方文档\n![ddd](./imgs/2023813-30209.jpg)\n### profile 控制台例子\n```bash\n# 内存分配分析\ngo tool pprof --alloc_objects  http://localhost:8080/debug/pprof/heap\n\n## 10s的CPU时间消耗分析\ngo tool pprof http://localhost:8080/debug/pprof/profile?seconds=10\n```\n\n查看消耗前10  \n\n```bash\nflat：当前函数自身消耗  \nflat%：当前函数自身消耗的比例   \nsum%：累计到当前函数的消耗和比例       \ncum：当前函数以及子函数的消耗   \ncum%：当前函数以及子函数的消耗比例\n```\n```bash\n\nPS E:\\github\\go-text-filter-server\u003e go tool pprof --alloc_objects  http://localhost:8080/debug/pprof/heap                       \nFetching profile over HTTP from http://localhost:8080/debug/pprof/heap\nSaved profile in C:\\Users\\24707\\pprof\\pprof.___go_build_go_text_filter_server_cmd_v2.exe.alloc_objects.alloc_space.inuse_objects.inuse_space.012.pb.gz\nFile: ___go_build_go_text_filter_server_cmd_v2.exe\nBuild ID: C:\\Users\\24707\\AppData\\Local\\JetBrains\\GoLand2023.2\\tmp\\GoLand\\___go_build_go_text_filter_server_cmd_v2.exe2023-08-12 21:18:09.0289114 +0800 CST\nType: alloc_objects\nTime: Aug 13, 2023 at 2:48am (CST)\nEntering interactive mode (type \"help\" for commands, \"o\" for options)\n(pprof) top 10\nShowing nodes accounting for 138899709, 55.81% of 248881666 total\nDropped 144 nodes (cum \u003c= 1244408)\nShowing top 10 nodes out of 125\n      flat  flat%   sum%        cum   cum%\n  32870118 13.21% 13.21%   32874303 13.21%  regexp.(*Regexp).replaceAll\n  26689541 10.72% 23.93%   26689541 10.72%  github.com/kamilsk/tracer.(*Trace).Start\n  13943347  5.60% 29.53%   13943347  5.60%  github.com/prometheus/client_golang/prometheus.constrainLabelValues\n  12784611  5.14% 34.67%   12784611  5.14%  net/textproto.readMIMEHeader\n  12488081  5.02% 39.69%   30832902 12.39%  go-text-filter-server/modules/analyzer/internal.(*Model).CleanInvalidCharset\n  11065972  4.45% 44.13%   43940275 17.66%  regexp.(*Regexp).ReplaceAllString\n   9706697  3.90% 48.03%  193189821 77.62%  go-text-filter-server/pkg/network/internal.GinWitheRequestId.func1\n   6777571  2.72% 50.76%   31185327 12.53%  net/http.(*conn).readRequest\n   6443751  2.59% 53.35%    6443751  2.59%  net/http.Header.Clone (inline)\n   6130020  2.46% 55.81%    6130020  2.46%  strings.(*Builder).grow (inline)\n```\n\n查看函数分配详情\n```bash\n(pprof) list CleanInvalidCharset  \nTotal: 248881666\nROUTINE ======================== go-text-filter-server/modules/analyzer/internal.(*Model).CleanInvalidCharset in E:\\github\\go-text-filter-server\\modules\\analyzer\\internal\\model.go\n  12488081   30832902 (flat, cum) 12.39% of Total\n         .          .    136:func (model *Model) CleanInvalidCharset(ctx context.Context, content string, cleanEmoji bool) (string, []string) {\n         .    4290348    137:   defer ctxhelper.FetchTrace(ctx).Start().Stop()\n         .          .    138:\n         .          .    139:   charset := configs.GetCharset()\n         .          .    140:   if charset == nil || len(charset) == 0 {\n         .          .    141:           return content, nil\n         .          .    142:   }\n         .          .    143:\n         .          .    144:   // 全角转半角\n         .          .    145:   oldContent := content\n         .    8297580    146:   content = width.Narrow.String(content)\n         .          .    155:\n         .          .    156:   // 去除不在字符集里的字符\n      6620       6620    157:   runesContent := []rune(content)\n         .          .    158:   var hitCharset []string\n   4109289    4109289    159:   retContent := make([]rune, 0, len(runesContent))\n         .          .    160:   for _, char := range runesContent {\n         .          .    161:           if _, ok := charset[char]; !ok {\n   4613060    4613060    162:                   hitCharset = append(hitCharset, string(char))\n         .          .    163:                   continue\n         .          .    164:           }\n         .          .    165:\n         .          .    166:           retContent = append(retContent, char)\n         .          .    167:   }\n         .          .    168:\n   3759112    3759112    169:   return string(retContent), hitCharset\n         .          .    170:}\n         .          .    171:\n         .          .    172:// AnalyzeCN 分析中文 （比如：你妈）\n         .          .    173:func (model *Model) AnalyzeCN(ctx context.Context, content string, dangerous bool) (bool, string) {\n         .          .    174:   defer ctxhelper.FetchTrace(ctx).Start().Stop()\n\n```\n\n### profile 网页例子\n先安装[graphviz](https://graphviz.org/download/)，在环境变量加上graphviz:  \n![dd](./imgs/2023813-21914.jpg)\n```bash\n# 内存分配分析\ngo tool pprof -http=\"127.0.0.1:8082\" --alloc_objects  http://localhost:8080/debug/pprof/heap\n\n## 10s的CPU时间消耗分析\ngo tool pprof -http=\"127.0.0.1:8082\" \"http://localhost:8080/debug/pprof/profile?seconds=10\"\n```\n调用关系图\n![ddd](./imgs/2023813-22121.jpg)\n\n消耗前30\n![ddd](./imgs/2023813-22419.jpg)\n\n火焰图\n![ddd](./imgs/2023813-22612.jpg)\n\n代码分析\n![ddd](./imgs/2023813-23935.jpg)\n\n### trace\n运行时的事件追踪\n```bash\nPS E:\\github\\go-text-filter-server\u003e curl \"http://localhost:8080/debug/pprof/trace?seconds=5\" -o trace.out\nPS E:\\github\\go-text-filter-server\u003e go tool trace .\\trace.out\n```\n\n![dd](./imgs/2023813-30904.jpg)\n\n## 慢查询和进程内trace\n参考 [model.go](modules%2Fanalyzer%2Finternal%2Fmodel.go), 在函数里加:\n```golang\nfunc (model *Model) F(ctx context.Contextn) bool {\n\t// 添加追踪\n    defer ctxhelper.FetchTrace(ctx).Start().Stop()\n\t\n\t// 其他功能实现\n\t// ......\n}\n```\n\n日志里就会出现，调用栈的耗时:\n```json\n{\"L\":\"WARN\",\"T\":\"2023-07-30 14:42:36.948\",\"C\":\"E:/github/go-text-filter-server/modules/say/internal/controller.go:44\",\"M\":\"slow call\",\"module\":\"analyzer\",\"function\":\"analyze\",\"requestId\":\"48e1749f45e593d7da716fc80dbd9272c8b985b574d865f7243d528c0d27b92c\",\"trace\":\"allocates at call stack: 0, detailed call stack:\\n\\tcall internal.GinWitheRequestId.func1 [48e1749f45e593d7da716fc80dbd9272c8b985b574d865f7243d528c0d27b92c]: 5.043792s, allocates: 0\\n\\tcall internal.(*Model).AllowWhitelist: 0s, allocates: 0\\n\\tcall internal.(*Model).CleanInvalidCharacter: 0s, allocates: 0\\n\\tcall internal.(*Model).AnalyzeByDanger: 0s, allocates: 0\\n\\tcall internal.(*Model).AnalyzeEN: 0s, allocates: 0\\n\\tcall internal.(*Model).AnalyzeCN: 0s, allocates: 0\\n\\tcall internal.(*Model).AnalyzeByNormal: 0s, allocates: 0\\n\\tcall internal.(*Model).AnalyzeEN: 0s, allocates: 0\\n\\tcall internal.(*Model).AnalyzeCN: 0s, allocates: 0\\n\\tcall internal.(*Model).AnalyzeByCacheWithTimeout: 3.0056157s, allocates: 0\\n\\tcall internal.(*Model).AnalyzeByNLPWithTimeout: 2.0378617s, allocates: 0\"}\n```\n```bash\nallocates at call stack: 0, detailed call stack:\n\tcall internal.GinWitheRequestId.func1 [a975f0e155c3643a146b94dd5d387cd73f5aac85d0a5b9db77f4449e77aa0bd8]: 5.0606849s, allocates: 0\n\tcall internal.(*Model).AllowWhitelist: 0s, allocates: 0\n\tcall internal.(*Model).CleanInvalidCharacter: 0s, allocates: 0\n\tcall internal.(*Model).AnalyzeByDanger: 0s, allocates: 0\n\tcall internal.(*Model).AnalyzeEN: 0s, allocates: 0\n\tcall internal.(*Model).AnalyzeCN: 0s, allocates: 0\n\tcall internal.(*Model).AnalyzeByCacheWithTimeout: 3.0070433s, allocates: 0\n\tcall internal.(*Model).AnalyzeByNLPWithTimeout: 2.0532953s, allocates: 0\n\tcall internal.(*Model).AnalyzeByNormal: 0s, allocates: 0\n\tcall internal.(*Model).AnalyzeEN: 0s, allocates: 0\n\tcall internal.(*Model).AnalyzeCN: 0s, allocates: 0\n```\n\n## 分布式trace\nTODO\n\n## 配置加载\nTODO   \n自动轮询远端仓库判断文件是否发生变化，有就加载  \n\n手动加载：  \n    ![dd](./imgs/screenshot-20230807-121530.png)\n\n## 磁盘io阻塞导致慢查询\n生产环境配置改成production，只打印error信息哈，磁盘对于系统调用来说总是就绪，所以就算磁盘请求排队了也是一查就是就绪，go会在这种状态下生成很多线程，后续的请求不再卡在上面，但是这个请求就会被挂起好久  \n生产环境走kafka，不打本地日志  \n\n\n## 日志\n自动切割，zap的error等级的日志会输出堆栈，所以可以在关键位置输出堆栈\n\n## 告警和日志扫描\nTODO prometheus grafana zap\n\n\n## 文档\nTODO swagger页面接口\nTODO godoc","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdan-and-dna%2Fespresso","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdan-and-dna%2Fespresso","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdan-and-dna%2Fespresso/lists"}