{"id":42177663,"url":"https://github.com/mangenotwork/gathertool","last_synced_at":"2026-01-26T21:55:27.025Z","repository":{"id":41268613,"uuid":"360864492","full_name":"mangenotwork/gathertool","owner":"mangenotwork","description":"gathertool是golang脚本化开发库，目的是提高对应场景程序开发的效率；轻量级爬虫库，接口测试\u0026压力测试库，DB操作库等。","archived":false,"fork":false,"pushed_at":"2025-03-11T09:38:00.000Z","size":8466,"stargazers_count":53,"open_issues_count":0,"forks_count":13,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-11T10:39:22.993Z","etag":null,"topics":["crawl","gathertool","golang","grpc-go","proxy-server","spider","spider-framework"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mangenotwork.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-23T11:42:11.000Z","updated_at":"2025-03-11T09:38:04.000Z","dependencies_parsed_at":"2023-01-29T13:33:27.405Z","dependency_job_id":"752b082a-82a8-418b-9ac7-83545f77fc52","html_url":"https://github.com/mangenotwork/gathertool","commit_stats":null,"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"purl":"pkg:github/mangenotwork/gathertool","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mangenotwork%2Fgathertool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mangenotwork%2Fgathertool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mangenotwork%2Fgathertool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mangenotwork%2Fgathertool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mangenotwork","download_url":"https://codeload.github.com/mangenotwork/gathertool/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mangenotwork%2Fgathertool/sbom","scorecard":{"id":614962,"data":{"date":"2025-08-11","repo":{"name":"github.com/mangenotwork/gathertool","commit":"4cfa7b8a0acb0a8c06f9de1f53b858df39b1983c"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":1.7,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Maintained","score":0,"reason":"1 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'main'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":0,"reason":"10 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GO-2022-0635","Warn: Project is vulnerable to: GO-2022-0646","Warn: Project is vulnerable to: GO-2020-0017 / GHSA-w73w-5m7g-f7qc","Warn: Project is vulnerable to: GO-2023-2402 / GHSA-45x7-px36-x8w8","Warn: Project is vulnerable to: GO-2024-3321 / GHSA-v778-237x-gjrc","Warn: Project is vulnerable to: GO-2025-3487 / GHSA-hcg3-q754-cr77","Warn: Project is vulnerable to: GO-2024-2687 / GHSA-4v7x-pqxf-cx7m","Warn: Project is vulnerable to: GO-2024-3333","Warn: Project is vulnerable to: GO-2025-3503 / GHSA-qxp5-gwg8-xv66","Warn: Project is vulnerable to: GO-2025-3595 / GHSA-vvgc-356p-c3xw"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-21T03:39:48.235Z","repository_id":41268613,"created_at":"2025-08-21T03:39:48.235Z","updated_at":"2025-08-21T03:39:48.235Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28789405,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-26T21:49:50.245Z","status":"ssl_error","status_checked_at":"2026-01-26T21:48:29.455Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawl","gathertool","golang","grpc-go","proxy-server","spider","spider-framework"],"created_at":"2026-01-26T21:55:26.895Z","updated_at":"2026-01-26T21:55:27.010Z","avatar_url":"https://github.com/mangenotwork.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"## 简介\n- gathertool是golang脚本化开发集成库，目的是提高爬虫，数据处理，接口测试，模拟请求等对应场景脚本程序开发的效率；\n- gathertool也是一款轻量级爬虫库，特色是抽象了请求事件，通俗点理解就是对请求过程状态进行事件处理。\n- gathertool也是接口测试\u0026压力测试库，在接口测试脚本开发上有明显的效率优势，\n- gathertool还集成了对三方中间件的操作，DB操作等。\n\n---\n\n## 使用\n- go get github.com/mangenotwork/gathertool\n\n---\n\n## 介绍\n gathertool是一个高度封装工具库，包含了http/s的请求，Mysql数据库方法，数据类型处理方法，数据提取方法，websocket相关方法，\nTCP|UDP相关方法，NoSql相关方法，开发常用方法等;  可以用于爬虫程序，接口\u0026压力测试程序，常见网络协议调试程序，数据提取与存储程序等； \n\n- gathertool的请求特点: 会在请求阶段执行各个事件如请求失败后的重试事件,请求前后的事件，请求成功事件等等, 可以根据请求状态码自定义这些事件；\n\n- gathertool还拥有很好的可扩展性， 适配传入任意自定义http请求对象， 能适配各种代理对象等等；\n\n- gathertool还拥有抓取数据存储功能, 存储到mysql；配合 https://github.com/mangenotwork/dbHelper 可存储到更多的数据存储上; 还有很多创新的地方文档会根据具体方法进行介绍；\n\n- gathertool还封装了消息队列接口，支持Nsq,Kafka,rabbitmq,redis等消息队列中间件\n\n## 文档： \n[开发文档](https://github.com/mangenotwork/gathertool/blob/main/_doc/develop.md)\n\n[pkg.go.dev](https://pkg.go.dev/github.com/mangenotwork/gathertool)\n\n\n## 开始使用\n\u003e go get github.com/mangenotwork/gathertool\n\n\n#### 简单的get请求\n```go\nimport gt \"github.com/mangenotwork/gathertool\"\n\nfunc main(){\n    ctx, err := gt.Get(\"https://www.baidu.com\")\n    if err != nil {\n        log.Println(err)\n    }\n    log.Println(ctx.RespBodyString)\n}\n```\n\n#### 含请求事件请求\n\n- StartFunc 开始事件，请求开始前执行的方法类型\n\n- SucceedFunc 成功事件，请求成功后执行的方法类型\n\n- FailedFunc 失败事件，请求失败执行的方法类型\n\n- RetryFunc 重试事件，重试请求前执行的方法类型，是否重试来自上次请求的状态码来确定，见StatusCodeMap;\n\n- EndFunc 结束事件，请求流程结束后执行的方法类型\n\n```go\nimport gt \"github.com/mangenotwork/gathertool\"\n\nfunc main(){\n\n    gt.NewGet(`http://192.168.0.1`).SetStartFunc(func(ctx *gt.Context){\n            log.Println(\"请求前： 添加代理等等操作\")\n            ctx.Client.Transport = \u0026http.Transport{Proxy: http.ProxyURL(proxyUrl)}\n        }\n    ).SetSucceedFunc(func(ctx *gt.Context){\n            log.Println(\"请求成功： 处理数据或存储等等\")\n            log.Println(ctx.RespBodyString())\n        }\n    ).SetFailedFunc(func(ctx *gt.Context){\n            log.Println(\"请求失败： 记录失败等等\")\n            log.Println(ctx.Err)\n        }\n    ).SetRetryFunc(func(ctx *gt.Context){\n             log.Println(\"请求重试： 更换代理或添加等待时间等等\")\n             ctx.Client.Transport = \u0026http.Transport{Proxy: http.ProxyURL(proxyUrl)}\n        }\n    ).SetEndFunc(func(ctx *gt.Context){\n             log.Println(\"请求结束： 记录结束，处理数据等等\")\n             log.Println(ctx)\n        }\n    ).Do()\n    \n}\n```\n\n#### 事件方法复用\n```go\nfunc main(){\n    gt.NewGet(`http://192.168.0.1`).SetSucceedFunc(succeed).SetFailedFunc(failed).SetRetryFunc(retry).Do()\n    gt.NewGet(`http://www.baidu.com`).SetSucceedFunc(baiduSucceed).SetFailedFunc(failed).SetRetryFunc(retry).Do()\n}\n// 请求成功： 处理数据或存储等等\nfunc succeed(ctx *gt.Context){\n    log.Println(ctx.RespBodyString())\n}\n// 请求失败： 记录失败等等\nfunc failed(ctx *gt.Context){\n    log.Println(ctx.Err)\n}\n// 请求重试： 更换代理或添加等待时间等等\nfunc retry(ctx *gt.Context){\n    ctx.Client.Transport = \u0026http.Transport{Proxy: http.ProxyURL(proxyUrl)}\n}\n// 百度请求成功后\nfunc baiduSucceed(ctx *gt.Context){\n    log.Println(ctx.RespBodyString())\n}\n```\n\n#### post请求\n```go\n    // FormData\n    postData := gt.FormData{\n        \"aa\":\"aa\",\t\n    }\n    \n    // header\n    header := gt.NewHeader(map[string]string{\n        \"Accept\":\"*/*\",\n        \"X-MicrosoftAjax\":\"Delta=true\",\n        \"Accept-Encoding\":\"gzip, deflate\",\n        \"XMLHttpRequest\":\"XMLHttpRequest\",\n        \"Content-Type\":\"application/x-www-form-urlencoded; charset=UTF-8\",\n    })\n    \n    // cookie\n    cookie := gt.Cookie{\n        \"aa\":\"a\"\n    }\n    \n    // 随机休眠 2~6秒\n    sleep := gt.SetSleep(2,6)\n    c := gt.NewPostForm(caseUrl, postData, header, cookie, sleep)\n    c.Do()\n    html := c.RespBodyString()\n    log.Print(html)\n\n```\n\n#### 数据存储到mysql\n```go\nvar (\n    host   = \"192.168.0.100\"\n    port      = 3306\n    user      = \"root\"\n    password  = \"root123\"\n    dbName  = \"dbName\"\n    db,_ = gt.NewMysql(host, port, user, password, dbName)\n)\n\n//.... 执行抓取\ndata1 := \"data1\"\ndata2 := \"data2\"\n\ninputdata := map[string]interface{} {\n    \"data1\" : data1,\n    \"data2\" : data2,\n}\n\ntableName := \"data\"\ndb.Spider2022DB.InsertAt(tableName, inputdata)\n```\n\n#### HTML数据提取\n```go\nfunc main(){\n\tdate := \"2022-07-05\"\n\tcaseUrl := \"***\"\n\tctx, _ := gt.Get(fmt.Sprintf(caseUrl, date))\n    datas, _ := gt.GetPointHTML(ctx.Html, \"div\", \"id\", \"domestic\")\n\tData(datas, date, \"内期表\", \"备注：内期表=国内期货主力合约表\")\n\tdatas, _ = gt.GetPointHTML(ctx.Html, \"div\", \"id\", \"overseas\")\n\tData(datas, date, \"外期表\", \"备注：外期表=国外期货主力合约表\")\n}\n\nfunc Data(datas []string, date, typeName, note string) {\n\tfor _, data := range datas {\n\t\ttable, _ := gt.GetPointHTML(data, \"table\", \"id\", \"fdata\")\n\t\tif len(table) \u003e 0 {\n\t\t\ttrList := gt.RegHtmlTr(table[0])\n\t\t\tjys := \"\"\n\t\t\tfor _, tr := range trList {\n\t\t\t\ttd := gt.RegHtmlTd(tr)\n\t\t\t\tlog.Println(\"td = \", td, len(td))\n\t\t\t\tif len(td) == 1 {\n\t\t\t\t\tjys = gt.RegHtmlTdTxt(td[0])[0]\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\t\t\t\tname := gt.RegHtmlTdTxt(td[0])[0]\n\t\t\t\tif strings.Index(name, \"商品名称\") != -1 {\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\t\t\t\tzlhy := gt.RegHtmlTdTxt(td[1])[0]\n\t\t\t\tjsj := gt.RegHtmlTdTxt(td[2])[0]\n\t\t\t\tzd := gt.RegDelHtml(gt.RegHtmlTdTxt(td[3])[0])\n\t\t\t\tcjj := gt.RegHtmlTdTxt(td[4])[0]\n\t\t\t\tccl := gt.RegHtmlTdTxt(td[5])[0]\n\t\t\t\tdw := gt.RegHtmlTdTxt(td[6])[0]\n\t\t\t\tlog.Println(\"日期 = \", date)\n\t\t\t\tlog.Println(\"机构 = \", jys)\n\t\t\t\tlog.Println(\"商品名称 = \", name)\n\t\t\t\tlog.Println(\"主力合约 = \", zlhy)\n\t\t\t\tlog.Println(\"结算价 = \", jsj)\n\t\t\t\tlog.Println(\"涨跌 = \", zd)\n\t\t\t\tlog.Println(\"成交量 = \", cjj)\n\t\t\t\tlog.Println(\"持仓量 = \", ccl)\n\t\t\t\tlog.Println(\"单位 = \", dw)\n\t\t\t}\n\t\t}\n\t}\n}\n```\n\n#### Json数据提取\n```go\nfunc main(){\n\ttxt := `{\n    \"reason\":\"查询成功!\",\n    \"result\":{\n        \"city\":\"苏州\",\n        \"realtime\":{\n            \"temperature\":\"17\",\n            \"humidity\":\"69\",\n            \"info\":\"阴\",\n            \"wid\":\"02\",\n            \"direct\":\"东风\",\n            \"power\":\"2级\",\n            \"aqi\":\"30\"\n        },\n        \"future\":[\n            {\n                \"date\":\"2021-10-25\",\n                \"temperature\":\"12\\/21℃\",\n                \"weather\":\"多云\",\n                \"wid\":{\n                    \"day\":\"01\",\n                    \"night\":\"01\"\n                },\n                \"direct\":\"东风\"\n            },\n            {\n                \"date\":\"2021-10-26\",\n                \"temperature\":\"13\\/21℃\",\n                \"weather\":\"多云\",\n                \"wid\":{\n                    \"day\":\"01\",\n                    \"night\":\"01\"\n                },\n                \"direct\":\"东风转东北风\"\n            },\n            {\n                \"date\":\"2021-10-27\",\n                \"temperature\":\"13\\/22℃\",\n                \"weather\":\"多云\",\n                \"wid\":{\n                    \"day\":\"01\",\n                    \"night\":\"01\"\n                },\n                \"direct\":\"东北风\"\n            }\n        ]\n    },\n    \"error_code\":0\n}`\n\n\tjx1 := \"/result/future/[0]/date\"\n\tjx2 := \"/result/future/[0]\"\n\tjx3 := \"/result/future\"\n\n\tlog.Println(gt.JsonFind(txt, jx1))\n\tlog.Println(gt.JsonFind2Json(txt, jx2))\n\tlog.Println(gt.JsonFind2Json(txt, jx3))\n\tlog.Println(gt.JsonFind2Map(txt, jx2))\n\tlog.Println(gt.JsonFind2Arr(txt, jx3))\n\n}\n```\n\n......\n\n## 实例\n-  [Get请求](https://github.com/mangenotwork/gathertool/tree/main/_examples/get)\n-  [阳光高考招生章程抓取](https://github.com/mangenotwork/gathertool/tree/main/_examples/get_yggk)\n-  [ip地址信息抓取](https://github.com/mangenotwork/gathertool/tree/main/_examples/ip_bczs_cn)\n-  [压力测试](https://github.com/mangenotwork/gathertool/tree/main/_examples/stress_testing)\n-  [文件下载](https://github.com/mangenotwork/gathertool/tree/main/_examples/upload_file)\n-  [无登录微博抓取](https://github.com/mangenotwork/gathertool/tree/main/_examples/weibo)\n-  [百度题库抓取](https://github.com/mangenotwork/gathertool/tree/main/_examples/baidu_tk)\n-  [搭建http/s代理与抓包](https://github.com/mangenotwork/gathertool/tree/main/_examples/intercept)\n-  [搭建socket5代理](https://github.com/mangenotwork/gathertool/tree/main/_examples/socket5)\n-  [商品报价信息抓取](https://github.com/mangenotwork/gathertool/tree/main/_examples/baojia)\n-  [期货信息抓取](https://github.com/mangenotwork/gathertool/tree/main/_examples/qihuo)\n-  [行业信息网行业分类](https://github.com/mangenotwork/gathertool/tree/main/_examples/cnlinfo)\n\n## JetBrains 开源证书支持\n\n`gathertool` 项目一直以来都是在 JetBrains 公司旗下的 `GoLand` 集成开发环境中进行开发，基于 **free JetBrains Open Source license(s)** 正版免费授权，在此表达我的谢意。\n\n\u003ca href=\"https://www.jetbrains.com/?from=gathertool\" target=\"_blank\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/moonD4rk/staticfiles/master/picture/jetbrains-variant-4.png\" width=\"256\" align=\"middle\"/\u003e\u003c/a\u003e\n\n\n## 使用注意\u0026不足之处\n1. Mysql相关方法封装使用的字符串拼接，如抓取无需关心Sql注入这类场景放心使用，生产环境使用会出现安全风险；\n2. 变量类型转换使用反射，性能方面会差一些，追求性能请使用其他；\n\n\n## 经验积累\n从2022年v0.3.4版本到至今已经应用在我的实际工作中，给边边角角的工作带来了巨大的脚步编写效率，经过不断的积累和总结质量也在不断的提示；感兴趣的伙伴可以fork代码，\n将你的宝贵经验提交进来。\n\n## 配合使用\n- 数据存储与读取 https://github.com/mangenotwork/dbHelper\n- 图片处理  https://github.com/mangenotwork/imgHelper\n\n## TODO\n- Redis连接方法改为连接池, 干掉redis队列\n- 关闭重试\n- 提供通过html定位加正则获取内容的方法\n- 引入请求工作与队列的功能，在广度爬取场景使用\n\n## BUG\n- 固定map调用不了  .\\main.go:124:16: undefined: gt.OrderMap\n- panic\n```azure\npanic: runtime error: index out of range [1] with length 1\n\ngoroutine 1 [running]:\ngithub.com/mangenotwork/gathertool.RegFindAllTxt({0xb042f8?, 0x1?}, {0xc00020c000, 0x3fe5})\n        D:/go/pkg/mod/github.com/mangenotwork/gathertool@v0.4.7/reg.go:32 +0x135\n```\n\n\n## 三方引用 感谢这些开源项目\n- github.com/Shopify/sarama \n- github.com/dgrijalva/jwt-go \n- github.com/go-sql-driver/mysql \n- github.com/nsqio/go-nsq \n- github.com/streadway/amqp \n- github.com/xuri/excelize/v2 \n- golang.org/x/crypto \n- golang.org/x/net \n- golang.org/x/text \n- gopkg.in/gomail.v2\n- gopkg.in/yaml.v3\n\n## 里程碑\n\n#### v0.4.8\n```\n1. 升级到go版本1.19并进行优化\n2. 多类型重复的方法改为泛型，减轻代码量  \u003ctodo\u003e\n3. 增加测试代码和整理实例 \u003ctodo\u003e\n4. 优化项目文件结构，规范库的注释\n5. 更新和优化文档  \u003ctodo\u003e\n6. 修复 panic \u003ctodo\u003e\n7. 修复问题: \u003ctodo\u003e\n  - Any2Float64 不支持字符串转float64 \u003ctodo\u003e\n  - 下载的函数名错误 Upload 执行GET下载请求 \u003ctodo\u003e\n  - 缺少url提取文件名，文件后缀的函数 \u003ctodo\u003e\n  - 类型转换缺少空值判断 \u003ctodo\u003e\n8. 新增:  \u003ctodo\u003e\n  - 正则提取增加  section, dl, dt, dd 标签 \u003ctodo\u003e\n  - 去掉 gorm.io/gorm, mongo-driver, github.com/gomodule/redigo\n  - html解析需要引入属性值包含关系 \u003ctodo\u003e\n  - 下载资源需要打印下载url, 以及检查下载大小，如果是0则抛出错误 \u003ctodo\u003e\n```\n\n#### v0.4.7\n```\n1. Redis连接方法改为连接池\n2. 增加关闭重试方法\n3. 增加Whois查询\n4. 测试与优化\n```\n\n#### v0.4.4 ~ v0.4.6\n```\n1. 移除 grpc相关方法\n2. 新增DNS查询\n3. 新增证书信息获取\n4. 新增Url扫描\n5. 新增邮件发送\n6. 优化ICMP Ping的方法\n```\n\n#### v0.4.3\n```\n1. 新增网站链接采集应用场景的方法 \n2. 修复v0.4.2的json导入bug \n3. 修复redis封装方法的bug \n4. 请求日志添加请求地址信息\n5. 优化抓取实例\n```\n\n#### v0.4.2\n```\n新增:\n1. redis, nsq, rabbitmq, kafka 消息队列方法\n2. 新增开发文档\n3. 新增redis相关方法\n```\n\n#### v0.4.1\n```\n新增:\n1. 文件相关处理\n2. 文件压缩解压\n3. 新增抓取实列 _examples/cnlinfo\n```\n\n#### v0.2.1 ~ v0.3.9\n```\n旧的迭代版本，不稳定，请使用 \u003e v0.3.4的版本\n- 新增代理与抓包\n- 启动一个代理服务，并抓包对数据包进行处理\n- 新增socket5代理\n\nv0.3.6 \n新增:\n1. func SimpleServer(addr string, g *grpc.Server) : grpc服务端\n2. func NewTask() *Task : task的实例方法\n3. func (task *Task) SetUrl(urlStr string) *Task \u0026 func (task *Task) SetJsonParam(jsonStr string) *Task\n\n修复:\n1. Context对象，Task为空的问题\n\nv0.3.7\n新增:\n1. 新增html解析，指定html内容提取\n2. 新增抓取实例\n3. 优化部分方法\n\nv0.3.9\n新增:\n1. 新增配置，支持yaml\n2. 优化部分方法\n```\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmangenotwork%2Fgathertool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmangenotwork%2Fgathertool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmangenotwork%2Fgathertool/lists"}