{"id":13840765,"url":"https://github.com/boy-hack/goWhatweb","last_synced_at":"2025-07-11T09:33:11.998Z","repository":{"id":45237084,"uuid":"162811962","full_name":"boy-hack/goWhatweb","owner":"boy-hack","description":"[学习GO] go语言写的web指纹识别 - Identify websites by go language","archived":false,"fork":false,"pushed_at":"2021-12-29T06:15:30.000Z","size":5935,"stargazers_count":129,"open_issues_count":0,"forks_count":29,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-08-05T17:25:42.026Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/boy-hack.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-12-22T13:07:46.000Z","updated_at":"2024-07-11T01:41:29.000Z","dependencies_parsed_at":"2022-08-29T00:40:27.739Z","dependency_job_id":null,"html_url":"https://github.com/boy-hack/goWhatweb","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/boy-hack%2FgoWhatweb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/boy-hack%2FgoWhatweb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/boy-hack%2FgoWhatweb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/boy-hack%2FgoWhatweb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/boy-hack","download_url":"https://codeload.github.com/boy-hack/goWhatweb/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225712695,"owners_count":17512460,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-04T17:00:55.289Z","updated_at":"2024-11-21T10:30:43.701Z","avatar_url":"https://github.com/boy-hack.png","language":"Go","funding_links":[],"categories":["Go","Go (531)"],"sub_categories":[],"readme":"# goWhatweb\ngo语言编写的指纹识别，指纹数据1800+，大量指纹跑完需要花费大量时间，用go的目的是尽可能的节省时间(单机上)和尽可能的发挥最大效率。\n## Useage\n- `./goWhatweb test.txt`\n- 目录下`goWhatweb`是在`MAC`下编译好的\n- 但是根据`go`的特性，可以静态编译成二进制运行到任何平台\n## Some thinking\n    - 如何快速识别指纹？\n        1. 为指纹添加更多的筛选条件，如php|asp|aspx类指纹\n        2. 基于决策的指纹识别，每次决策增加一定权值\n这当然会在'一定程度'上提高识别效率，但是却要收集到更多的指纹。对于不想如此大费周章的懒人来说（比如我），最好的优化效率方法就是”爆破“的方式，在提升程序运行的速度上进行改进。对于目前的1800+指纹，我认为单纯的增加筛选条件也\b并不会比”纯爆破“类型强很多。\n\n## Feature\u0026\u0026Checklist\n1. 内存加载指纹,在内存中对每个指纹的命中率进行统计，优先使用命中率高的指纹。（就是对排序数组中的路径hash做一次位置替换）\n\n2. 根据统计，指纹可能达到了1800+，累计访问路径会达到1600,如果访问一个不知名的cms至少会请求1600+次，减少请求\u0026提高效率的方法：\n    1. [x] 判断网站连通性，在请求之前，会访问一次首页，存储时间间隔来用于下一部判断，若时间间隔较长，则取消该次请求，标记该URL失败原因为“请求首页时间间隔过长”\n    2. [x] 优先使用命中率高的指纹，其次是排序(同一个访问地址有不同的cms指纹情况)中较高的指纹，在识别成功后立即关闭其他请求。\n    3. [x] 优先使用head请求，head为200才会进行下一步get请求。\n    4. [ ] 每发包200次会随机延时片刻，在请求首页是否正常，若不正常则延时更大片刻，延时超过1分钟则取消该URL其他请求，标记该URL失败原因为“请求量过大 {}次请求后网页无法访问”\n    5. [x] 将请求UA伪装为随机爬虫\n    6. [x] WAF识别，识别到WAF后，自动减缓发包频率\n    7. [x] 针对单个URL的并发默认会达到7次（同时针对该URL的GET请求7个），当有waf时，这个数值为1\n    8. [x] 单个URL超时次数超过阈值（默认200），则该URL整个响应请求暂停，标记失败原因为\"超时次数过多\"\n3. 上述针对单个URL的指纹识别可能无法发挥机器的最大效率，所以设计上goWhatweb对多个URL同时进行指纹分析，并执行上述规则，每个URL的指纹分析是独立的，互不干扰的。所以，更多的URL会将机器的性能和宽带性能发挥到最大。","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fboy-hack%2FgoWhatweb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fboy-hack%2FgoWhatweb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fboy-hack%2FgoWhatweb/lists"}