{"id":13416743,"url":"https://github.com/andeya/pholcus","last_synced_at":"2025-05-14T05:10:23.885Z","repository":{"id":34554585,"uuid":"38499784","full_name":"andeya/pholcus","owner":"andeya","description":"Pholcus is a distributed high-concurrency crawler software written in pure golang","archived":false,"fork":false,"pushed_at":"2022-11-08T06:29:38.000Z","size":24694,"stargazers_count":7584,"open_issues_count":7,"forks_count":1695,"subscribers_count":454,"default_branch":"master","last_synced_at":"2025-04-13T14:59:18.761Z","etag":null,"topics":["crowler","spider"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/andeya.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-07-03T15:40:41.000Z","updated_at":"2025-04-12T20:10:50.000Z","dependencies_parsed_at":"2023-01-15T07:45:56.615Z","dependency_job_id":null,"html_url":"https://github.com/andeya/pholcus","commit_stats":null,"previous_names":["henrylee2cn/pholcus"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andeya%2Fpholcus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andeya%2Fpholcus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andeya%2Fpholcus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andeya%2Fpholcus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/andeya","download_url":"https://codeload.github.com/andeya/pholcus/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254076849,"owners_count":22010611,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crowler","spider"],"created_at":"2024-07-30T22:00:21.187Z","updated_at":"2025-05-14T05:10:23.852Z","avatar_url":"https://github.com/andeya.png","language":"Go","funding_links":[],"categories":["Popular","Go"],"sub_categories":[],"readme":"# Pholcus [![GitHub release](https://img.shields.io/github/release/henrylee2cn/pholcus.svg?style=flat-square)](https://github.com/henrylee2cn/pholcus/releases) [![report card](https://goreportcard.com/badge/github.com/henrylee2cn/pholcus?style=flat-square)](http://goreportcard.com/report/henrylee2cn/pholcus) [![github issues](https://img.shields.io/github/issues/henrylee2cn/pholcus.svg?style=flat-square)](https://github.com/henrylee2cn/pholcus/issues?q=is%3Aopen+is%3Aissue) [![github closed issues](https://img.shields.io/github/issues-closed-raw/henrylee2cn/pholcus.svg?style=flat-square)](https://github.com/henrylee2cn/pholcus/issues?q=is%3Aissue+is%3Aclosed) [![GoDoc](https://img.shields.io/badge/godoc-reference-blue.svg?style=flat-square)](http://godoc.org/github.com/henrylee2cn/pholcus)\r\n\r\nPholcus（幽灵蛛）是一款纯 Go 语言编写的支持分布式的高并发爬虫软件，仅用于编程学习与研究。\r\n\r\n它支持单机、服务端、客户端三种运行模式，拥有Web、GUI、命令行三种操作界面；规则简单灵活、批量任务并发、输出方式丰富（mysql/mongodb/kafka/csv/excel等）；另外它还支持横纵向两种抓取模式，支持模拟登录和任务暂停、取消等一系列高级功能。\r\n\r\n![image](https://github.com/henrylee2cn/pholcus/raw/master/doc/icon.png)\r\n\r\n## 免责声明\r\n\r\n**本软件仅用于学术研究，使用者需遵守其所在地的相关法律法规，请勿用于非法用途！！\r\n如在中国大陆频频爆出爬虫开发者涉诉与违规的[新闻](https://github.com/HiddenStrawberry/Crawler_Illegal_Cases_In_China)。\u003cbr\u003e\r\n\u003cb\u003e郑重声明\u003c/b\u003e：因违法违规使用造成的一切后果，使用者自行承担！！**\r\n\r\n## 爬虫原理\r\n\r\n![image](https://github.com/henrylee2cn/pholcus/raw/master/doc/module.png)\r\n\r\n\u0026nbsp;\r\n\r\n![image](https://github.com/henrylee2cn/pholcus/raw/master/doc/project.png)\r\n\r\n\u0026nbsp;\r\n\r\n![image](https://github.com/henrylee2cn/pholcus/raw/master/doc/distribute.png)\r\n\r\n\r\n## 框架特点\r\n\r\n- 为具备一定Go或JS编程基础的用户提供只需关注规则定制、功能完备的重量级爬虫工具；\r\n- 支持单机、服务端、客户端三种运行模式；\r\n- GUI(Windows)、Web、Cmd 三种操作界面，可通过参数控制打开方式；\r\n- 支持状态控制，如暂停、恢复、停止等；\r\n- 可控制采集量；\r\n- 可控制并发协程数；\r\n- 支持多采集任务并发执行；\r\n- 支持代理IP列表，可控制更换频率；\r\n- 支持采集过程随机停歇，模拟人工行为；\r\n- 根据规则需求，提供自定义配置输入接口\r\n- 有mysql、mongodb、kafka、csv、excel、原文件下载共五种输出方式；\r\n- 支持分批输出，且每批数量可控；\r\n- 支持静态Go和动态JS两种采集规则，支持横纵向两种抓取模式，且有大量Demo；\r\n- 持久化成功记录，便于自动去重；\r\n- 序列化失败请求，支持反序列化自动重载处理；\r\n- 采用surfer高并发下载器，支持 GET/POST/HEAD 方法及 http/https 协议，同时支持固定UserAgent自动保存cookie与随机大量UserAgent禁用cookie两种模式，高度模拟浏览器行为，可实现模拟登录等功能；\r\n- 服务器/客户端模式采用Teleport高并发SocketAPI框架，全双工长连接通信，内部数据传输格式为JSON。\r\n\r\n## 下载安装\r\n\r\n```\r\ngo get -u -v github.com/henrylee2cn/pholcus\r\n```\r\n\r\n## 创建项目\r\n\r\n```\r\npackage main\r\n\r\nimport (\r\n    \"github.com/henrylee2cn/pholcus/exec\"\r\n    // _ \"pholcus_lib_pte\" // 同样你也可以自由添加自己的规则库\r\n)\r\n\r\nfunc main() {\r\n    // 设置运行时默认操作界面，并开始运行\r\n    // 运行软件前，可设置 -a_ui 参数为\"web\"、\"gui\"或\"cmd\"，指定本次运行的操作界面\r\n    // 其中\"gui\"仅支持Windows系统\r\n    exec.DefaultRun(\"web\")\r\n}\r\n```\r\n\u0026nbsp;\r\n\r\n## 编译运行\r\n正常编译方法\r\n```\r\ncd {{replace your gopath}}/src/github.com/henrylee2cn/pholcus\r\ngo install 或者 go build\r\n```\r\nWindows下隐藏cmd窗口的编译方法\r\n```\r\ncd {{replace your gopath}}/src/github.com/henrylee2cn/pholcus\r\ngo install -ldflags=\"-H=windowsgui -linkmode=internal\" 或者 go build -ldflags=\"-H=windowsgui -linkmode=internal\"\r\n```\r\n查看可选参数: \r\n```\r\npholcus -h\r\n```\r\n![image](https://github.com/henrylee2cn/pholcus/raw/master/doc/help.jpg)\r\n\r\n\u0026nbsp;\r\n\r\n\u003e *\u003cfont size=\"2\"\u003eWeb版操作界面截图如下：*\r\n\r\n![image](https://github.com/henrylee2cn/pholcus/raw/master/doc/webshow_1.png)\r\n\r\n\u0026nbsp;\r\n\r\n\u003e *\u003cfont size=\"2\"\u003eGUI版操作界面之模式选择界面截图如下*\r\n\r\n![image](https://github.com/henrylee2cn/pholcus/raw/master/doc/guishow_0.jpg)\r\n\r\n\u0026nbsp;\r\n\r\n\u003e *\u003cfont size=\"2\"\u003eCmd版运行参数设置示例如下*\r\n\r\n```\r\n$ pholcus -_ui=cmd -a_mode=0 -c_spider=3,8 -a_outtype=csv -a_thread=20 -a_dockercap=5000 -a_pause=300\r\n-a_proxyminute=0 -a_keyins=\"\u003cpholcus\u003e\u003cgolang\u003e\" -a_limit=10 -a_success=true -a_failure=true\r\n```\r\n\r\n\u0026nbsp;\r\n\r\n*注意：*Mac下如使用代理IP功能，请务必获取root用户权限，否则无法通过`ping`获取可以代理！\r\n\r\n\u0026nbsp;\r\n\r\n## 运行时目录文件\r\n\r\n```\r\n├─pholcus 软件\r\n│\r\n├─pholcus_pkg 运行时文件目录\r\n│  ├─config.ini 配置文件\r\n│  │\r\n│  ├─proxy.lib 代理IP列表文件\r\n│  │\r\n│  ├─spiders 动态规则目录\r\n│  │  └─xxx.pholcus.html 动态规则文件\r\n│  │\r\n│  ├─phantomjs 程序文件\r\n│  │\r\n│  ├─text_out 文本数据文件输出目录\r\n│  │\r\n│  ├─file_out 文件结果输出目录\r\n│  │\r\n│  ├─logs 日志目录\r\n│  │\r\n│  ├─history 历史记录目录\r\n│  │\r\n└─└─cache 临时缓存目录\r\n```\r\n\r\n\u0026nbsp;\r\n\r\n## 动态规则示例\r\n\r\n特点：动态加载规则，无需重新编译软件，书写简单，添加自由，适用于轻量级的采集项目。\r\n\u003cbr/\u003e\r\nxxx.pholcus.html\r\n```\r\n\u003cSpider\u003e\r\n    \u003cName\u003eHTML动态规则示例\u003c/Name\u003e\r\n    \u003cDescription\u003eHTML动态规则示例 [Auto Page] [http://xxx.xxx.xxx]\u003c/Description\u003e\r\n    \u003cPausetime\u003e300\u003c/Pausetime\u003e\r\n    \u003cEnableLimit\u003efalse\u003c/EnableLimit\u003e\r\n    \u003cEnableCookie\u003etrue\u003c/EnableCookie\u003e\r\n    \u003cEnableKeyin\u003efalse\u003c/EnableKeyin\u003e\r\n    \u003cNotDefaultField\u003efalse\u003c/NotDefaultField\u003e\r\n    \u003cNamespace\u003e\r\n        \u003cScript\u003e\u003c/Script\u003e\r\n    \u003c/Namespace\u003e\r\n    \u003cSubNamespace\u003e\r\n        \u003cScript\u003e\u003c/Script\u003e\r\n    \u003c/SubNamespace\u003e\r\n    \u003cRoot\u003e\r\n        \u003cScript param=\"ctx\"\u003e\r\n        console.log(\"Root\");\r\n        ctx.JsAddQueue({\r\n            Url: \"http://xxx.xxx.xxx\",\r\n            Rule: \"登录页\"\r\n        });\r\n        \u003c/Script\u003e\r\n    \u003c/Root\u003e\r\n    \u003cRule name=\"登录页\"\u003e\r\n        \u003cAidFunc\u003e\r\n            \u003cScript param=\"ctx,aid\"\u003e\r\n            \u003c/Script\u003e\r\n        \u003c/AidFunc\u003e\r\n        \u003cParseFunc\u003e\r\n            \u003cScript param=\"ctx\"\u003e\r\n            console.log(ctx.GetRuleName());\r\n            ctx.JsAddQueue({\r\n                Url: \"http://xxx.xxx.xxx\",\r\n                Rule: \"登录后\",\r\n                Method: \"POST\",\r\n                PostData: \"username=44444444@qq.com\u0026amp;password=44444444\u0026amp;login_btn=login_btn\u0026amp;submit=login_btn\"\r\n            });\r\n            \u003c/Script\u003e\r\n        \u003c/ParseFunc\u003e\r\n    \u003c/Rule\u003e\r\n    \u003cRule name=\"登录后\"\u003e\r\n        \u003cParseFunc\u003e\r\n            \u003cScript param=\"ctx\"\u003e\r\n            console.log(ctx.GetRuleName());\r\n            ctx.Output({\r\n                \"全部\": ctx.GetText()\r\n            });\r\n            ctx.JsAddQueue({\r\n                Url: \"http://accounts.xxx.xxx/member\",\r\n                Rule: \"个人中心\",\r\n                Header: {\r\n                    \"Referer\": [ctx.GetUrl()]\r\n                }\r\n            });\r\n            \u003c/Script\u003e\r\n        \u003c/ParseFunc\u003e\r\n    \u003c/Rule\u003e\r\n    \u003cRule name=\"个人中心\"\u003e\r\n        \u003cParseFunc\u003e\r\n            \u003cScript param=\"ctx\"\u003e\r\n            console.log(\"个人中心: \" + ctx.GetRuleName());\r\n            ctx.Output({\r\n                \"全部\": ctx.GetText()\r\n            });\r\n            \u003c/Script\u003e\r\n        \u003c/ParseFunc\u003e\r\n    \u003c/Rule\u003e\r\n\u003c/Spider\u003e\r\n```\r\n\r\n## 静态规则示例\r\n\r\n特点：随软件一同编译，定制性更强，效率更高，适用于重量级的采集项目。\r\n\u003cbr/\u003e\r\nxxx.go\r\n\r\n```\r\nfunc init() {\r\n    Spider{\r\n        Name:        \"静态规则示例\",\r\n        Description: \"静态规则示例 [Auto Page] [http://xxx.xxx.xxx]\",\r\n        // Pausetime: 300,\r\n        // Limit:   LIMIT,\r\n        // Keyin:   KEYIN,\r\n        EnableCookie:    true,\r\n        NotDefaultField: false,\r\n        Namespace:       nil,\r\n        SubNamespace:    nil,\r\n        RuleTree: \u0026RuleTree{\r\n            Root: func(ctx *Context) {\r\n                ctx.AddQueue(\u0026request.Request{Url: \"http://xxx.xxx.xxx\", Rule: \"登录页\"})\r\n            },\r\n            Trunk: map[string]*Rule{\r\n                \"登录页\": {\r\n                    ParseFunc: func(ctx *Context) {\r\n                        ctx.AddQueue(\u0026request.Request{\r\n                            Url:      \"http://xxx.xxx.xxx\",\r\n                            Rule:     \"登录后\",\r\n                            Method:   \"POST\",\r\n                            PostData: \"username=123456@qq.com\u0026password=123456\u0026login_btn=login_btn\u0026submit=login_btn\",\r\n                        })\r\n                    },\r\n                },\r\n                \"登录后\": {\r\n                    ParseFunc: func(ctx *Context) {\r\n                        ctx.Output(map[string]interface{}{\r\n                            \"全部\": ctx.GetText(),\r\n                        })\r\n                        ctx.AddQueue(\u0026request.Request{\r\n                            Url:    \"http://accounts.xxx.xxx/member\",\r\n                            Rule:   \"个人中心\",\r\n                            Header: http.Header{\"Referer\": []string{ctx.GetUrl()}},\r\n                        })\r\n                    },\r\n                },\r\n                \"个人中心\": {\r\n                    ParseFunc: func(ctx *Context) {\r\n                        ctx.Output(map[string]interface{}{\r\n                            \"全部\": ctx.GetText(),\r\n                        })\r\n                    },\r\n                },\r\n            },\r\n        },\r\n    }.Register()\r\n}\r\n```\r\n\r\n\u0026nbsp;\r\n\r\n## 代理IP\r\n\r\n- 代理IP写在`/pholcus_pkg/proxy.lib`文件，格式如下，一行一个IP：\r\n\r\n```\r\nhttp://183.141.168.95:3128\r\nhttps://60.13.146.92:8088\r\nhttp://59.59.4.22:8090\r\nhttps://180.119.78.78:8090\r\nhttps://222.178.56.73:8118\r\nhttp://115.228.57.254:3128\r\nhttp://49.84.106.160:9000\r\n```\r\n\r\n- 在操作界面选择“代理IP更换频率”或命令行设置`-a_proxyminute`参数，进行使用\r\n\r\n- *注意：*Mac下如使用代理IP功能，请务必获取root用户权限，否则无法通过ping获取可以代理！\r\n\r\n\u0026nbsp;\r\n\r\n## FAQ\r\n\r\n请求队列中，重复的URL是否会自动去重？\r\n```\r\nurl默认情况下是去重的，但是可以通过设置Request.Reloadable=true忽略重复。\r\n```\r\n\r\nURL指向的页面内容若有更新，框架是否有判断的机制？\r\n```\r\nurl页面内容的更新，框架无法直接支持判断，但是用户可以自己在规则中自定义支持。\r\n```\r\n\r\n请求成功是依据web头的状态码判断？\r\n```\r\n不是判断状态，而是判断服务器有无响应流返回。即，404页面同样属于成功。\r\n```\r\n\r\n请求失败后的重新请求机制？\r\n```\r\n每个url尝试下载指定次数之后，若依然失败，则将该请求追加到一个类似defer性质的特殊队列中。  \r\n在当前任务正常结束后，将自动添加至下载队列，再次进行下载。如果依然有没下载成功的，则保存至失败历史记录。  \r\n当下次执行该条爬虫规则时，可通过选择继承历史失败记录，把这些失败请求自动加入defer性质的特殊队列……（后面是重复步骤）\r\n```\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandeya%2Fpholcus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandeya%2Fpholcus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandeya%2Fpholcus/lists"}