{"id":37019455,"url":"https://github.com/xbynet/crawler","last_synced_at":"2026-01-14T02:09:59.825Z","repository":{"id":57724123,"uuid":"92580335","full_name":"xbynet/crawler","owner":"xbynet","description":"A simple and flexible web crawler framework for java.","archived":false,"fork":false,"pushed_at":"2018-04-22T02:40:24.000Z","size":263,"stargazers_count":19,"open_issues_count":1,"forks_count":5,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-07-03T21:44:47.015Z","etag":null,"topics":["crawler","httpclient","java","jsoup","spider"],"latest_commit_sha":null,"homepage":"https://github.com/xbynet/crawler","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xbynet.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-05-27T07:29:15.000Z","updated_at":"2023-09-08T17:25:36.000Z","dependencies_parsed_at":"2022-09-02T06:44:33.901Z","dependency_job_id":null,"html_url":"https://github.com/xbynet/crawler","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/xbynet/crawler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xbynet%2Fcrawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xbynet%2Fcrawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xbynet%2Fcrawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xbynet%2Fcrawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xbynet","download_url":"https://codeload.github.com/xbynet/crawler/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xbynet%2Fcrawler/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28408711,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T01:52:23.358Z","status":"online","status_checked_at":"2026-01-14T02:00:06.678Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","httpclient","java","jsoup","spider"],"created_at":"2026-01-14T02:09:59.230Z","updated_at":"2026-01-14T02:09:59.820Z","avatar_url":"https://github.com/xbynet.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# crawler\nA simple and flexible web crawler framework for java.\n\n## Features:\n1、Code  is easy to understand and customized (代码简单易懂，可定制性强)     \n2、Api is simple and easy to use         \n3、Support File download、Content part fetch.(支持文件下载、分块抓取)          \n4、Request And Response support much options、strong customizable.(请求和响应支持的内容和选项比较丰富、每个请求可定制性强)   \n5、Support do your own operation before or after network request in downloader(支持网络请求前后执行自定义操作)        \n6、Selenium+PhantomJS support     \n7、Redis support      \n\n## Future:\n1、Complete the code comment and test(完善代码注释和完善测试代码)     \n\n## Install:\n\nThe only module that must be added is crawler-core\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.github.xbynet\u003c/groupId\u003e\n    \u003cartifactId\u003ecrawler-core\u003c/artifactId\u003e\n    \u003cversion\u003e0.3.0\u003c/version\u003e\n\u003c/dependency\n```\nBut if you want to use selenium support:\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.github.xbynet\u003c/groupId\u003e\n    \u003cartifactId\u003ecrawler-selenium\u003c/artifactId\u003e\n    \u003cversion\u003e0.3.0\u003c/version\u003e\n\u003c/dependency\n```\n\nModule crawler-server is now a experimental attempt, and now has more work to do on it.\n\n## Demo:\n\n```java\nimport com.github.xbynet.crawler.http.DefaultDownloader;\nimport com.github.xbynet.crawler.http.FileDownloader;\nimport com.github.xbynet.crawler.http.HttpClientFactory;\nimport com.github.xbynet.crawler.parser.JsoupParser;\nimport com.github.xbynet.crawler.scheduler.DefaultScheduler;\n\npublic class GithubCrawler extends Processor {\n\t@Override\n\tpublic void process(Response resp) {\n\t\tString currentUrl = resp.getRequest().getUrl();\n\t\tSystem.out.println(\"CurrentUrl:\" + currentUrl);\n\t\tint respCode = resp.getCode();\n\t\tSystem.out.println(\"ResponseCode:\" + respCode);\n\t\tSystem.out.println(\"type:\" + resp.getRespType().name());\n\t\tString contentType = resp.getContentType();\n\t\tSystem.out.println(\"ContentType:\" + contentType);\n\t\tMap\u003cString, List\u003cString\u003e\u003e headers = resp.getHeaders();\n\t\tSystem.out.println(\"ResonseHeaders:\");\n\t\tfor (String key : headers.keySet()) {\n\t\t\tList\u003cString\u003e values=headers.get(key);\n\t\t\tfor(String str:values){\n\t\t\t\tSystem.out.println(key + \":\" +str);\n\t\t\t}\n\t\t}\n\t\tJsoupParser parser = resp.html();\n\t\t// suppport parted ,分块抓取是会有个parent response来关联所有分块response\n\t\t// System.out.println(\"isParted:\"+resp.isPartResponse());\n\t\t// Response parent=resp.getParentResponse();\n\t\t// resp.addPartRequest(null);\n\t\t//Map\u003cString,Object\u003e extras=resp.getRequest().getExtras();\n\n\t\tif (currentUrl.equals(\"https://github.com/xbynet\")) {\n\t\t\tString avatar = parser.single(\"img.avatar\", \"src\");\n\t\t\tString dir = System.getProperty(\"java.io.tmpdir\");\n\t\t\tString savePath = Paths.get(dir, UUID.randomUUID().toString())\n\t\t\t\t\t.toString();\n\t\t\tboolean avatarDownloaded = download(avatar, savePath);\n\t\t\tSystem.out.println(\"avatar:\" + avatar + \", saved:\" + savePath);\n\t\t\t// System.out.println(\"avtar downloaded status:\"+avatarDownloaded);\n\t\t\tString name = parser.single(\".vcard-names \u003e .vcard-fullname\",\n\t\t\t\t\t\"text\");\n\t\t\tSystem.out.println(\"name:\" + name);\n\t\t\tList\u003cString\u003e reponames = parser.list(\n\t\t\t\t\t\".pinned-repos-list .repo.js-repo\", \"text\");\n\t\t\tList\u003cString\u003e repoUrls = parser.list(\n\t\t\t\t\t\".pinned-repo-item .d-block \u003ea\", \"href\");\n\t\t\tSystem.out.println(\"reponame:url\");\n\t\t\tif (reponames != null) {\n\t\t\t\tfor (int i = 0; i \u003c reponames.size(); i++) {\n\t\t\t\t\tString tmpUrl=\"https://github.com\"+repoUrls.get(i);\n\t\t\t\t\tSystem.out.println(reponames.get(i) + \":\"+tmpUrl);\n\t\t\t\t\tRequest req=new Request(tmpUrl).putExtra(\"name\", reponames.get(i));\n\t\t\t\t\tresp.addRequest(req);\n\t\t\t\t}\n\t\t\t}\n\t\t}else{\n\t\t\tMap\u003cString,Object\u003e extras=resp.getRequest().getExtras();\n\t\t\tString name=extras.get(\"name\").toString();\n\t\t\tSystem.out.println(\"repoName:\"+name);\n\t\t\tString shortDesc=parser.single(\".repository-meta-content\",\"allText\");\n\t\t\tSystem.out.println(\"shortDesc:\"+shortDesc);\n\t\t}\n\t}\n\n\tpublic void start() {\n\t\tSite site = new Site();\n\t\tSpider spider = Spider.builder(this).threadNum(5).site(site)\n\t\t\t\t.urls(\"https://github.com/xbynet\").build();\n\t\tspider.run();\n\t}\n  \n\tpublic static void main(String[] args) {\n\t\tnew GithubCrawler().start();\n\t}\n  \n  \n\tpublic void startCompleteConfig() {\n\t\tString pcUA = \"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36\";\n\t\tString androidUA = \"Mozilla/5.0 (Linux; Android 5.1.1; Nexus 6 Build/LYZ28E) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.23 Mobile Safari/537.36\";\n\n\t\tSite site = new Site();\n\t\tsite.setEncoding(\"UTF-8\").setHeader(\"Referer\", \"https://github.com/\")\n\t\t\t\t.setRetry(3).setRetrySleep(3000).setSleep(50).setTimeout(30000)\n\t\t\t\t.setUa(pcUA);\n\n\t\tRequest request = new Request(\"https://github.com/xbynet\");\n\t\tHttpClientContext ctx = new HttpClientContext();\n\t\tBasicCookieStore cookieStore = new BasicCookieStore();\n\t\tctx.setCookieStore(cookieStore);\n\t\trequest.setAction(new RequestAction() {\n\t\t\t@Override\n\t\t\tpublic void before(CloseableHttpClient client, HttpUriRequest req) {\n\t\t\t\tSystem.out.println(\"before-haha\");\n\t\t\t}\n\n\t\t\t@Override\n\t\t\tpublic void after(CloseableHttpClient client,\n\t\t\t\t\tCloseableHttpResponse resp) {\n\t\t\t\tSystem.out.println(\"after-haha\");\n\t\t\t}\n\t\t}).setCtx(ctx).setEncoding(\"UTF-8\")\n\t\t\t\t.putExtra(\"somekey\", \"I can use in the response by your own\")\n\t\t\t\t.setHeader(\"User-Agent\", pcUA).setMethod(Const.HttpMethod.GET)\n\t\t\t\t.setPartRequest(null).setEntity(null)\n\t\t\t\t.setParams(\"appkeyqqqqqq\", \"1213131232141\").setRetryCount(5)\n\t\t\t\t.setRetrySleepTime(10000);\n\n\t\tSpider spider = Spider.builder(this).threadNum(5)\n\t\t\t\t.name(\"Spider-github-xbynet\")\n\t\t\t\t.defaultDownloader(new DefaultDownloader())\n\t\t\t\t.fileDownloader(new FileDownloader())\n\t\t\t\t.httpClientFactory(new HttpClientFactory()).ipProvider(null)\n\t\t\t\t.listener(null).pool(null).scheduler(new DefaultScheduler())\n\t\t\t\t.shutdownOnComplete(true).site(site).build();\n\t\tspider.run();\n\t}\n\n\n}\n\n```\n## Examples:\n\n- Github(github个人项目信息)\n- OSChinaTweets(开源中国动弹)\n- Qiushibaike(醜事百科)\n- Neihanshequ(内涵段子)   \n- ZihuRecommend(知乎推荐)   \n \n**More Examples:** Please see [here](https://github.com/xbynet/crawler/tree/master/crawler-core/src/test/java/net/xby1993/crawler)  \n\n## Thanks: \n[webmagic](https://github.com/code4craft/webmagic):本项目借鉴了webmagic多处代码，设计上也作了较多参考，非常感谢。     \n[xsoup](https://github.com/code4craft/xsoup)：本项目使用xsoup作为底层xpath处理器      \n[JsonPath](https://github.com/json-path/JsonPath)：本项目使用JsonPath作为底层jsonpath处理器    \n[Jsoup](https://jsoup.org/) 本项目使用Jsoup作为底层HTML/XML处理器      \n[HttpClient](http://hc.apache.org/) 本项目使用HttpClient作为底层网络请求工具    \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxbynet%2Fcrawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxbynet%2Fcrawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxbynet%2Fcrawler/lists"}