{"id":13464531,"url":"https://github.com/CrawlScript/WebCollector","last_synced_at":"2025-03-25T11:31:48.127Z","repository":{"id":18556690,"uuid":"21758436","full_name":"CrawlScript/WebCollector","owner":"CrawlScript","description":"WebCollector is an open source web crawler framework based on Java.It provides some simple interfaces for crawling the Web,you can setup a multi-threaded web crawler in less than 5 minutes.","archived":false,"fork":false,"pushed_at":"2025-01-11T11:37:27.000Z","size":291228,"stargazers_count":3075,"open_issues_count":62,"forks_count":1451,"subscribers_count":327,"default_branch":"master","last_synced_at":"2025-03-20T14:08:27.464Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://github.com/CrawlScript/WebCollector","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CrawlScript.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-07-12T03:16:12.000Z","updated_at":"2025-03-19T05:03:50.000Z","dependencies_parsed_at":"2022-07-21T04:34:58.051Z","dependency_job_id":"114ab4fd-d1a8-48c2-8419-41080db8e0b0","html_url":"https://github.com/CrawlScript/WebCollector","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CrawlScript%2FWebCollector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CrawlScript%2FWebCollector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CrawlScript%2FWebCollector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CrawlScript%2FWebCollector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CrawlScript","download_url":"https://codeload.github.com/CrawlScript/WebCollector/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245454033,"owners_count":20617966,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T14:00:45.611Z","updated_at":"2025-03-25T11:31:43.108Z","avatar_url":"https://github.com/CrawlScript.png","language":"Java","funding_links":[],"categories":["All","Java"],"sub_categories":[],"readme":"# WebCollector\r\nWebCollector is an open source web crawler framework based on Java.It provides\r\n  some simple interfaces for crawling the Web,you can setup a\r\n  multi-threaded web crawler in less than 5 minutes.\r\n\r\n\r\nIn addition to a general crawler framework, WebCollector also integrates __CEPF__, a well-designed state-of-the-art web content extraction algorithm proposed by Wu, et al.:\r\n+ Wu GQ, Hu J, Li L, Xu ZH, Liu PC, Hu XG, Wu XD. Online Web news extraction via tag path feature fusion. Ruan Jian Xue Bao/Journal of Software, 2016,27(3):714-735 (in Chinese). http://www.jos.org.cn/1000-9825/4868.htm\r\n\r\n\r\n## HomePage\r\n[https://github.com/CrawlScript/WebCollector](https://github.com/CrawlScript/WebCollector)\r\n\r\n\r\n## Installation\r\n\r\n\r\n### Using Maven\r\n\r\n```xml\r\n\u003cdependency\u003e\r\n    \u003cgroupId\u003ecn.edu.hfut.dmic.webcollector\u003c/groupId\u003e\r\n    \u003cartifactId\u003eWebCollector\u003c/artifactId\u003e\r\n    \u003cversion\u003e2.73-alpha\u003c/version\u003e\r\n\u003c/dependency\u003e\r\n```\r\n\r\n\r\n### Without Maven\r\nWebCollector jars are available on the [HomePage](https://github.com/CrawlScript/WebCollector).\r\n\r\n+ __webcollector-version-bin.zip__ contains core jars.\r\n\r\n\r\n\r\n## Example Index\r\n\r\nAnnotation versions are named with `DemoAnnotatedxxxxxx.java`.\r\n\r\n### Basic\r\n\r\n+ [DemoAutoNewsCrawler.java](src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoAutoNewsCrawler.java) | [DemoAnnotatedAutoNewsCrawler.java](src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoAnnotatedAutoNewsCrawler.java)\r\n+ [DemoManualNewsCrawler.java](src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoManualNewsCrawler.java) | [DemoAnnotatedManualNewsCrawler.java](src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoAnnotatedManualNewsCrawler.java)\r\n+ [DemoExceptionCrawler.java](src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoExceptionCrawler.java)\r\n\r\n### CrawlDatum and MetaData\r\n\r\n+ [DemoMetaCrawler.java](src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoMetaCrawler.java)\r\n+ [DemoAnnotatedMatchTypeCrawler.java](src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoAnnotatedMatchTypeCrawler.java)\r\n+ [DemoAnnotatedDepthCrawler.java](src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoAnnotatedDepthCrawler.java)\r\n+ [DemoBingCrawler.java](src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoBingCrawler.java) | [DemoAnnotatedBingCrawler.java](src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoAnnotatedBingCrawler.java)\r\n+ [DemoAnnotatedDepthCrawler.java](src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoAnnotatedDepthCrawler.java)\r\n\r\n### Http Request and Javascript\r\n\r\n+ [DemoCookieCrawler.java](src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoCookieCrawler.java)\r\n+ [DemoRedirectCrawler.java](src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoRedirectCrawler.java)  | [DemoAnnotatedRedirectCrawler.java](src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoAnnotatedRedirectCrawler.java)\r\n+ [DemoPostCrawler.java](src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoPostCrawler.java)\r\n+ [DemoRandomProxyCrawler.java](src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoRandomProxyCrawler.java)\r\n+ [AbuyunDynamicProxyRequester.java](src/main/java/cn/edu/hfut/dmic/webcollector/example/AbuyunDynamicProxyRequester.java)\r\n+ [DemoSeleniumCrawler.java](src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoSeleniumCrawler.java)\r\n\r\n### NextFilter\r\n\r\n+ [DemoNextFilter.java](src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoNextFilter.java)\r\n+ [DemoHashSetNextFilter.java](src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoHashSetNextFilter.java)\r\n\r\n\r\n\r\n\r\n\r\n\r\n## Quickstart\r\nLets crawl some news from github news.This demo prints out the titles and contents extracted from news of github news.\r\n\r\n### Automatically Detecting URLs\r\n\r\n[DemoAutoNewsCrawler.java](src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoAutoNewsCrawler.java):\r\n\r\n```java\r\n\r\nimport cn.edu.hfut.dmic.webcollector.model.CrawlDatums;\r\nimport cn.edu.hfut.dmic.webcollector.model.Page;\r\nimport cn.edu.hfut.dmic.webcollector.plugin.rocks.BreadthCrawler;\r\n\r\n/**\r\n * Crawling news from github news\r\n *\r\n * @author hu\r\n */\r\npublic class DemoAutoNewsCrawler extends BreadthCrawler {\r\n    /**\r\n     * @param crawlPath crawlPath is the path of the directory which maintains\r\n     *                  information of this crawler\r\n     * @param autoParse if autoParse is true,BreadthCrawler will auto extract\r\n     *                  links which match regex rules from pag\r\n     */\r\n    public DemoAutoNewsCrawler(String crawlPath, boolean autoParse) {\r\n        super(crawlPath, autoParse);\r\n        /*start pages*/\r\n        this.addSeed(\"https://blog.github.com/\");\r\n        for(int pageIndex = 2; pageIndex \u003c= 5; pageIndex++) {\r\n            String seedUrl = String.format(\"https://blog.github.com/page/%d/\", pageIndex);\r\n            this.addSeed(seedUrl);\r\n        }\r\n\r\n        /*fetch url like \"https://blog.github.com/2018-07-13-graphql-for-octokit/\" */\r\n        this.addRegex(\"https://blog.github.com/[0-9]{4}-[0-9]{2}-[0-9]{2}-[^/]+/\");\r\n        /*do not fetch jpg|png|gif*/\r\n        //this.addRegex(\"-.*\\\\.(jpg|png|gif).*\");\r\n        /*do not fetch url contains #*/\r\n        //this.addRegex(\"-.*#.*\");\r\n\r\n        setThreads(50);\r\n        getConf().setTopN(100);\r\n\r\n        //enable resumable mode\r\n        //setResumable(true);\r\n    }\r\n\r\n    @Override\r\n    public void visit(Page page, CrawlDatums next) {\r\n        String url = page.url();\r\n        /*if page is news page*/\r\n        if (page.matchUrl(\"https://blog.github.com/[0-9]{4}-[0-9]{2}-[0-9]{2}[^/]+/\")) {\r\n\r\n            /*extract title and content of news by css selector*/\r\n            String title = page.select(\"h1[class=lh-condensed]\").first().text();\r\n            String content = page.selectText(\"div.content.markdown-body\");\r\n\r\n            System.out.println(\"URL:\\n\" + url);\r\n            System.out.println(\"title:\\n\" + title);\r\n            System.out.println(\"content:\\n\" + content);\r\n\r\n            /*If you want to add urls to crawl,add them to nextLink*/\r\n            /*WebCollector automatically filters links that have been fetched before*/\r\n            /*If autoParse is true and the link you add to nextLinks does not match the \r\n              regex rules,the link will also been filtered.*/\r\n            //next.add(\"http://xxxxxx.com\");\r\n        }\r\n    }\r\n\r\n    public static void main(String[] args) throws Exception {\r\n        DemoAutoNewsCrawler crawler = new DemoAutoNewsCrawler(\"crawl\", true);\r\n        /*start crawl with depth of 4*/\r\n        crawler.start(4);\r\n    }\r\n\r\n}\r\n\r\n```\r\n\r\n\r\n### Manually Detecting URLs\r\n\r\n\r\n[DemoManualNewsCrawler.java](src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoManualNewsCrawler.java):\r\n\r\n```java\r\n\r\nimport cn.edu.hfut.dmic.webcollector.model.CrawlDatums;\r\nimport cn.edu.hfut.dmic.webcollector.model.Page;\r\nimport cn.edu.hfut.dmic.webcollector.plugin.rocks.BreadthCrawler;\r\n\r\n/**\r\n * Crawling news from github news\r\n *\r\n * @author hu\r\n */\r\npublic class DemoManualNewsCrawler extends BreadthCrawler {\r\n    /**\r\n     * @param crawlPath crawlPath is the path of the directory which maintains\r\n     *                  information of this crawler\r\n     * @param autoParse if autoParse is true,BreadthCrawler will auto extract\r\n     *                  links which match regex rules from pag\r\n     */\r\n    public DemoManualNewsCrawler(String crawlPath, boolean autoParse) {\r\n        super(crawlPath, autoParse);\r\n        // add 5 start pages and set their type to \"list\"\r\n        //\"list\" is not a reserved word, you can use other string instead\r\n        this.addSeedAndReturn(\"https://blog.github.com/\").type(\"list\");\r\n        for(int pageIndex = 2; pageIndex \u003c= 5; pageIndex++) {\r\n            String seedUrl = String.format(\"https://blog.github.com/page/%d/\", pageIndex);\r\n            this.addSeed(seedUrl, \"list\");\r\n        }\r\n\r\n        setThreads(50);\r\n        getConf().setTopN(100);\r\n\r\n        //enable resumable mode\r\n        //setResumable(true);\r\n    }\r\n\r\n    @Override\r\n    public void visit(Page page, CrawlDatums next) {\r\n        String url = page.url();\r\n\r\n        if (page.matchType(\"list\")) {\r\n            /*if type is \"list\"*/\r\n            /*detect content page by css selector and mark their types as \"content\"*/\r\n            next.add(page.links(\"h1.lh-condensed\u003ea\")).type(\"content\");\r\n        }else if(page.matchType(\"content\")) {\r\n            /*if type is \"content\"*/\r\n            /*extract title and content of news by css selector*/\r\n            String title = page.select(\"h1[class=lh-condensed]\").first().text();\r\n            String content = page.selectText(\"div.content.markdown-body\");\r\n\r\n            //read title_prefix and content_length_limit from configuration\r\n            title = getConf().getString(\"title_prefix\") + title;\r\n            content = content.substring(0, getConf().getInteger(\"content_length_limit\"));\r\n\r\n            System.out.println(\"URL:\\n\" + url);\r\n            System.out.println(\"title:\\n\" + title);\r\n            System.out.println(\"content:\\n\" + content);\r\n        }\r\n\r\n    }\r\n\r\n    public static void main(String[] args) throws Exception {\r\n        DemoManualNewsCrawler crawler = new DemoManualNewsCrawler(\"crawl\", false);\r\n\r\n        crawler.getConf().setExecuteInterval(5000);\r\n\r\n        crawler.getConf().set(\"title_prefix\",\"PREFIX_\");\r\n        crawler.getConf().set(\"content_length_limit\", 20);\r\n\r\n        /*start crawl with depth of 4*/\r\n        crawler.start(4);\r\n    }\r\n\r\n}\r\n\r\n```\r\n\r\n## CrawlDatum\r\n\r\nCrawlDatum is an important data structure in WebCollector, which corresponds to url of webpages. Both crawled urls and detected urls are maintained as CrawlDatums.\r\n\r\nThere are some differences between CrawlDatum and url:\r\n\r\n+ A CrawlDatum contains a key and a url. The key is the url by default. You can set the key manually by CrawlDatum.key(\"xxxxx\") so that CrawlDatums with the same url may have different keys. This is very useful in some tasks like crawling data by api, which often request different data by the same url with different post parameters.\r\n+ A CrawlDatum may contain metadata, which could maintain some information besides the url. \r\n\r\n## Manually Detecting URLs\r\n\r\nIn both `void visit(Page page, CrawlDatums next)` and `void execute(Page page, CrawlDatums next)`, the second parameter `CrawlDatum next` is a container which you should put the detected URLs in:\r\n\r\n```java\r\n//add one detected URL\r\nnext.add(\"detected URL\");\r\n//add one detected URL and set its type\r\nnext.add(\"detected URL\", \"type\");\r\n//add one detected URL\r\nnext.add(new CrawlDatum(\"detected URL\"));\r\n//add detected URLs\r\nnext.add(\"detected URL list\");\r\n//add detected URLs\r\nnext.add((\"detected URL list\",\"type\");\r\n//add detected URLs\r\nnext.add(new CrawlDatums(\"detected URL list\"));\r\n\r\n//add one detected URL and return the added URL(CrawlDatum)\r\n//and set its key and type\r\nnext.addAndReturn(\"detected URL\").key(\"key\").type(\"type\");\r\n//add detected URLs and return the added URLs(CrawlDatums)\r\n//and set their type and meta info\r\nnext.addAndReturn(\"detected URL list\").type(\"type\").meta(\"page_num\",10);\r\n\r\n//add detected URL and return next\r\n//and modify the type and meta info of all the CrawlDatums in next,\r\n//including the added URL\r\nnext.add(\"detected URL\").type(\"type\").meta(\"page_num\", 10);\r\n//add detected URLs and return next\r\n//and modify the type and meta info of all the CrawlDatums in next,\r\n//including the added URLs\r\nnext.add(\"detected URL list\").type(\"type\").meta(\"page_num\", 10);\r\n```\r\n\r\nYou don't need to consider how to filter duplicated URLs, the crawler will filter them automatically.\r\n\r\n## Plugins\r\n\r\nPlugins provide a large part of the functionality of WebCollector. There are several kinds of plugins:\r\n\r\n+ Executor: Plugins which define how to download webpages, how to parse webpages and how to detect new CrawlDatums(urls)\r\n+ DBManager: Plugins which maintains the crawling history\r\n+ GeneratorFilter: Plugins which generate CrawlDatums(urls) which will be crawled\r\n+ NextFilter: Plugins which filter CrawlDatums(urls) which detected by the crawler\r\n\r\nSome BreadthCrawler and RamCrawler are the most used crawlers which extends AutoParseCrawler. The following plugins only work in crawlers which extend AutoParseCrawler:\r\n\r\n+ Requester: Plugins which define how to do http request\r\n+ Visitor: Plugins which define how to parse webpages and how to detect new CrawlDatums(urls)\r\n\r\nPlugins can be mounted as follows:\r\n\r\n```java\r\ncrawler.setRequester(xxxxx);\r\ncrawler.setVisitor(xxxxx);\r\ncrawler.setNextFilter(xxxxx);\r\ncrawler.setGeneratorFilter(xxxxx);\r\ncrawler.setExecutor(xxxxx);\r\ncrawler.setDBManager(xxxxx);\r\n```\r\n\r\nAutoParseCrawler is also an Executor plugin, a Requester plugin and a Visitor plugin. By default it use itsself as the Executor plugin, Request Plugin and Visitor plugin. So if you want to write a plugin for AutoParseCrawler, you have two ways:\r\n\r\n+ Just override the corresponding methods of your AutoParseCrawler. For example, if you are using BreadthCrawler, all you have to do is override the `Page getResponse(CrawlDatum crawlDatum)` method.\r\n+ Create a new class which implements Requester interface and implement the `Page getResponse(CrawlDatum crawlDatum)` method of the class. Instantiate the class and use `crawler.setRequester(the instance)` to mount the plugin to the crawler.\r\n\r\n\r\n## Customizing Requester Plugin\r\n\r\nCreating a Requester plugin is very easy. You just need to create a new class which implements Requester interface and implement the `Page getResponse(CrawlDatum crawlDatum)` method of the class. [OkHttpRequester](https://github.com/CrawlScript/WebCollector/blob/master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/plugin/net/OkHttpRequester.java) is a Requester Plugin provided by WebCollector. You can find the code here: [OkHttpRequester.class](https://github.com/CrawlScript/WebCollector/blob/master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/plugin/net/OkHttpRequester.java).\r\n\r\nMost of the time, you don't need to write a Requester plugin from the scratch. Creating a Requester plugin by extending the OkHttpRequester is a convenient way.\r\n\r\n    \r\n## Configuration Details\r\n\r\nConfiguration mechanism of WebCollector is redesigned in version 2.70. The above example [ManualNewsCrawler.java](https://github.com/CrawlScript/WebCollector/blob/master/ManualNewsCrawler.java) also shows how to use configuration to customize your crawler.\r\n\r\nBefore version 2.70, configuration is maintained by static variables in class `cn.edu.hfut.dmic.webcollector.util.Config`, hence it's cumbersome to assign different configurations to different crawlers.\r\n\r\nSince version 2.70, each crawler can has its own configuration. You can use `crawler.getConf()` to get it or `crawler.setConf(xxx)` to set it. By default, all crawlers use a singleton default configuration, which could be get by Configuration.getDefault(). So in the above example [ManualNewsCrawler.java](https://github.com/CrawlScript/WebCollector/blob/master/ManualNewsCrawler.java), `crawler.getConf().set(\"xxx\", \"xxx\")` would affect the default configuration, which may be used by other crawlers.\r\n\r\nIf you want to change the configuration of a crawler without affecting other crawlers, you should manually create a configuration and specify it to the crawler. For example:\r\n\r\n```java\r\nConfiguration conf = Configuration.copyDefault();\r\n\r\nconf.set(\"test_string_key\", \"test_string_value\");\r\ncrawler.getConf().setReadTimeout(1000 * 5);\r\n\r\ncrawler.setConf(conf);\r\n\r\ncrawler.getConf().set(\"test_int_key\", 10);\r\ncrawler.getConf().setConnectTimeout(1000 * 5);\r\n```\r\n\r\n`Configuration.copyDefault()` is suggested, because it creates a copy of the singleton default configuration, which contains some necessary key-value pairs, while `new Configuration()` creates an empty configuration.\r\n\r\n\r\n## Resumable Crawling\r\n\r\nIf you want to stop a crawler and continue crawling the next time, you should do two things:\r\n\r\n+ Add `crawler.setResumable(true)` to your code.\r\n+ Don't delete the history directory generated by the crawler, which is specified by the crawlPath parameter.\r\n\r\nWhen you call `crawler.start(depth)`, the crawler will delete the history if you set resumable to false, which is false by default. So if you forget to put 'crawler.setResumable(true)' in your code before the first time you start your crawler, it doesn't matter, because you have no history directory. \r\n\r\n\r\n## Content Extraction\r\nWebCollector could automatically extract content from news web-pages:\r\n\r\n```java\r\nNews news = ContentExtractor.getNewsByHtml(html, url);\r\nNews news = ContentExtractor.getNewsByHtml(html);\r\nNews news = ContentExtractor.getNewsByUrl(url);\r\n\r\nString content = ContentExtractor.getContentByHtml(html, url);\r\nString content = ContentExtractor.getContentByHtml(html);\r\nString content = ContentExtractor.getContentByUrl(url);\r\n\r\nElement contentElement = ContentExtractor.getContentElementByHtml(html, url);\r\nElement contentElement = ContentExtractor.getContentElementByHtml(html);\r\nElement contentElement = ContentExtractor.getContentElementByUrl(url);\r\n```\r\n\r\n## WeChat Group\r\n\r\n\u003cp align=\"center\"\u003e\r\n  \u003cimg width=\"400\"  src=\"http://cdn.zhuanzhi.ai/other/wc_wx2.jpg\"\u003e\r\n\u003c/p\u003e\r\n\r\n\r\n\r\n\r\n\u003c!--\r\n\r\n## Other Documentation\r\n\r\n+ [中文文档](https://github.com/CrawlScript/WebCollector/blob/master/README.zh-cn.md)\r\n--\u003e\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FCrawlScript%2FWebCollector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FCrawlScript%2FWebCollector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FCrawlScript%2FWebCollector/lists"}