{"id":15043340,"url":"https://github.com/zhangyingwei/cockroach","last_synced_at":"2025-04-10T00:40:58.776Z","repository":{"id":57724443,"uuid":"99882402","full_name":"zhangyingwei/cockroach","owner":"zhangyingwei","description":"又一个号称高性能的  java 爬虫工具/爬虫框架","archived":false,"fork":false,"pushed_at":"2019-10-29T02:45:40.000Z","size":425,"stargazers_count":122,"open_issues_count":3,"forks_count":19,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-03-24T02:22:43.225Z","etag":null,"topics":["cockroach","java","java-8","java-spider","spider","spiders"],"latest_commit_sha":null,"homepage":"https://zhangyingwei.github.io/cockroach/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zhangyingwei.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-08-10T04:48:17.000Z","updated_at":"2024-10-21T02:19:54.000Z","dependencies_parsed_at":"2022-09-02T07:12:01.297Z","dependency_job_id":null,"html_url":"https://github.com/zhangyingwei/cockroach","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhangyingwei%2Fcockroach","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhangyingwei%2Fcockroach/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhangyingwei%2Fcockroach/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhangyingwei%2Fcockroach/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zhangyingwei","download_url":"https://codeload.github.com/zhangyingwei/cockroach/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248137997,"owners_count":21053775,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cockroach","java","java-8","java-spider","spider","spiders"],"created_at":"2024-09-24T20:48:52.667Z","updated_at":"2025-04-10T00:40:58.762Z","avatar_url":"https://github.com/zhangyingwei.png","language":"Java","readme":"# cockroach 爬虫：又一个 java 爬虫实现\n\n[![](https://travis-ci.org/zhangyingwei/cockroach.svg?branch=master)](https://travis-ci.org/zhangyingwei/cockroach)\n[![](https://img.shields.io/badge/language-java-orange.svg)]()\n[![](https://img.shields.io/badge/jdk-1.8-green.svg)]()\n[![License](http://img.shields.io/:license-apache-blue.svg)](http://www.apache.org/licenses/LICENSE-2.0.html)\n\n![](http://util.zhangyingwei.com//cockroach/1/carbon.png)\n\n重构了\n[cockroach2](https://github.com/zhangyingwei/cockroach2)\n\n## 简介\n\ncockroach[小强] 当时不知道为啥选了这么个名字，又长又难记，导致编码的过程中因为单词的拼写问题耽误了好长时间。\n\n这个项目算是我的又一个坑吧，算起来挖的坑多了去了，多一个不多少一个不少。\n\n一个小巧、灵活、健壮的内容（pa）获取（chong）框架，暂且叫做框架吧。\n\n简单到什么程度呢，几句话就可以创建一个内容（pa）获取（chong）程序。\n\n### 依赖部分\n\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.github.zhangyingwei\u003c/groupId\u003e\n  \u003cartifactId\u003ecockroach-core\u003c/artifactId\u003e\n  \u003cversion\u003e1.0.6-Beta\u003c/version\u003e\n\u003c/dependency\u003e\n\u003c!-- https://mvnrepository.com/artifact/com.github.zhangyingwei/cockroach-annotation --\u003e\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.github.zhangyingwei\u003c/groupId\u003e\n    \u003cartifactId\u003ecockroach-annotation\u003c/artifactId\u003e\n    \u003cversion\u003e1.0.6-Beta\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n### 代码部分：\n\n```java\n@EnableAutoConfiguration\npublic class CockroachApplicationTest {\n    public static void main(String[] args) throws Exception {\n        TaskQueue queue = TaskQueue.of();\n        queue.push(new Task(\"http://blog.zhangyingwei.com\"));\n        CockroachApplication.run(CockroachApplicationTest.class,queue);\n    }\n}\n```\n没错，就是这么简单。这个内容（pa）获取（chong）程序就是获（pa）取 `http://blog.zhangyingwei.com` 这个页面的内容并将结果打印出来。\n在结果处理这个问题上，程序中默认使用 PringStore 这个类将所有结果打印出来。\n\n## scala \u0026 kotlin\n\n作为目前使用的 jvm 系语言几大巨头，scala 与 kotlin 这里基本上对跟 java 的互调做的很好，但是这里还是给几个 demo。\n\n### scala\n\n```scala\n/**\n  * Created by zhangyw on 2017/12/25.\n  */\nclass TTTStore extends IStore{\n    override def store(taskResponse: TaskResponse): Unit = {\n        println(\"ttt store\")\n    }\n}\n\nobject TTTStore{}\n```\n\n```scala\n/**\n  * Created by zhangyw on 2017/12/25.\n  */\n@EnableAutoConfiguration\n@ThreadConfig(num = 1)\n@Store(classOf[TTTStore])\nobject MainApplication {\n    def main(args: Array[String]): Unit = {\n        println(\"hello scala spider\")\n        val queue = TaskQueue.of()\n        queue.push(new Task(\"http://blog.zhangyingwei.com\"))\n        CockroachApplication.run(MainApplication.getClass(),queue)\n    }\n}\n```\n\n### kotlin\n\n```kotlin\nclass TTTStore :IStore{\n    override fun store(response: TaskResponse) {\n        print(\"ttt store\")\n    }\n}\n```\n\n```kotlin\n\n/**\n * Created by zhangyw on 2017/12/25.\n */\n@EnableAutoConfiguration\n@ThreadConfig(num = 1)\n@Store(TTTStore::class)\nobject MainApplication {\n    @JvmStatic\n    fun main(args: Array\u003cString\u003e) {\n        print(\"hello kotlin spider\")\n        val queue = TaskQueue.of()\n        queue.push(Task(\"http://blog.zhangyingwei.com\"))\n        CockroachApplication.run(MainApplication::class.java, queue)\n    }\n}\n```\n\n## 联系方式\n* 邮箱： zhangyw001@gmail.com\n* 微信： fengche361\n\n## Lisence\n\nLisenced under [Apache 2.0 lisence](./LICENSE)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhangyingwei%2Fcockroach","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzhangyingwei%2Fcockroach","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhangyingwei%2Fcockroach/lists"}