{"id":15523749,"url":"https://github.com/arloor/easyspider","last_synced_at":"2025-04-23T07:22:00.999Z","repository":{"id":37171017,"uuid":"175587555","full_name":"arloor/EasySpider","owner":"arloor","description":"A spider framework powered by groovy","archived":false,"fork":false,"pushed_at":"2022-09-01T23:04:11.000Z","size":32,"stargazers_count":5,"open_issues_count":5,"forks_count":2,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-03-29T22:41:15.763Z","etag":null,"topics":["spider","spider-framework"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/arloor.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-03-14T09:15:55.000Z","updated_at":"2023-06-24T12:14:33.000Z","dependencies_parsed_at":"2023-01-17T13:52:26.020Z","dependency_job_id":null,"html_url":"https://github.com/arloor/EasySpider","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arloor%2FEasySpider","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arloor%2FEasySpider/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arloor%2FEasySpider/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arloor%2FEasySpider/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/arloor","download_url":"https://codeload.github.com/arloor/EasySpider/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250387471,"owners_count":21422143,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["spider","spider-framework"],"created_at":"2024-10-02T10:46:46.244Z","updated_at":"2025-04-23T07:22:00.976Z","avatar_url":"https://github.com/arloor.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![](https://travis-ci.org/arloor/EasySpider.svg?branch=master)](https://travis-ci.org/arloor/EasySpider)\n[![](https://img.shields.io/github/last-commit/arloor/EasySpider.svg?style=flat)](https://github.com/arloor/EasySpider/commit/master)\n![](https://img.shields.io/github/languages/code-size/arloor/EasySpider.svg?style=flat)\n\n# EasySpider 一个（并不）简单的爬虫框架\n\n![](http://arloor.com/img/%E7%88%AC%E8%99%AB%E7%B3%BB%E7%BB%9F%E9%83%A8%E7%BD%B2%E5%9B%BE.png)\n\n# 目前进度\n\ngroovy执行引擎完成，spider雏形实现。\n\n# proxy-local和proxy-server\n\n使用自己实现的httpproxy。[项目详情](https://github.com/arloor/HttpProxy)\n\n# 为什么能称之为框架？\n\n因为做好了骨架，只需要修改`taskmanager\\src\\main\\resources`下的groovy脚本就可以构建自己的爬虫任务。\n\n目前还是雏形，需要改进的地方是：\n\n1. 将保存任务的queue使用消息队列保存\n2. 缺少一个网关来提交任务开始的标志\n\n# [Release v1.0](https://github.com/arloor/EasySpider/tree/f987ff7a71e92a2fd69562db44dd8d27bd5ff71b)\n\n这是EasySpider的第一版。还很简单，源码读起来也轻松，要看代码，从这一版开始吧。之后就会引入分布式了亲。\n\n！！！这里对groovy脚本的使用和“任务上下文”的定义是EasySpider的精髓。\n\n## 代码结构\n\n\n分为两个模块：\n\n- spider：封装了一个httpclient，Main函数会开启一个Graber线程，循环从队列中取“任务上下文”，消费该上下文，并发送http请求、处理http响应。\n- taskmanager： 这一模块是关于上面提到的“任务上下文”。所谓任务上下文就是当前任务需要执行什么操作、当前任务的数据、和当前任务的状态（cookie），这些都定义在stepContext类中。StepContainer类负责加载groovy脚本生成对象。groovy脚本都在resources下，groovy脚本就是用于定义“当前任务执行操作”的，实现了Step接口，定义如何创建http请求和如何解析http响应。\n\n## 其他注意点\n\nhttpclient本身可以自己管理cookie。但我选择了自己去维护cookie，而httpclient开启了IGNORE_COOKIE。原因是，httpclient的cookie是绑定在httpclient上，在分布式环境中会存在多个httpclient。这样，一些httpclient执行了关键比如登陆操作，而其他的httpclient没有执行，这样就导致其他httpclient没有登陆之后拿到的cookie。这肯定不行，所以我将cookie信息保存在了“任务上下文”中，这个任务上下文可以在分布式环境中传递，保证关键cookie不丢失。\n\n## 如何使用这个版本的框架\n\n只需要创建自己的groovy脚本就好。groovy脚本实现Step接口，创建请求，解析响应。","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farloor%2Feasyspider","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farloor%2Feasyspider","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farloor%2Feasyspider/lists"}