{"id":13487040,"url":"https://github.com/xuxueli/xxl-crawler","last_synced_at":"2025-05-15T02:07:27.954Z","repository":{"id":43175183,"uuid":"107214157","full_name":"xuxueli/xxl-crawler","owner":"xuxueli","description":"A lightweight web crawler framework.（Java爬虫框架）","archived":false,"fork":false,"pushed_at":"2025-01-05T15:26:24.000Z","size":460,"stargazers_count":718,"open_issues_count":5,"forks_count":309,"subscribers_count":49,"default_branch":"master","last_synced_at":"2025-04-14T06:49:22.588Z","etag":null,"topics":["crawler","distributed","flexible","java","object-oriented","spider","web","xxl-crawler"],"latest_commit_sha":null,"homepage":"http://www.xuxueli.com/xxl-crawler","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xuxueli.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-10-17T03:39:46.000Z","updated_at":"2025-04-10T01:40:27.000Z","dependencies_parsed_at":"2025-01-19T16:00:23.377Z","dependency_job_id":"c3c2083c-2861-4268-a283-dc9c61bb9626","html_url":"https://github.com/xuxueli/xxl-crawler","commit_stats":{"total_commits":138,"total_committers":5,"mean_commits":27.6,"dds":"0.28985507246376807","last_synced_commit":"27d1e4ea9255e72572e46a679253835396fdc80a"},"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xuxueli%2Fxxl-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xuxueli%2Fxxl-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xuxueli%2Fxxl-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xuxueli%2Fxxl-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xuxueli","download_url":"https://codeload.github.com/xuxueli/xxl-crawler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254259383,"owners_count":22040820,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","distributed","flexible","java","object-oriented","spider","web","xxl-crawler"],"created_at":"2024-07-31T18:00:54.683Z","updated_at":"2025-05-15T02:07:27.873Z","avatar_url":"https://github.com/xuxueli.png","language":"Java","funding_links":[],"categories":["Java","网络爬虫"],"sub_categories":["语音合成"],"readme":"\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://www.xuxueli.com/doc/static/xxl-job/images/xxl-logo.jpg\" width=\"150\"\u003e\n    \u003ch3 align=\"center\"\u003eXXL-CRAWLER\u003c/h3\u003e\n    \u003cp align=\"center\"\u003e\n        XXL-CRAWLER, a lightweight web crawler framework.\n        \u003cbr\u003e\n        \u003ca href=\"https://www.xuxueli.com/xxl-crawler/\"\u003e\u003cstrong\u003e-- Home Page --\u003c/strong\u003e\u003c/a\u003e\n        \u003cbr\u003e\n        \u003cbr\u003e\n        \u003ca href=\"https://maven-badges.herokuapp.com/maven-central/com.xuxueli/xxl-crawler/\"\u003e\n            \u003cimg src=\"https://maven-badges.herokuapp.com/maven-central/com.xuxueli/xxl-crawler/badge.svg\" \u003e\n        \u003c/a\u003e\n         \u003ca href=\"https://github.com/xuxueli/xxl-crawler/releases\"\u003e\n             \u003cimg src=\"https://img.shields.io/github/release/xuxueli/xxl-crawler.svg\" \u003e\n         \u003c/a\u003e\n         \u003cimg src=\"https://img.shields.io/github/license/xuxueli/xxl-crawler.svg\" \u003e\n         \u003ca href=\"https://www.xuxueli.com/page/donate.html\"\u003e\n            \u003cimg src=\"https://img.shields.io/badge/%24-donate-ff69b4.svg?style=flat-square\" \u003e\n         \u003c/a\u003e\n    \u003c/p\u003e    \n\u003c/p\u003e\n\n\n## Introduction\n\nXXL-CRAWLER is a lightweight web crawler framework. A line of code to develop a multi-threaded crawler, fully annotated way to collect page data to Java objects, with \"multi-threaded, fully annotated, JS rendering, proxy, distributed extension\" and other features;\n\nXXL-CRAWLER 是一个轻量级Java爬虫框架。一行代码开发一个多线程爬虫，全注解方式采集页面数据至Java对象，拥有\"多线程、全注解、JS渲染、代理、分布式扩展\"等特性；\n\n\n## Documentation\n- [中文文档](https://www.xuxueli.com/xxl-crawler/)\n\n\n## Features\n- 1、简洁：API直观简洁，可快速上手；\n- 2、轻量级：底层实现仅强依赖jsoup，简洁高效；\n- 3、模块化：模块化的结构设计，可轻松扩展；\n- 4、全注解：支持通过注解提取页面数据，高效映射页面数据到PageVO对象，底层自动完成PageVO对象的数据抽取和封装返回；单个页面支持抽取一个或多个PageVO；\n- 5、多线程：线程池方式运行，提高采集效率；\n- 6、扩散全站：支持以现有URL为起点扩散爬取整站；\n- 7、JS渲染：通过扩展 \"PageLoader\" 模块，支持采集JS动态渲染数据。原生提供 Jsoup(非JS渲染，速度更快)、Selenium+ChromeDriver(JS渲染，兼容性高) 等多种实现，支持自由扩展其他实现；\n- 8、代理IP：对抗反采集策略规则WAF；\n- 9、动态代理：支持运行时动态调整代理池，以及自定义代理池路由策略；\n- 10、失败重试：请求失败后重试，并支持设置重试次数；\n- 11、异步：支持同步、异步两种方式运行；\n- 12、幂等去重：防止重复爬取；\n- 13、URL扩散过滤：支持设置页面白名单正则，过滤URL；\n- 14、分布式支持：通过扩展 \"RunUrlPool\" 模块，并结合Redis或DB共享运行数据可实现分布式。默认提供LocalRunUrlPool单机版爬虫；\n- 15、自定义请求信息，如：请求参数、Cookie、Header、UserAgent轮询、Referrer等；\n- 16、动态参数：支持运行时动态调整请求参数；\n- 17、超时控制：支持设置爬虫请求的超时时间；\n- 18、主动停顿：爬虫线程处理完页面之后进行主动停顿，避免过于频繁被拦截；\n\n\n## Example code\n\u003e注意：仅供学习测试使用，如有侵犯请联系删除\n\n如下测试代码可以前往仓库查看：[测试代码目录](https://github.com/xuxueli/xxl-crawler/tree/master/src/test/java/com/xxl/crawler/test)\n\n| 序号 | 爬虫名称                        | 功能描述                                                                                | 测试用例代码文件         |\n|----|-----------------------------|-------------------------------------------------------------------------------------|------------------|\n| 1  | Gitee高星项目数据爬虫【页面提取数据】       | 一行代码启动多线程爬虫，分页方式扩散爬取“Gitee开源项目列表”，通过“注解式”自动提取页面数据，封装成PageVo输出；                      | XxlCrawlerTest01 |\n| 2  | Gitee页面下载爬虫【页面下载】           | 爬取“Gitee开源项目列表”，获取相关页面html原始数据，下载本地生成html文件；                                        | XxlCrawlerTest02 |\n| 3  | 网易图片下载爬虫【图片下载】              | 爬取“网易新闻文章图片”，下载图片文件至本地；                                                             | XxlCrawlerTest03 |\n| 4  | 百度新闻爬虫【接口提取数据】              | 爬取非Web页面，本案例为JSON接口，直接输出响应数据                                                        | XxlCrawlerTest04 |\n| 5  | 电商商品价格爬虫【JS渲染方式；Selenium集成】 | 爬虫获取电商商品价格，由于价格异步渲染；该方案使用 Selenium + ChromeDriver 方式JS渲染，模拟浏览器行为采集数据；               | XxlCrawlerTest05 |\n| 6  | 代理方式爬取数据【Proxy代理方式】         | 爬取目标页面数据，通过代理进行；可突破访问限制、保障数据安全；                                                     | XxlCrawlerTest06 |\n| 7  | 集群方式爬取数据【Redis集群方式】       | 爬取目标页面数据，通过集群方式进行；集群中多个XxlCrawler共享RunUrlPool，协同扩散URL并消费待采集任务，提升采集效率。               | XxlCrawlerTest07 |\n\n\n## Communication\n\n- [社区交流](https://www.xuxueli.com/page/community.html)\n\n\n## Contributing\nContributions are welcome! Open a pull request to fix a bug, or open an [Issue](https://github.com/xuxueli/xxl-crawler/issues/) to discuss a new feature or change.\n\n欢迎参与项目贡献！比如提交PR修复一个bug，或者新建 [Issue](https://github.com/xuxueli/xxl-crawler/issues/) 讨论新特性或者变更。\n\n## 接入登记\n更多接入的公司，欢迎在 [登记地址](https://github.com/xuxueli/xxl-crawler/issues/1 ) 登记，登记仅仅为了产品推广。\n\n\n## Copyright and License\nThis product is open source and free, and will continue to provide free community technical support. Individual or enterprise users are free to access and use.\n\n- Licensed under the Apache License, Version 2.0.\n- Copyright (c) 2015-present, xuxueli.\n\n产品开源免费，并且将持续提供免费的社区技术支持。个人或企业内部可自由的接入和使用。\n\n\n## Donate\nNo matter how much the amount is enough to express your thought, thank you very much ：）     [To donate](https://www.xuxueli.com/page/donate.html )\n\n无论金额多少都足够表达您这份心意，非常感谢 ：）      [前往捐赠](https://www.xuxueli.com/page/donate.html )\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxuxueli%2Fxxl-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxuxueli%2Fxxl-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxuxueli%2Fxxl-crawler/lists"}