{"id":16099368,"url":"https://github.com/googtech/crawler-learning","last_synced_at":"2025-03-18T07:31:01.687Z","repository":{"id":37292594,"uuid":"195831871","full_name":"GoogTech/crawler-learning","owner":"GoogTech","description":"🕷 一个基于 HttpClient，Jsoup，WebMagic 的迷你版 JD 商城图书爬虫 ~","archived":false,"fork":false,"pushed_at":"2022-11-08T07:32:12.000Z","size":700,"stargazers_count":8,"open_issues_count":8,"forks_count":4,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-02-28T07:55:01.655Z","etag":null,"topics":["httpclient","jsoup","mybatis","mysql","springboot"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GoogTech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-07-08T14:43:02.000Z","updated_at":"2023-06-30T03:22:41.000Z","dependencies_parsed_at":"2023-01-21T10:34:59.755Z","dependency_job_id":null,"html_url":"https://github.com/GoogTech/crawler-learning","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogTech%2Fcrawler-learning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogTech%2Fcrawler-learning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogTech%2Fcrawler-learning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogTech%2Fcrawler-learning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GoogTech","download_url":"https://codeload.github.com/GoogTech/crawler-learning/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243910715,"owners_count":20367538,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["httpclient","jsoup","mybatis","mysql","springboot"],"created_at":"2024-10-09T18:27:03.058Z","updated_at":"2025-03-18T07:31:00.546Z","avatar_url":"https://github.com/GoogTech.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"## 🕷 JD商城图书爬虫\n\n\n### 版本介绍\n#### JDBookCrawler 1.0\n*涉及技术 : `HttpClient`，`Jsoup`，`MySQL`，详细介绍及使用指南见 : [`博客文章`](https://yubuntu0109.github.io/2019/07/14/%E5%B0%8F%E7%88%AC%E8%99%AB-JDBookCrawler-V1-0/)*\n\n#### JDBookCrawler 2.0\n*涉及技术 : `WebMagic`，`MyBatis`，`MySQL`，详细介绍及使用指南见 : [`博客文章`](https://yubuntu0109.github.io/2019/07/15/%E5%B0%8F%E7%88%AC%E8%99%AB-JDBookCrawler-V2-0/)*\n\n#### JDBookCrawler 3.0\n*涉及技术 : `Spring Boot`，`WebMagic`，`MyBatis`，`MySQL`，爬虫功能基于`JDBookCrawler 2.0`，前端设计参考项目[`springboot-beginner`](https://github.com/YUbuntu0109/springboot-beginner)，简单点说`v3.0`就等于`v2.0` + `springboot`，😅不尴尬嘿嘿嘿~*\n\n\n\n### 开发环境\n| 工具    | 版本或描述                |    \n| ------- | ------------------------ |    \n| `OS`    | Windows 10               | \n| `JDK`   |  1.8                     |    \n| `IDE`   | IntelliJ IDEA 2019.1     |    \n| `Maven` | 3.6.0                    |    \n| `MySQL` | 8.0.11                   |\n\n\u003e 本项目的数据库版本为`8.0.11`，请广大版本为`5.0.0+`的同学注意咯：可通过逐个复制表结构来创建该数据库哟 ~\n\n\n\n### 项目概述( `JDBookCrawler 3.0` )\n\u003e 除此项目概述外，你也可以参考 : [博客文章](https://yubuntu0109.github.io/2019/07/17/%E5%B0%8F%E7%88%AC%E8%99%AB-JDBookCrawler-V3-0/)\n\n#### 项目结构图\n```\n├─bookcrawler-v3.0\n│  │  crawler.sql\n│  │  pom.xml\n│  │\n│  │\n│  └─src\n│      └─main\n│          ├─java\n│          │  └─pers\n│          │      └─huangyuhui\n│          │          └─bookcrawler\n│          │              │  BookcrawlerApplication.java\n│          │              │\n│          │              ├─controller\n│          │              │      BookController.java\n│          │              │\n│          │              ├─crawler\n│          │              │  │  BookCrawlerTest.java\n│          │              │  │\n│          │              │  ├─dao\n│          │              │  │      BookDao.java\n│          │              │  │\n│          │              │  ├─mapper\n│          │              │  │      BookMapper.xml\n│          │              │  │\n│          │              │  ├─pojo\n│          │              │  │      Book.java\n│          │              │  │\n│          │              │  ├─resources\n│          │              │  │      db.properties\n│          │              │  │      log4j.properties\n│          │              │  │      mybatis-config.xml\n│          │              │  │\n│          │              │  ├─task\n│          │              │  │      BookProcessor.java\n│          │              │  │\n│          │              │  └─util\n│          │              │          FileUtils.java\n│          │              │          HttpUtils.java\n│          │              │          MyBatisUtils.java\n│          │              │\n│          │              ├─dao\n│          │              │      BookMapper.java\n│          │              │      BookMapper.xml\n│          │              │\n│          │              ├─pojo\n│          │              │      Book.java\n│          │              │\n│          │              └─service\n│          │                  │  BookService.java\n│          │                  │\n│          │                  └─impl\n│          │                          BookServiceImpl.java\n│          │\n│          └─resources\n│              │  application.properties\n│              │\n│              ├─static\n│              │  │  exist.txt\n│              │  │\n│              │  └─easyui\n│              │      │  jquery.easyui.min.js\n│              │      │  jquery.min.js\n│              │      │\n│              │      ├─css\n│              │      │      default.css\n│              │      │      demo.css\n│              │      │\n│              │      ├─js\n│              │      │      outlook2.js\n│              │      │      validateExtends.js\n│              │      │\n│              │      └─themes\n|              |         |\n│              │         |(略..)\n│              │\n|              |        \n│              └─templates\n│                      bookList.html\n│                      intro.html\n│                      main.html\n│\n└─demonstration-images\n        BookCrawler-V3.0-bookList.PNG\n        BookCrawler-V3.0-Intro.PNG\n```\n\n\u003e `crawler`包存放的是`JDBookCrawler-v2.0`，既`3.0`使用了`2.0`的爬虫功能，其本身仅加入了页面可视化功能\n\n\n\n#### 运行指南\n1. *crawler.sql : 数据库文件*\n```\nBookCrawler-v3.0/bookcrawler-v3.0/crawler.sql\n```\n\n2. *BookCrawlerTest.java : 爬虫启动程序*\n```java\npackage pers.huangyuhui.bookcrawler.crawler;\n\n\nimport pers.huangyuhui.bookcrawler.crawler.task.BookProcessor;\nimport pers.huangyuhui.bookcrawler.crawler.util.FileUtils;\n\n/**\n * @project: bookcrawler\n * @description: 爬虫测试程序\n * @author: 黄宇辉\n * @date: 7/11/2019-9:12 PM\n * @version: 2.0\n * @website: https://yubuntu0109.github.io/\n */\npublic class BookCrawlerTest {\n    //指定图书关键字\n    private static final String KEY_WORD = \"网络爬虫\";\n    //指定页码数,每页可爬取三十条数据( 注:下一页的页码数为前一页的页码数加二 )\n    private static final int END_PAGE_NUM = 2;\n    private static final int CURRENT_PAGE_NUM = 1;\n    //指定项目下存储书籍图片的文件夹路径\n    private static final String IMAGE_PATH = FileUtils.getDirPath(\"/static/download/bookImage/\");\n    //指定JD商城书籍列表页面的链接\n    private static final String URL = \"https://search.jd.com/Search?keyword=\" + KEY_WORD + \"\u0026enc=utf-8\u0026page=\";\n\n    /**\n     * @description: 启动爬虫程序\n     * @date: 2019-07-15 4:09 PM\n     */\n    public static void main(String[] args) {\n        new BookProcessor(URL, IMAGE_PATH, CURRENT_PAGE_NUM, END_PAGE_NUM).run();\n    }\n}\n```\n\n\u003e 启动爬虫后其控制台输出的日志信息请参考 : [`JDBookCrawler v2.0`](https://yubuntu0109.github.io/2019/07/15/%E5%B0%8F%E7%88%AC%E8%99%AB-JDBookCrawler-V2-0/)\n\n3. *BookcrawlerApplication.java : 项目启动类( 爬虫程序成功运行完成后，就可以启动该`springboot`项目啦，这操作不尴尬哈哈哈 ~ )*\n```java\npackage pers.huangyuhui.bookcrawler;\n\nimport org.mybatis.spring.annotation.MapperScan;\nimport org.springframework.boot.SpringApplication;\nimport org.springframework.boot.autoconfigure.SpringBootApplication;\n\n/**\n * @project: bookcrawler\n * @description: springboot项目启动类\n * @author: 黄宇辉\n * @date: 7/11/2019-9:16 PM\n * @version: 3.0\n * @website: https://yubuntu0109.github.io/\n */\n@SpringBootApplication\n@MapperScan(\"pers.huangyuhui.bookcrawler.dao\") //扫描Mapper接口\npublic class BookcrawlerApplication {\n\n    public static void main(String[] args) { SpringApplication.run(BookcrawlerApplication.class, args); }\n}\n```\n\n4. *项目成功启动后，其项目主页，图书数据管理页如下图所示 :*\n\n![](https://raw.githubusercontent.com/YUbuntu0109/Crawler-learning/master/BookCrawler-v3.0/demonstration-images/BookCrawler-V3.0-Intro.PNG)\n\n![](https://raw.githubusercontent.com/YUbuntu0109/Crawler-learning/master/BookCrawler-v3.0/demonstration-images/BookCrawler-V3.0-bookList.PNG)\n\n\n\n*:books:更多有趣项目及详细学习笔记请前往我的个人博客哟（づ￣3￣）づ╭❤～ : https://yubuntu0109.github.io/*\n\n*👩‍💻学习笔记已全部开源 : https://github.com/YUbuntu0109/YUbuntu0109.github.io*\n \n*:coffee: Look forward to your contribution, if you need any help, please contact me~ QQ : 3083968068*\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogtech%2Fcrawler-learning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogtech%2Fcrawler-learning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogtech%2Fcrawler-learning/lists"}