Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/huangcongqing/spider
爬虫python3 (request,BeautifulSoup,xpath,re,Selenium,wordcloud等模块)
https://github.com/huangcongqing/spider
bf4 charles lxml python3 python3x re request requests selenium spider spiders xpath
Last synced: 2 months ago
JSON representation
爬虫python3 (request,BeautifulSoup,xpath,re,Selenium,wordcloud等模块)
- Host: GitHub
- URL: https://github.com/huangcongqing/spider
- Owner: HuangCongQing
- License: mit
- Created: 2018-05-26T13:54:47.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2024-08-24T18:16:49.000Z (4 months ago)
- Last Synced: 2024-10-04T15:35:17.105Z (3 months ago)
- Topics: bf4, charles, lxml, python3, python3x, re, request, requests, selenium, spider, spiders, xpath
- Language: HTML
- Homepage:
- Size: 19.9 MB
- Stars: 15
- Watchers: 3
- Forks: 12
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# spider
python3 各种爬虫技术
**个人爬虫笔记：https://www.yuque.com/huangzhongqing/spider**
@[双愚](https://github.com/HuangCongQing/Spider) , 若fork或star请注明来源
### note笔记
* 爬虫介绍：https://www.yuque.com/docs/share/edb944f3-880a-4a48-a053-df2953be56b4?# 《爬虫基础学习（总结）》
* [notes/01数据爬取requests_note](notes/01数据爬取requests_note)
* [notes/02数据解析note](notes/02数据解析note)
### 模块库
1. [package/1request](package/1request)
2. [package/1request-advanced](package/1request-advanced): cookie&代理
3. [package/2BeautifulSoup4](package/2BeautifulSoup4)
4. [package/3xpath](package/3xpath)
5. [package/4re正则表达式](package/4re正则表达式)
1. [re.findall](package/4re正则表达式/re基础/findall.py)
2. [re.search](package/4re正则表达式/re基础/search.py)
6. [package/5selenium](package/5selenium)
7. [package/6wordcloud&jieba](package/6wordcloud&jieba) 词云
| 功能 | **包名** | **作用** |
| - | - | - |
| 数据获取 | request | 爬取网页 |
| 数据 解析 | re | 正则表达式 |
|
| BeautifulSoup |
|
|
| xpath | xpath语法来进行文件格式解析 |
|
| lxml | lxml库结合libxml2快速强大的特性，使用xpath语法来进行文件格式解析，与Beautiful相比，效率更高。 |
| 模拟浏览器 | Selenium | 用于测试网站的自动化测试工具，支持各种浏览器包括Chrome、Firefox、Safari等主流界面浏览器，同时也支持phantomJS无界面浏览器。模拟点击 |
|
| PhantomJS | 无界面浏览器 |
|
| pandas |
|
|
| jieba | 使用结巴分词进行中文分词 |
|
| pandas |
|
|
| wordcloud | 词云包 |
|
| matplotlib | 绘制图表 |
| | random |
|
[]()[]()
### 通用代码(输出|表格|)
* [common.ipynb](common.ipynb)
### 爬虫实战
1. [practice/01复仇者联盟3豆瓣影评爬虫](practice/01复仇者联盟3豆瓣影评爬虫)
2. [practice/02分析豆瓣中最新电影的影评（词云显示）《超时空同居》](practice/02分析豆瓣中最新电影的影评（词云显示）《超时空同居》)
3. [practice/03王菊微博评论数据抓取jupyter](practice/03王菊微博评论数据抓取jupyter)
4. [practice/04python模拟登录带验证码的网站](practice/04python模拟登录带验证码的网站)
5. [practice/05抓取得到App音频数据](practice/05抓取得到App音频数据)
6. [practice/06python爬取公众号文章](practice/06python爬取公众号文章)
7. [practice/07通过关键词爬取csdn博客文章](practice/07通过关键词爬取csdn博客文章)
8. [practice/08百度搜狗百科关键词爬取](practice/08百度搜狗百科关键词爬取)
9. [practice/09大学排行榜榜单爬取](practice/09大学排行榜榜单爬取)
10. [practice/10bilibili视频爬取下载](practice/10bilibili视频爬取下载)
11.
### 文件操作
读取保存excel，txt等文件
1. [文件操作/excel](文件操作/excel)
2. [文件操作/json](文件操作/json)【todo】
3. [文件操作/txt](文件操作/txt)
### LICENSE
本项目全部内容遵守 MIT 许可协议.