https://github.com/cwjokaka/bilibili_member_crawler

B站用户爬虫 好耶~是爬虫
https://github.com/cwjokaka/bilibili_member_crawler

bilibili crawler multithreading mysql python python3 queue requests spider web

Last synced: 13 days ago
JSON representation

B站用户爬虫 好耶~是爬虫

• Host: GitHub
• URL: https://github.com/cwjokaka/bilibili_member_crawler
• Owner: cwjokaka
• License: mit
• Created: 2019-07-13T08:52:46.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2022-12-08T05:52:42.000Z (almost 2 years ago)
• Last Synced: 2024-10-13T09:42:59.593Z (27 days ago)
• Topics: bilibili, crawler, multithreading, mysql, python, python3, queue, requests, spider, web
• Language: Python
• Homepage:
• Size: 33.2 KB
• Stars: 147
• Watchers: 3
• Forks: 21
• Open Issues: 3
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-seeds - bilibili_member_crawler

README

        
# bilibili_member_crawler 

B站用户信息爬虫 (求Star\(^o^)/~

仅供娱乐学习使用

### 环境

* python 3.6+

* mysql 5.7+

### 下载安装

* 下载源码:

```shell

git clone [email protected]:cwjokaka/bilibili_member_crawler.git

或者在https://github.com/cwjokaka/bilibili_member_crawler 下载zip文件

```

* 安装相关依赖:

```shell

pip install -r requirements.txt

```

### 文件介绍

* `bilibili_member_crawler.py`：爬虫入口

* `distributor.py`：任务分发器,负责生成任务到任务队列

* `worker.py`：工作线程,负责从任务队列拉取任务,并把B站用户信息持久化到mysql

* `res_manager.py`：资源管理,用于管理任务队列

* `variable.py`: 配置文件, 包含代理、数据库、线程设置等

* `sql/bilibili.sql`：数据库初始化文件

* `user-agents.txt`：浏览器agent列表文件

* `exception/*`：各类异常

### 注意

* 请控制好您的车速(由variable.py的线程数、爬取时间间隔决定)

* 代理PROXIES需要定期更换(2019/9/4已加入外部代理池)

PS:有时间会做相关统计 (溜了溜了