Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/wangy8961/python3-concurrency
Python3爬虫系列的理论验证，首先研究I/O模型，分别用Python实现了blocking I/O、nonblocking I/O、I/O multiplexing各模型下的TCP服务端和客户端。然后，研究同步I/O操作（依序下载、多进程并发、多线程并发）和异步I/O（asyncio）之间的效率差别
https://github.com/wangy8961/python3-concurrency
aiohttp asyncio concurrency coroutine futures processing progressbar python3 requests threadpool
Last synced: 9 days ago
JSON representation
Python3爬虫系列的理论验证，首先研究I/O模型，分别用Python实现了blocking I/O、nonblocking I/O、I/O multiplexing各模型下的TCP服务端和客户端。然后，研究同步I/O操作（依序下载、多进程并发、多线程并发）和异步I/O（asyncio）之间的效率差别
- Host: GitHub
- URL: https://github.com/wangy8961/python3-concurrency
- Owner: wangy8961
- Created: 2018-08-27T02:25:35.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2019-07-09T08:05:49.000Z (over 5 years ago)
- Last Synced: 2023-03-02T09:33:33.563Z (over 1 year ago)
- Topics: aiohttp, asyncio, concurrency, coroutine, futures, processing, progressbar, python3, requests, threadpool
- Language: Python
- Homepage: http://www.madmalls.com/blog/category/python3-spider/
- Size: 15.6 KB
- Stars: 52
- Watchers: 2
- Forks: 23
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# [python3-concurrency](http://www.madmalls.com/blog/category/python3-spider/)
[](https://www.python.org/)
[](https://aiohttp.readthedocs.io/en/stable/)
[](https://pypi.org/project/beautifulsoup4/)
[](http://docs.python-requests.org/en/master/)
[](https://pypi.org/project/pymongo/)
[](https://pypi.org/project/progressbar2/)
# 1. 爬虫系列
- [Python 3 爬虫｜第1章：I/O Models 阻塞/非阻塞 同步/异步](https://madmalls.com/blog/post/io-models/)
- [Python 3 爬虫｜第2章：Python 并发编程](https://madmalls.com/blog/post/concurrent-programming-for-python/)
- [Python 3 爬虫｜第3章：同步阻塞下载](https://madmalls.com/blog/post/sequential-download-for-python/)
- [Python 3 爬虫｜第4章：多进程并发下载](https://madmalls.com/blog/post/multi-process-for-python3/)
- [Python 3 爬虫｜第5章：多线程并发下载](https://madmalls.com/blog/post/multi-thread-for-python/)
- [Python 3 爬虫｜第6章：可迭代对象 / 迭代器 / 生成器](https://madmalls.com/blog/post/iterable-iterator-and-generator-in-python/)
- [Python 3 爬虫｜第7章：协程 Coroutines](https://madmalls.com/blog/post/coroutine-in-python/)
- [Python 3 爬虫｜第8章：使用 asyncio 模块实现并发](https://madmalls.com/blog/post/asyncio-howto-in-python3/)
- [Python 3 爬虫｜第9章：使用 asyncio + aiohttp 并发下载](https://madmalls.com/blog/post/aiohttp-howto-in-python3/)
- [Python 3 爬虫｜第10章：爬取少量妹子图](https://madmalls.com/blog/post/python3-concurrency-pics-01/)
- [Python 3 爬虫｜第11章：爬取海量妹子图](https://madmalls.com/blog/post/python3-concurrency-pics-02/)
# 2. 使用方法
## 2.1 Server
为防止DDoS攻击，本次测试需要在本地搭建一个HTTP测试服务器，具体方法参考 [Python3爬虫系列03 (实验) - 同步阻塞下载](http://www.madmalls.com/blog/post/sequential-download-for-python/)
## 2.2 Client
### (1) 下载代码
```bash
[root@CentOS ~]# git clone https://github.com/wangy8961/python3-concurrency.git
[root@CentOS ~]# cd python3-concurrency/
```
### (2) 准备虚拟环境
如果你的操作系统是`Linux`:
```bash
[root@CentOS python3-concurrency]# python3 -m venv venv3
[root@CentOS python3-concurrency]# source venv3/bin/activate
```
> `Windows`激活虚拟环境的命令是: `venv3\Scripts\activate`
### (3) 安装依赖包
如果你的操作系统是`Linux`:
```bash
(venv3) [root@CentOS python3-concurrency]# pip install -r requirements-linux.txt
```
如果你的操作系统是`Windows`（不会使用`uvloop`）:
```bash
(venv3) C:\Users\wangy> pip install -r requirements-win32.txt
```
### (4) 测试
依序下载：
```python
(venv3) [root@CentOS python3-concurrency]# python sequential.py
```
多进程下载：
```python
(venv3) [root@CentOS python3-concurrency]# python processpool.py
```
多线程下载：
```python
(venv3) [root@CentOS python3-concurrency]# python threadpool.py
```
异步下载：
```python
(venv3) [root@CentOS python3-concurrency]# python asynchronous.py
```