{"id":19569583,"url":"https://github.com/wangy8961/python3-concurrency","last_synced_at":"2025-09-11T04:37:13.925Z","repository":{"id":49376671,"uuid":"146236511","full_name":"wangy8961/python3-concurrency","owner":"wangy8961","description":"Python3爬虫系列的理论验证，首先研究I/O模型，分别用Python实现了blocking I/O、nonblocking I/O、I/O multiplexing各模型下的TCP服务端和客户端。然后，研究同步I/O操作（依序下载、多进程并发、多线程并发）和异步I/O（asyncio）之间的效率差别","archived":false,"fork":false,"pushed_at":"2019-07-09T08:05:49.000Z","size":16,"stargazers_count":56,"open_issues_count":0,"forks_count":27,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-15T12:05:04.508Z","etag":null,"topics":["aiohttp","asyncio","concurrency","coroutine","futures","processing","progressbar","python3","requests","threadpool"],"latest_commit_sha":null,"homepage":"http://www.madmalls.com/blog/category/python3-spider/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wangy8961.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-08-27T02:25:35.000Z","updated_at":"2025-04-14T02:56:45.000Z","dependencies_parsed_at":"2022-08-25T13:30:30.457Z","dependency_job_id":null,"html_url":"https://github.com/wangy8961/python3-concurrency","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/wangy8961/python3-concurrency","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wangy8961%2Fpython3-concurrency","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wangy8961%2Fpython3-concurrency/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wangy8961%2Fpython3-concurrency/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wangy8961%2Fpython3-concurrency/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wangy8961","download_url":"https://codeload.github.com/wangy8961/python3-concurrency/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wangy8961%2Fpython3-concurrency/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274578317,"owners_count":25310955,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-11T02:00:13.660Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aiohttp","asyncio","concurrency","coroutine","futures","processing","progressbar","python3","requests","threadpool"],"created_at":"2024-11-11T06:11:08.395Z","updated_at":"2025-09-11T04:37:13.850Z","avatar_url":"https://github.com/wangy8961.png","language":"Python","readme":"# [python3-concurrency](http://www.madmalls.com/blog/category/python3-spider/)\n\n[![Python](https://img.shields.io/badge/python-v3.4%2B-blue.svg)](https://www.python.org/)\n[![aiohttp](https://img.shields.io/badge/aiohttp-v3.3.2-brightgreen.svg)](https://aiohttp.readthedocs.io/en/stable/)\n[![BeautifulSoup4](https://img.shields.io/badge/BeautifulSoup4-v4.6.3-orange.svg)](https://pypi.org/project/beautifulsoup4/)\n[![requests](https://img.shields.io/badge/requests-v2.19.1-yellow.svg)](http://docs.python-requests.org/en/master/)\n[![pymongo](https://img.shields.io/badge/pymongo-v3.7.1-red.svg)](https://pypi.org/project/pymongo/)\n[![progressbar2](https://img.shields.io/badge/progressbar2-v3.38.0-lightgrey.svg)](https://pypi.org/project/progressbar2/)\n\n\n![](http://www.madmalls.com/api/medias/uploaded/python3-concurrency-1016d526.png)\n\n\n# 1. 爬虫系列\n\n- [Python 3 爬虫｜第1章：I/O Models 阻塞/非阻塞 同步/异步](https://madmalls.com/blog/post/io-models/)\n- [Python 3 爬虫｜第2章：Python 并发编程](https://madmalls.com/blog/post/concurrent-programming-for-python/)\n- [Python 3 爬虫｜第3章：同步阻塞下载](https://madmalls.com/blog/post/sequential-download-for-python/)\n- [Python 3 爬虫｜第4章：多进程并发下载](https://madmalls.com/blog/post/multi-process-for-python3/)\n- [Python 3 爬虫｜第5章：多线程并发下载](https://madmalls.com/blog/post/multi-thread-for-python/)\n- [Python 3 爬虫｜第6章：可迭代对象 / 迭代器 / 生成器](https://madmalls.com/blog/post/iterable-iterator-and-generator-in-python/)\n- [Python 3 爬虫｜第7章：协程 Coroutines](https://madmalls.com/blog/post/coroutine-in-python/)\n- [Python 3 爬虫｜第8章：使用 asyncio 模块实现并发](https://madmalls.com/blog/post/asyncio-howto-in-python3/)\n- [Python 3 爬虫｜第9章：使用 asyncio + aiohttp 并发下载](https://madmalls.com/blog/post/aiohttp-howto-in-python3/)\n- [Python 3 爬虫｜第10章：爬取少量妹子图](https://madmalls.com/blog/post/python3-concurrency-pics-01/)\n- [Python 3 爬虫｜第11章：爬取海量妹子图](https://madmalls.com/blog/post/python3-concurrency-pics-02/)\n\n\n# 2. 使用方法\n\n## 2.1 Server\n\n为防止DDoS攻击，本次测试需要在本地搭建一个HTTP测试服务器，具体方法参考 [Python3爬虫系列03 (实验) - 同步阻塞下载](http://www.madmalls.com/blog/post/sequential-download-for-python/)\n\n## 2.2 Client\n\n### (1) 下载代码\n\n```bash\n[root@CentOS ~]# git clone https://github.com/wangy8961/python3-concurrency.git\n[root@CentOS ~]# cd python3-concurrency/\n```\n\n### (2) 准备虚拟环境\n\n如果你的操作系统是`Linux`:\n\n```bash\n[root@CentOS python3-concurrency]# python3 -m venv venv3\n[root@CentOS python3-concurrency]# source venv3/bin/activate\n```\n\n\u003e `Windows`激活虚拟环境的命令是: `venv3\\Scripts\\activate`\n\n### (3) 安装依赖包\n\n如果你的操作系统是`Linux`:\n\n```bash\n(venv3) [root@CentOS python3-concurrency]# pip install -r requirements-linux.txt\n```\n\n如果你的操作系统是`Windows`（不会使用`uvloop`）:\n\n```bash\n(venv3) C:\\Users\\wangy\u003e pip install -r requirements-win32.txt\n```\n\n### (4) 测试\n\n依序下载：\n\n```python\n(venv3) [root@CentOS python3-concurrency]# python sequential.py\n```\n\n多进程下载：\n\n```python\n(venv3) [root@CentOS python3-concurrency]# python processpool.py\n```\n\n多线程下载：\n\n```python\n(venv3) [root@CentOS python3-concurrency]# python threadpool.py\n```\n\n异步下载：\n\n```python\n(venv3) [root@CentOS python3-concurrency]# python asynchronous.py\n```","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwangy8961%2Fpython3-concurrency","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwangy8961%2Fpython3-concurrency","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwangy8961%2Fpython3-concurrency/lists"}