{"id":19569579,"url":"https://github.com/wangy8961/python3-concurrency-pics-01","last_synced_at":"2025-06-10T10:34:19.175Z","repository":{"id":96848061,"uuid":"146240439","full_name":"wangy8961/python3-concurrency-pics-01","owner":"wangy8961","description":"爬虫多线程或异步下载 http://gank.io/api/data/%E7%A6%8F%E5%88%A9/1000/1 所分享的美女图片","archived":false,"fork":false,"pushed_at":"2019-06-14T04:04:51.000Z","size":11,"stargazers_count":12,"open_issues_count":0,"forks_count":7,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-04T20:36:15.572Z","etag":null,"topics":["aiohhtp","asyncio","coroutine","crawler","progressbar","python3","requests","threadpool"],"latest_commit_sha":null,"homepage":"http://www.madmalls.com/blog/post/python3-concurrency-pics-01/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wangy8961.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-08-27T03:07:00.000Z","updated_at":"2025-03-20T16:49:43.000Z","dependencies_parsed_at":"2023-05-09T14:46:25.513Z","dependency_job_id":null,"html_url":"https://github.com/wangy8961/python3-concurrency-pics-01","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wangy8961%2Fpython3-concurrency-pics-01","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wangy8961%2Fpython3-concurrency-pics-01/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wangy8961%2Fpython3-concurrency-pics-01/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wangy8961%2Fpython3-concurrency-pics-01/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wangy8961","download_url":"https://codeload.github.com/wangy8961/python3-concurrency-pics-01/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251085147,"owners_count":21533821,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aiohhtp","asyncio","coroutine","crawler","progressbar","python3","requests","threadpool"],"created_at":"2024-11-11T06:11:06.965Z","updated_at":"2025-04-27T03:30:47.155Z","avatar_url":"https://github.com/wangy8961.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# [python3-concurrency-pics-01](https://madmalls.com/blog/post/python3-concurrency-pics-01/)\n\n[![Python](https://img.shields.io/badge/python-v3.4%2B-blue.svg)](https://www.python.org/)\n[![aiohttp](https://img.shields.io/badge/aiohttp-v3.3.2-brightgreen.svg)](https://aiohttp.readthedocs.io/en/stable/)\n[![BeautifulSoup4](https://img.shields.io/badge/BeautifulSoup4-v4.6.3-orange.svg)](https://pypi.org/project/beautifulsoup4/)\n[![requests](https://img.shields.io/badge/requests-v2.19.1-yellow.svg)](http://docs.python-requests.org/en/master/)\n[![pymongo](https://img.shields.io/badge/pymongo-v3.7.1-red.svg)](https://pypi.org/project/pymongo/)\n[![progressbar2](https://img.shields.io/badge/progressbar2-v3.38.0-lightgrey.svg)](https://pypi.org/project/progressbar2/)\n\n\n\n# 1. 进度条\n\n![](https://wx3.sinaimg.cn/large/007xgOh4ly1g40jnjpn5hj30s504tjrs.jpg)\n\n# 2. 截图\n\n![](https://madmalls.com/api/medias/uploaded/gank-02-643b167c.png)\n\n# 3. 使用方法\n\n## 3.1 下载代码\n\n```bash\n[root@CentOS ~]# git clone https://github.com/wangy8961/python3-concurrency-pics-01.git\n[root@CentOS ~]# cd python3-concurrency-pics-01/\n```\n\n## 3.2 准备虚拟环境\n\n如果你的操作系统是`Linux`:\n\n```bash\n[root@CentOS python3-concurrency-pics-01]# python3 -m venv venv3\n[root@CentOS python3-concurrency-pics-01]# source venv3/bin/activate\n```\n\n\u003e `Windows`激活虚拟环境的命令是: `venv3\\Scripts\\activate`\n\n## 3.3 安装依赖包\n\n如果你的操作系统是`Linux`:\n\n```bash\n(venv3) [root@CentOS python3-concurrency-pics-01]# pip install -r requirements-linux.txt\n```\n\n如果你的操作系统是`Windows`（不会使用`uvloop`）:\n\n```bash\n(venv3) C:\\Users\\wangy\u003e pip install -r requirements-win32.txt\n```\n\n## 3.4 测试\n\n### (1) 依序下载\n\n```python\n(venv3) [root@CentOS python3-concurrency-pics-01]# python sequential.py\n```\n\n### (2) 多线程下载\n\n```python\n(venv3) [root@CentOS python3-concurrency-pics-01]# python threadpool.py\n```\n\n### (3) 异步下载\n\n```python\n(venv3) [root@CentOS python3-concurrency-pics-01]# python asynchronous.py\n```\n\n# 4. 爬虫系列\n\n## 4.1 理论\n\n- [Python3爬虫系列01 (理论) - I/O Models 阻塞 非阻塞 同步 异步](https://madmalls.com/blog/post/io-models/)\n- [Python3爬虫系列02 (理论) - Python并发编程](https://madmalls.com/blog/post/concurrent-programming-for-python/)\n- [Python3爬虫系列06 (理论) - 可迭代对象、迭代器、生成器](https://madmalls.com/blog/post/iterable-iterator-and-generator-in-python/)\n- [Python3爬虫系列07 (理论) - 协程](https://madmalls.com/blog/post/coroutine-in-python/)\n- [Python3爬虫系列08 (理论) - 使用asyncio模块实现并发](https://madmalls.com/blog/post/asyncio-howto-in-python3/)\n\n\n## 4.2 实验\n\n- [Python3爬虫系列03 (实验) - 同步阻塞下载](https://madmalls.com/blog/post/sequential-download-for-python/)\n- [Python3爬虫系列04 (实验) - 多进程并发下载](https://madmalls.com/blog/post/multi-process-for-python3/)\n- [Python3爬虫系列05 (实验) - 多线程并发下载](https://madmalls.com/blog/post/multi-thread-for-python/)\n- [Python3爬虫系列09 (实验) - 使用asyncio+aiohttp并发下载](https://madmalls.com/blog/post/aiohttp-howto-in-python3/)\n\n\n## 4.3 实战\n\n- [Python3爬虫系列10 (实战) - 爬取妹子图 第一弹](https://madmalls.com/blog/post/python3-concurrency-pics-01/)\n- [Python3爬虫系列11 (实战) - 爬取妹子图 第二弹](https://madmalls.com/blog/post/python3-concurrency-pics-02/)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwangy8961%2Fpython3-concurrency-pics-01","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwangy8961%2Fpython3-concurrency-pics-01","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwangy8961%2Fpython3-concurrency-pics-01/lists"}