{"id":13464524,"url":"https://github.com/gnemoug/distribute_crawler","last_synced_at":"2026-03-10T22:09:47.667Z","repository":{"id":8080528,"uuid":"9493725","full_name":"gnemoug/distribute_crawler","owner":"gnemoug","description":"使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现","archived":false,"fork":false,"pushed_at":"2017-04-18T08:00:51.000Z","size":12514,"stargazers_count":3257,"open_issues_count":26,"forks_count":1580,"subscribers_count":363,"default_branch":"master","last_synced_at":"2025-12-20T14:04:10.945Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gnemoug.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-04-17T09:14:02.000Z","updated_at":"2025-12-06T03:44:08.000Z","dependencies_parsed_at":"2022-09-13T22:02:01.133Z","dependency_job_id":null,"html_url":"https://github.com/gnemoug/distribute_crawler","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/gnemoug/distribute_crawler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gnemoug%2Fdistribute_crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gnemoug%2Fdistribute_crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gnemoug%2Fdistribute_crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gnemoug%2Fdistribute_crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gnemoug","download_url":"https://codeload.github.com/gnemoug/distribute_crawler/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gnemoug%2Fdistribute_crawler/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30357622,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-10T21:41:54.280Z","status":"ssl_error","status_checked_at":"2026-03-10T21:40:59.357Z","response_time":106,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T14:00:45.530Z","updated_at":"2026-03-10T22:09:47.645Z","avatar_url":"https://github.com/gnemoug.png","language":"Python","funding_links":[],"categories":["All","Python"],"sub_categories":[],"readme":"distribute_crawler\n==================\n\n使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,\n爬虫状态显示使用graphite实现。\n\n这个工程是我对垂直搜索引擎中分布式网络爬虫的探索实现，它包含一个针对http://www.woaidu.org/ 网站的spider，\n将其网站的书名，作者，书籍封面图片，书籍概要，原始网址链接，书籍下载信息和书籍爬取到本地：\n* 分布式使用redis实现，redis中存储了工程的request，stats信息，能够对各个机器上的爬虫实现集中管理，这样可以\n解决爬虫的性能瓶颈，利用redis的高效和易于扩展能够轻松实现高效率下载：当redis存储或者访问速度遇到瓶颈时，可以\n通过增大redis集群数和爬虫集群数量改善。\n* 底层存储实现了两种方式：\n  * 将书名，作者，书籍封面图片文件系统路径，书籍概要，原始网址链接，书籍下载信息，书籍文件系统路径保存到mongodb\n中，此时mongodb使用单个服务器,对图片采用图片的url的hash值作为文件名进行存储，同时可以定制生成各种大小尺寸的缩略\n图，对文件动态获得文件名，将其下载到本地，存储方式和图片类似，这样在每次下载之前会检查图片和文件是否曾经下载，对\n已经下载的不再下载；\n  * 将书名，作者，书籍封面图片文件系统路径，书籍概要，原始网址链接，书籍下载信息，书籍保存到mongodb中，此时mongodb\n采用mongodb集群进行存储，片键和索引的选择请看代码，文件采用mongodb的gridfs存储,图片仍然存储在文件系统中,在每次下载\n之前会检查图片和文件是否曾经下载，对已经下载的不再下载；\n* 避免爬虫被禁的策略：\n  * 禁用cookie\n  * 实现了一个download middleware，不停的变user-aget\n  * 实现了一个可以访问google cache中的数据的download middleware(默认禁用)\n* 调试策略的实现：\n  * 将系统log信息写到文件中\n  * 对重要的log信息(eg:drop item,success)采用彩色样式终端打印\n* 文件，信息存储：\n  * 实现了FilePipeline可以将指定扩展名的文件下载到本地\n  * 实现了MongodbWoaiduBookFile可以将文件以gridfs形式存储在mongodb集群中\n  * 实现了SingleMongodbPipeline和ShardMongodbPipeline，用来将采集的信息分别以单服务器和集群方式保存到mongodb中\n* 访问速度动态控制:\n  * 跟据网络延迟，分析出scrapy服务器和网站的响应速度，动态改变网站下载延迟\n  * 配置最大并行requests个数，每个域名最大并行请求个数和并行处理items个数\n* 爬虫状态查看：\n  * 将爬虫stats信息(请求个数，文件下载个数，图片下载个数等)保存到redis中\n  * 实现了一个针对分布式的stats collector，并将其结果用graphite以图表形式动态实时显示\n* mongodb集群部署：在commands目录下有init_sharding_mongodb.py文件，可以方便在本地部署\n\n需要的其他的库\n==============\n\n* scrapy(最好是最新版)\n* graphite(针对他的配置可以参考：statscol/graphite.py)\n* redis\n* mongodb\n\n可重用的组件\n============\n\n* 终端彩色样式显示(utils/color.py)\n* 在本地建立一个mongodb集群(commands/init_sharding_mongodb.py),使用方法:\n```\n   sudo python init_sharding_mongodb.py --path=/usr/bin\n```\n* 单机graphite状态收集器(statscol.graphite.GraphiteStatsCollector)\n* 基于redis分布式的graphite状态收集器(statscol.graphite.RedisGraphiteStatsCollector)\n* scrapy分布式处理方案(scrapy_redis)\n* rotate user-agent download middleware(contrib.downloadmiddleware.rotate_useragent.RotateUserAgentMiddleware)\n* 访问google cache的download middleware(contrib.downloadmiddleware.google_cache.GoogleCacheMiddleware)\n* 下载指定文件类型的文件并实现避免重复下载的pipeline(pipelines.file.FilePipeline)\n* 下载制定文件类型的文件并提供mongodb gridfs存储的pipeline(pipelines.file.MongodbWoaiduBookFile)\n* item mongodb存储的pipeline(pipelines.mongodb.SingleMongodbPipeline and ShardMongodbPipeline)\n\n使用方法\n========\n\n#mongodb集群存储\n* 安装scrapy\n* 安装redispy\n* 安装pymongo\n* 安装graphite(如何配置请查看：statscol/graphite.py)\n* 安装mongodb\n* 安装redis\n* 下载本工程\n* 启动redis server\n* 搭建mongodb集群\n```\n      cd woaidu_crawler/commands/\n      sudo python init_sharding_mongodb.py --path=/usr/bin\n```\n* 在含有log文件夹的目录下执行:\n```\n      scrapy crawl woaidu\n```\n* 打开http://127.0.0.1/ 通过图表查看spider实时状态信息\n* 要想尝试分布式，可以在另外一个目录运行此工程\n\n#mongodb\n* 安装scrapy\n* 安装redispy\n* 安装pymongo\n* 安装graphite(如何配置请查看：statscol/graphite.py)\n* 安装mongodb\n* 安装redis\n* 下载本工程\n* 启动redis server\n* 搭建mongodb服务器\n```\n      cd woaidu_crawler/commands/\n      python init_single_mongodb.py \n```\n* 设置settings.py：\n\n```python\n      ITEM_PIPELINES = ['woaidu_crawler.pipelines.cover_image.WoaiduCoverImage',\n          'woaidu_crawler.pipelines.bookfile.WoaiduBookFile',\n          'woaidu_crawler.pipelines.drop_none_download.DropNoneBookFile',\n          'woaidu_crawler.pipelines.mongodb.SingleMongodbPipeline',\n          'woaidu_crawler.pipelines.final_test.FinalTestPipeline',]\n```\n* 在含有log文件夹的目录下执行:\n```\n      scrapy crawl woaidu\n```\n* 打开http://127.0.0.1/ (也就是你运行的graphite-web的url) 通过图表查看spider实时状态信息\n* 要想尝试分布式，可以在另外一个目录运行此工程\n\n注意\n====\n\n每次运行完之后都要执行commands/clear_stats.py文件来清除redis中的stats信息\n```\n     python clear_stats.py\n```\n\nScreenshots\n===========\n\n![graphite](https://raw.github.com/gnemoug/distribute_crawler/master/woaidu_crawler/woaidu_crawler/screenshots/graphite/Screenshot-2013-04-17%2016:32:51.png)\n![book_files](https://raw.github.com/gnemoug/distribute_crawler/master/woaidu_crawler/woaidu_crawler/screenshots/media/book_files/Screenshot-2013-04-17%2016:50:29.png)\n![book_files](https://raw.github.com/gnemoug/distribute_crawler/master/woaidu_crawler/woaidu_crawler/screenshots/media/book_files/Screenshot-2013-04-17%2016:52:03.png)\n![images](https://raw.github.com/gnemoug/distribute_crawler/master/woaidu_crawler/woaidu_crawler/screenshots/media/image/Screenshot-2013-04-17%2016:48:15.png)\n![terminal](https://raw.github.com/gnemoug/distribute_crawler/master/woaidu_crawler/woaidu_crawler/screenshots/runing/Screenshot-2013-04-17%2016:13:03.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgnemoug%2Fdistribute_crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgnemoug%2Fdistribute_crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgnemoug%2Fdistribute_crawler/lists"}