{"id":16347063,"url":"https://github.com/ospoon/scrapydemo","last_synced_at":"2025-11-09T02:30:26.752Z","repository":{"id":105557411,"uuid":"110806850","full_name":"OSpoon/ScrapyDemo","owner":"OSpoon","description":"Scrapy 第一步","archived":false,"fork":false,"pushed_at":"2017-12-07T03:00:53.000Z","size":372,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-12-28T06:09:40.563Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OSpoon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-11-15T08:37:22.000Z","updated_at":"2017-11-15T08:37:46.000Z","dependencies_parsed_at":null,"dependency_job_id":"58bae588-dd7f-4d2b-a91b-3299e5823b4d","html_url":"https://github.com/OSpoon/ScrapyDemo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSpoon%2FScrapyDemo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSpoon%2FScrapyDemo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSpoon%2FScrapyDemo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSpoon%2FScrapyDemo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OSpoon","download_url":"https://codeload.github.com/OSpoon/ScrapyDemo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239565664,"owners_count":19660158,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T00:39:28.868Z","updated_at":"2025-11-09T02:30:26.696Z","avatar_url":"https://github.com/OSpoon.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"问题描述\n\n当前环境win10，python_3.6.1，64位。\n在windows下，在dos中运行pip install Scrapy报错：\n\nbuilding 'twisted.test.raiser' extension\nerror: Microsoft Visual C++ 14.0 is required. Get it with \"Microsoft Visual C++ Build Tools\": http://landinghub.visualstudio.com/visual-cpp-build-tools\n解决方案\n\nhttp://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted 下载twisted对应版本的whl文件（如我的Twisted‑17.5.0‑cp36‑cp36m‑win_amd64.whl），cp后面是python版本，amd64代表64位，运行命令：\n\npip install C:\\Users\\CR\\Downloads\\Twisted-17.5.0-cp36-cp36m-win_amd64.whl\n其中install后面为下载的whl文件的完整路径名\n安装完成后，再次运行：\n\npip install Scrapy\n即可成功。\n\nscrapy startproject tutorial\n\ntutorial/\n    scrapy.cfg            # 部署配置文件\n\n    tutorial/             # Python模块,代码写在这个目录下\n        __init__.py\n\n        items.py          # 项目项定义文件\n\n        pipelines.py      # 项目管道文件\n\n        settings.py       # 项目设置文件\n\n        spiders/          # 我们的爬虫/蜘蛛 目录\n            __init__.py\n\n\n如何运行我们爬虫\n\n进入项目根目录，也就是上面的tutorial目录\ncd tutorial\n执行爬虫：\nscrapy crawl quotes\n\n\nMicrosoft Windows [版本 6.1.7601]\n版权所有 (c) 2009 Microsoft Corporation。保留所有权利。\n\nC:\\Users\\zhanxiaolin-n22\\PycharmProjects\\ScrapyDemo\u003escrapy startproject tutorial\n'scrapy' 不是内部或外部命令，也不是可运行的程序\n或批处理文件。\n\nC:\\Users\\zhanxiaolin-n22\\PycharmProjects\\ScrapyDemo\u003escrapy startproject tutorial\n'scrapy' 不是内部或外部命令，也不是可运行的程序\n或批处理文件。\n\nC:\\Users\\zhanxiaolin-n22\\PycharmProjects\\ScrapyDemo\u003escrapy startproject tutorial\n'scrapy' 不是内部或外部命令，也不是可运行的程序\n或批处理文件。\n\nC:\\Users\\zhanxiaolin-n22\\PycharmProjects\\ScrapyDemo\u003eScrapy\n'Scrapy' 不是内部或外部命令，也不是可运行的程序\n或批处理文件。\n\nC:\\Users\\zhanxiaolin-n22\\PycharmProjects\\ScrapyDemo\u003eScrapy\n'Scrapy' 不是内部或外部命令，也不是可运行的程序\n或批处理文件。\n\nC:\\Users\\zhanxiaolin-n22\\PycharmProjects\\ScrapyDemo\u003eScrapy\n'Scrapy' 不是内部或外部命令，也不是可运行的程序\n或批处理文件。\n\nC:\\Users\\zhanxiaolin-n22\\PycharmProjects\\ScrapyDemo\u003escrapy startproject tutorial\nNew Scrapy project 'tutorial', using template directory 'c:\\\\users\\\\zhanxiaolin-n22\\\\appdata\\\\local\\\\programs\\\\python\\\\python36\\\\lib\\\\site-packages\\\\scrapy\\\\templates\\\\project', created in:\n    C:\\Users\\zhanxiaolin-n22\\PycharmProjects\\ScrapyDemo\\tutorial\n\nYou can start your first spider with:\n    cd tutorial\n    scrapy genspider example example.com\n\nC:\\Users\\zhanxiaolin-n22\\PycharmProjects\\ScrapyDemo\u003ecd tutorial\n\nC:\\Users\\zhanxiaolin-n22\\PycharmProjects\\ScrapyDemo\\tutorial\u003escrapy crawl quotes\n2017-11-15 16:17:15 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: tutorial)\n2017-11-15 16:17:15 [scrapy.utils.log] INFO: Overridden settings: {'BOT_NAME': 'tutorial', 'NEWSPIDER_MODULE': 'tutorial.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['tutorial.spiders']}\n2017-11-15 16:17:15 [scrapy.middleware] INFO: Enabled extensions:\n['scrapy.extensions.corestats.CoreStats',\n 'scrapy.extensions.telnet.TelnetConsole',\n 'scrapy.extensions.logstats.LogStats']\n\nUnhandled error in Deferred:\n2017-11-15 16:17:15 [twisted] CRITICAL: Unhandled error in Deferred:\n\n2017-11-15 16:17:15 [twisted] CRITICAL:\nTraceback (most recent call last):\n  File \"c:\\users\\zhanxiaolin-n22\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\twisted\\internet\\defer.py\", line 1386, in _inlineCallbacks\n    result = g.send(result)\n  File \"c:\\users\\zhanxiaolin-n22\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\scrapy\\crawler.py\", line 77, in crawl\n    self.engine = self._create_engine()\n  File \"c:\\users\\zhanxiaolin-n22\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\scrapy\\crawler.py\", line 102, in _create_engine\n    return ExecutionEngine(self, lambda _: self.stop())\n  File \"c:\\users\\zhanxiaolin-n22\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\scrapy\\core\\engine.py\", line 69, in __init__\n    self.downloader = downloader_cls(crawler)\n  File \"c:\\users\\zhanxiaolin-n22\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\scrapy\\core\\downloader\\__init__.py\", line 88, in __init__\n    self.middleware = DownloaderMiddlewareManager.from_crawler(crawler)\n  File \"c:\\users\\zhanxiaolin-n22\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\scrapy\\middleware.py\", line 58, in from_crawler\n    return cls.from_settings(crawler.settings, crawler)\n  File \"c:\\users\\zhanxiaolin-n22\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\scrapy\\middleware.py\", line 34, in from_settings\n    mwcls = load_object(clspath)\n  File \"c:\\users\\zhanxiaolin-n22\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\scrapy\\utils\\misc.py\", line 44, in load_object\n    mod = import_module(module)\n  File \"c:\\users\\zhanxiaolin-n22\\appdata\\local\\programs\\python\\python36\\lib\\importlib\\__init__.py\", line 126, in import_module\n    return _bootstrap._gcd_import(name[level:], package, level)\n  File \"\u003cfrozen importlib._bootstrap\u003e\", line 994, in _gcd_import\n  File \"\u003cfrozen importlib._bootstrap\u003e\", line 971, in _find_and_load\n  File \"\u003cfrozen importlib._bootstrap\u003e\", line 955, in _find_and_load_unlocked\n  File \"\u003cfrozen importlib._bootstrap\u003e\", line 665, in _load_unlocked\n  File \"\u003cfrozen importlib._bootstrap_external\u003e\", line 678, in exec_module\n  File \"\u003cfrozen importlib._bootstrap\u003e\", line 219, in _call_with_frames_removed\n  File \"c:\\users\\zhanxiaolin-n22\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\scrapy\\downloadermiddlewares\\retry.py\", line 20, in \u003cmodule\u003e\n    from twisted.web.client import ResponseFailed\n  File \"c:\\users\\zhanxiaolin-n22\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\twisted\\web\\client.py\", line 42, in \u003cmodule\u003e\n    from twisted.internet.endpoints import HostnameEndpoint, wrapClientTLS\n  File \"c:\\users\\zhanxiaolin-n22\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\twisted\\internet\\endpoints.py\", line 41, in \u003cmodule\u003e\n    from twisted.internet.stdio import StandardIO, PipeAddress\n  File \"c:\\users\\zhanxiaolin-n22\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\twisted\\internet\\stdio.py\", line 30, in \u003cmodule\u003e\n    from twisted.internet import _win32stdio\n  File \"c:\\users\\zhanxiaolin-n22\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\twisted\\internet\\_win32stdio.py\", line 9, in \u003cmodule\u003e\n    import win32api\nModuleNotFoundError: No module named 'win32api'\n\nC:\\Users\\zhanxiaolin-n22\\PycharmProjects\\ScrapyDemo\\tutorial\u003e\n\n    安装win32compat\n\n1. 基本命令\n\n1. scrapy startproject 项目名称\n   - 在当前目录中创建中创建一个项目文件（类似于Django）\n\n2. scrapy genspider [-t template] \u003cname\u003e \u003cdomain\u003e\n   - 创建爬虫应用\n   如：\n      scrapy gensipider -t basic oldboy oldboy.com\n      scrapy gensipider -t xmlfeed autohome autohome.com.cn\n   PS:\n      查看所有命令：scrapy gensipider -l\n      查看模板命令：scrapy gensipider -d 模板名称\n\n3. scrapy list\n   - 展示爬虫应用列表\n\n4. scrapy crawl 爬虫应用名称\n   - 运行单独爬虫应用\n\n\n文件说明：\n\nscrapy.cfg  项目的主配置信息。（真正爬虫相关的配置信息在settings.py文件中）\nitems.py    设置数据存储模板，用于结构化数据，如：Django的Model\npipelines    数据处理行为，如：一般结构化的数据持久化\nsettings.py 配置文件，如：递归的层数、并发数，延迟下载等\nspiders      爬虫目录，如：创建文件，编写爬虫规则","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fospoon%2Fscrapydemo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fospoon%2Fscrapydemo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fospoon%2Fscrapydemo/lists"}