{"id":13632157,"url":"https://github.com/bowenpay/wechat-spider","last_synced_at":"2025-12-29T22:26:00.937Z","repository":{"id":37733343,"uuid":"63926271","full_name":"bowenpay/wechat-spider","owner":"bowenpay","description":"微信公众号爬虫","archived":false,"fork":false,"pushed_at":"2021-08-10T03:59:44.000Z","size":2581,"stargazers_count":3103,"open_issues_count":65,"forks_count":1009,"subscribers_count":206,"default_branch":"master","last_synced_at":"2024-02-15T01:32:27.797Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://wechatspider.0fenbei.com/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bowenpay.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-07-22T05:35:47.000Z","updated_at":"2024-02-07T14:30:52.000Z","dependencies_parsed_at":"2022-07-14T00:50:41.271Z","dependency_job_id":null,"html_url":"https://github.com/bowenpay/wechat-spider","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bowenpay%2Fwechat-spider","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bowenpay%2Fwechat-spider/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bowenpay%2Fwechat-spider/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bowenpay%2Fwechat-spider/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bowenpay","download_url":"https://codeload.github.com/bowenpay/wechat-spider/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249414253,"owners_count":21267724,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T22:02:53.798Z","updated_at":"2025-12-29T22:26:00.908Z","avatar_url":"https://github.com/bowenpay.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"注意：搜狗微信在2019.10.29下线相关功能过后，该项目废弃 ！！！\n\n# 微信爬虫\n一个爬取微信公众号文章的爬虫。 \n\n# 微信爬虫的由来\n零分贝是一家帮助中国5000万贫困人口与社会公益组织对接的公司。\n\n我们通过国家和地方政府的“建档立卡”系统，获取到了一手的贫困户数据，目前有100万左右，总数为5000万，目前每个月都在增长。\n\n为了帮助这部分贫困户对接公益机构，我写了这个微信爬虫，从微信公众号发布的文章中上找出最新的公益项目。\n\n这种找项目的方式的可行性，我们还在试验中。 \n\n起初，为了快速上线，本爬虫的代码是基于我的另一个 [通用爬虫项目](https://github.com/yijingping/unicrawler) 开发的，我也希望任何对本项目感兴趣的人联系我，与我一同改进这个项目。\n\n联系方式：在本项目中新建一个issue留言。\n\n# 界面预览\n\n1） 要爬取的微信公众号列表\n\n![](docs/images/1.jpg?raw=true)\n\n2） 要爬取的文章关键字列表\n\n![](docs/images/2.png?raw=true)\n\n3） 已经爬取的微信文章\n\n![](docs/images/3.png?raw=true)\n\n4） 查看文章，并标记是否可用\n\n![](docs/images/4.jpg?raw=true)\n\n5） 控制爬取进程数\n\n![](docs/images/5.png?raw=true)\n\n\n# 安装\n\n1）python环境, 检查python的版本，是否为2.7.x，如果不是，安装2.7.6。\n\n如果是centos 6.x，升级python2.6到python2.7，参考教程 http://ruiaylin.github.io/2014/12/12/python%20update/\n\n如果是centos 7.x，默认就是python2.7,不用升级\n\n如果是mac osx，可以使用virtualenv，安装python2.7\n\n2）安装依赖包, clone代码\n安装Mysql-python依赖\n```\nyum install python-devel mysql-devel gcc\n```\n\n安装lxml依赖\n```\nyum install libxslt-devel libxml2-devel\n```\n\n安装浏览器环境 selenium依赖.(如果是mac环境，仅需安装firefox， 但确保版本是 firefox 36.0，使用最新的版本会报错)\n```\nyum install xorg-x11-server-Xvfb\nyum upgrade glib2 # 确保glib2版本大于2.42.2，否则firefox启动会报错 \nyum install firefox # centos下安装最新的firefox版本\n```\n\nclone代码,安装依赖python库\n```\n$ git clone https://github.com/bowenpay/wechat-spider.git\n$ cd wechat-spider\n$ pip install -r requirements.txt\n```\n\n3) 创建mysql数据库\n\n创建数据库wechatspider，默认采用utf8编码。（如果系统支持，可以采用utf8mb4，以兼容emoji字符）\n\n```\nmysql\u003e CREATE DATABASE `wechatspider` CHARACTER SET utf8;\n```\n\n4) 安装和运行Redis \n\n```shell\n$ wget http://download.redis.io/releases/redis-2.8.3.tar.gz\n$ tar xzvf redis-2.8.3.tar.gz\n$ cd redis-2.8.3\n$ make\n$ make install\n$ redis-server\n```\n\n5) 更新配置文件local_settings \n\n在 wechatspider 目录下,添加 `local_settings.py` 文件,配置如下:\n```\n# -*- coding: utf-8 -*-\n\nSECRET_KEY=\"xxxxxx\"\n\nCRAWLER_DEBUG = True\n\n# aliyun oss2, 可以将图片和视频存储到阿里云，也可以选择不存储，爬取速度会更快。 默认不存储。\n#OSS2_ENABLE = True\n#OSS2_CONFIG = {\n#    \"ACCESS_KEY_ID\": \"XXXXXXXXXXXXXX\",\n#    \"ACCESS_KEY_SECRET\": \"YYYYYYYYYYYYYYYYYYYYYY\",\n#    \"ENDPOINT\": \"\",\n#    \"BUCKET_DOMAIN\": \"oss-cn-hangzhou.aliyuncs.com\",\n#    \"BUCKET_NAME\": \"XXXXX\",\n#    \"IMAGES_PATH\": \"images/\",\n#    \"VIDEOS_PATH\": \"videos/\",\n#    \"CDN_DOMAIN\": \"XXXXXX.oss-cn-hangzhou.aliyuncs.com\"\n#}\n# mysql 数据库配置\nDATABASES = {\n    'default': {\n        'ENGINE': 'django.db.backends.mysql',\n        'HOST': '127.0.0.1',\n        'NAME': 'wechatspider',\n        'USER': 'root',\n        'PASSWORD': '',\n        'OPTIONS':{\n            'charset': 'utf8mb4',\n        },\n    }\n}\n# redis配置,用于消息队列和k-v存储\nREDIS_OPTIONS = {\n    'host': 'localhost',\n    'port': 6379,\n    'password': '',\n    'db': 4\n}\n\n```\n\n6) 初始化表\n```\n$ python manage.py migrate\n```\n\n7）启动网站\n\n```\npython manage.py runserver 0.0.0.0:8001\n```\n访问 http://localhost:8001/。 \n\n\n6) 创建超级管理员账号,访问后台，并配置要爬取的公众号和关键字\n```\npython manage.py createsuperuser\n```\n\n\n8）启动爬虫\n\n```shell\n$ python bin/scheduler.py\n$ python bin/downloader.py\n$ python bin/extractor.py\n$ python bin/processor.py\n```\n\n以上步骤执行成功，并能爬取文章后。可以参考以下部分配置生产环境。\n\n# 部署nginx\n前期先用nginx将域名www.mydomain.com转发到8001端口。\n\n# 部署supervisor脚本\n参考文件 `supervisord.conf`\n\n# 部署crontab脚本\n参考文件 `crontab`\n\n# 系统使用文档\n\n\n# API接口文档\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbowenpay%2Fwechat-spider","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbowenpay%2Fwechat-spider","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbowenpay%2Fwechat-spider/lists"}