{"id":13488681,"url":"https://github.com/szcf-weiya/SinaSpider","last_synced_at":"2025-03-28T01:37:16.572Z","repository":{"id":111915675,"uuid":"81031736","full_name":"szcf-weiya/SinaSpider","owner":"szcf-weiya","description":"动态IP解决新浪的反爬虫机制，快速抓取内容。","archived":false,"fork":false,"pushed_at":"2017-09-10T00:05:41.000Z","size":5334,"stargazers_count":141,"open_issues_count":0,"forks_count":54,"subscribers_count":15,"default_branch":"master","last_synced_at":"2024-08-01T18:39:15.443Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://zhuanlan.zhihu.com/p/25448083","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/szcf-weiya.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-02-06T00:12:37.000Z","updated_at":"2024-07-18T07:30:44.000Z","dependencies_parsed_at":null,"dependency_job_id":"37d68f76-ed9d-44ad-8807-4d058c18db5a","html_url":"https://github.com/szcf-weiya/SinaSpider","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/szcf-weiya%2FSinaSpider","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/szcf-weiya%2FSinaSpider/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/szcf-weiya%2FSinaSpider/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/szcf-weiya%2FSinaSpider/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/szcf-weiya","download_url":"https://codeload.github.com/szcf-weiya/SinaSpider/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222333976,"owners_count":16968058,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T18:01:19.981Z","updated_at":"2025-03-28T01:37:16.566Z","avatar_url":"https://github.com/szcf-weiya.png","language":"Python","funding_links":[],"categories":["Python","Python (1887)"],"sub_categories":[],"readme":"# SinaSpider\n动态IP解决新浪的反爬虫机制，快速抓取微博内容。\n\n## Background\n抓取1000个公司（在companyList.py文件中）五年内相关的微博，进而统计评论数、转发数、点赞数等等。\n\n## Environment\n- Python2.7\n- winxp服务器（通过某宝购买，关键是ADSL拨号功能，不然无法实现动态IP，也就解决不了新浪的反爬虫机制）\n\n## Results\n- 每个公司五年内的微博（通过sqlite3存储）\n\n  下面截图为company0000.db的微博。\n\n  ![](db.png)\n\n- 所有公司微博评论数、转发数、点赞数的统计（excel形式呈现）\n  ![](results.png)\n\n## References\n\n刚刚(Sun Sep 10 07:51:46 CST 2017)在整理浏览器的书签，因为自己习惯性会把觉得有用的网页存储为书签，所以日积月累，书签的数量已经十分庞大，决定清理一下。清理之前把那些与此项目的网页书签贴在这里吧\n\n1. [模拟登录新浪微博（Python)](https://www.douban.com/note/201767245/)\n2. [Python验证码识别处理实例 - Python - 伯乐在线](http://python.jobbole.com/83945/)\n3. [Python验证码识别处理实例 - 林炳文Evankaka的专栏 - CSDN博客](http://blog.csdn.net/evankaka/article/details/49533493)\n4. [爬虫怎么解决封IP？ - 知乎](https://www.zhihu.com/question/26018679)\n5. [爬虫ip代理服务器的简要思路 - 京东放养的爬虫 - CSDN博客](http://blog.csdn.net/djd1234567/article/details/51741557)\n6. [关于使用动态轮训切换ip防止爬虫被封杀](http://xiaorui.cc/2015/01/12/%E5%85%B3%E4%BA%8E%E4%BD%BF%E7%94%A8%E5%8A%A8%E6%80%81%E8%BD%AE%E8%AE%AD%E5%88%87%E6%8D%A2ip%E9%98%B2%E6%AD%A2%E7%88%AC%E8%99%AB%E8%A2%AB%E5%B0%81%E6%9D%80/)\n7. [python爬虫-爬取代理IP并通过多线程快速验证](http://www.oschina.net/code/snippet_2463131_51169)\n8. [OpenCV-Python教程（5、初级滤波内容）](http://blog.csdn.net/sunny2038/article/details/9155893)\n9. [字符型图片验证码识别完整过程及Python实现](http://www.cnblogs.com/beer/p/5672678.html)\n10. [Linux IP代理筛选系统（shell+proxy）](http://blog.csdn.net/ithomer/article/details/7639385)\n11. [SQLite 连接两个字符串](http://www.cnblogs.com/AngelLee2009/p/3208223.html)\n12. [取得sqlite数据库里所有的表名 \u0026复制表](http://blog.csdn.net/vlily/article/details/9096909)\n13. [python - Beautifulsoup and AJAX-table problem - Stack Overflow](https://stackoverflow.com/questions/5913280/beautifulsoup-and-ajax-table-problem)\n14. [python - How to enable digits only in pytesser? - Stack Overflow](https://stackoverflow.com/questions/9466694/how-to-enable-digits-only-in-pytesser)\n15. [Python 文件读写操作实例详解](http://www.jb51.net/article/47999.htm)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fszcf-weiya%2FSinaSpider","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fszcf-weiya%2FSinaSpider","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fszcf-weiya%2FSinaSpider/lists"}