{"id":20502393,"url":"https://github.com/skygongque/spider","last_synced_at":"2025-08-21T15:32:54.234Z","repository":{"id":46095095,"uuid":"254666360","full_name":"skygongque/Spider","owner":"skygongque","description":"爬虫合集","archived":false,"fork":false,"pushed_at":"2024-01-12T13:09:22.000Z","size":13449,"stargazers_count":119,"open_issues_count":5,"forks_count":43,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-12-10T15:51:29.899Z","etag":null,"topics":["nodejs","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/skygongque.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-10T15:15:06.000Z","updated_at":"2024-12-05T09:17:39.000Z","dependencies_parsed_at":"2024-11-15T19:32:30.822Z","dependency_job_id":null,"html_url":"https://github.com/skygongque/Spider","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skygongque%2FSpider","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skygongque%2FSpider/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skygongque%2FSpider/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skygongque%2FSpider/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/skygongque","download_url":"https://codeload.github.com/skygongque/Spider/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230520394,"owners_count":18238948,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["nodejs","python3"],"created_at":"2024-11-15T19:21:53.135Z","updated_at":"2024-12-20T01:16:01.353Z","avatar_url":"https://github.com/skygongque.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![pic](https://img.shields.io/badge/python-v3.7-green)\n![pic](https://img.shields.io/badge/node-v12.16-green)\n# 爬虫项目整理\n\u003e 声明：本项目的所有代码仅用于科研学习之用，禁止商用，否则造成的后果与本人无关\n\n## [网易云音乐下载](./01-网易音乐（execjs或express使用）/readme.md)\nnodejs express  \nparams和encSecKey参数  \n直接扣代码补全缺少的环境即可  \nCrypto-js库用npm安装  \n## [网易云音乐评论爬虫](./02-网易音乐评论(AES-RSA)/readme.md)\nparams和encSecKey参数  \nAES/CBC  \n`pad = lambda s: s + (16 - len(s) % 16) * chr(16 - len(s) % 16)`  \nRSA/nopading \n## [微博模拟登录](./03-微博登录(RSA)/readme.md)\nRSA/PKCS1_v1_5  \n用python的pycryptodome或rsa库都可以实现  \n下载验证码图片手动输入结果 可以接入深度学习的[验证码识别模型](https://github.com/skygongque/captcha-weibo)准确率98%以上  \n## [微博图片批量下载](./04-微博图片/readme.md)\n用asyncio + aiohttp异步下载图片  \n\n## 知乎x-zse-96(jsvmp 插桩还原算法)\n\nmd5 + 位运算 索引固定字符串\n\n## [抖音web signature](https://github.com/skygongque/douyin_signature)\n\n~~目前官方自己的**web端**也无法访问用户主页，无法得到tac参数，**所以该方法已经失效**，可以尝试逆向抖音的app。~~\n\n~~通过补环境的方式得到可以在node环境单独运行的签名算法，需要tac参数（网页端源码中获取）。~~\n\n相关环境的补充\n\n```\nAudio = {}\ndocument = {\n    createElement: function() {\n        return canvas\n    }\n};\ncanvas = {\n    getContext: function getContext() {\n        return CanvasRenderingContext2D\n    },\n    toDataURL: function toDataURL() {\n        return \"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAADAAAAAQCAYAAABQrvyxAAACyUlEQVRIS8XWTYhVZRgH8N8dkPFqJqgI4sKmTUSgaBCKihCI5dZFUEaLwHAThjujEnXjpkAUHSRc+AEtFJIWKWRBSR9QENgiggplNqkTlXBnhjveeM6cF955PXccFJ2zuPec99zz3v/X87+3ZfrRK67jslWvxb103vCxx770DN5sApSAloBLco+SzFKcxbZClmG8gw7uIdCkfnq+BDsXbmzEizhYg2p0oEn1PEK5II/SgTKPbXyE89iBt+oPDDcpWz7cbwZysq/h9Vqd9/E2fkWsn0GsJeVi/1DvE6xJQLJoNA1T7PN0wx73zMBMDsTGTXMQ1n6TAfkJ63ACr+IcDmekUr6D0FUkdb+uc18SCPCba4ILitlodGA2ik8R7Rmw2E4ThnR9oVsRjJyuxL/4FLdwJCOQE07O5CBjQONIRINk7l4iOOMQB4mkdn6eajWgM2mhYU855KhRXRMO6DqKD/Ay3sWygkCKUE6qJPAegmis3+5T0I0EEtj8PUVn+r2e+Zb52HqjPqti8ptFnnPHPhzH6lq5+KIcbOwX4AJkAtjkQK50Pi9pfWfEKR/ifjVaOjC1wcLq9ZoPXbTLZfyi7Vljlep/4nSd8ZzA9bpNokUuzYJASTSBT3Mzcr8qbO77CNBSk0ZNGrPHoO+r1plnta4YxhxcTiAAbMcPs3QgXLpSC1EmKfbd38+B/jMQ4NvuWoTlPnfNKfyIES3P142U12Y/B6oI1Kj6RSgcOIZX6lqe0YHyL0Su/tT5gJ5JbSt03PSfrl34vco//2i522fgHma5/M1Ie20KZ0oHEsj5Fhireqijra2j4wlL3HHMGrv9bC2+tKUGf1vLxMOgfNBnp89AT8sGq3xrHE/WVRo0ojQHvWSHK/Y6ab83qhz/gZsYr34V5uAoCQwYstV1F6zzlXlaxg0aMeQvq7zgqu+qno+WuYG/5xJ86PU//5znHgwZKlQAAAAASUVORK5CYII=\"\n    },\n}\n\nCanvasRenderingContext2D = {\n    arc: function arc() {},\n    stroke: function stroke() {},\n    fillText: function fillText() {},\n}\nwindow = {\n    document: {\n        location:{},\n        _zid:2\n    },\n}\n```\n\n\n\n~~nodejs express puppeteer   \n1.得到指定uid请求返回的tac  \n2.puppeteer驱动chromium  \n在console中执行签名算法得到signature~~ \n\n## [喜马拉雅有声书爬虫](./06-喜马拉雅(自研加密算法%20直接扣代码)/readme.md)\n\n纯JavaScript爬虫  \n通过fileId和ep等参数还原真实地址  \n直接扣代码即可  \n\n## [喜马拉雅模拟登录](https://github.com/skygongque/login-ximalaya)\nopencv-python 识别滑块验证码的缺口识别成功率约 60-70%  \n参考腾讯防水墙缺口的识别算法简单修改\n因为喜马拉雅滑块没有检测轨迹，只对post的缺口位置经过简单的偏移  \n可以实现此滑动验证码的突破\n\n## [58同城](./14-58同城登录（RSA%20fingerprint）/../14-58同城登录（RSA%20fingerprint）/58loginEncryptPassword.js)\n密码加密RSA\n有浏览器指纹  \n\n## [豆瓣app](./08-豆瓣app%20（Frida%20hook实战）/readme.md) _sig\n\u003e frida 简单使用  \n\n[安装frida环境](./08-豆瓣app%20（Frida%20hook实战）/installFrida.md)   \n[豆瓣app _sig 分析](./08-豆瓣app%20（Frida%20hook实战）/readme.md)    \n\n## [懒人听书app](./10-懒人听书APP（MD5签名）/readme.md)\n\u003e md5 哈希\n\u003e frida hook 简单使用 \n\n## 猫眼电影字体反爬\n[猫眼电影字体反爬](https://github.com/skygongque/knn-font)  \n使用knn（k近邻算法）  \n字体文件的标注使用font creator 软件 \n\n## 中国知网\n[中国知网](./17-中国知网（CNKI）(英数验证码)/readme.MD)\n\n## 百度翻译\n[百度翻译](./19-百度翻译(自研签名算法%20直接扣代码)/readme.md)  \n\n## 百度登录RSA免扣调用实战记录\n[百度登录RSA免扣调用实战记录](./21-RPC百度登录免扣实现RSA实战记录/readme.md)  \n\n\u003e 文章可能首次发表于吾爱破解论坛，以下账户也是本人  \n\u003e https://www.52pojie.cn/home.php?mod=space\u0026uid=965057\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fskygongque%2Fspider","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fskygongque%2Fspider","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fskygongque%2Fspider/lists"}