{"id":13840709,"url":"https://github.com/gudegg/yunSpider","last_synced_at":"2025-07-11T09:32:56.031Z","repository":{"id":49850275,"uuid":"69833752","full_name":"gudegg/yunSpider","owner":"gudegg","description":"百度云网盘爬虫","archived":false,"fork":false,"pushed_at":"2020-04-05T13:17:13.000Z","size":43,"stargazers_count":806,"open_issues_count":0,"forks_count":296,"subscribers_count":47,"default_branch":"user_conf","last_synced_at":"2024-08-05T17:25:34.686Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gudegg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-10-03T01:52:22.000Z","updated_at":"2024-07-29T10:41:07.000Z","dependencies_parsed_at":"2022-08-29T00:30:19.642Z","dependency_job_id":null,"html_url":"https://github.com/gudegg/yunSpider","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gudegg%2FyunSpider","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gudegg%2FyunSpider/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gudegg%2FyunSpider/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gudegg%2FyunSpider/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gudegg","download_url":"https://codeload.github.com/gudegg/yunSpider/tar.gz/refs/heads/user_conf","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225712657,"owners_count":17512451,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-04T17:00:53.148Z","updated_at":"2024-11-21T10:30:40.751Z","avatar_url":"https://github.com/gudegg.png","language":"Go","funding_links":[],"categories":["Go","Go (531)"],"sub_categories":[],"readme":"百度云爬虫\n-----\n#### 安装使用\n1. [安装go与设置gopath](https://github.com/astaxie/build-web-application-with-golang/blob/master/zh/01.2.md)\n2. clone项目到gopath目录\n3. 安装依赖\n  * go get github.com/go-sql-driver/mysql\n  * go get github.com/siddontang/go/log\n  * go get github.com/garyburd/redigo/redis\n  * go get github.com/Unknwon/goconfig\n4. 修改config.ini配置文件中你自己的MySQL和Redis配置(可以不配Redis),Redis不配或者错误会默认将已经爬取的uk保存到hasIndexKeys.txt文件中,这样做防止重复爬同一位用户多次\n5. 创建database名为baidu,然后执行baidu.sql\n6. go run spider.go (当然也可以编译后运行,此时config.ini记得也复制到编译后的文件目录下,然后运行)\n\n    \n#### 直接下载编译好的使用\n1. [下载](https://github.com/gudegg/yunSpider/releases)适合自己系统的版本\n2. 修改config.ini配置文件，并且放到程序运行的目录下\n3. linux: chmod +x spider,然后直接./spider运行或者nohup ./spider 1\u003elog.out 2\u003eerr.out \u0026以后台方式运行;win:直接点击spider.exe运行\n\n#### 爬到数据使用\n- 只需替换链接中uk,shareid或者album_id即可访问\n    * 分享专辑链接类型：https://pan.baidu.com/wap/album/info?uk=1112219283\u0026album_id=129732510768224935\n    * 普通文件或者文件夹类型：https://pan.baidu.com/wap/link?uk=1112219283\u0026shareid=2109459878\n    \n\n- 获取用户订阅:\n http://yun.baidu.com/pcloud/friend/getfollowlist?query_uk=%s\u0026limit=24\u0026start=%s\u0026bdstoken=e6f1efec456b92778e70c55ba5d81c3d\u0026channel=chunlei\u0026clienttype=0\u0026web=1\u0026logid=MTQ3NDA3NDg5NzU4NDAuMzQxNDQyMDY2MjA5NDA4NjU=  \n    (query_uk limit start是必须参数)\n- 获取用户粉丝: \nhttp://pan.baidu.com/pcloud/friend/getfanslist?query_uk=%s\u0026limit=24\u0026start=%s\u0026bdstoken=null\u0026channel=chunlei\u0026clienttype=0\u0026web=1\u0026logid=MTQ3NDAzNjQwNzg3OTAuNzM1MzMxMDUyMDczMjYxNA==\n    (query_uk limit start是必须参数)\n- 获取用户分享: \nhttp://pan.baidu.com/pcloud/feed/getsharelist?t=1474202771918\u0026category=0\u0026auth_type=1\u0026request_location=share_home\u0026start=0\u0026limit=60\u0026query_uk=224534490\u0026channel=chunlei\u0026clienttype=0\u0026web=1\u0026logid=MTQ3NDIwMjc3MTkxOTAuMzA1NjAzMzQ4MTM1MDc0MTc=\u0026bdstoken=e6f1efec456b92778e70c55ba5d81c3d\n    (query_uk limit start auth_type是必须参数)\n\n \n\u003e 上面3个连接请求必须带上Referer`(\"Referer\", \"https://yun.baidu.com/share/home?uk=23432432#category/type=0\")`,否则请求不到json数据,\n获取用户订阅和获取用户粉丝每次请求一次休眠2s的话可以无限制请求,对ip没要求,获取用户分享超坑,一个ip只能请求10次,并且休眠也没用.\n因为没有那么多ip,我就去研究手机版的用户分享,手机版获取用户分享可以一次性连续请求60次,60次后必须休眠35s左右在继续请求就可以,不会像pc版那样必须换ip,\n但是手机版只能请求网页源码,然后用正则进行匹配.\n\n- 手机版分享:\n http://pan.baidu.com/wap/share/home?uk=2889076181\u0026start=%s\u0026adapt=pc\u0026fr=ftw (uk:**每个百度网盘用户的唯一标示**,start:用户可能有上百上千个分享,必须分页,start默认从0开始,手机版默认分页是20个每页)\n \n\n\n \n#### License\n\nyunSpide source code is licensed under the Apache Licence, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0.html).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgudegg%2FyunSpider","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgudegg%2FyunSpider","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgudegg%2FyunSpider/lists"}