{"id":13600578,"url":"https://github.com/cnbattle/douyin","last_synced_at":"2025-05-14T18:05:24.386Z","repository":{"id":35943754,"uuid":"204860917","full_name":"cnbattle/douyin","owner":"cnbattle","description":"抖音推荐/搜索页视频列表视频爬虫方案,基于app(虚拟机或真机) 相关技术 golang adb","archived":false,"fork":false,"pushed_at":"2025-05-08T01:06:39.000Z","size":38693,"stargazers_count":1157,"open_issues_count":1,"forks_count":241,"subscribers_count":29,"default_branch":"master","last_synced_at":"2025-05-08T02:21:51.454Z","etag":null,"topics":["douyin","go","golang"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cnbattle.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-08-28T06:12:41.000Z","updated_at":"2025-05-08T01:06:36.000Z","dependencies_parsed_at":"2024-01-14T04:45:06.395Z","dependency_job_id":"3c804fd1-8fe8-428f-b625-3dda7b9342fc","html_url":"https://github.com/cnbattle/douyin","commit_stats":{"total_commits":135,"total_committers":6,"mean_commits":22.5,"dds":0.5777777777777777,"last_synced_commit":"f036af9e85a2647a388036a21f9e0107e5beab08"},"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cnbattle%2Fdouyin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cnbattle%2Fdouyin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cnbattle%2Fdouyin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cnbattle%2Fdouyin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cnbattle","download_url":"https://codeload.github.com/cnbattle/douyin/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254198514,"owners_count":22030965,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["douyin","go","golang"],"created_at":"2024-08-01T18:00:43.400Z","updated_at":"2025-05-14T18:05:19.377Z","avatar_url":"https://github.com/cnbattle.png","language":"Go","funding_links":[],"categories":["Go","Repositories"],"sub_categories":[],"readme":"# 抖音推荐/搜索页视频列表视频爬虫方案\n\n\u003e 最近测试可用时间：`2024.06.20`\n\n\u003e adb暂未实现进入搜索页的操作，请根据自身技术栈实现相关点击操作及键入关键词等\n\n\u003e 老版本请切换到`old`分支查看，`old`分支使用anyproxy抓取，更适合大多数人使用\n\n基于APP爬取\n\n技术栈：`golang` `adb`\n\n![](example/example.gif)\n\n## 特点\n- 可设置仅抓取大于xx赞的视频\n- 可自定义设置是否下载远程文件到本地\n\n## 使用\n\n1. 安装`mitm-proxy.crt`证书到相关设备，`虚拟机或真机`\n\n2. 复制 `config.example.toml` 为 `config.toml`,并根据自己需求修改参数\n\n3. 运行本项目程序 `go run main.go` 或 编辑运行\n\n4. 若开启下载会生成一个 `database.db`的sqlite3数据库文件,字符详见`model/videos.go`文件,静态文件(用户头像,视频封面图,视频文件)将放在`download/[avatar,cover,video]`目录下\n\n5. 最后根据个人需要上传处理即可\n\n## 待优化的地方\n\n- 一直滑动请求推荐列表,APP会异常退出,所以暂时的方法是每爬取一段时间(配置文件里可自定义)后,强制关闭APP,重新打开继续爬取\n\n- 目前个人在用的`雷电模拟器`,长时间运行会卡死,暂不清楚是系统原因还是模拟器原因,有测试能稳定运行环境的朋友,方便的话请告诉我,谢谢\n\n## 最后说明\n\n- 个人能力一般,有很多编码不规范的地方请包涵\n- 有能力的朋友可以根据个人需求修改,如果可以请提交pr\n- 如果使用有问题,请提交`issues` 或加我同名微信号,请备注github过来的,谢谢\n\n## Stargazers over time\n\n[![Stargazers over time](https://starchart.cc/cnbattle/douyin.svg)](https://starchart.cc/cnbattle/douyin)\n      \n\n## Thanks\n\n```\ngorm.io/gorm\ngithub.com/ouqiang/goproxy\ngithub.com/spf13/viper\n```\n## Development Tool (IDE) For Jetbrains\n\nThanks for [Jetbrains Open Source Licenses](https://www.jetbrains.com/community/opensource/#support).\n\n## 声明\n\n本项目只做个人学习研究之用，不得用于商业用途！\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcnbattle%2Fdouyin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcnbattle%2Fdouyin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcnbattle%2Fdouyin/lists"}