{"id":17207028,"url":"https://github.com/angelkitty/bilibili-smallvideo","last_synced_at":"2025-02-24T05:31:11.997Z","repository":{"id":48967558,"uuid":"152376283","full_name":"AngelKitty/bilibili-smallvideo","owner":"AngelKitty","description":"🕷️用于爬取B站前top100的小视频","archived":false,"fork":false,"pushed_at":"2018-10-10T07:19:02.000Z","size":1362,"stargazers_count":139,"open_issues_count":3,"forks_count":59,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-02-22T04:11:13.341Z","etag":null,"topics":["bilibili","bilibili-smallvideo","python","spider"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AngelKitty.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-10-10T06:49:16.000Z","updated_at":"2025-02-15T14:57:43.000Z","dependencies_parsed_at":"2022-09-17T08:34:37.196Z","dependency_job_id":null,"html_url":"https://github.com/AngelKitty/bilibili-smallvideo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AngelKitty%2Fbilibili-smallvideo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AngelKitty%2Fbilibili-smallvideo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AngelKitty%2Fbilibili-smallvideo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AngelKitty%2Fbilibili-smallvideo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AngelKitty","download_url":"https://codeload.github.com/AngelKitty/bilibili-smallvideo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240425422,"owners_count":19799328,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bilibili","bilibili-smallvideo","python","spider"],"created_at":"2024-10-15T02:44:30.621Z","updated_at":"2025-02-24T05:31:11.556Z","avatar_url":"https://github.com/AngelKitty.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## bilibili-smallvideo\n\n### 如何手动写一个 `Python` 脚本自动爬取 `Bilibili` 小视频\n\n![bilibili](./image/bilibili.jpeg)\n\n国庆结束之余，某个不务正业的码农不好好干活，在B站瞎逛着，毕竟国庆嘛，还让不让人休息了诶～～\n\n我身边的很多小伙伴们在朋友圈里面晒着出去游玩的照片，简直了，人多的不要不要的，长城被堵到水泄不通，老实人想想啊，既然人这么多，哪都不去也是件好事，没事还可以刷刷 `B` 站 `23333` 。这时候老实人也有了一个大胆地想法，能不能让这些在旅游景点排队的小伙伴们更快地打发时间呢？考虑到视频的娱乐性和大众观看量，我决定对 `B` 站新推出的小视频功能下手，于是我跑到B站去找API接口，果不起然，B站在小视频功能处提供了 `API` 接口，小伙伴们有福了哟！\n\n![vcbilibili](./image/vcbilibili.png)\n\n`B` 站小视频网址在这里哦：\n\n```\nhttp://vc.bilibili.com/p/eden/rank#/?tab=全部\n```\n\n此次实验，我们爬取的是每日的小视频排行榜前 `top100`\n\n我们该如何去爬取呢？？？\n\n### 实验环境准备\n\n- `Chrome` 浏览器 (能使用开发者模式的浏览器都行)\n- `Vim` (编辑器任选，老实人比较喜欢Vim界面，所以才用这个啦)\n- `Python3` 开发环境\n- `Kali Linux` (其实随便一个操作系统都行啦)\n\n### `API` 寻找 `\u0026\u0026` 提取\n\n我们通过 `F12` 打开开发者模式，然后在 `Networking -\u003e  Name` 字段下找到这个链接：\n\n```\nhttp://api.vc.bilibili.com/board/v1/ranking/top?page_size=10\u0026next_offset=\u0026tag=%E4%BB%8A%E6%97%A5%E7%83%AD%E9%97%A8\u0026platform=pc\n```\n\n我们查看一下  `Headers` 属性\n\n![requestsurl](./image/requestsurl.png)\n\n我们可以看到Request URL这个属性值，我们向下滑动加载视频的过程中，发现只有这段url是不变的。\n\n```\nhttp://api.vc.bilibili.com/board/v1/ranking/top?\n```\n\n`next_offset` 会一直变化，我们可以猜测，这个可能就是获取下一个视频序号，我们只需要把这部分参数取出来，把 `next_offset` 写成变量值，用 `JSON` 的格式返回到目标网页即可。\n\n![next_offset](./image/next_offset.png)\n\n### 代码实现\n\n我们通过上面的尝试写了段代码，发现 `B` 站在一定程度上做了反爬虫操作，所以我们需要先获取 `headers` 信息，否则下载下来的视频是空的，然后定义 `params` 参数存储 `JSON` 数据，然后通过 `requests.get` 去获取其参数值信息，用 `JSON` 的格式返回到目标网页即可，实现代码如下：\n\n```python\ndef get_json(url):\n    headers = {\n        'User-Agent': \n        'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36'\n    }\n\n    params = {\n        'page_size': 10,\n        'next_offset': str(num),\n        'tag': '今日热门',\n        'platform': 'pc'\n    }\n\n    try:\n        html = requests.get(url,params=params,headers=headers)\n        return html.json()\n\n    except BaseException:\n        print('request error')\n        pass\n\n```\n\n为了能够清楚的看到我们下载的情况，我们折腾了一个下载器上去，实现代码如下：\n\n```python\ndef download(url,path):\n    start = time.time() # 开始时间\n    size = 0\n    headers = {\n        'User-Agent': \n        'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36'\n    }\n\n    response = requests.get(url,headers=headers,stream=True) # stream属性必须带上\n    chunk_size = 1024 # 每次下载的数据大小\n    content_size = int(response.headers['content-length']) # 总大小\n    if response.status_code == 200:\n        print('[文件大小]:%0.2f MB' %(content_size / chunk_size / 1024)) # 换算单位\n        with open(path,'wb') as file:\n            for data in response.iter_content(chunk_size=chunk_size):\n                file.write(data)\n                size += len(data) # 已下载的文件大小\n```\n\n效果如下：\n\n![download](./image/download.png)\n\n将上面的代码进行汇总，整个实现过程如下：\n\n```python\n#!/usr/bin/env python\n#-*-coding:utf-8-*-\nimport requests\nimport random\nimport time\ndef get_json(url):\n    headers = {\n        'User-Agent': \n        'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36'\n    }\n\n    params = {\n        'page_size': 10,\n        'next_offset': str(num),\n        'tag': '今日热门',\n        'platform': 'pc'\n    }\n\n    try:\n        html = requests.get(url,params=params,headers=headers)\n        return html.json()\n\n    except BaseException:\n        print('request error')\n        pass\n\ndef download(url,path):\n    start = time.time() # 开始时间\n    size = 0\n    headers = {\n        'User-Agent': \n        'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36'\n    }\n\n    response = requests.get(url,headers=headers,stream=True) # stream属性必须带上\n    chunk_size = 1024 # 每次下载的数据大小\n    content_size = int(response.headers['content-length']) # 总大小\n    if response.status_code == 200:\n        print('[文件大小]:%0.2f MB' %(content_size / chunk_size / 1024)) # 换算单位\n        with open(path,'wb') as file:\n            for data in response.iter_content(chunk_size=chunk_size):\n                file.write(data)\n                size += len(data) # 已下载的文件大小\n\n    \n\nif __name__ == '__main__':\n    for i in range(10):\n        url = 'http://api.vc.bilibili.com/board/v1/ranking/top?'\n        num = i*10 + 1\n        html = get_json(url)\n        infos = html['data']['items']\n        for info in infos:\n            title = info['item']['description'] # 小视频的标题\n            video_url = info['item']['video_playurl'] # 小视频的下载链接\n            print(title)\n\n            # 为了防止有些视频没有提供下载链接的情况\n            try:\n                download(video_url,path='%s.mp4' %title)\n                print('成功下载一个!')\n                \n            except BaseException:\n                print('凉凉,下载失败')\n                pass\n\n        time.sleep(int(format(random.randint(2,8)))) # 设置随机等待时间\n\n```\n\n爬取效果图如下：\n\n![download](./image/download.png)\n\n![watch](./image/watch.png)\n\n似乎爬取的效果还可以，当然喜欢的朋友不要忘记点赞分享转发哦。\n\n### 项目链接\n\n- [Github](https://github.com/AngelKitty/bilibili-smallvideo)\n\n我的公众号是**IT老实人(IThonest)**，一个有故事的公众号，欢迎大家来这里讨论，共同进步，不断学习才能不断进步。扫下面的二维码或者收藏下面的二维码关注吧(长按下面的二维码图片、并选择识别图中的二维码)\n\n\u003cp\u003e\u003cimg style=\"margin-left: auto; margin-right: auto; display: block\" src=\"./image/qrcode_for_gh_d5c9921b015a_258.jpg\"\u003e\u003c/p\u003e","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fangelkitty%2Fbilibili-smallvideo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fangelkitty%2Fbilibili-smallvideo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fangelkitty%2Fbilibili-smallvideo/lists"}