{"id":28240761,"url":"https://github.com/flute/coub-crawler","last_synced_at":"2026-03-01T03:01:17.259Z","repository":{"id":77565307,"uuid":"150250819","full_name":"flute/coub-crawler","owner":"flute","description":"coub.com crawler, download all videos.","archived":false,"fork":false,"pushed_at":"2018-09-25T11:06:20.000Z","size":14,"stargazers_count":7,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-09-09T06:58:06.285Z","etag":null,"topics":["coub","coub-com-crawler","coub-crawler","crawler"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flute.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2018-09-25T10:56:48.000Z","updated_at":"2022-04-01T14:59:50.000Z","dependencies_parsed_at":"2023-03-01T20:30:53.585Z","dependency_job_id":null,"html_url":"https://github.com/flute/coub-crawler","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/flute/coub-crawler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flute%2Fcoub-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flute%2Fcoub-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flute%2Fcoub-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flute%2Fcoub-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flute","download_url":"https://codeload.github.com/flute/coub-crawler/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flute%2Fcoub-crawler/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29959284,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-01T01:47:18.291Z","status":"online","status_checked_at":"2026-03-01T02:00:07.437Z","response_time":124,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["coub","coub-com-crawler","coub-crawler","crawler"],"created_at":"2025-05-19T04:11:58.778Z","updated_at":"2026-03-01T03:01:17.250Z","avatar_url":"https://github.com/flute.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"## coub.com 内容抓取\n\n### 运行\n\n修改 `app.js`:\n\n* `time` 为每个分类要抓取内容的时间段。`daily`、`weekly`、 `monthly`、 `quarter`、`half`\n* `per_page` 为每页返回的数据条数，最大为25\n* `fastMode` 为true时下载速度快，但视频可能出现音频问题。为false时速度慢，视频音频正常。\n\n执行`node app.js`运行代码或使用pm2守护进程：`pm2 start app.js --name 'coub'`。\n\n下载完的视频及JSON文件存储在`downloads`对应目录下，完整日志在`logs`目录。\n\n### 程序说明\n\n1、总共17个分类。\n\n2、数据获取\n\n* url：`https://coub.com/api/v2/timeline/hot/movies/half?per_page=25`\n* 说明：`movies` 为分类。 `per_page` 为每页返回的数据量[1,25]。首次获取只需传入 `page=1` 即为第一页的数据。下次请求附带字段 `anchor` 为上次请求返回的 `next` 参数即可。\n\n3、每个资源的属性：\n\n* 唯一标志： id、permalink\n* 资源描述： titile\n\n\n4、下载\n\ncoub.com的音频和视频是分开的，下载的时候需要将音视频分别下载，然后使用FFmpeg合并。\n下载及合并使用开源项目 `https://github.com/TeeSeal/coub-dl`\n\n\n5、分类数组\n\n```js\n[\"animals-pets\", \"mashup\", \"anime\", \"movies\", \"gaming\", \"cartoons\", \"art\", \"music\", \"sports\", \"science-technology\", \"celebrity\", \"nature-travel\", \"fashion\", \"dance\", \"cars\", \"nsfw\"]\n```\n\n6、请求返回数据格式\n\n```js\n{\n    \"page\": 1,\n    \"per_page\": 25,\n    \"total_pages\": 8,\n    \"next\": 12663692,\n    \"coubs\": [\n        {\n            \"flag\": null,\n            \"abuses\": null,\n            \"recoubs_by_users_channels\": null,\n            \"favourite\": false,\n            \"recoub\": null,\n            \"like\": null,\n            \"in_my_best2015\": false,\n            \"id\": 84085654,\n            \"type\": \"Coub::Simple\",\n            \"permalink\": \"1dnm2b\",\n            \"title\": \"How to ride\",\n            \"visibility_type\": \"public\",\n            \"original_visibility_type\": \"public\",\n            \"channel_id\": 4572400,\n            \"created_at\": \"2018-09-02T04:09:28Z\",\n            \"updated_at\": \"2018-09-19T09:03:07Z\",\n            \"is_done\": true,\n            \"views_count\": 110268,\n            \"cotd\": null,\n            \"cotd_at\": null,\n            \"published\": true,\n            \"published_at\": \"2018-09-02T04:09:28Z\",\n            \"reversed\": true,\n            \"from_editor_v2\": true,\n            \"is_editable\": true,\n            \"original_sound\": false,\n            \"has_sound\": false,\n            \"recoub_to\": null,\n            \"file_versions\": {\n                \"html5\": {\n                    \"video\": {\n                        \"high\": {\n                            \"url\": \"https://coubsecure-s.akamaihd.net/get/b133/p/coub/simple/cw_file/fa92b77cd25/8bb4c6b9ee76c9b7b3648/muted_mp4_big_size_1535861379_muted_big.mp4\",\n                            \"size\": 3059375\n                        },\n                        \"med\": {\n                            \"url\": \"https://coubsecure-s.akamaihd.net/get/b133/p/coub/simple/cw_file/fa92b77cd25/8bb4c6b9ee76c9b7b3648/muted_mp4_med_size_1535861379_muted_med.mp4\",\n                            \"size\": 877351\n                        }\n                    },\n                    \"audio\": {\n                        \"high\": {\n                            \"url\": \"https://coubsecure-s.akamaihd.net/get/b133/p/coub/simple/cw_looped_audio/4d466be3536/557a27d624abf3032d767/high_1535861396_high.mp3\",\n                            \"size\": 4194209\n                        },\n                        \"med\": {\n                            \"url\": \"https://coubsecure-s.akamaihd.net/get/b133/p/coub/simple/cw_looped_audio/4d466be3536/557a27d624abf3032d767/high_1535861396_high.mp3\",\n                            \"size\": 4194209\n                        }\n                    }\n                },\n                \"mobile\": {\n                    \"gifv\": \"Ahr0Chm6lY9JB3vIC2vJDxjLlxmUywTHBwfPAgqUBMv0l2DLDc9ImtmZl3aVy291yI9ZAw1WBguVy3DFzMLSzs9MytKYyJC3y2qYns84yMi0yZzIowvLnZzJowi3yJm2ndGVz2LMDL8Xntm1odyXmZC5xZiWndK3x2DPzNyUBxa0\",\n                    \"audio\": [\n                        \"https://coubsecure-s.akamaihd.net/get/b133/p/coub/simple/cw_looped_audio/4d466be3536/557a27d624abf3032d767/med_m4a_1535861396_med.m4a\",\n                        \"https://coubsecure-s.akamaihd.net/get/b133/p/coub/simple/cw_looped_audio/4d466be3536/557a27d624abf3032d767/high_1535861396_high.mp3\"\n                    ]\n                }\n            },\n            \"audio_versions\": {\n                \"template\": \"https://coubsecure-s.akamaihd.net/get/b127/p/audio_track/cw_normalized_copy/c030ec098c2/99af4293448999330b402/mid_1536706207_16ex7y2_normalized_1535861211_audio.mp3\",\n                \"versions\": [\n                    \"mid\",\n                    \"low\"\n                ],\n                \"chunks\": {\n                    \"template\": \"https://coubsecure-s.akamaihd.net/get/b127/p/audio_track/cw_normalized_copy/c030ec098c2/99af4293448999330b402/mp3_%{version}_c%{chunk}_1536706207_16ex7y2_normalized_1535861211_audio.mp3\",\n                    \"versions\": [\n                        \"mid\",\n                        \"low\"\n                    ],\n                    \"chunks\": [\n                        1,\n                        2,\n                        3,\n                        4\n                    ]\n                }\n            },\n            \"image_versions\": {\n                \"template\": \"https://coubsecure-s.akamaihd.net/get/b133/p/coub/simple/cw_image/4ebdc8e286a/643e256926c9d08d7e083/%{version}_1535861411_00038.jpg\",\n                \"versions\": [\n                    \"micro\",\n                    \"tiny\",\n                    \"age_restricted\",\n                    \"ios_large\",\n                    \"ios_mosaic\",\n                    \"big\",\n                    \"med\",\n                    \"small\",\n                    \"pinterest\"\n                ]\n            },\n            \"first_frame_versions\": {\n                \"template\": \"https://coubsecure-s.akamaihd.net/get/b133/p/coub/simple/cw_timeline_pic/1a0e612c5b9/3cadd8ed1b15cd803cb28/%{version}_1535861410_image.jpg\",\n                \"versions\": [\n                    \"big\",\n                    \"med\",\n                    \"small\",\n                    \"ios_large\"\n                ]\n            },\n            \"dimensions\": {\n                \"big\": [\n                    1280,\n                    720\n                ],\n                \"med\": [\n                    640,\n                    360\n                ]\n            },\n            \"site_w_h\": [\n                640,\n                360\n            ],\n            \"page_w_h\": [\n                640,\n                360\n            ],\n            \"site_w_h_small\": [\n                310,\n                174\n            ],\n            \"age_restricted\": false,\n            \"age_restricted_by_admin\": false,\n            \"not_safe_for_work\": false,\n            \"allow_reuse\": false,\n            \"dont_crop\": false,\n            \"banned\": false,\n            \"global_safe\": true,\n            \"audio_file_url\": \"https://coubsecure-s.akamaihd.net/get/b127/p/audio_track/cw_normalized_copy/c030ec098c2/99af4293448999330b402/low_1536706207_16ex7y2_normalized_1535861211_audio.mp3\",\n            \"external_download\": false,\n            \"application\": null,\n            \"channel\": {\n                \"id\": 4572400,\n                \"permalink\": \"d4369596ec4c87b3cc411eba9b29cbdd\",\n                \"title\": \"Saigofu\",\n                \"description\": null,\n                \"followers_count\": 21,\n                \"following_count\": 1,\n                \"avatar_versions\": {\n                    \"template\": \"https://coubsecure-s.akamaihd.net/get/b14/p/channel/cw_avatar/33160a13e38/ebc7ba4b1e05d2f64aa6f/%{version}_1533405536_tumblr_inline_n0bl06TqUk1syef2f.png\",\n                    \"versions\": [\n                        \"medium\",\n                        \"medium_2x\",\n                        \"profile_pic\",\n                        \"profile_pic_new\",\n                        \"profile_pic_new_2x\",\n                        \"tiny\",\n                        \"tiny_2x\",\n                        \"small\",\n                        \"small_2x\",\n                        \"ios_large\",\n                        \"ios_small\"\n                    ]\n                }\n            },\n            \"file\": null,\n            \"picture\": \"https://coubsecure-s.akamaihd.net/get/b133/p/coub/simple/cw_image/4ebdc8e286a/643e256926c9d08d7e083/med_1535861411_00038.jpg\",\n            \"timeline_picture\": \"https://coubsecure-s.akamaihd.net/get/b133/p/coub/simple/cw_timeline_pic/1a0e612c5b9/3cadd8ed1b15cd803cb28/ios_large_1535861410_image.jpg\",\n            \"small_picture\": \"https://coubsecure-s.akamaihd.net/get/b133/p/coub/simple/cw_image/4ebdc8e286a/643e256926c9d08d7e083/ios_mosaic_1535861411_00038.jpg\",\n            \"sharing_picture\": null,\n            \"percent_done\": 100,\n            \"tags\": [\n                {\n                    \"id\": 52997,\n                    \"title\": \"horse racing\",\n                    \"value\": \"horse%20racing\"\n                },\n                {\n                    \"id\": 344904,\n                    \"title\": \"horseback riding\",\n                    \"value\": \"horseback%20riding\"\n                },\n                {\n                    \"id\": 12335,\n                    \"title\": \"riding\",\n                    \"value\": \"riding\"\n                },\n                {\n                    \"id\": 5453,\n                    \"title\": \"jennifer connelly\",\n                    \"value\": \"jennifer%20connelly\"\n                },\n                {\n                    \"id\": 184,\n                    \"title\": \"sexy\",\n                    \"value\": \"sexy\"\n                },\n                {\n                    \"id\": 183,\n                    \"title\": \"girl\",\n                    \"value\": \"girl\"\n                },\n                {\n                    \"id\": 3171,\n                    \"title\": \"horse\",\n                    \"value\": \"horse\"\n                }\n            ],\n            \"categories\": [\n                {\n                    \"id\": 19,\n                    \"title\": \"Movies \u0026 TV\",\n                    \"permalink\": \"movies\",\n                    \"visible\": true\n                }\n            ],\n            \"recoubs_count\": 357,\n            \"remixes_count\": 5,\n            \"likes_count\": 1786,\n            \"raw_video_id\": \"\",\n            \"uploaded_by_ios_app\": false,\n            \"uploaded_by_android_app\": false,\n            \"media_blocks\": {\n                \"uploaded_raw_videos\": [],\n                \"external_raw_videos\": [],\n                \"remixed_from_coubs\": [\n                    {\n                        \"id\": 27927193,\n                        \"title\": \"Racing\",\n                        \"url\": \"https://coub.com/coubs/1dd7bk/remix\",\n                        \"image\": \"https://coubsecure-s.akamaihd.net/get/b55/p/media_block/cw_image/e260a04e3b7/89f6b60d8fd7e995476ea/video_1535436309_1535436298_00032.jpg\",\n                        \"image_retina\": \"https://coubsecure-s.akamaihd.net/get/b55/p/media_block/cw_image/e260a04e3b7/89f6b60d8fd7e995476ea/video_retina_1535436309_1535436298_00032.jpg\",\n                        \"meta\": {\n                            \"duration\": \"10.0\"\n                        },\n                        \"duration\": null,\n                        \"coub_channel_title\": \"Emma Watson\",\n                        \"coub_channel_permalink\": \"hermione\",\n                        \"coub_views_count\": 220780,\n                        \"coub_permalink\": \"1dd7bk\"\n                    }\n                ]\n            },\n            \"raw_video_thumbnail_url\": \"\",\n            \"raw_video_title\": \"\",\n            \"video_block_banned\": false,\n            \"duration\": 12.08,\n            \"promo_winner\": false,\n            \"promo_winner_recoubers\": null,\n            \"editorial_info\": {},\n            \"promo_hint\": null,\n            \"beeline_best_2014\": null,\n            \"from_web_editor\": true,\n            \"normalize_sound\": true,\n            \"best2015_addable\": false,\n            \"ahmad_promo\": null,\n            \"promo_data\": null,\n            \"audio_copyright_claim\": null,\n            \"ads_disabled\": false,\n            \"position_on_page\": 1\n        }\n    ]\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflute%2Fcoub-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflute%2Fcoub-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflute%2Fcoub-crawler/lists"}