{"id":27433143,"url":"https://github.com/hypervapor/bilibili-crawler","last_synced_at":"2025-04-14T17:14:17.868Z","repository":{"id":287347199,"uuid":"964435219","full_name":"HYPERVAPOR/bilibili-crawler","owner":"HYPERVAPOR","description":"根据关键字列表爬取 Bilibili 视频信息的后端应用 / Backend application for crawling Bilibili video information based on a list of keywords.","archived":false,"fork":false,"pushed_at":"2025-04-11T09:49:24.000Z","size":43,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-14T17:14:13.329Z","etag":null,"topics":["bilibili","crawler","express","nodejs"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/HYPERVAPOR.png","metadata":{"files":{"readme":"README.en.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-11T07:58:02.000Z","updated_at":"2025-04-11T09:56:46.000Z","dependencies_parsed_at":"2025-04-11T10:22:21.492Z","dependency_job_id":"f7402cdb-266e-4a2f-a4bb-b374e8eef8c8","html_url":"https://github.com/HYPERVAPOR/bilibili-crawler","commit_stats":null,"previous_names":["hypervapor/bilibili-crawler"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HYPERVAPOR%2Fbilibili-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HYPERVAPOR%2Fbilibili-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HYPERVAPOR%2Fbilibili-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HYPERVAPOR%2Fbilibili-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/HYPERVAPOR","download_url":"https://codeload.github.com/HYPERVAPOR/bilibili-crawler/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248923767,"owners_count":21183953,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bilibili","crawler","express","nodejs"],"created_at":"2025-04-14T17:14:17.314Z","updated_at":"2025-04-14T17:14:17.851Z","avatar_url":"https://github.com/HYPERVAPOR.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Project Introduction\n\nThis project is a backend application that scrapes Bilibili video information based on a list of keywords. It uses Axios for HTTP requests and Cheerio to parse the returned HTML data. The video information that can be scraped includes:\n\n- **Title**\n- **Link**\n- **Cover Image**\n- **Publisher**\n- **View Count**\n- **Duration**\n\nAfter running this project, it listens on port **3000** by default. You can retrieve data by sending a POST request to `http://127.0.0.1:3000/api/scrape`.\n\n### Example Request Body\n\n```json\n{\n  \"tags\": [\"Artificial Intelligence\", \"Blockchain\"]\n}\n```\n\n### Example Expected Response\n\n```json\n[\n  {\n    \"tag\": \"Artificial Intelligence\",\n    \"result\": [\n      {\n        \"title\": \"History of Artificial Intelligence: The Evolution of AI\",\n        \"link\": \"https://www.bilibili.com/video/BV15q421A7tB/\",\n        \"cover\": \"https://i2.hdslb.com/bfs/archive/53659a990006e6a54aab0e1572ffe2f2110583aa.jpg@672w_378h_1c_!web-search-common-cover\",\n        \"author\": \"Pulsar Technology\",\n        \"views\": \"59,000\",\n        \"duration\": \"08:47\",\n        \"source\": \"bilibili\"\n      },\n      {\n        \"title\": \"[AI Edition] Ten Layers of Understanding AI: A Comprehensive Look at AI's Past, Present, and Future.\",\n        \"link\": \"https://www.bilibili.com/video/BV15P6GYcEc3/\",\n        \"cover\": \"https://i1.hdslb.com/bfs/archive/31f71401cacc3490b11e9d1eb19efa7505171da6.jpg@672w_378h_1c_!web-search-common-cover\",\n        \"author\": \"Floating World Three Thousand Questions\",\n        \"views\": \"231,000\",\n        \"duration\": \"21:16\",\n        \"source\": \"bilibili\"\n      }\n    ]\n  },\n  {\n    \"tag\": \"Blockchain\",\n    \"result\": [\n      {\n        \"title\": \"Peking University Professor Xiao Zhen's Open Course on Blockchain Technology and Applications\",\n        \"link\": \"https://www.bilibili.com/video/BV1Vt411X7JF/\",\n        \"cover\": \"https://i1.hdslb.com/bfs/archive/cb62713060c5c3a4058436fd5beba55f5894b70d.jpg@672w_378h_1c_!web-search-common-cover\",\n        \"author\": \"Xiao Zhen, Department of Computer Science, Peking University\",\n        \"views\": \"1,501,000\",\n        \"duration\": \"20:01:19\",\n        \"source\": \"bilibili\"\n      },\n      {\n        \"title\": \"[Blockchain General Course] From Bitcoin to Blockchain - A Lecture by Top 10 Chinese University Graduate Instructors and PhD Students\",\n        \"link\": \"https://www.bilibili.com/video/BV1me411J7Sk/\",\n        \"cover\": \"https://i2.hdslb.com/bfs/archive/5f86a3c9a13ae5d79f23019c73079efdae83761e.png@672w_378h_1c_!web-search-common-cover\",\n        \"author\": \"Approaching Web3\",\n        \"views\": \"1,389\",\n        \"duration\": \"06:22:59\",\n        \"source\": \"bilibili\"\n      }\n    ]\n  }\n]\n```\n\n# Quick Start\n\nPlease follow the steps below to quickly start the project:\n\n```bash\n# Clone this project\ngit clone https://github.com/HYPERVAPOR/bilibili-crawler\n\n# Enter this directory\ncd bilibili-crawler\n\n# Install dependencies (recommended to use pnpm)\npnpm i\n\n# Start the service\npnpm start\n```\n\nNext, you can use **POSTMAN** or your frontend application to send requests for debugging.\n\n# FAQ\n\n## How to Change the Default Listening Port\n\nIn the `bin/www` file, modify the following code:\n\n```javascript\nvar port = normalizePort(process.env.PORT || \"3000\");\n```\n\n## How to Change the Number of Returned Video Information\n\nIn the `routes/scrape.js` file, modify the logic of the function `function determineK(length)`:\n\n```javascript\n// The default function returns around 20 videos based on the number of tags\nfunction determineK(length) {\n  if (length === 1) return 20;\n  if (length === 2) return 10;\n  if (length === 3) return 7;\n  if (length === 4) return 5;\n  if (length === 5 || length === 6) return 4;\n  if (length \u003e= 7 \u0026\u0026 length \u003c= 9) return 3;\n  if (length \u003e= 10) return 2;\n  return 5; // Default value\n}\n```\n\n# Disclaimer\n\nUsing this project indicates that you have read, understood, and agreed to the following terms:\n\n1. **Usage Restrictions**  \n   This project is for educational and communication purposes only and is strictly prohibited for any commercial use. Users may not use this project for profit-making activities in any form.\n\n2. **Legal Compliance**  \n   Users commit not to use this project for any illegal or criminal activities, including but not limited to infringing on others' intellectual property rights, spreading malware, or conducting cyber attacks.\n\n3. **Platform Protection**  \n   Users must not in any way affect or interfere with the normal operation of the Bilibili platform, including but not limited to excessive requests, scraping data, or other actions that may lead to service interruptions.\n\n4. **Liability Statement**  \n   The developers of this project are not responsible for any direct, indirect, incidental, special, or consequential damages arising from the use of this project. Any legal responsibility arising from the use of this project shall be borne by the user.\n\n5. **Terms Changes**  \n   This disclaimer may be updated periodically, and users should regularly review this disclaimer to ensure they are aware of the latest terms.\n\nBy using this project, you agree to comply with all the above terms. If you disagree, please stop using this project immediately.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhypervapor%2Fbilibili-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhypervapor%2Fbilibili-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhypervapor%2Fbilibili-crawler/lists"}