{"id":15188231,"url":"https://github.com/eric2788/platformscrawler","last_synced_at":"2025-10-27T09:30:18.949Z","repository":{"id":50480696,"uuid":"450056205","full_name":"eric2788/PlatformsCrawler","owner":"eric2788","description":"多平台爬蟲 + 模塊化管理，用於搜集資料並經 redis pubsub 發送","archived":false,"fork":false,"pushed_at":"2023-07-13T06:20:41.000Z","size":184,"stargazers_count":5,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-01T04:31:36.525Z","etag":null,"topics":["bilibili","crawler","crawling","pubsub","redis","twitter","youtube"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eric2788.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-20T10:44:55.000Z","updated_at":"2024-09-06T04:01:10.000Z","dependencies_parsed_at":"2024-06-19T19:01:38.874Z","dependency_job_id":"a5581ca7-d27c-4de1-90be-a287b13aedb1","html_url":"https://github.com/eric2788/PlatformsCrawler","commit_stats":{"total_commits":59,"total_committers":2,"mean_commits":29.5,"dds":"0.11864406779661019","last_synced_commit":"c1238a7362fab8cdbf2bb20075e5e0858ca2c538"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eric2788%2FPlatformsCrawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eric2788%2FPlatformsCrawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eric2788%2FPlatformsCrawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eric2788%2FPlatformsCrawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eric2788","download_url":"https://codeload.github.com/eric2788/PlatformsCrawler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238471939,"owners_count":19478132,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bilibili","crawler","crawling","pubsub","redis","twitter","youtube"],"created_at":"2024-09-27T19:02:05.719Z","updated_at":"2025-10-27T09:30:18.626Z","avatar_url":"https://github.com/eric2788.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Platform Crawlers\n\n多平台爬蟲 + 模塊化管理，透過 Redis Pubsub 推送\n\n## 目前內置爬蟲\n\n### Youtube\n- 預定直播\n- 開播\n\n### Twitter\n- 推文/轉發推文\n- 回復\n\n### Bilibili\n- 所有直播數據流 (with [biligo-live-ws](https://github.com/eric2788/biligo-live-ws))\n\n### Valorant\n- 玩家的最新對戰資料追蹤\n\n## 原理\n\n透過隔秒檢查 Redis pubsub 內符合 pattern 的 channels，經過去重過濾取出要新增監聽和中止監聽的頻道，然後基於機制:\n\n- 為 `EachCrawling` 則各自啟動和中止\n- 為 `OnceCrawling` 則先關閉再啟動以刷新\n\n實現: [crawling.go](/crawling/crawling.go)\n\n\n## 部署\n\n你需要:\n- redis 伺服器\n- biligo-live-ws 以推送 bilibili 直播數據流\n\n參數:\n- `--debug` 啟動 debug level (默認: false)\n- `--port` 啟動 rest api 的 port (默認: 8989)\n\nDocker 部署: `docker.io/eric1008818/platform-crawlers`\n\n## 使用\n\n- 開啟 platform-crawlers 程序後關閉，設置好 `config` 內所有 yaml 再重開\n- 首先把 platform-crawlers 與 監聽程序 連接到同一個 redis 伺服器\n- 在 監聽程序 訂閱 格式為 `[prefix]:[room]` 的 topic (eg. `blive:22671795` 將監聽房間號為 22671795 的B站直播)\n- 訂閱後，將會開始自動接收推送\n\n## 新增新的推送\n\n由於此程序採用模塊化管理，其新增新的推送極其簡單(需要使用 `golang`):\n\n- 創建一個 `struct` 並實現 `Crawler`\n- 根據你的監聽方式實現 `EachCrawling` 或 `OnceCrawling` 二選一\n- 在 `init` 方法中使用 `crawling.RegisterCrawler(Tag, crawler instance, logger)`\n- 最後，在 `main.go` 透過 `_` import 你的 package 即可 \n- 如果想禁用某些推送，可以在 `application.yml` 中的 `disabledCrawlers` 屬性中加入你想要禁用的推送 Tag\n\n### 此爬蟲目前主要負責用於我的私群專用機器人上，詳見 [mirai-val-bot](https://github.com/eric2788/miraivalbot)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feric2788%2Fplatformscrawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feric2788%2Fplatformscrawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feric2788%2Fplatformscrawler/lists"}