{"id":49287226,"url":"https://github.com/siveci/javdb_magnet_spider","last_synced_at":"2026-06-06T09:01:28.004Z","repository":{"id":350895108,"uuid":"1208686378","full_name":"SiVeci/JavDB_magnet_Spider","owner":"SiVeci","description":"基于 Python 的 JavDB 磁力链接自动化爬虫。采用 curl_cffi 完美模拟浏览器 TLS 指纹绕过 Cloudflare 防火墙。支持多页列表抓取，根据“无码/中字/高清”等标签及文件大小，自动筛选并导出最优的磁力链接至 CSV 文件。","archived":false,"fork":false,"pushed_at":"2026-06-05T17:02:55.000Z","size":1619,"stargazers_count":11,"open_issues_count":0,"forks_count":2,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-05T18:46:53.720Z","etag":null,"topics":["crawler","data-extraction","javdb","magnet-links","python","python3","scraper","spider"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SiVeci.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-12T16:06:54.000Z","updated_at":"2026-06-05T17:03:00.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/SiVeci/JavDB_magnet_Spider","commit_stats":null,"previous_names":["siveci/javdb_magnet_spider"],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/SiVeci/JavDB_magnet_Spider","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SiVeci%2FJavDB_magnet_Spider","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SiVeci%2FJavDB_magnet_Spider/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SiVeci%2FJavDB_magnet_Spider/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SiVeci%2FJavDB_magnet_Spider/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SiVeci","download_url":"https://codeload.github.com/SiVeci/JavDB_magnet_Spider/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SiVeci%2FJavDB_magnet_Spider/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33975476,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-06T02:00:07.033Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","data-extraction","javdb","magnet-links","python","python3","scraper","spider"],"created_at":"2026-04-25T22:03:41.494Z","updated_at":"2026-06-06T09:01:27.999Z","avatar_url":"https://github.com/SiVeci.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"spider_core/frontend/logo.png\" alt=\"JavDB Magnet Spider Logo\" width=\"180\"\u003e\n  \u003ch1\u003eJavDB Magnet Spider\u003c/h1\u003e\n  \u003cp\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Platform-Android%20%7C%20Docker%20%7C%20PC-brightgreen\" alt=\"Platform\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Python-3.12-blue\" alt=\"Python Version\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Framework-FastAPI%20%7C%20Chaquopy-orange\" alt=\"Framework\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/License-MIT-lightgrey\" alt=\"License\"\u003e\n  \u003c/p\u003e\n\u003c/div\u003e\n\n---\n\n## 🌟 项目简介\n\n跨平台的自动化JavDB爬虫工具，支持在Docker、PC、Android 移动端运行。本项目实现了在手机上**手动过盾、后台静默爬取、WebUI 远程控制**的闭环体验，能批量抓取指定目录，并根据自定义规则（中文字幕、无码、高清等）自动挑选出质量最高的磁力链接。\n\n## ✨ 核心特性\n\n- **Android 原生支持**：基于 Chaquopy 框架，在手机内运行完整的 FastAPI + Uvicorn 后端。\n- **Android 端**：利用原生 WebView 手动登录，Cookie 自动无缝同步。\n- **PC/Docker 端**：采用 `curl_cffi` 模拟 TLS 指纹，稳定穿透防护。\n- **Web 控制台**：适配移动＆桌面屏幕尺寸，支持实时日志查看与任务管理。\n- **自动选择磁力**：基于权重算法（无码+100 \u003e 高清+10 \u003e 字幕+1）自动筛选出最高质量资源。\n- **断点续传系统**：支持任务挂起与救援，Cookie 失效后补充即可接力。\n- **Docker 化部署**\n\n---\n\n## 🚀 快速开始\n\n### 方案一：Android 手机端\n1. **下载安装**：前往 [Releases] 下载最新的 APK 文件。\n2. **三步走启动**：\n   - **第一步**：点击 `1. 手动登录过盾`。在弹出的网页中完成 JavDB 登录并过掉 CF 验证，找到需要爬取的目标页面后点击 **「复制当前链接」**，最后点击关闭（Cookie 将自动接管）。\n   - **第二步**：点击 `2. 启动爬虫引擎`。按提示授予**通知权限**与**悬浮窗权限**。\n   - **第三步**：点击 `3. 打开 WebUI`。系统将跳转至浏览器访问 `127.0.0.1:8000`，即可开始配置爬取任务。\n\n### 方案二：Docker 部署\n```bash\ndocker run -d \\\n  --name=javdb-spider \\\n  -p 8090:8000 \\\n  -e JAVDB_AUTH_TOKEN=请替换为你的访问令牌 \\\n  -v /你的路径/appdata/javdb_spider/data:/app/data \\\n  --restart=unless-stopped \\\n  ghcr.io/你的用户名/javdb_spider:latest\n```\n*访问地址：`http://NAS_IP:8090`*\n\n### 方案三：本地 Python 运行\n```bash\n# 1. 安装依赖\npip install -r requirements.txt\n\n# 2. 进入核心目录并启动\n# (可选) 设置 JAVDB_AUTH_TOKEN 环境变量启用 API 鉴权保护\ncd spider_core\npython -m uvicorn main:app --host 0.0.0.0 --port 8000\n```\n\n---\n\n## 🛠️ 使用指南\n\n### 1. 爬取逻辑配置\n- **起始页面 URL**：进入老师主页，筛选好想要抓取的 Tag（如“单体”、“高清”），复制浏览器地址栏链接（支持在WebUI中动态获取过滤标签）。\n  - **Android 端**：建议直接在“手动登录过盾”的内置浏览器中浏览，并点击「复制当前链接」，然后直接在 WebUI 中粘贴，全程无需切换外部浏览器以防 Cookie 失效。\n- **Cookie \u0026 UA**：\n  - **Android 端**：自动接管，无需手动填写（注意不要留空，随便填，避免触发前台检测）。\n  - **PC/Docker 端**：按 `F5` 刷新页面，在开发者工具 `Network` 选项卡的第一个请求头中获取。\n\n### 2. 电池优化 (Android)\n为了防止安卓系统在息屏后杀掉爬虫进程，请进行以下设置：\n- **开启悬浮窗权限**（核心要求）。\n- 在“系统设置” -\u003e “应用管理”中，将本 App 的电池策略设为**无限制**。\n- 在多任务界面为 App **加锁**。\n\n---\n\n## 📁 项目结构 (Monorepo)\n```text\n├── app/                # Android 原生 Java 代码与资源\n├── spider_core/        # 核心 Python 逻辑 (三端共享)\n│   ├── frontend/       # WebUI 静态资源\n│   ├── main.py         # FastAPI 路由入口\n│   └── spider_engine.py # 爬虫核心引擎\n├── Dockerfile          # Docker 构建脚本\n├── build.gradle        # 安卓构建配置\n└── requirements.txt    # Python 依赖清单\n```\n\n---\n\n## ⚙️ 系统要求\n\n| 维度 | 要求 |\n| :--- | :--- |\n| **Android** | Android 8.0+ (推荐 11+), arm64-v8a 架构 |\n| **Docker** | 支持 linux/amd64, linux/arm64 (支持树莓派) |\n| **Python** | 推荐 3.12.x |\n\n---\n\n## ⚠️ 免责声明\n1. **仅供学习**：本项目仅用于 Python 爬虫技术研究与 Chaquopy 框架实践，请勿用于非法用途。\n2. **合规使用**：请遵守目标网站的 `robots.txt` 协议，合理控制爬取频率。\n3. **隐私安全**：本项目为开源软件，不收集任何用户 Cookie 信息。用户需妥善保管个人凭据。\n---\n\n## v1.3 安全说明\n\n- Docker 镜像默认启用 API 访问保护，请通过 `JAVDB_AUTH_TOKEN` 设置访问令牌；WebUI 首次访问时会提示输入令牌。\n- WebUI 的 Cookie 默认只保存在当前浏览器会话中；勾选“记住 Cookie”后才会写入 `localStorage`。\n- Android WebView 抓取增加 45 秒超时保护，避免页面卡住时 Python 线程永久阻塞。\n- 引入文件写入与防路径穿越机制，提高底层文件的安全性。\n- 本版本暂不迁移 Android release 签名，仍保留现有本地编译方式；后续建议再迁移到安全的 Secrets 流程。\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsiveci%2Fjavdb_magnet_spider","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsiveci%2Fjavdb_magnet_spider","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsiveci%2Fjavdb_magnet_spider/lists"}