{"id":20148862,"url":"https://github.com/dev-chenxing/jjwxc-crawler","last_synced_at":"2025-04-09T19:52:06.958Z","repository":{"id":227014245,"uuid":"770133754","full_name":"dev-chenxing/jjwxc-crawler","owner":"dev-chenxing","description":"基于Scrapy开发的晋江爬虫，根据书号下载小说非V章节，生成可编辑的Word文档 | A simple tool to scrape and download non-V chapters of any novel from jjwxc.net in .docx format, built with Python and Scrapy","archived":false,"fork":false,"pushed_at":"2025-02-27T00:41:06.000Z","size":7403,"stargazers_count":13,"open_issues_count":1,"forks_count":4,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-23T21:46:01.525Z","etag":null,"topics":["chinese","cli","crawler","docx","download","jjwxc","open-source","python","scraping","scrapy","terminal","word"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dev-chenxing.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-11T01:34:06.000Z","updated_at":"2025-03-01T11:36:28.000Z","dependencies_parsed_at":"2024-05-17T13:52:23.926Z","dependency_job_id":"06fc9a3a-e89b-46bc-bfa1-231d03f32993","html_url":"https://github.com/dev-chenxing/jjwxc-crawler","commit_stats":null,"previous_names":["amaliegay/jjwxc-scrape","amaliegay/jjwxc-crawler","dev-chenxing/jjwxc-crawler"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dev-chenxing%2Fjjwxc-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dev-chenxing%2Fjjwxc-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dev-chenxing%2Fjjwxc-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dev-chenxing%2Fjjwxc-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dev-chenxing","download_url":"https://codeload.github.com/dev-chenxing/jjwxc-crawler/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248103903,"owners_count":21048244,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chinese","cli","crawler","docx","download","jjwxc","open-source","python","scraping","scrapy","terminal","word"],"created_at":"2024-11-13T22:39:44.489Z","updated_at":"2025-04-09T19:52:06.952Z","avatar_url":"https://github.com/dev-chenxing.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\u003ca href=\"https://www.jjwxc.net//\"\u003e\u003cimg src=\"public/logo.png\" alt=\"jjwxc-logo\" title=\"jjwxc\" width=\"220\"\u003e\u003c/a\u003e\u003c/div\u003e\r\n\r\n\u003cdiv\u003e\r\n  \u003ch1 align=\"center\"\u003e\r\n    《重生之我在绿江爪爪巴》\r\n  \u003c/h1\u003e\r\n  \u003cp align=\"center\"\u003e\r\n    一键下载\r\n    \u003ca href=\"https://www.jjwxc.net\"\u003e晋江文学城 (https://www.jjwxc.net)\u003c/a\u003e \r\n    网站小说非 V 章节\r\n  \u003c/p\u003e\r\n  \u003cp align=\"center\"\u003e\r\n      \u003cimg alt=\"language: python\" src=\"https://img.shields.io/badge/language-Python-118629\"\u003e\r\n      \u003ca href=\"https://www.github.com/labuladong\" target=\"_blank\"\u003e\u003cimg src=\"https://img.shields.io/badge/作者-@陈刑-689e75.svg?logo=GitHub\"\u003e\u003c/a\u003e\r\n      \u003cimg alt=\"release version\" src=\"https://img.shields.io/badge/release-v1.0.0-9ccca4\"\u003e\r\n      \u003cimg alt=\"last commit\" src=\"https://img.shields.io/github/last-commit/dev-chenxing/jjwxc-crawler?color=7fbc87\"\u003e\r\n  \u003c/p\u003e\r\n\u003c/div\u003e\r\n\r\n\u003ch4 align=\"center\"\u003e\r\n    \u003cp\u003e\r\n        \u003cb\u003e简体中文\u003c/b\u003e |\r\n        \u003ca href=\"https://github.com/dev-chenxing/jjwxc-crawler/blob/main/README_en.md\"\u003eEnglish\u003c/a\u003e\r\n    \u003c/p\u003e\r\n\u003c/h4\u003e\r\n\r\n### 特点功能\r\n\r\n-   命令行界面\r\n-   支持输出 DOCX 和 TXT 格式\r\n-   可自定义输出路径\r\n-   ...................\r\n\r\n有建议或 bug 可以提 issue.\r\n\r\n命令行界面使用命令行 UI 库[Rich](https://github.com/Textualize/rich)编写。\r\n\r\n界面样例：\r\n\r\n\u003cdiv align=\"center\"\u003e\r\n  \u003cimg src=\"public/preview.gif\" width=\"800px\"/\u003e\r\n\u003c/div\u003e\r\n\r\n# 安装文档\r\n\r\n### 下载文件\r\n\r\n点击 Code - Download ZIP，下载后解压缩得到文件夹，建议重命名为`jjwxc-crawler`\r\n\r\n### 环境配置\r\n\r\n-   Python 3.9.15\r\n-   Windows\r\n\r\n安装 Python 后，第一步，打开所在目录的命令行，输入以下命令创建并激活虚拟环境\r\n\r\n```powershell\r\npython -m venv venv   # 创建名为venv的Python虚拟环境\r\nvenv\\Scripts\\activate # Windows系统下激活虚拟环境venv\r\n```\r\n\r\n在Linux系统下，\r\n\r\n```bash\r\nchmod +x venv/bin/activate \r\nsource venv/bin/activate \r\n```\r\n\r\n此时命令行前应显示有`(venv)`，表示当前已激活虚拟环境`venv`\r\n\r\n第二步，在虚拟环境内安装 Scrapy 和其他依赖\r\n\r\n```powershell\r\npip install -r requirements.txt\r\n```\r\n\r\n### 运行小程序\r\n\r\n```powershell\r\n# 进入程序所在目录\r\ncd jjcrawler\r\n\r\n# 运行爬虫命令，其中ID为书号\r\nscrapy crawl novel -a id=ID\r\n\r\n# 例如，我要下载书号为2的测试文，则运行以下命令行\r\nscrapy crawl novel -a id=2\r\n```\r\n\r\n下载章节将保存至根目录下的 novels 文件夹\r\n\r\n默认输出格式为.docx，如果要更改为.txt 格式输出，可编辑`\\jjcrawler\\jjcrawler\\spiders\\config.py`中参数\r\n\r\n```python\r\n# docx | txt\r\nformat = \"txt\"\r\n```\r\n\r\n下载一整页的小说\r\n\r\n```bash\r\n# 无CP-女主视角-仙侠修真标签\r\nscrapy crawl novellist -a xx=5 -a mainview=2 -a bq=68\r\n# 无CP-女主视角-古色古香-仙侠类型\r\nscrapy crawl novellist -a xx=5 -a mainview=2 -a sd=2 -a lx=4\r\n# 衍生-百合-武侠\r\nscrapy crawl novellist -a yc=2 -a xx=3 -a bq=11\r\n# 标题含有\"神雕\"的百合小说 (Experiental option, still in development)\r\nscrapy crawl novellist -a title=神雕 -a xx=3\r\n```\r\n\r\n**[⬆ 回到顶部](#特点功能)**\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdev-chenxing%2Fjjwxc-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdev-chenxing%2Fjjwxc-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdev-chenxing%2Fjjwxc-crawler/lists"}