{"id":25878268,"url":"https://github.com/djchanahcjd/osintool","last_synced_at":"2026-02-23T08:34:54.140Z","repository":{"id":280261503,"uuid":"941451432","full_name":"DJChanahCJD/OSINTool","owner":"DJChanahCJD","description":"OSINTool 是一个自动化爬虫客户端，包含一个Flask后端和一个Electron客户端，用于执行和管理爬虫任务。","archived":false,"fork":false,"pushed_at":"2025-03-02T11:06:24.000Z","size":98814,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-02T11:27:12.589Z","etag":null,"topics":["apscheduler","electron","element-ui","flask","html-css-javascript","pandas","playwright","requests","tinydb","vue2"],"latest_commit_sha":null,"homepage":"https://github.com/DJChanahCJD/OSINTool","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DJChanahCJD.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-02T10:23:56.000Z","updated_at":"2025-03-02T11:06:27.000Z","dependencies_parsed_at":"2025-03-02T11:27:34.297Z","dependency_job_id":"87993dba-a51b-44b0-ba6b-abd9094c1070","html_url":"https://github.com/DJChanahCJD/OSINTool","commit_stats":null,"previous_names":["djchanahcjd/osintool"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DJChanahCJD%2FOSINTool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DJChanahCJD%2FOSINTool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DJChanahCJD%2FOSINTool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DJChanahCJD%2FOSINTool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DJChanahCJD","download_url":"https://codeload.github.com/DJChanahCJD/OSINTool/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241504376,"owners_count":19973231,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apscheduler","electron","element-ui","flask","html-css-javascript","pandas","playwright","requests","tinydb","vue2"],"created_at":"2025-03-02T12:24:04.404Z","updated_at":"2025-10-27T22:15:16.167Z","avatar_url":"https://github.com/DJChanahCJD.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"## 项目概述\nOSINTool 是一个自动化爬虫客户端，包含一个Flask后端和一个Electron客户端，用于执行和管理爬虫任务。\n\u003e [网页端仓库](https://github.com/DJChanahCJD/OSINTool/tree/osintool-web)\n\n\n## 快速开始\n\n### 安装依赖\n\n在`frontend`目录下运行以下命令安装**前端依赖**：\n\n```bash\nnpm install\n```\n\n在项目根目录下运行以下命令安装**后端依赖**：\n\n```bash\npip install flask flask-cors flask[async] tinydb shortuuid requests lxml pandas apscheduler playwright\nplaywright install\n```\n\n### 启动前后端\n\n在`根目录`启动**Flask后端**：\n\n```bash\npython app.py\n```\n\n在`frontend`目录启动**Electron客户端**：\n\n\n```bash\nnpm start\n```\n\n\u003e 请确保已经安装了 Playwright 所需的浏览器和驱动，可通过 `playwright install` 命令进行安装。\n\n\n\n## 技术栈\n### 前端\n- 页面构建：HTML/CSS/JS\n- 客户端开发：Electron\n- 组件库：ElementUI\n\n### 后端\n- 本地服务器：Flask\n- 数据爬取：Playwirght, Requests\n- 定时任务：APSchedule\n- 数据库：TinyDB\n\n\n\n## 项目结构\n\n```\n├── frontend\n│   ├── edit.html \t \t\t# 编辑页\n│   ├── index.html  \t\t\t# 列表页\n│   ├── main.js \t\t\t# Electron主进程\n│   ├── package.json\n│   ├── preload.js  \t\t\t# 预加载脚本（含API）\n├── utils\n│   ├── parsers  \t\t\t# 解析器\n│   │   ├── base.py\n│   │   ├── csv_parser.py\n│   │   ├── html_parser.py\n│   │   └── txt_parser.py\n│   ├── common.py  \t\t\t# 通用函数\n│   ├── logger.py  \t\t\t# 日志模块\n│   ├── parser_factory.py  \t        # 解析器工厂\n├── app.py  \t\t\t\t# Flask后端（含API）\n├── db.json  \t\t\t\t# TinyDB数据库\n```\n\n\n\n## 预览图\n\n### 任务列表页（index.html）\n\n![image-20250301183926480](https://github.com/user-attachments/assets/7e8be9f8-efa2-4b7f-8429-9a4974c89b31)\n\n\n### 任务编辑页（edit.html）\n\n![image-20250301183707849](https://github.com/user-attachments/assets/0b32050e-dc01-40e5-af55-4ee804f3d2bd)\n\n\n\n\n## Todo\n\n- 支持CSV输出\n- 处理OCR/滑动验证码等人机验证\n- CI/CD\n- 支持网页端部署\n\n## SWOT分析\n\n- **Strengths（优势）**：支持多类型数据解析，自定义程度高 （登录、翻页等），任务调度灵活。\n- **Weaknesses（劣势）**：学习成本高，需网页分析与正则基础；可视化差，没有直观界面；暂时无法应对人机验证等反爬技术。\n- **Opportunities（机会）**：未来可融合AI实现智能爬取，集成如Excel等工具方便数据分析。\n- **Threats（威胁）**：市场已有八爪鱼、EasySpider 等成熟可视化爬取工具；网页反爬技术迭代快 ，需持续更新爬取技术。\n\n\n\n## 参考项目\n- [八爪鱼采集器](https://www.bazhuayu.com/)\n- [EasySpider](https://www.easyspider.net/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdjchanahcjd%2Fosintool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdjchanahcjd%2Fosintool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdjchanahcjd%2Fosintool/lists"}