https://github.com/qiannianshuangxue/safe_message_spider
About Solve CTF offline disconnection problem - based on python3's small crawler, support keyword search and local map bed establishment, currently support Jianshu, xianzhi,anquanke,freebuf,seebug
https://github.com/qiannianshuangxue/safe_message_spider
Last synced: 4 months ago
JSON representation
About Solve CTF offline disconnection problem - based on python3's small crawler, support keyword search and local map bed establishment, currently support Jianshu, xianzhi,anquanke,freebuf,seebug
- Host: GitHub
- URL: https://github.com/qiannianshuangxue/safe_message_spider
- Owner: qiannianshuangxue
- Created: 2021-11-29T12:17:44.000Z (almost 4 years ago)
- Default Branch: main
- Last Pushed: 2023-05-17T09:08:17.000Z (over 2 years ago)
- Last Synced: 2024-11-21T18:39:49.577Z (12 months ago)
- Language: Python
- Size: 314 KB
- Stars: 42
- Watchers: 1
- Forks: 10
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome-hacking-lists - qiannianshuangxue/safe_message_spider - About Solve CTF offline disconnection problem - based on python3's small crawler, support keyword search and local map bed establishment, currently support Jianshu, xianzhi,anquanke,freebuf,seebug (Python)
README
## 前言
某次CTF线下比赛断网,想着搞点资料,而后参考了书鱼大佬的代码,搞了一套获取资料的小工具。
### 在原作者的基础上补充了安全客,freebuf,seebug完善了先知的部分代码
原作者项目地址 https://github.com/bkfish/html2markdown_Spider
### 参数说明
- -s 需要查找的关键字，空代表按时间顺序爬取
- -c 需要文章的数目，默认30
- -p 是否需要建立本地图床，默认0 0代表不需要 1代表需要
### 安装使用
安装如下
```bash
git clone https://github.com/qiannianshuangxue/safe_message_spider
cd xianzhiSpider
pip install -r requirements.txt
```
#### 先知
使用前注意设置headers的cookie
```
python3 xianzhiSpider.py -s CTF -c 5 -p 1
```
#### seebug
无需使用其他参数
```
python3 seebug_Spider.py
```
#### freebuf
无需使用其他参数
```
python3 freebufSpider.py
```
#### anquanke
无需使用其他参数
```
python3 anquankeSpider.py
```
### 查看下载的文档
由于我个人习惯使用VSCODE来查看以及搜索关键词,因此并没有开发一套查看工具。
### 结果
包括图床一共也只有26个G左右
效果图如下
仅供学习，禁止商用行为
### 2023年4月更新
略微修改爬虫，适应部分更新后的网站。