Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/byjrk/pal-dos-datebase
https://github.com/byjrk/pal-dos-datebase
crawler python
Last synced: 17 days ago
JSON representation
- Host: GitHub
- URL: https://github.com/byjrk/pal-dos-datebase
- Owner: BYJRK
- Created: 2021-10-23T08:09:40.000Z (over 3 years ago)
- Default Branch: master
- Last Pushed: 2021-10-23T08:10:30.000Z (over 3 years ago)
- Last Synced: 2024-11-25T10:40:47.176Z (3 months ago)
- Topics: crawler, python
- Language: Python
- Homepage:
- Size: 6.84 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: readme.md
Awesome Lists containing this project
README
# PAL web crawler
抓取[外塞之雾](http://www.whistlerwmz.my-place.us/)网站上的仙剑相关的数据（目前只做了怪物信息的抓取）
## 使用方法
首先搭建环境，安装 `requirements.txt` 中列出的包。其中，`playwright` 需要额外的安装，以添加 web driver。
```shell
pip install -r requirements.txt
python -m playwright install
```
然后运行 `retrieve.py` 将数据抓取到本地（我已经提交了抓取好的 `.json` 文件，直接使用即可，网站上应该也不会有什么更新了）。
之后再运行 `display.py` 即可查看结果。
```shell
# optional
python retrieve.py
# see the results
python display.py
```