https://github.com/exp-codes/python-crawler-template

Python 爬虫开发模板
https://github.com/exp-codes/python-crawler-template

crawler programming template

Last synced: 8 months ago
JSON representation

Python 爬虫开发模板

• Host: GitHub
• URL: https://github.com/exp-codes/python-crawler-template
• Owner: EXP-Codes
• License: apache-2.0
• Created: 2021-11-07T06:36:22.000Z (about 4 years ago)
• Default Branch: master
• Last Pushed: 2023-01-16T17:51:44.000Z (almost 3 years ago)
• Last Synced: 2025-03-30T19:23:19.096Z (9 months ago)
• Topics: crawler, programming, template
• Language: Python
• Homepage: https://lyy289065406.github.io/python-crawler-template/
• Size: 122 KB
• Stars: 3
• Watchers: 2
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Funding: .github/FUNDING.yml
  • License: LICENSE

Awesome Lists containing this project

README

          
# python-crawler-template

> Python 爬虫开发模板

------

## 运行环境

![](https://img.shields.io/badge/Python-3.8%2B-brightgreen.svg) ![](https://img.shields.io/badge/Platform-Linux%20amd64-brightgreen.svg) ![](https://img.shields.io/badge/Platform-Windows%20x64-brightgreen.svg)

## 使用说明

1. 创建 Github Repository 时选择这个仓库做模板

2. 修改 [`script/crawler-create.sql`](./script/crawler-create.sql) 建库脚本

3. 运行 [`python gen_pdm.py`](./gen_pdm.py) 脚本生成数据库 pdm 代码

4. 修改 [`src`](./src) 下的爬虫代码（`FIXME` 和 `TODO` 标记的都需要根据场景修改），运行 [`python main.py`](./main.py) 启动爬虫

5. 修改 [`autorun.yml`](./.github/workflows/autorun.yml)，可通过 Github Actions 自动运行

6. 开启 Github Pages，指定目录为 master/docs