https://github.com/imfht/webhelper
https://github.com/imfht/webhelper
Last synced: 4 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/imfht/webhelper
- Owner: imfht
- Created: 2022-04-24T09:29:31.000Z (over 3 years ago)
- Default Branch: master
- Last Pushed: 2022-06-15T06:52:31.000Z (over 3 years ago)
- Last Synced: 2024-08-05T17:44:16.661Z (over 1 year ago)
- Language: Python
- Size: 24.4 KB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome-hacking-lists - imfht/webhelper - (Python)
README
# WebHelper
Web 爬虫小工具
## 安装
```bash
pip install web-helper
```
## 使用方法
```python
from web_helper import *
title = get_html(f'https://baidu.com', obey_robot=False).to_soup().find('title').text
assert title == '百度一下，你就知道'
```
## 使用代理
```python
from web_helper import *
set_global_random_proxy('ws_token') # 设置全局代理,TOKEN 前往: https://proxy.webshare.io/subscription/ 获取
title = get_html(f'https://baidu.com', obey_robot=False).to_soup().find('title').text
assert title == '百度一下，你就知道'
```
## 设置日志级别
```python
from web_helper import *
import sys
logger.remove()
logger.add(sys.stdout, level="TRACE")
title = get_html(f'https://baidu.com', obey_robot=False).to_soup().find('title').text
assert title == '百度一下，你就知道'
```