https://github.com/miskler/human-requests
Asynchronous library for browser‑like HTTP scenarios with controlled offline rendering and two‑way state transfer.
https://github.com/miskler/human-requests
anti-bot antibot asyncio curl-cffi playwright python3 scaping scraper
Last synced: 19 days ago
JSON representation
Asynchronous library for browser‑like HTTP scenarios with controlled offline rendering and two‑way state transfer.
- Host: GitHub
- URL: https://github.com/miskler/human-requests
- Owner: Miskler
- License: mit
- Created: 2025-09-03T10:48:40.000Z (about 1 month ago)
- Default Branch: main
- Last Pushed: 2025-09-13T21:53:43.000Z (21 days ago)
- Last Synced: 2025-09-13T22:09:40.067Z (21 days ago)
- Topics: anti-bot, antibot, asyncio, curl-cffi, playwright, python3, scaping, scraper
- Language: Python
- Homepage: https://pypi.org/project/human-requests/
- Size: 793 KB
- Stars: 1
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# 🧰 Human Requests
*Asynchronous library for browser‑like HTTP scenarios with controlled offline rendering and two‑way state transfer.*
[](https://miskler.github.io/human-requests/tests/tests-report.html)
[](https://miskler.github.io/human-requests/coverage/)
[](https://python.org)
[](https://pypi.org/project/human-requests/)
[](LICENSE)
[](https://github.com/psf/black)
[](https://mypy.readthedocs.io/en/stable/index.html)
[](https://discord.gg/UnJnGHNbBp)
[](https://t.me/miskler_dev)
**[⭐ Star us on GitHub](https://github.com/Miskler/human-requests)** | **[📚 Read the Docs](https://miskler.github.io/human-requests/quick_start)** | **[🐛 Report a Bug](https://github.com/Miskler/human-requests/issues)**
## ✨ Features
- **HTTP by default.** Direct requests via `curl_cffi` in *impersonate* mode + real browser headers generation.
- **Browser on demand.** Offline render of an already received response (no repeated HTTP) and JS execution.
- **Unified state.** Two‑way transfer of **cookies** and **`localStorage`** between HTTP and the browser (storage_state ⇄ session).
- **Async by design.** Native `asyncio` for predictable concurrency.
## 🚀 Quick Start
### Installation
```bash
pip install human-requests[playwright-stealth]
playwright install
```
### Direct request *(pretend to be a browser)*
```python
import asyncio
from human_requests import Session, HttpMethod
async def main():
async with Session(headless=True, browser="camoufox") as s:
resp = await s.request(HttpMethod.GET, "https://target.example/")
print(resp.status_code, len(resp.text))
asyncio.run(main())
```
### Render an already received response *(without another request)*
```python
# resp — the result of an HTTP request
async with resp.render(wait_until="networkidle") as page:
await page.wait_for_selector("#content")
# after exiting:
# - cookies and localStorage are synced back into the session
```
### Warm‑up: inject `localStorage` BEFORE page start
```python
origin = "https://target.example"
async with Session(headless=True, browser="camoufox") as s:
# prepare storage_state in advance
s.local_storage.setdefault(origin, {})
s.local_storage[origin]["seen"] = "1"
s.local_storage[origin]["ab_variant"] = "B"
# the browser starts with the required values already in place
async with s.goto_page(f"{origin}/", wait_until="networkidle"):
pass
```
### Accessing state
```python
# Cookies:
print(s.cookies.storage)
# LocalStorage:
print(s.local_storage.get("https://target.example", {}))
```
## Key Characteristics
- HTTP impersonation: `curl_cffi` + browser‑grade headers on every request.
- Offline render: first response interception (fulfill) and soft reloads without recreating contexts.
- State as a first‑class citizen: cookies and `localStorage` sync both ways.
- Unified proxy layer: single proxy format → for `curl_cffi` and Playwright.
- Clean stack: no external Go binaries.
## Comparison: human-requests vs hrequests
| Aspect | human-requests | hrequests |
|---|---|---|
| Execution model | `asyncio` (native) | sync + threads/gevent |
| HTTP impersonation | `curl_cffi` impersonate + per‑request browser headers | `tls-client` (Go backend) |
| Offline `Response` render | Yes (fulfill + soft‑reload; no repeated HTTP) | Yes (post‑render with cookies/content update) |
| Cookies ↔ HTTP/Browser | Two‑way transfer | Two‑way transfer |
| `localStorage` ↔ HTTP/Browser | First‑class (storage_state ⇄ session) | Via `page.evaluate(...)` |
| Typing | mypy‑friendly | — |
| Dependencies | No Go binaries | Go backend (`tls-client`) |
| Built‑in HTML parser | — | `selectolax` |
> The focus of human-requests is a **controlled** anti‑bot pipeline in `asyncio`: HTTP by default, a browser only where needed, with state hand‑off.
## 🛠️ Development
### Setup
```bash
git clone https://github.com/Miskler/human-requests.git
cd human-requests
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
make build
make install-dev
```
### Commands
```bash
# Checks
pytest # tests + coverage
make lint # ruff/flake8/isort/black (if enabled)
make type-check # mypy/pyright
# Actions
make format # formatting
make docs # build documentation
```
### Dev: local test server
```bash
# from the test_server/ folder
make serve # foreground (Ctrl+C to stop)
make stop # stop background process
```