https://github.com/miskler/human-requests
Asynchronous library for browser‑like HTTP scenarios with controlled offline rendering and two‑way state transfer.
https://github.com/miskler/human-requests
anti-bot antibot asyncio curl-cffi playwright python3 scaping scraper
Last synced: 28 days ago
JSON representation
Asynchronous library for browser‑like HTTP scenarios with controlled offline rendering and two‑way state transfer.
- Host: GitHub
- URL: https://github.com/miskler/human-requests
- Owner: Miskler
- License: mit
- Created: 2025-09-03T10:48:40.000Z (10 months ago)
- Default Branch: main
- Last Pushed: 2025-09-13T21:53:43.000Z (9 months ago)
- Last Synced: 2025-09-13T22:09:40.067Z (9 months ago)
- Topics: anti-bot, antibot, asyncio, curl-cffi, playwright, python3, scaping, scraper
- Language: Python
- Homepage: https://pypi.org/project/human-requests/
- Size: 793 KB
- Stars: 1
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Human Requests
*Asynchronous Playwright wrappers for browser-like HTTP scenarios, controlled render flow, and API autotest integration.*
[](https://miskler.github.io/human-requests/tests/tests-report.html)
[](https://miskler.github.io/human-requests/coverage/)
[](https://python.org)
[](https://pypi.org/project/human-requests/)
[](LICENSE)
[](https://github.com/psf/black)
[](https://mypy.readthedocs.io/en/stable/index.html)
[](https://discord.gg/UnJnGHNbBp)
[](https://t.me/miskler_dev)
**[Star us on GitHub](https://github.com/Miskler/human-requests)** | **[Read the Docs](https://miskler.github.io/human-requests/quick_start)** | **[Report a Bug](https://github.com/Miskler/human-requests/issues)**
## Features
- Typed wrappers over Playwright primitives:
- `HumanBrowser`
- `HumanContext`
- `HumanPage`
- `HumanPage.fetch(...)`: execute HTTP requests from page context and get structured `FetchResponse`.
- `HumanPage.goto_render(...)`: render already available response payloads without duplicate upstream request.
- Storage helpers:
- `HumanContext.local_storage()` for full context snapshot
- `HumanPage.local_storage()` for current page origin
- `HumanPage.cookies()` convenience alias
- Fingerprint snapshot collection: `HumanContext.fingerprint(...)`.
- Built-in pytest autotest plugin for API clients (`@autotest`, hooks, params, dependencies).
## Installation
Base package:
```bash
pip install human-requests
playwright install chromium
```
Optional autotest addon dependencies:
```bash
pip install human-requests[autotest] pytest pytest-anyio pytest-jsonschema-snapshot pytest-subtests
```
If you run with Camoufox, install it separately:
```bash
pip install camoufox
camoufox fetch
```
## Quick Start
### Wrap a Playwright browser
```python
import asyncio
from playwright.async_api import async_playwright
from human_requests import HumanBrowser
async def main() -> None:
async with async_playwright() as p:
pw_browser = await p.chromium.launch(headless=True)
browser = HumanBrowser.replace(pw_browser)
ctx = await browser.new_context()
page = await ctx.new_page()
await page.goto("https://httpbin.org/html", wait_until="domcontentloaded")
print(page.url)
await browser.close()
asyncio.run(main())
```
### Direct request in page context (`fetch`)
```python
resp = await page.fetch("https://httpbin.org/json")
print(resp.status_code)
print(resp.json())
```
### Render previously fetched response (`goto_render`)
```python
challenge = await page.fetch("https://example.com/challenge")
await page.goto_render(challenge, wait_until="networkidle")
```
### Auto Screenshot on Handler Failure
If you want a screenshot to be taken when errors like `playwright.Error` (including timeouts) occur,
you can set `page.on_error_screenshot_path = "screenshot.png"` (this setting is page-specific; by default, screenshots are disabled).
### State helpers
```python
cookies = await page.cookies()
context_storage = await ctx.local_storage()
page_storage = await page.local_storage()
print(len(cookies), context_storage.keys(), page_storage.keys())
```
### Fingerprint snapshot
```python
fingerprint = await ctx.fingerprint(origin="https://example.com")
print(fingerprint.user_agent)
print(fingerprint.browser_name, fingerprint.browser_version)
```
## API Tree Boilerplate Helper
To avoid repetitive `_parent` and `__post_init__` wiring in SDK-style clients
(like `fixprice_api` / `perekrestok_api`), use:
- `ApiChild[ParentType]`
- `ApiParent`
- `api_child_field(...)`
```python
from dataclasses import dataclass
from human_requests import ApiChild, ApiParent, api_child_field
class ClassCatalog(ApiChild["ShopApi"]):
async def tree(self):
...
class ClassGeolocation(ApiChild["ShopApi"]):
async def cities_list(self):
...
@dataclass
class ShopApi(ApiParent):
Catalog: ClassCatalog = api_child_field(ClassCatalog)
Geolocation: ClassGeolocation = api_child_field(ClassGeolocation)
```
`ApiParent` initializes all `api_child_field(...)` values in `__post_init__`
automatically, so manual assignments are no longer needed.
Nested chains are supported as well (`Root -> Child -> Child`):
```python
@dataclass
class BranchApi(ApiChild["RootApi"], ApiParent):
Catalog: ClassCatalog = api_child_field(ClassCatalog)
@dataclass
class RootApi(ApiParent):
Branch: BranchApi = api_child_field(BranchApi)
```
## API Autotest Addon (pytest)
`human-requests` ships with a pytest plugin that can auto-run API methods marked with `@autotest` and validate payloads via `schemashot` from `pytest-jsonschema-snapshot`.
With `pytest-subtests`, each discovered `@autotest` method and `@autotest_data`
provider is shown as a separate subtest inside `test_autotest_api_methods`.
Minimal `pytest.ini`:
```ini
[pytest]
anyio_mode = auto
autotest_start_class = your_package.StartClass
autotest_typecheck = warn
```
`autotest_typecheck` modes:
- `off` (default): no runtime type checks for params provider arguments
- `warn`: emit `RuntimeWarning` on annotation mismatch
- `strict`: fail test case with `TypeError` on mismatch
Minimal fixtures:
```python
import pytest
from your_package import StartClass
@pytest.fixture(scope="session")
def anyio_backend() -> str:
return "asyncio"
@pytest.fixture(scope="session")
async def api() -> StartClass:
async with StartClass() as client:
yield client
```
Business code only marks methods:
```python
from human_requests import autotest
class Catalog:
@autotest
async def tree(self):
...
```
Test layer adds hooks and params:
```python
from human_requests import autotest_depends_on, autotest_hook, autotest_params
from human_requests.autotest import AutotestCallContext, AutotestContext
@autotest_hook(target=Catalog.tree)
def _capture_category(_resp, data, ctx: AutotestContext) -> None:
ctx.state["category_id"] = data["items"][0]["id"]
@autotest_depends_on(Catalog.tree)
@autotest_params(target=Catalog.feed)
def _feed_params(ctx: AutotestCallContext) -> dict[str, int]:
return {"category_id": ctx.state["category_id"]}
```
Parent-specific registration is supported:
```python
@autotest_hook(target=Child.method, parent=ParentA)
def _only_for_parent_a(_resp, data, ctx):
...
```
For a complete guide, see `docs/source/autotest.rst`.
## Development
Setup:
```bash
git clone https://github.com/Miskler/human-requests.git
cd human-requests
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -r requirements.txt
pip install -e .
```
Commands:
```bash
pytest
make lint
make type-check
make format
make docs
```