https://github.com/jayelbotvibe-web/webgecko
Pluck what you need. Climb through the web. — Fast, stealthy web scraping for AI agents.
https://github.com/jayelbotvibe-web/webgecko
ai-agents crawler curl-cffi html-parser lxml markdown-converter python scraping stealth tls-fingerprinting web-scraping
Last synced: about 1 month ago
JSON representation
Pluck what you need. Climb through the web. — Fast, stealthy web scraping for AI agents.
- Host: GitHub
- URL: https://github.com/jayelbotvibe-web/webgecko
- Owner: jayelbotvibe-web
- License: mit
- Created: 2026-06-29T04:56:51.000Z (about 1 month ago)
- Default Branch: master
- Last Pushed: 2026-06-29T05:51:30.000Z (about 1 month ago)
- Last Synced: 2026-06-29T07:05:20.512Z (about 1 month ago)
- Topics: ai-agents, crawler, curl-cffi, html-parser, lxml, markdown-converter, python, scraping, stealth, tls-fingerprinting, web-scraping
- Language: Python
- Homepage: https://pypi.org/project/webgecko/
- Size: 27.3 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Web Gecko
> Pluck what you need. Climb through the web.
[](https://github.com/jayelbotvibe-web/webgecko/actions/workflows/test.yml)
[](https://pypi.org/project/webgecko/)
[](https://pypi.org/project/webgecko/)
[](https://github.com/jayelbotvibe-web/webgecko/blob/master/LICENSE)
**Web Gecko** is a point-solution library — stealthy HTTP + HTML parsing for when you need to scrape a site that blocks plain `curl` or `requests` but doesn't require JavaScript. Built for AI agents that want structured data without writing parsing loops.
**What it does:** TLS impersonation, CSS/XPath with pseudo-elements, markdown conversion, structured extraction, concurrent crawling.
**What it doesn't:** Browser automation, JavaScript rendering, proxy rotation, CAPTCHA solving. This is a library, not a scraping framework.
```python
from gecko import fetch
r = fetch("https://httpbin.org/html", impersonate="chrome131")
print(r.page.title)
print(r.page.markdown[:200])
```
## Install
```bash
pip install webgecko
```
Python 3.10+. Depends on `lxml`, `cssselect`, `curl_cffi`, `anyio`.
## API
### Fetch
```python
from gecko import fetch, Session, AsyncSession
r = fetch("https://example.com", impersonate="chrome131")
r.status # 200
r.page # parsed Page (HTML) or placeholder (JSON)
r.json # parsed JSON body, or None
r.headers # case-insensitive dict
with Session(impersonate="firefox124") as s:
r = s.get("https://example.com")
async with AsyncSession() as s:
r = await s.get("https://example.com")
```
### Parse
```python
from gecko import Page
page = Page("...")
# Queries return Elements (iterable, indexable, .get() for first)
page.css(".title") # Elements
page.xpath("//h2") # Elements
page.find("Hello") # exact text
page.find("Hel", partial=True) # substring
page.find_all("div", class_="foo") # by tag + attrs
# Pseudo-elements — extract strings directly
page.css(".title::text").get() # "Widget A"
page.css("a::attr(href)").get() # "/buy/a"
# Element properties
el = page.css(".product")[0]
el.text # text content
el.tag # "div"
el.html # inner HTML
el.attr("href") # attribute value
```
### Agent-friendly shortcuts
```python
page.title # "My Page" — text
page.markdown # full page as markdown
page.links() # [{"text": "Link A", "href": "/a"}, ...]
page.jsonld() # [{"@type": "WebSite", ...}] — JSON-LD data
```
### Extract (agent-friendly structured output)
```python
# Map CSS selectors → field names. One call per group of elements.
page.css(".product").extract({
"name": ".title::text",
"price": ".price::text",
"link": "a::attr(href)",
})
# → [{"name": "Widget A", "price": "$9.99", "link": "/buy/a"}, ...]
# first=True returns a single dict or None
page.css("h1").extract({"title": "::text"}, first=True)
# → {"title": "Welcome"}
```
### Gecko
```python
class QuotesGecko(Gecko):
start_urls = ["https://quotes.toscrape.com/"]
concurrency = 4
def parse(self, response: Response):
yield from response.page.css(".quote").extract({
"text": ".text::text",
"author": ".author::text",
})
next_link = response.page.css(".next a::attr(href)").get()
if next_link:
yield response.follow(next_link, callback=self.parse)
result = QuotesGecko().run() # 100 quotes, 10 pages, ~3s
result.save("quotes.json")
```
## License
MIT