{"id":51432876,"url":"https://github.com/jayelbotvibe-web/webgecko","last_synced_at":"2026-07-05T05:02:44.006Z","repository":{"id":368121828,"uuid":"1283619463","full_name":"jayelbotvibe-web/webgecko","owner":"jayelbotvibe-web","description":"Pluck what you need. Climb through the web. — Fast, stealthy web scraping for AI agents.","archived":false,"fork":false,"pushed_at":"2026-06-29T05:51:30.000Z","size":28,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-06-29T07:05:20.512Z","etag":null,"topics":["ai-agents","crawler","curl-cffi","html-parser","lxml","markdown-converter","python","scraping","stealth","tls-fingerprinting","web-scraping"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/webgecko/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jayelbotvibe-web.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-29T04:56:51.000Z","updated_at":"2026-06-29T05:51:34.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/jayelbotvibe-web/webgecko","commit_stats":null,"previous_names":["jayelbotvibe-web/webgecko"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/jayelbotvibe-web/webgecko","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jayelbotvibe-web%2Fwebgecko","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jayelbotvibe-web%2Fwebgecko/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jayelbotvibe-web%2Fwebgecko/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jayelbotvibe-web%2Fwebgecko/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jayelbotvibe-web","download_url":"https://codeload.github.com/jayelbotvibe-web/webgecko/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jayelbotvibe-web%2Fwebgecko/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35143802,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-05T02:00:06.290Z","response_time":100,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agents","crawler","curl-cffi","html-parser","lxml","markdown-converter","python","scraping","stealth","tls-fingerprinting","web-scraping"],"created_at":"2026-07-05T05:02:43.450Z","updated_at":"2026-07-05T05:02:43.994Z","avatar_url":"https://github.com/jayelbotvibe-web.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Web Gecko\n\n\u003e Pluck what you need. Climb through the web.\n\n[![Tests](https://github.com/jayelbotvibe-web/webgecko/actions/workflows/test.yml/badge.svg)](https://github.com/jayelbotvibe-web/webgecko/actions/workflows/test.yml)\n[![PyPI](https://img.shields.io/pypi/v/webgecko)](https://pypi.org/project/webgecko/)\n[![Python](https://img.shields.io/pypi/pyversions/webgecko)](https://pypi.org/project/webgecko/)\n[![License](https://img.shields.io/github/license/jayelbotvibe-web/webgecko)](https://github.com/jayelbotvibe-web/webgecko/blob/master/LICENSE)\n\n**Web Gecko** is a point-solution library — stealthy HTTP + HTML parsing for when you need to scrape a site that blocks plain `curl` or `requests` but doesn't require JavaScript. Built for AI agents that want structured data without writing parsing loops.\n\n**What it does:** TLS impersonation, CSS/XPath with pseudo-elements, markdown conversion, structured extraction, concurrent crawling.\n\n**What it doesn't:** Browser automation, JavaScript rendering, proxy rotation, CAPTCHA solving. This is a library, not a scraping framework.\n\n![Demo](demo.svg)\n\n```python\nfrom gecko import fetch\n\nr = fetch(\"https://httpbin.org/html\", impersonate=\"chrome131\")\nprint(r.page.title)\nprint(r.page.markdown[:200])\n```\n\n## Install\n\n```bash\npip install webgecko\n```\n\nPython 3.10+. Depends on `lxml`, `cssselect`, `curl_cffi`, `anyio`.\n\n## API\n\n### Fetch\n\n```python\nfrom gecko import fetch, Session, AsyncSession\n\nr = fetch(\"https://example.com\", impersonate=\"chrome131\")\nr.status          # 200\nr.page            # parsed Page (HTML) or placeholder (JSON)\nr.json            # parsed JSON body, or None\nr.headers         # case-insensitive dict\n\nwith Session(impersonate=\"firefox124\") as s:\n    r = s.get(\"https://example.com\")\n\nasync with AsyncSession() as s:\n    r = await s.get(\"https://example.com\")\n```\n\n### Parse\n\n```python\nfrom gecko import Page\n\npage = Page(\"\u003chtml\u003e...\u003c/html\u003e\")\n\n# Queries return Elements (iterable, indexable, .get() for first)\npage.css(\".title\")                 # Elements\npage.xpath(\"//h2\")                 # Elements\npage.find(\"Hello\")                 # exact text\npage.find(\"Hel\", partial=True)     # substring\npage.find_all(\"div\", class_=\"foo\") # by tag + attrs\n\n# Pseudo-elements — extract strings directly\npage.css(\".title::text\").get()     # \"Widget A\"\npage.css(\"a::attr(href)\").get()    # \"/buy/a\"\n\n# Element properties\nel = page.css(\".product\")[0]\nel.text            # text content\nel.tag             # \"div\"\nel.html            # inner HTML\nel.attr(\"href\")    # attribute value\n```\n\n### Agent-friendly shortcuts\n\n```python\npage.title         # \"My Page\" — \u003ctitle\u003e text\npage.markdown      # full page as markdown\npage.links()       # [{\"text\": \"Link A\", \"href\": \"/a\"}, ...]\npage.jsonld()      # [{\"@type\": \"WebSite\", ...}] — JSON-LD data\n```\n\n### Extract (agent-friendly structured output)\n\n```python\n# Map CSS selectors → field names. One call per group of elements.\npage.css(\".product\").extract({\n    \"name\": \".title::text\",\n    \"price\": \".price::text\",\n    \"link\": \"a::attr(href)\",\n})\n# → [{\"name\": \"Widget A\", \"price\": \"$9.99\", \"link\": \"/buy/a\"}, ...]\n\n# first=True returns a single dict or None\npage.css(\"h1\").extract({\"title\": \"::text\"}, first=True)\n# → {\"title\": \"Welcome\"}\n```\n\n### Gecko\n\n```python\nclass QuotesGecko(Gecko):\n    start_urls = [\"https://quotes.toscrape.com/\"]\n    concurrency = 4\n\n    def parse(self, response: Response):\n        yield from response.page.css(\".quote\").extract({\n            \"text\": \".text::text\",\n            \"author\": \".author::text\",\n        })\n        next_link = response.page.css(\".next a::attr(href)\").get()\n        if next_link:\n            yield response.follow(next_link, callback=self.parse)\n\nresult = QuotesGecko().run()  # 100 quotes, 10 pages, ~3s\nresult.save(\"quotes.json\")\n```\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjayelbotvibe-web%2Fwebgecko","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjayelbotvibe-web%2Fwebgecko","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjayelbotvibe-web%2Fwebgecko/lists"}