{"id":51232182,"url":"https://github.com/xcrap-dev/xcrap-python","last_synced_at":"2026-06-28T17:02:06.553Z","repository":{"id":340219095,"uuid":"1164524754","full_name":"xcrap-dev/xcrap-python","owner":"xcrap-dev","description":"A modern, declarative, and modular web scraping framework for Python.","archived":false,"fork":false,"pushed_at":"2026-04-13T11:55:51.000Z","size":381,"stargazers_count":0,"open_issues_count":5,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-06-24T01:33:41.864Z","etag":null,"topics":["alternative","declarative","framework","modular","python","scraping","scrapy","web"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/xcrap/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xcrap-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-23T07:20:47.000Z","updated_at":"2026-03-12T03:33:43.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/xcrap-dev/xcrap-python","commit_stats":null,"previous_names":["xcrap-cloud/xcrap-python"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/xcrap-dev/xcrap-python","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xcrap-dev%2Fxcrap-python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xcrap-dev%2Fxcrap-python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xcrap-dev%2Fxcrap-python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xcrap-dev%2Fxcrap-python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xcrap-dev","download_url":"https://codeload.github.com/xcrap-dev/xcrap-python/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xcrap-dev%2Fxcrap-python/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34896652,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-28T02:00:05.809Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alternative","declarative","framework","modular","python","scraping","scrapy","web"],"created_at":"2026-06-28T17:02:05.710Z","updated_at":"2026-06-28T17:02:06.545Z","avatar_url":"https://github.com/xcrap-dev.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Xcrap for Python\n\nXcrap é um framework originalmente feito para Node.js, mas, digamos que eu também sou um desenvolvedor Python e, estava um pouco entediado; por isso, resolvi fazer uma versão para Python.\n\nAinda está em fase experimental, mas já conta com funcionalidades poderosas portadas da versão original em TypeScript, garantindo paridade entre os ecossistemas.\n\n## 🚀 Como funciona\n\nA ideia é ser declarativo e fácil. Veja um exemplo de como você já pode usar o `xcrap` para extrair dados estruturados:\n\n```python\nfrom xcrap.extractor import HtmlExtractionModel, HtmlBaseField, HtmlNestedField, css\nfrom xcrap.clients import HttpxClient\nimport asyncio\n\n# 1. Defina seus modelos de forma declarativa\nclass AuthorModel(HtmlExtractionModel):\n    name = HtmlBaseField(query=css(\"small.author::text\"))\n    link = HtmlBaseField(query=css(\"a::attr(href)\"))\n\nclass QuoteModel(HtmlExtractionModel):\n    text = HtmlBaseField(query=css(\"span.text::text\"))\n    # Modelos aninhados sem esforço!\n    author = HtmlNestedField(model=AuthorModel) \n    tags = HtmlBaseField(query=css(\"div.tags a.tag::text\"), multiple=True)\n\nclass QuotesPageModel(HtmlExtractionModel):\n    quotes = HtmlNestedField(query=css(\"div.quote\"), model=QuoteModel, multiple=True)\n\nasync def main():\n    client = HttpxClient()\n    \n    # 2. Busque a página\n    response = await client.fetch(url=\"http://quotes.toscrape.com\")\n\n    # 3. Transforme em um parser e extraia os dados\n    parser = response.as_html_parser()\n    data = parser.extract_model(QuotesPageModel)\n\n    for quote in data[\"quotes\"][:3]:\n        print(f\"Quote: {quote['text']}\")\n        print(f\"Author: {quote['author']['name']}\")\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```\n\n## ✨ Funcionalidades Principais\n\n### 🔒 Descriptografia Automática\nSuporte integrado para descriptografar respostas HTTP (AES-CBC/ECB).\n\n```python\nfrom xcrap.core import decrypt_client\nfrom xcrap.utils.decryption import DecryptConfig, DecryptKeyConfig\n\nconfig = DecryptConfig(\n    algorithm=\"aes-256-cbc\",\n    key=DecryptKeyConfig(value=\"sua-chave-aqui\"),\n    iv=DecryptKeyConfig(value=\"seu-iv-aqui\")\n)\n\n@decrypt_client(config)\nclass MySecureClient(HttpxClient):\n    pass\n\n# Todas as respostas deste cliente agora virão descriptografadas!\n```\n\n### 🏭 Sistema de Factories\nO sistema de Factory permite a criação dinâmica de componentes a partir de arquivos de configuração (JSON/Dict), facilitando a manutenção de bots sem alteração de código.\n\n#### 1. Model Factory\nConstrói modelos complexos de forma recursiva:\n\n```python\nfrom xcrap.factory import create_extraction_model\nfrom xcrap.extractor import HtmlExtractionModel\n\nconfig = {\n    \"type\": \"html\",\n    \"model\": {\n        \"title\": {\"query\": {\"type\": \"css\", \"value\": \"h1::text\"}},\n        \"items\": {\n            \"query\": {\"type\": \"css\", \"value\": \"li\"},\n            \"multiple\": true,\n            \"nested\": {\n                \"type\": \"html\",\n                \"model\": {\"name\": {\"query\": {\"type\": \"css\", \"value\": \"span::text\"}}}\n            }\n        }\n    }\n}\n\nmodel = create_extraction_model(config, allowed_models={\"html\": HtmlExtractionModel})\n```\n\n#### 2. Client Factory\nInstancia clientes com configurações específicas:\n\n```python\nfrom xcrap.factory import create_client\nfrom xcrap.clients import HttpxClient\n\nclient = create_client(\n    client_type=\"httpx\",\n    allowed_clients={\"httpx\": HttpxClient},\n    options={\"user_agent\": \"CustomAgent\", \"proxy_url\": \"http://...\"}\n)\n```\n\n#### 3. Extractor Factory\nCria extratores a partir de strings, suportando argumentos:\n\n```python\nfrom xcrap.factory import create_extractor\n\n# 'attr:src' chamará o gerador 'attr' com o argumento 'src'\nget_src = create_extractor(\"attr:src\", allowed_extractors={\"attr\": lambda a: lambda el: el.attrib.get(a)})\n```\n\n### 🛠️ Extratores Customizados\nVocê pode passar funções customizadas para processar campos individualmente:\n\n```python\nclass MyModel(HtmlExtractionModel):\n    # 'extractor' permite transformar o dado logo após a captura\n    price = HtmlBaseField(\n        query=css(\".price::text\"), \n        extractor=lambda text: float(text.replace(\"$\", \"\"))\n    )\n```\n\n### 📊 Extração de JSON (JMESPath)\nTambém suportamos extração de JSON usando JMESPath:\n\n```python\nfrom xcrap.extractor import JsonExtractionModel, JsonBaseField, jmes_path, JsonParser\n\nclass ProjectModel(JsonExtractionModel):\n    name = JsonBaseField(query=jmes_path(\"title\"))\n    tags = JsonBaseField(query=jmes_path(\"tags\"), multiple=True)\n\ncontent = '{\"title\": \"Xcrap\", \"tags\": [\"python\", \"scraping\"]}'\nparser = JsonParser(content)\ndata = parser.extract_model(ProjectModel) # {'name': 'Xcrap', 'tags': ['python', 'scraping']}\n```\n\n## 🛠️ Desenvolvimento\n\nPara rodar os testes e verificar a cobertura:\n\n```bash\npoetry run pytest --cov=xcrap --cov-report=term-missing\n```\n\nAtualmente o projeto conta com **100% de cobertura de código**, garantindo a confiabilidade de todas as funcionalidades.\n\n---\nFeito com ❤️ por Marcuth \u003ccontact@marcuth.dev\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxcrap-dev%2Fxcrap-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxcrap-dev%2Fxcrap-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxcrap-dev%2Fxcrap-python/lists"}