{"id":14064940,"url":"https://github.com/mike-gee/webtranspose","last_synced_at":"2025-04-29T17:31:21.450Z","repository":{"id":205375369,"uuid":"708227182","full_name":"mike-gee/webtranspose","owner":"mike-gee","description":"Web scraping API for building AI applications.","archived":false,"fork":false,"pushed_at":"2024-01-24T03:30:56.000Z","size":1502,"stargazers_count":41,"open_issues_count":12,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-19T05:25:42.731Z","etag":null,"topics":["chatbots","crawling","crawling-python","python","scraping","scraping-python","web-crawling","web-scraping","web-scraping-python"],"latest_commit_sha":null,"homepage":"https://webtranspose.com/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mike-gee.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.rst","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-10-21T22:53:22.000Z","updated_at":"2024-12-12T23:22:37.000Z","dependencies_parsed_at":"2023-11-18T05:24:04.222Z","dependency_job_id":"d7bf10ed-3bc5-4388-aff2-9e3885a28e6d","html_url":"https://github.com/mike-gee/webtranspose","commit_stats":null,"previous_names":["mike-gee/webtranspose"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mike-gee%2Fwebtranspose","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mike-gee%2Fwebtranspose/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mike-gee%2Fwebtranspose/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mike-gee%2Fwebtranspose/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mike-gee","download_url":"https://codeload.github.com/mike-gee/webtranspose/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251549212,"owners_count":21607368,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chatbots","crawling","crawling-python","python","scraping","scraping-python","web-crawling","web-scraping","web-scraping-python"],"created_at":"2024-08-13T07:04:11.497Z","updated_at":"2025-04-29T17:31:20.020Z","avatar_url":"https://github.com/mike-gee.png","language":"Python","readme":"\u003ca href=\"https://webtranspose.com\"\u003e\n  \u003cimg alt=\"Web Transpose. Simple APIs to get data from the internet.\" src=\"img/web-transpose-cover.png\" width=\"100%\" /\u003e\n  \u003ch1 align=\"center\"\u003eWeb Transpose\u003c/h1\u003e\n  \u003cp align=\"center\"\u003e\u003cb\u003e\n    Web Crawler \u0026 AI Web Scraper APIs for building new web experiences.\n  \u003c/b\u003e\u003c/p\u003e\n\u003c/a\u003e\n\n```bash\npip install webtranspose\n```\n\n\u003ch4 align=\"center\"\u003e\n  \u003ca href=\"https://twitter.com/mikegeecmu\"\u003e\n    \u003cimg src=\"https://img.shields.io/twitter/follow/mikegeecmu?style=flat\u0026label=%40mikegeecmu\u0026logo=twitter\u0026color=0bf\u0026logoColor=fff\" alt=\"X\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/mike-gee/webtranspose/blob/master/LICENSE.rst\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/LICENSE-GNU%20AGPLv3-blue\" alt=\"License\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/mikegeecmu/webtranspose/blob/main/LICENSE\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/docs-Web%20Transpose-blue\" alt=\"License\" /\u003e\n  \u003c/a\u003e\n\u003c/h4\u003e\n\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#introduction\"\u003e\u003cstrong\u003eIntroduction\u003c/strong\u003e\u003c/a\u003e ·\n  \u003ca href=\"#installation\"\u003e\u003cstrong\u003eInstallation\u003c/strong\u003e\u003c/a\u003e ·\n  \u003ca href=\"https://docs.webtranspose.com\"\u003e\u003cstrong\u003eDocs\u003c/strong\u003e\u003c/a\u003e\n\u003c/p\u003e\n\u003cbr/\u003e\n\n## Introduction\n\nIn the near future, **nobody will open websites**. Instead, we will be directly served the information we are seeking. New web experiences will combine the information from many websites into a single, unified experience.\n\n**Web Transpose** is a collection of API tools that allow building these new web experiences simple.\n\n- [Webᵀ Crawl: Distributed Web Crawler](#crawl)\n- [Webᵀ Scrape: AI Web Scraper](#scrape)\n\n\n### Crawl\n\n```python\nimport webtranspose as webt\n\nimport os\nos.environ['WEBTRANSPOSE_API_KEY'] = \"YOUR WEBT API KEY\"\n\ncrawl = webt.Crawl(\n    \"https://www.example.com\",\n    max_pages=100,\n    render_js=True,\n)\nawait crawl.crawl() # crawl.queue_crawl() for async\n```\n\n## Scrape\n\n```python\nimport webtranspose as webt\n\nimport os\nos.environ['WEBTRANSPOSE_API_KEY'] = \"YOUR WEBT API KEY\"\n\nschema = {\n    \"Merchant Name\": \"string\",\n    \"Title of Product\": \"string\",\n    \"Product Photo URL\": \"string\",\n}\n\nscraper = webt.Scraper(\n    schema, \n    render_js=True, \n)\nout_json = scraper.scrape(\"https://www.example.com\")\n```\n\n## Web Search (AI SERP API)\n\n```python\nimport webtranspose as webt\n\nimport os\nos.environ['WEBTRANSPOSE_API_KEY'] = \"YOUR WEBT API KEY\"\n\nresults = webt.search(\"what caused the fourth great ninja war?\")\n# results.keys()\n# ['results']\n\n# AI Filter\nresults = webt.search_filter(\"Paul Graham's Blog\")\n# results.keys()\n# ['results', 'filtered_results']\n```\n\n\n## Installation\n\nNon-Python Users: [📄 API Docs](https://docs.webtranspose.com).\n\nThis repo contains a local **lite** installation of Web Transpose. This is a good option if you want to run Web Transpose locally on your machine for quick use cases. \n\n```shell\npip install webtranspose\n```\n\nHowever, if you wish to leverage the full tools of Web Transpose and use in production, you should add your API key to add the **full** version.\n\n```python\nos.environ[\"WEBTRANSPOSE_API_KEY\"] = \"YOUR_API_KEY_HERE\"\n```\n\n\n## Enterprise Support\n\nWeb Transpose serves enterprises small and large. We partner with companies for the long term with hands-on support and custom solutions.\n\nPlease email me directly at mike@webtranspose.com for enquiries.\n","funding_links":[],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmike-gee%2Fwebtranspose","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmike-gee%2Fwebtranspose","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmike-gee%2Fwebtranspose/lists"}