Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/mike-gee/webtranspose
Web scraping API for building AI applications.
https://github.com/mike-gee/webtranspose
chatbots crawling crawling-python python scraping scraping-python web-crawling web-scraping web-scraping-python
Last synced: about 1 month ago
JSON representation
Web scraping API for building AI applications.
- Host: GitHub
- URL: https://github.com/mike-gee/webtranspose
- Owner: mike-gee
- License: other
- Created: 2023-10-21T22:53:22.000Z (about 1 year ago)
- Default Branch: master
- Last Pushed: 2024-01-24T03:30:56.000Z (11 months ago)
- Last Synced: 2024-10-13T15:14:42.873Z (2 months ago)
- Topics: chatbots, crawling, crawling-python, python, scraping, scraping-python, web-crawling, web-scraping, web-scraping-python
- Language: Python
- Homepage: https://webtranspose.com/
- Size: 1.43 MB
- Stars: 40
- Watchers: 1
- Forks: 2
- Open Issues: 12
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE.rst
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
- jimsghstars - mike-gee/webtranspose - Web scraping API for building AI applications. (Python)
README
Web Transpose
Web Crawler & AI Web Scraper APIs for building new web experiences.
```bash
pip install webtranspose
```
Introduction Ā·
Installation Ā·
Docs
## Introduction
In the near future, **nobody will open websites**. Instead, we will be directly served the information we are seeking. New web experiences will combine the information from many websites into a single, unified experience.
**Web Transpose** is a collection of API tools that allow building these new web experiences simple.
- [Webįµ Crawl: Distributed Web Crawler](#crawl)
- [Webįµ Scrape: AI Web Scraper](#scrape)
### Crawl
```python
import webtranspose as webt
import os
os.environ['WEBTRANSPOSE_API_KEY'] = "YOUR WEBT API KEY"
crawl = webt.Crawl(
"https://www.example.com",
max_pages=100,
render_js=True,
)
await crawl.crawl() # crawl.queue_crawl() for async
```
## Scrape
```python
import webtranspose as webt
import os
os.environ['WEBTRANSPOSE_API_KEY'] = "YOUR WEBT API KEY"
schema = {
"Merchant Name": "string",
"Title of Product": "string",
"Product Photo URL": "string",
}
scraper = webt.Scraper(
schema,
render_js=True,
)
out_json = scraper.scrape("https://www.example.com")
```
## Web Search (AI SERP API)
```python
import webtranspose as webt
import os
os.environ['WEBTRANSPOSE_API_KEY'] = "YOUR WEBT API KEY"
results = webt.search("what caused the fourth great ninja war?")
# results.keys()
# ['results']
# AI Filter
results = webt.search_filter("Paul Graham's Blog")
# results.keys()
# ['results', 'filtered_results']
```
## Installation
Non-Python Users: [š API Docs](https://docs.webtranspose.com).
This repo contains a local **lite** installation of Web Transpose. This is a good option if you want to run Web Transpose locally on your machine for quick use cases.
```shell
pip install webtranspose
```
However, if you wish to leverage the full tools of Web Transpose and use in production, you should add your API key to add the **full** version.
```python
os.environ["WEBTRANSPOSE_API_KEY"] = "YOUR_API_KEY_HERE"
```
## Enterprise Support
Web Transpose serves enterprises small and large. We partner with companies for the long term with hands-on support and custom solutions.
Please email me directly at [email protected] for enquiries.