Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/semanser/jsongenius
Get structured JSON data from any page.
https://github.com/semanser/jsongenius
api go golang gpt gpt-3 gpt-4 scraper scraping web-scraping
Last synced: 17 days ago
JSON representation
Get structured JSON data from any page.
- Host: GitHub
- URL: https://github.com/semanser/jsongenius
- Owner: semanser
- License: apache-2.0
- Created: 2023-10-05T11:38:32.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2023-10-11T16:27:07.000Z (about 1 year ago)
- Last Synced: 2024-10-11T18:12:22.941Z (about 1 month ago)
- Topics: api, go, golang, gpt, gpt-3, gpt-4, scraper, scraping, web-scraping
- Language: Go
- Homepage: https://singleapi.co
- Size: 1.05 MB
- Stars: 171
- Watchers: 6
- Forks: 10
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# JsonGenius
JsonGenius is a self-hosted scraping API that extracts structured data described by a [JSON Schema](https://json-schema.org). Provide any URL and a desired JSON Schema, and JsonGenius will return the structured data from the website.
## Demo
## Prerequisites
- [Docker Compose](https://docs.docker.com/compose/install/)
- `OPEN_AI_KEY` - An API key for [OpenAI](https://openai.com/). You can get one for free [here](https://platform.openai.com/account/api-keys). This should be set as an environment variable.
## Usage
#### Docker Compose (recommended)
```bash
git clone https://github.com/semanser/jsongenius
cd jsongenius
export OPEN_AI_KEY=
docker compose up
```
The API will be available at http://localhost:3001. You can change the port by editing the `docker-compose.yml` file.
#### Compile from source
```bash
git clone https://github.com/semanser/jsongenius
cd jsongenius
export OPEN_AI_KEY=
go build .
./jsongenius
```
## API
### POST /lookup
This endpoint accepts a JSON body with the following fields:
- `url`: The URL of the website to scrape
- `schema`: The JSON Schema to use to extract data from the website. The schema must be a valid JSON Schema object. Read more about JSON Schema [here](https://json-schema.org/).
#### Example
```bash
curl -X POST -H "Content-Type: application/json" -d '{
"url": "https://www.amazon.com/s?k=gaming+headsets",
"schema": {
"type": "object",
"properties": {
"products": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string",
"description": "The product name"
},
"price": {
"type": "number",
"description": "The price of the product in USD"
}
}
}
}
}
}
}' http://localhost:3001/lookup
```
### FAQ
- **Does it work with JS heavy websites?** Yes! JsonGenius uses Chromium to render the page, so it can handle any website that a normal browser can.
- **Can I bring my own Chromium instance?** Yes! You can set the `WS_URL` environment variable that points to a [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/) endpoint. JsonGenius will use that instead of spinning up its own Chromium instance.