https://github.com/sajjadanwar0/llm-scraper

https://github.com/sajjadanwar0/llm-scraper

Last synced: about 2 months ago
JSON representation

• Host: GitHub
• URL: https://github.com/sajjadanwar0/llm-scraper
• Owner: sajjadanwar0
• Created: 2024-05-16T21:59:50.000Z (8 months ago)
• Default Branch: master
• Last Pushed: 2024-05-16T22:12:40.000Z (8 months ago)
• Last Synced: 2024-05-16T23:25:07.875Z (8 months ago)
• Language: Python
• Size: 3.91 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
## Web-Scraper AI Agent

This Streamlit program allows you to scrape a website using OpenAI API and the scrapegraphai library. Simply submit your OpenAI API key, provide the URL of the website you want to scrape, and indicate what you want the AI agent to extract from the page.

### Features

- Scrape any website simply supplying the URL

- Utilize OpenAI's LLMs (GPT-3.5-turbo or GPT-4) for intelligent scraping

- Customize the scraping task by defining what you want the AI Agent to extract

#### Getting Started

 1. Clone the repo

 

https://github.com/sajjadanwar0/llm-scraper


2. Install dependencies

 
pip install -r requirements.txt


3. Get OpenAI API key

4. streamlit run ai_scrapper.py