https://github.com/shivah12/webscrapper-python
AI-Powered Web Scraper built with Streamlit that extracts tables, headings, and rows from any webpage. Users can input a URL and data type, visualize results, and export to CSV. Powered by BeautifulSoup, and Pandas.
https://github.com/shivah12/webscrapper-python
beautifulsoup matplotlib python3 streamlit webscraping
Last synced: 7 days ago
JSON representation
AI-Powered Web Scraper built with Streamlit that extracts tables, headings, and rows from any webpage. Users can input a URL and data type, visualize results, and export to CSV. Powered by BeautifulSoup, and Pandas.
- Host: GitHub
- URL: https://github.com/shivah12/webscrapper-python
- Owner: shivah12
- License: apache-2.0
- Created: 2025-08-01T16:14:46.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2025-08-01T17:45:02.000Z (11 months ago)
- Last Synced: 2025-09-03T17:47:04.107Z (10 months ago)
- Topics: beautifulsoup, matplotlib, python3, streamlit, webscraping
- Language: Python
- Homepage: https://webscrapper-python.streamlit.app/
- Size: 23.4 KB
- Stars: 0
- Watchers: 0
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# AI Web Scraper with Streamlit
This project is a web-based tool built using Streamlit and BeautifulSoup that allows users to extract specific data (tables, headings, or rows) from webpages. The tool enables quick preview, visualization, and CSV export of the scraped data.
## Features
- Input any webpage URL
- Select the type of data to extract: full table, headings, or specific rows
- Visualize the extracted data using basic charts
- Export the data to a CSV file
- Simple and user-friendly interface
## Getting Started
### 1. Clone the repository
```bash
git clone https://github.com/yourusername/ai-web-scraper.git
cd ai-web-scraper
````
### 2. Create and activate a virtual environment (recommended)
**On Windows:**
```bash
python -m venv venv
venv\Scripts\activate
```
**On macOS/Linux:**
```bash
python3 -m venv venv
source venv/bin/activate
```
### 3. Install dependencies
```bash
pip install -r requirements.txt
```
### 4. Run the app
```bash
streamlit run app.py
```
## Project Structure
```
app/
├── app.py # Main Streamlit app
├── scraper.py # Contains scraping logic
├── interpreter.py # AI prompt → selector logic (optional)
├── utils.py # CSV export, error handling
├── requirements.txt
```
## Secrets Setup (Optional)
If using an OpenAI API key or other environment variables, create `.streamlit/secrets.toml` like this:
```toml
[openai]
api_key = "your-openai-api-key"
```
Access it in your code as:
```python
st.secrets["openai"]["api_key"]
```
## Dependencies
* streamlit
* pandas
* beautifulsoup4
* lxml
* matplotlib
## Deployment
You can deploy this project on Streamlit Cloud. Just upload your code and add your `secrets.toml` in the Settings → Secrets section of your app.
Let me know if you’d like badges, Docker setup, or example screenshots added.
```