https://github.com/kgelli/web-scraper-python
Web scraper that extracts quotes from quotes.toscrape.com and stores them in a MySQL database using Docker containers.
https://github.com/kgelli/web-scraper-python
beautifulsoup container-orchestration data-extraction docker mysql python web-scraping
Last synced: 3 months ago
JSON representation
Web scraper that extracts quotes from quotes.toscrape.com and stores them in a MySQL database using Docker containers.
- Host: GitHub
- URL: https://github.com/kgelli/web-scraper-python
- Owner: kgelli
- Created: 2025-02-07T04:35:15.000Z (4 months ago)
- Default Branch: main
- Last Pushed: 2025-03-07T05:34:39.000Z (3 months ago)
- Last Synced: 2025-03-07T06:25:54.943Z (3 months ago)
- Topics: beautifulsoup, container-orchestration, data-extraction, docker, mysql, python, web-scraping
- Language: Python
- Homepage:
- Size: 5.86 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Dockerized Web Scraper with MySQL Integration using Python
This project demonstrates how to build a **Dockerized web scraper** using Python that scrapes movie quotes from a website and stores the data in a **MySQL database** running in a separate Docker container. The setup uses a **custom Docker network** with the `bridge` driver for container communication.
## 🚀 Features:
- **Web Scraping** with Python using the `requests` and `BeautifulSoup` libraries.
- **Dockerized MySQL** container to store the scraped data.
## 🛠️ Prerequisites:
- Docker installed on your machine.
- Basic understanding of Docker and Python.
- Python packages: `requests`, `BeautifulSoup`, `mysql-connector-python`.
SQL commands
docker run -d --name mysql-container -e MYSQL_ROOT_PASSWORD=redhat -e MYSQL_DATABASE=scraper_db -p 3306:3306 MySQL:latest
docker exec -it mysql_container mysql -u root -predhat
USE scraper_db;
CREATE TABLE quotes ( id INT AUTO_INCREMENT PRIMARY KEY, text TEXT NOT NULL, author VARCHAR(255) NOT NULL);