https://github.com/nothingnothings/mlscraper

Mercado Livre Website Scraper Built with Python
https://github.com/nothingnothings/mlscraper

docker mercado-livre python scraper scrapy sql

Last synced: 3 days ago
JSON representation

Mercado Livre Website Scraper Built with Python

• Host: GitHub
• URL: https://github.com/nothingnothings/mlscraper
• Owner: nothingnothings
• License: mit
• Created: 2024-08-26T20:50:08.000Z (4 months ago)
• Default Branch: master
• Last Pushed: 2024-10-03T15:18:58.000Z (3 months ago)
• Last Synced: 2024-11-05T10:20:08.128Z (about 2 months ago)
• Topics: docker, mercado-livre, python, scraper, scrapy, sql
• Language: Python
• Homepage:
• Size: 24.4 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
Mlscraper - A Web Scraper Built with Python



  

  


  This script is an example of a Web Scraper Built with

    
Python.

  




## Introduction

[![en](https://img.shields.io/badge/lang-en-red.svg?style=flat-square)](https://github.com/nothingnothings/mlscraper)

[![pt-br](https://img.shields.io/badge/lang-pt--br-green.svg?style=flat-square)](https://github.com/nothingnothings/mlscraper/blob/master/README.pt-br.md)

Script built with Python that extracts and stores data about the Mercado Livre website's data in a containerized SQL database.

The script uses scrapy for the website's scraping.

For more information about its usage, read the instructions below.

## Technologies Used

- **Python**: The primary programming language used.

- **Scrapy**: Library responsible for the scraping

- **PyMySQL**: Library for connecting to and interacting with MySQL databases.

- **MySQL**: Database management system for storing product data.

## Installation

1. Run `git clone` to clone the project into your local Git repository.

2. **Install Requirements**: Make sure you have Python installed and run:

```bash 

pip install scrapy pymysql

```

3. The docker-compose.yml file contains a ready-to-use SQL database. To initialize it, with Docker installed and running, type the following commands:

```

cd docker

docker-compose up -d

```

## Usage

To run the scraper, execute the following commands:

```

cd mlscraper

scrapy crawl mlscraper -a s=

```

## Example:

```

scrapy crawl mlscraper -a s=smartphone

```