https://github.com/arobyr/exchange-rate-scrapy

A simple web scraping for exchange rate of different banks from DR.
https://github.com/arobyr/exchange-rate-scrapy

playwright python3 scrapy selenium web-scraping

Last synced: 1 day ago
JSON representation

A simple web scraping for exchange rate of different banks from DR.

• Host: GitHub
• URL: https://github.com/arobyr/exchange-rate-scrapy
• Owner: ArobyR
• License: mit
• Created: 2022-08-15T14:46:47.000Z (over 2 years ago)
• Default Branch: main
• Last Pushed: 2024-11-07T16:30:09.000Z (about 2 months ago)
• Last Synced: 2024-11-07T17:33:46.269Z (about 2 months ago)
• Topics: playwright, python3, scrapy, selenium, web-scraping
• Language: Python
• Homepage:
• Size: 48.8 KB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Exchage rate scrapy

## Requirements

### Minimum required versions

- Python >= 3.7

- Scrapy >= 2.0 (!= 2.4.0)

- Playwright >= 1.15

[check scrapy playwright doc](https://github.com/scrapy-plugins/scrapy-playwright)

## Installation

Creating the environment:

    python -m venv env

Active the env.:

    source /path/to/venv/bin/activate

Download the packages:

    pip3 install -r requirements.txt

Move to the project, in my case the name is ratescrapy.

Install the required browsers

```bash

playwright install

```

It's also possible to install only a subset of the available browsers:

```bash

playwright install firefox chromium

```

## Run

You can run with the custom configuration:

```bash

scrapy crawl 

```

with params for save in .json, for example:

```bash

scrapy crawl  -o output.json

```

or create your own custom settings.

## Notes / Disclaimer

1. This scrapy script respect the ROBOTSTXT_OBEY (ROBOTSTXT_OBEY = True).

2. The repo need some updates, example: for a better env and performance you can delete selenium and just use playwright.