https://github.com/IaroslavR/scrapy-mysql-pipeline

scrapy mysql pipeline
https://github.com/IaroslavR/scrapy-mysql-pipeline

mysql python scrapy scrapy-mysql-pipeline

Last synced: about 2 months ago
JSON representation

scrapy mysql pipeline

• Host: GitHub
• URL: https://github.com/IaroslavR/scrapy-mysql-pipeline
• Owner: IaroslavR
• License: other
• Created: 2017-10-05T13:28:19.000Z (almost 7 years ago)
• Default Branch: master
• Last Pushed: 2022-01-15T09:04:21.000Z (over 2 years ago)
• Last Synced: 2024-07-23T15:44:28.904Z (about 2 months ago)
• Topics: mysql, python, scrapy, scrapy-mysql-pipeline
• Language: Python
• Homepage:
• Size: 10.7 KB
• Stars: 49
• Watchers: 2
• Forks: 17
• Open Issues: 13
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-scrapy - scrapy-mysql-pipeline

README

        
![Python 3.6](https://img.shields.io/badge/Python-3.6-blue.svg)  

# Pull requests are always welcome 

## scrapy-mysql-pipeline  

Asynchronous mysql [Scrapy](https://doc.scrapy.org/en/latest/) item pipeline  

#### Installation  

```bash

pip install scrapy-mysql-pipeline

```

#### Configuration  

Add pipeline  

```python

ITEM_PIPELINES = {

    'scrapy_mysql_pipeline.MySQLPipeline': 300,

}

```

Default values:  

```python

MYSQL_HOST = 'localhost'

MYSQL_PORT = 3306

MYSQL_USER = None

MYSQL_PASSWORD = ''

MYSQL_DB = None

MYSQL_TABLE = None

MYSQL_UPSERT = False

MYSQL_RETRIES = 3

MYSQL_CLOSE_ON_ERROR = True

MYSQL_CHARSET = 'utf8'

```

`MYSQL_USER`, `MYSQL_PASSWORD`, `MYSQL_DB` and `MYSQL_TABLE`,  variables must be set in settings.py