Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/IaroslavR/scrapy-mysql-pipeline
scrapy mysql pipeline
https://github.com/IaroslavR/scrapy-mysql-pipeline
mysql python scrapy scrapy-mysql-pipeline
Last synced: 25 days ago
JSON representation
scrapy mysql pipeline
- Host: GitHub
- URL: https://github.com/IaroslavR/scrapy-mysql-pipeline
- Owner: IaroslavR
- License: other
- Created: 2017-10-05T13:28:19.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2022-01-15T09:04:21.000Z (almost 3 years ago)
- Last Synced: 2024-11-02T03:46:31.909Z (about 1 month ago)
- Topics: mysql, python, scrapy, scrapy-mysql-pipeline
- Language: Python
- Homepage:
- Size: 10.7 KB
- Stars: 49
- Watchers: 2
- Forks: 17
- Open Issues: 13
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-scrapy - scrapy-mysql-pipeline
README
# Pull requests are always welcome
## scrapy-mysql-pipeline
Asynchronous mysql [Scrapy](https://doc.scrapy.org/en/latest/) item pipeline
#### Installation
```bash
pip install scrapy-mysql-pipeline
```
#### Configuration
Add pipeline
```python
ITEM_PIPELINES = {
'scrapy_mysql_pipeline.MySQLPipeline': 300,
}
```
Default values:
```python
MYSQL_HOST = 'localhost'
MYSQL_PORT = 3306
MYSQL_USER = None
MYSQL_PASSWORD = ''
MYSQL_DB = None
MYSQL_TABLE = None
MYSQL_UPSERT = False
MYSQL_RETRIES = 3
MYSQL_CLOSE_ON_ERROR = True
MYSQL_CHARSET = 'utf8'
```
`MYSQL_USER`, `MYSQL_PASSWORD`, `MYSQL_DB` and `MYSQL_TABLE`, variables must be set in settings.py