https://github.com/pceuropa/youtube-crawler
Youtube crawler & scraper based on scrapy. Written in Python3.
https://github.com/pceuropa/youtube-crawler
crawler csv mariadb python3 scraper scrapy sqlalchemy youtube
Last synced: 2 months ago
JSON representation
Youtube crawler & scraper based on scrapy. Written in Python3.
- Host: GitHub
- URL: https://github.com/pceuropa/youtube-crawler
- Owner: pceuropa
- Created: 2019-07-21T17:03:07.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2024-05-14T22:26:05.000Z (about 1 year ago)
- Last Synced: 2025-04-08T07:36:12.809Z (3 months ago)
- Topics: crawler, csv, mariadb, python3, scraper, scrapy, sqlalchemy, youtube
- Language: Python
- Homepage:
- Size: 179 KB
- Stars: 16
- Watchers: 1
- Forks: 6
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Youtube Movie and Channel Crawler
Youtube crawler & scraper based on scrapy. Write in Python3.
REQUIREMENTS
------------
Python 3.6+
SQLAlchemy (optional if save in MariaDB)
Pytest (optional to test unit)
CONFIGURATION
----------
Configuration is in ./yt/settings.py
Default save in CSV. Uncomment yt.pipelines.SqlAlchemy if you need store items in DB
```
ITEM_PIPELINES = {
# 'yt.pipelines.SqlAlchemy': 300,
'yt.pipelines.Csv': 300,
}
```
RUNING
----------
```
scrapy crawl youtube
```
TESTING Model
----------
```
pytest yt/test.py
```