https://github.com/pceuropa/youtube-crawler

Youtube crawler & scraper based on scrapy. Written in Python3.
https://github.com/pceuropa/youtube-crawler

crawler csv mariadb python3 scraper scrapy sqlalchemy youtube

Last synced: about 2 months ago
JSON representation

Youtube crawler & scraper based on scrapy. Written in Python3.

• Host: GitHub
• URL: https://github.com/pceuropa/youtube-crawler
• Owner: pceuropa
• Created: 2019-07-21T17:03:07.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2022-07-29T22:37:20.000Z (over 2 years ago)
• Last Synced: 2023-03-04T03:17:44.635Z (almost 2 years ago)
• Topics: crawler, csv, mariadb, python3, scraper, scrapy, sqlalchemy, youtube
• Language: Python
• Homepage:
• Size: 179 KB
• Stars: 7
• Watchers: 1
• Forks: 6
• Open Issues: 1
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Youtube Movie and Channel Crawler

Youtube crawler & scraper based on scrapy. Write in Python3.

REQUIREMENTS

------------

Python 3.6+

SQLAlchemy (optional if save in MariaDB)

Pytest (optional to test unit)

CONFIGURATION

----------

Configuration is in ./yt/settings.py

Default save in CSV. Uncomment yt.pipelines.SqlAlchemy if you need store items in DB

```

ITEM_PIPELINES = {

    # 'yt.pipelines.SqlAlchemy': 300,

    'yt.pipelines.Csv': 300,

}

```

RUNING

----------

```

scrapy crawl youtube

```

TESTING Model

----------

```

pytest yt/test.py

```