https://github.com/pymarcus/spiderbot
https://github.com/pymarcus/spiderbot
Last synced: 5 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/pymarcus/spiderbot
- Owner: PyMarcus
- Created: 2022-07-21T20:24:46.000Z (almost 4 years ago)
- Default Branch: main
- Last Pushed: 2022-07-22T13:14:35.000Z (almost 4 years ago)
- Last Synced: 2025-09-15T11:49:45.095Z (10 months ago)
- Language: HTML
- Size: 15.6 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
SCRAPY A FRAMEWORK TO BUILD WEBSCRAPING AND WEBCRAWLER
https://docs.scrapy.org/en/latest/intro/overview.html
Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like data mining, information processing or historical archival.
Even though Scrapy was originally designed for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler.
### start a project
scrapy startproject tutorial
### run the project (first time)
python -m venv venv
.\venv\Scripts\activate
pip install -r requirements
scrapy crawl quotes
### run if u already install the requirements:
scrapy crawl quotes
### save data how a json file
scrapy crawl quotes -O quotes.json
### run with cache:
scrapy crawl quotes -s HTTPCACHE_ENABLE=1
### create a new spider:
scrapy genspider name site.com
### list spiders
scrapy list