https://github.com/riju18/web-scraping
Scraping data from the website & saving it into different file formats.
https://github.com/riju18/web-scraping
python scrapy webscarpping
Last synced: about 2 months ago
JSON representation
Scraping data from the website & saving it into different file formats.
- Host: GitHub
- URL: https://github.com/riju18/web-scraping
- Owner: riju18
- Created: 2021-06-21T08:51:25.000Z (almost 4 years ago)
- Default Branch: master
- Last Pushed: 2021-06-27T03:47:57.000Z (almost 4 years ago)
- Last Synced: 2025-01-28T12:44:27.004Z (4 months ago)
- Topics: python, scrapy, webscarpping
- Language: Python
- Homepage:
- Size: 83 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## Learning Project
## Websites used to scrape data:
- https://www.imdb.com/search/title/?groups=top_250&sort=user_rating
- https://www.worldometers.info/world-population/population-by-country/
## configuration:
- imdb dir:
- Default User-Agent is changed
## Setup Environment:
- create a virtual environment;
- Install requirements.txt.;
- place the above two dirs into newly created venv;
## Run the project:
- Goto specific dir;
- cmd: scrapy crawl spider_name or scrapy crawl spider_name -o *.ext [ext: json, csv, xml]