https://github.com/zhangshyue/webscrapper-for-imdb-and-data-visualization
A scrapy webscrapper that can scrape information about the IMDb top box office films and use pandas and matplotlib to visualize the data
https://github.com/zhangshyue/webscrapper-for-imdb-and-data-visualization
json matplotlib mongodb pandas python3 regular-expression scrapy webscrapper
Last synced: 4 months ago
JSON representation
A scrapy webscrapper that can scrape information about the IMDb top box office films and use pandas and matplotlib to visualize the data
- Host: GitHub
- URL: https://github.com/zhangshyue/webscrapper-for-imdb-and-data-visualization
- Owner: zhangshyue
- Created: 2019-01-01T14:24:05.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2019-01-02T02:03:23.000Z (over 7 years ago)
- Last Synced: 2026-01-03T17:13:41.949Z (7 months ago)
- Topics: json, matplotlib, mongodb, pandas, python3, regular-expression, scrapy, webscrapper
- Language: Jupyter Notebook
- Homepage:
- Size: 117 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Webscrapper for IMDb and data visualization
# Running the spiders
### To create a scrapy project:
```
scrapy startproject imdbtopboxoffice
cd imdbtopboxoffice
scrapy genspider imdb imdb.com
```
### Start scrapping
```
scrapy crawl imdb
```
### Save the data in a json file
```
scrapy crawl imdb -o imdb.json
```
This will crawl imdb.com and save the data in a file called imdb.json.
# The structure of imdb spider
This spider can mainly scrape information about the rating and popularity of the films on the IMDb top box office page.
The extracted data of a Imdbtopboxofficesitem is in this form:
```
{
"name": "The Grinch",
"url": "https://www.imdb.com/title/tt2709692/?pf_rd_m=A2FGELUUNOQJNL&pf_rd_p=f9f31d04-fc22-4d12-86b4-f46e25aa2f6f&pf_rd_r=2V27JT0FAH5YK8Z5PEAJ&pf_rd_s=center-1&pf_rd_t=15506&pf_rd_i=boxoffice&ref_=cht_bo_10",
"user_rating": "6.4",
"num_of_user": "15802",
"metascore": "51",
"popularity": "3",
"budget": "75000000",
"length": "1h 26min"
}
```
There is also a log file that can be used for debug. Write anything that you want in the log file by importing logging:
```
import logging
```
# Data Visualization
Extract the data that are stored in MongoDB. Using jupyter notebook, pandas and matplotlib, we can analyze the data.
