https://github.com/dan3002/imdb-crawler
A powerful Python-based web crawler that collects comprehensive movie information from IMDb using both GraphQL API and web scraping techniques. This tool can gather detailed movie data including basic information, reviews, and ratings for any type of movies based on customizable filters.
https://github.com/dan3002/imdb-crawler
crawler imdb imdb-dataset selenium
Last synced: about 2 months ago
JSON representation
A powerful Python-based web crawler that collects comprehensive movie information from IMDb using both GraphQL API and web scraping techniques. This tool can gather detailed movie data including basic information, reviews, and ratings for any type of movies based on customizable filters.
- Host: GitHub
- URL: https://github.com/dan3002/imdb-crawler
- Owner: DAN3002
- License: apache-2.0
- Created: 2025-03-03T08:19:40.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-03-06T10:47:05.000Z (over 1 year ago)
- Last Synced: 2026-01-02T08:45:48.025Z (6 months ago)
- Topics: crawler, imdb, imdb-dataset, selenium
- Language: Python
- Homepage:
- Size: 367 KB
- Stars: 1
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# IMDb Movie Crawler
A powerful Python-based web crawler that collects comprehensive movie information from IMDb using both GraphQL API and web scraping techniques. This tool can gather detailed movie data including basic information, reviews, and ratings for any type of movies based on customizable filters.
## Features
- Advanced movie filtering capabilities similar to IMDb's search:
- By country of origin
- By year range
- By genre
- By rating range
- By number of votes
- By title type (movie, TV series, etc.)
- By language
- By user reviews count
- Collects detailed movie information including:
- Basic details (title, original title, year, runtime)
- Ratings and reviews
- Plot summaries
- Genre information
- Country of origin
- Popularity rankings
- Certificate ratings
- Comprehensive user review collection
- JSON to CSV conversion for easy data analysis
- Robust logging system
- Rate limiting to prevent server overload
- Progress saving and error handling
- Session management with automatic retry
## Project Structure
```
IMDb-Crawler/
├── imdb_crawler.py # Main crawler for basic movie information
├── movie_detail_crawler.py # Detailed movie information crawler
├── user_review_crawler.py # Movie reviews crawler
├── filter_movies.py # Movie filtering script
├── json_to_csv_converter.py # JSON to CSV conversion utility
├── utils/
│ └── logger.py # Logging utility
├── logs/ # Log files directory
├── output/ # Output files directory
└── error_logs/ # Error logging directory
```
## Requirements
- Python 3.x
- Chrome Browser
- Selenium WebDriver
## Installation
1. Clone the repository:
```
git clone https://github.com/DAN3002/IMDb-Crawler.git
cd IMDb-Crawler
```
2. Install required packages:
```
pip install -r requirements.txt
```
3. Install Chrome WebDriver for your Chrome browser version
## Usage
1. Basic Movie Crawling:
```bash
python imdb_crawler.py
```
2. Detailed Movie Information:
```bash
python movie_detail_crawler.py
```
3. User Reviews Collection:
```bash
python user_review_crawler.py
```
4. Custom Filtering:
```python
# Example in filter_movies.py
filter_criteria = {
'votes_min': 1000, # Minimum votes
'rating_min': 7.0, # Minimum rating
'year_range': (2000, 2024), # Year range
'countries': ['US', 'UK'], # Countries
'genres': ['Action', 'Drama'],# Genres
'reviews_min': 5 # Minimum reviews
}
```
5. Convert Results to CSV:
```bash
python json_to_csv_converter.py
```
## Customizing Filter Criteria When Crawling Movies
```python
# Example in imdb_crawler.py
variables = {
"first": self.PAGE_SIZE,
"locale": "vi-VN",
"originCountryConstraint": {
"anyPrimaryCountries": ["VN"]
},
"titleTypeConstraint":{"anyTitleTypeIds":["movie"],"excludeTitleTypeIds":[]},
"sortBy": "POPULARITY",
"sortOrder": "ASC"
}
```
## Output Formats
The crawler generates several output files:
- `movie_details.json`: Complete movie information
- `filtered_movies.json`: Filtered movie results
- `movie_reviews.json`: User reviews data
- Corresponding CSV files for each JSON file
## Error Handling
The crawler includes comprehensive error handling and logging:
- Automatic session refresh on connection issues
- Rate limiting to prevent IP blocking
- Progress saving for long-running crawls
- Detailed error logs in `error_logs` directory
## Author
This project is created by [@DAN3002](https://github.com/DAN3002).
## Contributing
1. Fork the repository
2. Create your feature branch (`git checkout -b feature/AmazingFeature`)
3. Commit your changes (`git commit -m 'Add some AmazingFeature'`)
4. Push to the branch (`git push origin feature/AmazingFeature`)
5. Open a Pull Request
## License
This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.
## Disclaimer
This tool is for educational purposes only. Please review IMDb's terms of service and robots.txt before using this crawler. Ensure you comply with IMDb's usage policies and implement appropriate rate limiting.