{"id":23136849,"url":"https://github.com/zhangshyue/webscrapper-for-imdb-and-data-visualization","last_synced_at":"2026-04-05T08:35:07.226Z","repository":{"id":130865143,"uuid":"163742342","full_name":"zhangshyue/webscrapper-for-IMDb-and-data-visualization","owner":"zhangshyue","description":"A scrapy webscrapper that can scrape information about the IMDb top box office films and use pandas and matplotlib to visualize the data","archived":false,"fork":false,"pushed_at":"2019-01-02T02:03:23.000Z","size":120,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-01-03T17:13:41.949Z","etag":null,"topics":["json","matplotlib","mongodb","pandas","python3","regular-expression","scrapy","webscrapper"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zhangshyue.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-01-01T14:24:05.000Z","updated_at":"2019-01-02T02:03:25.000Z","dependencies_parsed_at":null,"dependency_job_id":"f6a26416-fd2e-43d9-9db2-b962a5eb077d","html_url":"https://github.com/zhangshyue/webscrapper-for-IMDb-and-data-visualization","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/zhangshyue/webscrapper-for-IMDb-and-data-visualization","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhangshyue%2Fwebscrapper-for-IMDb-and-data-visualization","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhangshyue%2Fwebscrapper-for-IMDb-and-data-visualization/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhangshyue%2Fwebscrapper-for-IMDb-and-data-visualization/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhangshyue%2Fwebscrapper-for-IMDb-and-data-visualization/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zhangshyue","download_url":"https://codeload.github.com/zhangshyue/webscrapper-for-IMDb-and-data-visualization/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhangshyue%2Fwebscrapper-for-IMDb-and-data-visualization/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31430009,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-05T08:13:15.228Z","status":"ssl_error","status_checked_at":"2026-04-05T08:13:11.839Z","response_time":75,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["json","matplotlib","mongodb","pandas","python3","regular-expression","scrapy","webscrapper"],"created_at":"2024-12-17T12:26:00.083Z","updated_at":"2026-04-05T08:35:07.205Z","avatar_url":"https://github.com/zhangshyue.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Webscrapper for IMDb and data visualization\n\n# Running the spiders\n### To create a scrapy project:\n```\nscrapy startproject imdbtopboxoffice\ncd imdbtopboxoffice\nscrapy genspider imdb imdb.com\n```\n### Start scrapping\n```\nscrapy crawl imdb\n```\n### Save the data in a json file\n```\nscrapy crawl imdb -o imdb.json\n```\nThis will crawl imdb.com and save the data in a file called imdb.json.\n\n# The structure of imdb spider\nThis spider can mainly scrape information about the rating and popularity of the films on the IMDb top box office page.\n\nThe extracted data of a Imdbtopboxofficesitem is in this form:\n```\n{\n  \"name\": \"The Grinch\", \n  \"url\": \"https://www.imdb.com/title/tt2709692/?pf_rd_m=A2FGELUUNOQJNL\u0026pf_rd_p=f9f31d04-fc22-4d12-86b4-f46e25aa2f6f\u0026pf_rd_r=2V27JT0FAH5YK8Z5PEAJ\u0026pf_rd_s=center-1\u0026pf_rd_t=15506\u0026pf_rd_i=boxoffice\u0026ref_=cht_bo_10\", \n  \"user_rating\": \"6.4\", \n  \"num_of_user\": \"15802\", \n  \"metascore\": \"51\", \n  \"popularity\": \"3\", \n  \"budget\": \"75000000\", \n  \"length\": \"1h 26min\"\n}\n```\n\nThere is also a log file that can be used for debug. Write anything that you want in the log file by importing logging:\n```\nimport logging\n```\n\n# Data Visualization\nExtract the data that are stored in MongoDB. Using jupyter notebook, pandas and matplotlib, we can analyze the data. \n![Pandas](https://github.com/zhangshyue/webscrapper-for-IMDb-and-data-visualization/blob/master/img1.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhangshyue%2Fwebscrapper-for-imdb-and-data-visualization","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzhangshyue%2Fwebscrapper-for-imdb-and-data-visualization","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhangshyue%2Fwebscrapper-for-imdb-and-data-visualization/lists"}