{"id":13824291,"url":"https://github.com/MaartenGr/Reviewer","last_synced_at":"2025-07-08T19:31:30.511Z","repository":{"id":100714689,"uuid":"284610135","full_name":"MaartenGr/Reviewer","owner":"MaartenGr","description":"Tool for extracting and analyzing IMDB reviews","archived":false,"fork":false,"pushed_at":"2020-08-26T11:35:00.000Z","size":107951,"stargazers_count":11,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-11-14T18:44:51.530Z","etag":null,"topics":["bert","disney","imdb","ner","nlp","sentiment-analysis"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MaartenGr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-08-03T05:19:26.000Z","updated_at":"2024-10-16T04:40:07.000Z","dependencies_parsed_at":null,"dependency_job_id":"9e88ae10-081d-4e31-8292-4f726a1f6468","html_url":"https://github.com/MaartenGr/Reviewer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaartenGr%2FReviewer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaartenGr%2FReviewer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaartenGr%2FReviewer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaartenGr%2FReviewer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MaartenGr","download_url":"https://codeload.github.com/MaartenGr/Reviewer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225457763,"owners_count":17477351,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","disney","imdb","ner","nlp","sentiment-analysis"],"created_at":"2024-08-04T09:01:00.429Z","updated_at":"2024-11-20T02:30:36.681Z","avatar_url":"https://github.com/MaartenGr.png","language":"Jupyter Notebook","funding_links":[],"categories":["Jupyter Notebook"],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/MaartenGr/Reviewer/raw/master/images/wordclouds/result_0.png\" height=\"400\"/\u003e\n\u003c/p\u003e\n\n\u003e Code for scraping IMDB reviews and applying NER and Sentiment Analyses \n\u003eto extract Character popularity\n\n**Reviewer** can be used to scrape user reviews from IMDB, generate word clouds based\non a custom class-based TF-IDF, and extract popular characters/actors from reviews\nusing a combination of Named Entity Recognition and Sentiment Analyses.   \n\n\u003ca name=\"toc\"/\u003e\u003c/a\u003e\n## Table of Contents\n\n1. [Instructions](#instructions)\n\n    a. [Scrape](#instructions-scrape)\n    \n    b. [Word Cloud](#instructions-wordcloud)\n    \n    c. [Character Analysis](#instructions-character)\n    \n2. [c-TF-IDF](#ctf-idf)\n\n3. [Disney](#disney)\n\n4. [Sources](#sources)\n\n\u003ca name=\"instructions\"/\u003e\u003c/a\u003e\n## 1. Instructions\n[Back to ToC](#toc)\n\nI would advise you to start with the **notebooks/Overview.ipynb** notebook for a good \nintroduction before going to the command line. Moreover, scraping multiple movies\nis actually preferred as it allows you to use the class-based TF-IDF.\n\nInstead, you can **dowload** or **fork** this repo and start with the\ninstructions below. \n\n\u003ca name=\"instructions-scrape\"/\u003e\u003c/a\u003e\n#### 1.a Scrape\n\nTo scrape a single movie (e.g., Aladdin), simply run from the command line:\n```commandline\npython scraper.py --prefix aladdin --url https://www.imdb.com/title/tt0103639/reviews?ref_=tt_ov_rt\n```\n\nMake sure to select the url of the review page of the movie you want to scrape. \nThe `prefix` variable is the name used for saving the resulting .json file.\n\nNot only is the movie scraped, count data is also extracted if it is a single movie. If you want\nto apply the class-based TF-IDF, I would suggest to follow the instructions at **notebooks/Overview.ipynb**.\n\n\u003ca name=\"instructions-wordcloud\"/\u003e\u003c/a\u003e\n#### 1.b Word Cloud\n\nMake sure that you save an image that you want to be used as a mask. \nIt is important the background is white and the file saved as a .jpg. \n\nThen, after scraping the reviews, run the following from the command line:\n```commandline\npython scrape.py --path \"data/aladdin_count.json\" --mask your_mask.jpg --pixels 1200\n```\n\nThe data/aladdin_count.json is the file saved after running the scraper. The name\nthus depends on the prefix at the scrape stage. \n\nThe result is something like this:\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/MaartenGr/Reviewer/raw/master/images/wordclouds/result_1.png\" height=\"200\"/\u003e\n\u003cimg src=\"https://github.com/MaartenGr/Reviewer/raw/master/images/wordclouds/result_2.png\" height=\"200\"/\u003e\n\u003cimg src=\"https://github.com/MaartenGr/Reviewer/raw/master/images/wordclouds/result_3.png\" height=\"200\"/\u003e\n\u003cimg src=\"https://github.com/MaartenGr/Reviewer/raw/master/images/wordclouds/result_5.png\" height=\"200\"/\u003e\n\u003c/p\u003e\n\n\u003ca name=\"instructions-characters\"/\u003e\u003c/a\u003e\n#### 1.c Character Analysis\nWe want to extract, from the reviews, which characters and actors are often talked about. \nWe start by using Named Entity Recognition to extract the entity \"Person\" from reviews. \nThen, in the sentence where the entity \"Person\" is found, we apply sentiment analysis to \nunderstand the sentiment about that character. In other words, we extract often talked about characters \ncombined with how positive those characters are regarded.\n\nTo do this, I made use of **Named Entity Recognition** and **Sentiment Analysis** using pre-trained **BERT** models. \n\nAfter having scraped the review data, run the following from the command line:\n```commandline\npython char.py --movie Aladdin --extract True --fast True --prefix disney --rpath disney_reviews.json\n```\n\nThe result will be the following visualization: \n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/MaartenGr/Reviewer/raw/master/images/characters/aladdin_characters.png\" height=\"400\"/\u003e\n\u003c/p\u003e \n\n\u003ca name=\"ctf-idf\"/\u003e\u003c/a\u003e\n## 2. Class-based TF-IDF\n[Back to ToC](#toc)\n\nThis project uses a custom TF-IDF used for exploring words that are interesting based\non the differences between classes. In other words, words are only important if they are often\nmentioned in one class, but not so much in all other classes. I call it a class-based TF-IDF (c-TF-IDF):\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/MaartenGr/Reviewer/raw/master/images/ctfidf.gif\" height=\"50\"/\u003e\n\u003c/p\u003e\n\nThe above formula can best be explained as a TF-IDF formula adopted for \nmultiple classes by joining all documents per class. Thus, each class is converted \nto a single document instead of set of documents. Then, the frequency of words **t** \nare extracted for each class **i** and divided by the total number of words **w**. \n\nNext, the total, unjoined, number of documents across all classes **m** is divided by \nthe total sum of word **i** across all classes.\n\n\n\u003ca name=\"disney\"/\u003e\u003c/a\u003e\n## 3. Disney\n[Back to ToC](#toc)\n\nInitially, this project was meant for me to be used only for Disney and Pixar movies \n(as I enjoy those very much), but eventually I generalized the code to be used for,\nin principle, all movies. \n\nHowever, I also analyzed the most popular characters (by relative frequency in reviews)\nand created a visualization of it below:\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/MaartenGr/Reviewer/raw/master/images/disney_frequency.png\" height=\"400\"/\u003e\n\u003c/p\u003e\n\nMoreover, you will find some Disney snippets here and there that I purposefully did not remove as\nthere were some manual fixes to get the visualizations working and the pipeline running. See\n **notebooks/Overview.ipynb** for more information on how to run that code. \n\n\u003ca name=\"sources\"/\u003e\u003c/a\u003e      \n## 4. Sources\n[Back to ToC](#toc)\n\nAn overview of all sources used in this package (mainly images for masks).\nAll rights on these images belong to Disney, Pixar, and Marvel.  \n\n\u003cdetails\u003e\n\u003csummary\u003eMask Images\u003c/summary\u003e\n\n* Aladdin - https://www.amazon.com/Aladdin-Official-Lifesize-Cardboard-Fan/dp/B07QSZ5GC9\n* Coco - https://www.jing.fm/iclipt/mJTmmi/\n* Avengers - https://besthqwallpapers.com/films/ironman-4k-superheroes-iron-man-white-background-38148\n* Up - https://www.hiclipart.com/free-transparent-background-png-clipart-semci\n* Toy Story 3 - https://pixar.fandom.com/wiki/Lots-o%27-Huggin%27_Bear\n* Frozen - https://tvtropes.org/pmwiki/pmwiki.php/Characters/FrozenElsa\n* Moana - https://brooklynactivemama.com/11/2016/obsessed-disneys-moana-free-moana-movie-printables.html\n* Tangled - https://www.pngfuel.com/free-png/nfjxg\n* Toy Story (Woody) - http://www.allocine.fr/evenements/pixar/chapitre2/\n* Toy Story (Buzz) - https://heroes-and-villians.fandom.com/wiki/Buzz_Lightyear\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eDisney Popularity Images\u003c/summary\u003e\n \n* Simba - https://lionking.fandom.com/wiki/Simba\n* Basil - https://disney.fandom.com/wiki/Basil_of_Baker_Street\n* Mowgli - https://disney.fandom.com/wiki/Mowgli\n* Woody - https://www.vhv.rs/viewpic/hbRoomw_woody-toy-story-png-png-download-toy-story/\n* Woody - https://nl.disney.be/films/toy-story-4\n* Carl - https://pixar.fandom.com/wiki/Carl_Fredricksen\n* Mike - https://www.pngegg.com/en/png-emxle\n* Belle - https://i.pinimg.com/550x/89/e6/29/89e629b622a929e9b2e1b825c34a3c71.jpg\n* Rapunzel - https://picsart.com/i/318196865223211\n* Ariel - https://princess.disney.com/ariel\n\n\u003c/details\u003e","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMaartenGr%2FReviewer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMaartenGr%2FReviewer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMaartenGr%2FReviewer/lists"}