{"id":14979813,"url":"https://github.com/kostantinos-papadamou/pseudoscience-paper","last_synced_at":"2025-10-28T19:32:09.197Z","repository":{"id":129521581,"uuid":"336374420","full_name":"kostantinos-papadamou/pseudoscience-paper","owner":"kostantinos-papadamou","description":"Code for \"It is just a flu\": Assessing the Effect of Watch History on YouTube's Pseudoscientific Video Recommendations","archived":false,"fork":false,"pushed_at":"2022-04-15T15:24:39.000Z","size":5878,"stargazers_count":7,"open_issues_count":0,"forks_count":7,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-09-29T03:22:49.751Z","etag":null,"topics":["deep-learning","experiments","open-science","pseudoscience","pseudoscientific-videos","pseudoscientific-videos-detection","research-paper","youtube","youtube-api-v3","youtube-downloader-python","youtube-homepage","youtube-pseudoscience-classifier","youtube-random-walks","youtube-recommendations-section","youtube-recommender","youtube-search"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kostantinos-papadamou.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-02-05T19:54:58.000Z","updated_at":"2024-04-17T01:38:33.000Z","dependencies_parsed_at":"2023-06-12T01:00:22.409Z","dependency_job_id":null,"html_url":"https://github.com/kostantinos-papadamou/pseudoscience-paper","commit_stats":{"total_commits":23,"total_committers":2,"mean_commits":11.5,"dds":"0.30434782608695654","last_synced_commit":"199660fdaecb4664fd1f6e34bb63bd2e5150a86c"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kostantinos-papadamou%2Fpseudoscience-paper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kostantinos-papadamou%2Fpseudoscience-paper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kostantinos-papadamou%2Fpseudoscience-paper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kostantinos-papadamou%2Fpseudoscience-paper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kostantinos-papadamou","download_url":"https://codeload.github.com/kostantinos-papadamou/pseudoscience-paper/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219858899,"owners_count":16556039,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","experiments","open-science","pseudoscience","pseudoscientific-videos","pseudoscientific-videos-detection","research-paper","youtube","youtube-api-v3","youtube-downloader-python","youtube-homepage","youtube-pseudoscience-classifier","youtube-random-walks","youtube-recommendations-section","youtube-recommender","youtube-search"],"created_at":"2024-09-24T14:00:43.319Z","updated_at":"2025-10-28T19:32:03.679Z","avatar_url":"https://github.com/kostantinos-papadamou.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Code and Resources for \"It is just a flu\": Assessing the Effect of Watch History on YouTube's Pseudoscientific Video Recommendations\n\n### Authors: Kostantinos Papadamou, Savvas Zannettou, Jeremy Blackburn, Emiliano De Cristofaro, Gianluca Stringhini, and Michael Sirivianos\n\n## Abstract\nThe role played by YouTube's recommendation algorithm in unwittingly promoting misinformation and conspiracy theories is not entirely understood. Yet, this can have dire real-world consequences, especially when pseudoscientific content is promoted to users at critical times, such as the COVID-19 pandemic. In this paper, we set out to characterize and detect pseudoscientific misinformation on YouTube. We collect 6.6K videos related to COVID-19, the Flat Earth theory, as well as the anti-vaccination and anti-mask movements. Using crowdsourcing, we annotate them as pseudoscience, legitimate science, or irrelevant and train a deep learning classifier to detect pseudoscientific videos with an accuracy of 0.79.\n\nWe quantify user exposure to this content on various parts of the platform and how this exposure changes based on the user's watch history. We find that YouTube suggests more pseudoscientific content regarding traditional pseudoscientific topics (e.g., flat earth, anti-vaccination) than for emerging ones (like COVID-19). At the same time, these recommendations are more common on the search results page than on a user's homepage or in the recommendation section when actively watching videos. Finally, we shed light on how a user's watch history substantially affects the type of recommended videos.\n\nPreprint available \u003ca href=\"https://arxiv.org/abs/2010.11638\"\u003ehere\u003c/a\u003e.\nDataset available \u003ca href=\"https://zenodo.org/record/4769731\"\u003ehere\u003c/a\u003e.\n\n### What do we offer in this repository?\nThis repository makes publicly available to the research community, as well as the open-source community the folllowing tools and libraries:\n\n1. The [codebase](#part-1-detection-of-pseudoscientific-videos) of a Deep Learning Classifier for pseudoscientific videos detection on YouTube, and [examples](#13-training-the-classifier) on how to train and test it;\n\n2. A [library](#14-classifier-usage) that simplifies the usage of the trained classifier and implements all the required tasks for the classification of YouTube videos;\n\n3. An [open-source library](#part-2-youtube-recommendation-algorithm-audit-framework) that provides a unified framework for assessing the effects of personalization on YouTube video recommendations in multiple parts of the platform: a) the homepage; b) the search results page; and c) the video recommendations section (recommendations when watching videos).\n\n\n### Citing the Paper:\nIf you make use of any modules available in this codebase in your work, please cite the following paper:\n```latex\n@article{papadamou2020just,\n    title={'It is just a flu': Assessing the Effect of Watch History on YouTube's Pseudoscientific Video Recommendations},\n    author={Papadamou, Kostantinos and Zannettou, Savvas and Blackburn, Jeremy and De Cristofaro, Emiliano and Stringhini, Gianluca and Sirivianos, Michael},\n    journal={arXiv preprint arXiv:2010.11638},\n    year={2020}\n}\n```\n\n## Table of Contents\n\n- [Overview](#overview)\n- [Installation](#installation)\n- [Part 1: Detection of Pseudoscientific Videos](#part-1-detection-of-pseudoscientific-videos)\n  - [Classifier Architecture](#11-classifier-architecture)\n  - [Prerequisites](#12-prerequisites)\n  - [Classifier Codebase (Training the Classifier)](#13-training-the-classifier)\n  - [Classifier Usage](#14-classifier-usage)\n- [Part 2: YouTube Recommendation Algorithm Audit Framework](#part-2-youtube-recommendation-algorithm-audit-framework)\n  - [Framework Prerequisites](#21-framework-prerequisites)\n  - [User Profile Creation](#22-user-profile-creation)\n  - [Framework Usage (Running Audit Experiments)](#23-framework-usage)\n  - [Analyzing Results](#24-analyzing-audit-experiments-results)\n  - [Framework Common Issues](#25-framework-common-issues)\n- [Acknowledgements](#acknowledgements)\n- [LICENSE](#license)\n\n\n# Overview\nYouTube has revolutionized the way people discover and consume video content online.\nHowever, while YouTube facilitates easy access to hundreds of well-produced educational,entertaining, and trustworthy news videos, mistargeted and abhorrent content is also common.\nWhile the scientific community has repeatedly pinpointed the need for effectively moderating inappropriate content, the various types of inappropriate content on the platform are relatively unstudied. \nAt the same time, the role played by YouTube’s recommendation algorithm in unwittingly promoting such content is not entirely understood.\nIn our work, we study pseudoscientific misinformation on YouTube by focusing on quantifying user exposure to pseudoscientific misinformation on various parts of the platform, and how this exposure changes based on the user's watch history.\nTo do this, we first develop a deep learning model that detects pseudoscientific videos, as well as a methodology that allow us to simulate the behavior of logged-in and non-logged-in users with varying interests who casually browsing YouTube.\n\nIn this repository, we provide to the research community the source code of the developed classifier, as well as the source code of our methodology.\nIn particular, the ability to run this kind of experiments while taking into account users' viewing history will be beneficial to researchers focusing on demystifying YouTube’s recommendation algorithm—irrespective of the topic of interest. \nOur methodology and codebase are generic and can be used to study other topics besides pseudoscience, e.g., additional conspiracy theories.\nMore specifically, we make publicly available the following set of tools and libraries:\n\n1. The [codebase](#part-1-detection-of-pseudoscientific-videos) of a Deep Learning Classifier for pseudoscientific videos detection on YouTube, and [examples](#13-training-the-classifier) on how to train and test it;\n\n2. A [library](#14-classifier-usage) that simplifies the usage of the classifier and implements all the required tasks for the classification of YouTube videos;\n\n3. An [open-source library](#part-2-youtube-recommendation-algorithm-audit-framework) that provides a unified framework for assessing the effects of personalization on YouTube video recommendations in multiple parts of the platform: a) the homepage; b) the search results page; and c) the video recommendations section (recommendations when watching videos).\n\n\n\n# Installation\nFollow the steps below to install and configure all prerequisites for both the training and usage of the Pseudoscientific Content Detection Classifier (Part 1), and for using our YouTube Audit Framework (Part 2). \n\n### Create and activate Python \u003e=3.6 Virtual Environment\n```bash\npython3 -m venv virtualenv\n\nsource virtualenv/bin/activate\n```\n\n### Install required packages\n```bash\npip install -r requirements.txt\n```\n\n### Install MongoDB\nTo store the metadata of YouTube videos, as well as for other information we use MongoDB. Install MongoDB on your own system or server using:\n\n- **Ubuntu:** Follow instructions \u003ca href=\"https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/\"\u003ehere\u003c/a\u003e.\n- **Mac OS X:** Follow instructions \u003ca href=\"https://docs.mongodb.com/manual/tutorial/install-mongodb-on-os-x/\"\u003ehere\u003c/a\u003e.\n\n\n#### MongoDB Graphical User Interface:\nWe suggest the use of \u003ca href=\"https://robomongo.org/\"\u003eRobo3T\u003c/a\u003e as a user interface for interacting with your MongoDB instance.\n\n\n### Additional Requirements\n\n#### Install the youtube-dl package\n```bash\npip install youtube-dl\n```\n**Make use of ```youtube-dl``` wisely, carefully sending requests so that you do not spam YouTube with requests and get blocked.\n\n#### Install Google APIs Client library for Python\nThis is the library utilized to call the YouTube Data API from Python\n```bash\npip install --upgrade google-api-python-client\n```\n\n### HTTPS Proxies\nOur codebase uses HTTPS Proxies for multiple purposes: \n- For downloading the transcripts of YouTube videos; and \n- The YouTube Recommendation Algorithm Audit Framework uses an HTTPS Proxy for each one of the user profiles and browser instances that it maintains. \n  This is mainly to ensure that all User Profiles used in our framework have the same geolocation and avoid changes to our results due to geolocation personalization.\n\nYou can either use your own HTTPS Proxies or buy some online and set them in the following files:\n- ```youtubeauditframework/userprofiles/info/user_profiles_info.json```: Includes the HTTPS Proxies used to simulate distinct logged-in user profiles accessing YouTube from specific geolocations. \n  Preferrably, according to our Audit framework, all HTTPS Proxies set in this file MUST be from similar locations (e.g., \"US-San Fransisco-California\"). \n- ```youtubehelpers/config/YouTubeAPIConfig.py```: Includes the HTTPS Proxies used to download the transcript of YouTube videos using ```youtube-dl```.\n\n### YouTube Data API\nOur codebase uses the YouTube Data API to download video metadata and for many other purposes like searching YouTube. \nHence, it is important that you create an API key for the YouTube Data API and set it in the configuration files of our codebase.\nYou can enable the YouTube Data API for your Google account and obtain an API key following the steps \u003ca href=\"https://developers.google.com/youtube/v3/getting-started\"\u003ehere\u003c/a\u003e.\n\nOnce you have a **YouTube Data API Key**, please set the ```YOUTUBE_DATA_API_KEY``` variable in the following files:\n\n- ```youtubehelpers/config/YouTubeAPIConfig.py```\n  \n\n- ```youtubeauditframework/utils/YouTubeAuditFrameworkConfig.py```\n\n\n# Part 1: Detection of Pseudoscientific Videos\nWe implement a deep learning model geared to detect pseudoscientific YouTube videos. \nAs also described in our paper, to train and test our model we use the dataset available \u003ca href=\"https://zenodo.org/record/4558469#.YDlltl37Q6F\"\u003ehere\u003c/a\u003e.\n\n## 1.1. Classifier Architecture\n![Model Architecture Diagram](https://github.com/kostantinos-papadamou/pseudoscience-paper/blob/main/classifier/architecture/model_architecture.png)\n\n### Description\nOur classifier consists of four different branches, each processing a distinct input feature type: snippet, video tags, transcript, and the top 200 comments of a video. \nThen, all four branches' outputs are concatenated to form a five-layer, fully-connected neural network that merges their output and drives the final classification. \nThe classifier uses \u003ca href=\"https://fasttext.cc/\"\u003efastText\u003c/a\u003e, a library for efficient learning of word/document-level vector representations and sentence classification. \nWe use fastText to generate vector representations (embeddings) for all the available video metadata in text.\nFor each input feature, we use the \u003ca href=\"https://fasttext.cc/docs/en/english-vectors.html\"\u003epre-trained fastText models (1)\u003c/a\u003e and fine-tune them using each of our input features.\nThese models extract a 300-dimensional vector representation for each of the following input features of our dataset:\n- **Video Snippet:** Concatenation of the title and the description of the video.\n- **Video Tags:** Words defined by the uploader of a video to describe the content of the video.\n- **Transcript:** Naturally, this is one of the most important features, as it describes the video’s actual content. (It includes the subtitles uploaded by the creator of the video or auto-generated by YouTube.) \n  The classifier uses the fine-tuned model to learn a vector representation of the concatenated text of the transcript.\n- **Comments:** We consider the top 200 comments of the video as returned by the YouTube Data API. \n  We first concatenate each video’s comments and use them to fine-tune the fastText model and extract vector representations.\n\nThe second part of the classifier (\"Fusing Network\") is essentially a four-layer, fully-connected, dense neural network. \nWe use a Flatten utility layer to merge the outputs of the four branches of the first part of the classifier, creating a 1200-dimensional vector. \nThis vector is processed by the four subsequent layers comprising 256, 128, 64, and 32 units, respectively, with ReLU activation. \nTo avoid over-fitting, we regularize using the Dropout technique; at each fully-connected layer, we apply a Dropout level of d=0.5, i.e., during each iteration of training, half of each layer's units do not update their parameters. \nFinally, the Fusing Network output is fed to the last neural network of two units with softmax activation, which yields the probabilities that a particular video is pseudoscientific or not. \nWe implement our classifier using Keras with Tensorflow as the back-end.\n\n## 1.2. Prerequisites\n### 1.2.1.  Download pre-trained fastText word vectors that we fine-tune during feature engineering on our dataset: \n```bash\ncd pseudoscientificvideosdetection/models/feature_extraction\n\nwget https://dl.fbaipublicfiles.com/fasttext/vectors-english/wiki-news-300d-1M.vec.zip\n\nunzip wiki-news-300d-1M.vec.zip\n```\n\n### 1.2.2. Create MongoDB Database and Collections\n\n1. Create a MongoDB database called: ```youtube_pseudoscience_dataset``` either using Robo3T GUI or from the terminal.\n\n\n2. Create the following MongoDB collections under the ```youtube_pseudoscience_dataset``` database that you just created:\n- ```groundtruth_videos```\n- ```groundtruth_videos_comments```\n- ```groundtruth_videos_transcripts```\n\n**Note:** If you are using our \u003ca href=\"https://zenodo.org/record/4558469#.YDfBCmr7Rqs\"\u003edataset\u003c/a\u003e please make sure that you create the appropriate MongoDB collections and import the data in each collection. \n\n\n\n## 1.3. Training the Classifier\nBelow we describe how can you use the codebase to train our classifier using either our own data available \u003ca href=\"\"\u003ehere\u003c/a\u003e, \nor using your own dataset. \nWe note that, our model is optimized for the detection of pseudoscientific content related to COVID-19, Anti-vaccination, Anti-mask, and Flat Earth. \nHowever, feel free to extend/enhance the provided codebase implementing your own deep learning model optimized for your use case.\n\n### Step A. Fine-tune separate fastText models for each Video Metadata Type\nIn this step, we fine-tune four separate fastText models, one for each different video metadata type, \nwhich we use during the training of Deep Learning model to generate embeddings for each different video metadata type. \nThis step is only required to run once.\n\n### Step B. Train the Pseudoscience Deep Learning Model\nAt this step, we train and validate the Pseudoscientific Content Detection Deep Learning model using 10-fold cross-validation.\nAt the end of the training, the best model will be stored in: ```pseudoscientificvideosdetection\\models\\pseudoscience_model_final.hdf5```.\nWe provide below an example of how you can use our codebase to train the Pseudoscientific Content Detection classifier.\n\n### Classifier Training Example:\n```python\nfrom dataset.DatasetUtils import DatasetUtils\nfrom classifier.featureengineering.FeatureEngineeringModels import FeatureEngineeringModels\nfrom classifier.training.ClassifierTraining import ClassifierTraining\n\n# Create Objects\ndataset = DatasetUtils()\nfeatureEngineeringModels = FeatureEngineeringModels(dataset_object=dataset)\n\n\"\"\" Step 1: Fine-tune separate fastText models for each Video Metadata Type \"\"\"\n# Video Snippet\n# Generate Video Snippet fastText input features\nfeatureEngineeringModels.prepare_fasttext_data(model_type='video_snippet')\n# Fine-tune a fastText model for Video Snippet\nfeatureEngineeringModels.finetune_model(model_type='video_snippet')\n\n# Video Tags\n# Generate Video Tags fastText input features\nfeatureEngineeringModels.prepare_fasttext_data(model_type='video_tags')\n# Fine-tune a fastText model for Video Tags\nfeatureEngineeringModels.finetune_model(model_type='video_tags')\n\n# Video Transcript\n# Generate Video Transcript fastText input features\nfeatureEngineeringModels.prepare_fasttext_data(model_type='video_transcript')\n# Fine-tune a fastText model for Video Transcript\nfeatureEngineeringModels.finetune_model(model_type='video_transcript')\n\n# Video Comments\n# Generate Video Comments fastText input features\nfeatureEngineeringModels.prepare_fasttext_data(model_type='video_comments')\n# Fine-tune a fastText model for Video Comments\nfeatureEngineeringModels.finetune_model(model_type='video_comments')\n\n\"\"\" Step 2: Train the Pseudoscience Deep Learning Model \"\"\"\n# Create a Classifier Training Object\nclassifierTrainingObject = ClassifierTraining(dataset_object=dataset)\n# Train the Classifier\nclassifierTrainingObject.train_model()\n```\n\n\n## 1.4. Classifier Usage\nIn this repository, we also include a Python package that uses the trained classifier, which can be used by anyone who wants to train our classifier using our codebase and then use it to detect pseudoscientific videos on YouTube.\nThe available Python module accepts a YouTube Video ID as input, and implements all the necessary steps to download the required information of the given video, extract the required features, and classifies the video using the trained classifier.\nNote that, the classifier is supposed to be trained and used to detect pseudoscientific videos on YouTube related to the following topics: a) COVID-19; b) Anti-vaccination; c) Anti-mask; and d) Flat Earth.\n\nFinally, in order to use this package you first need to train the classifier using our classifier' codebase, and also to provide your own YouTube Data API key.\nIf case you want to train the classifier using our own dataset, you request access to it and download it from \u003ca href=\"https://zenodo.org/record/4558469#.YDfBCmr7Rqs\"\u003ehere\u003c/a\u003e.\n\n### Classifier Usage Example:\n```python\n# Import the PSeudoscientific Videos Detection package\nfrom pseudoscientificvideosdetection.PseudoscienceClassifier import PseudoscienceClassifier\nfrom youtubehelpers.YouTubeVideoDownloader import YouTubeVideoDownloader\n\n# Create an object of the classifier\npseudoscienceClassifier = PseudoscienceClassifier()\n# You need to provide your own YouTube Data API Key when creating an object of the YouTube Video Downloader helper class\nytDownloader = YouTubeVideoDownloader()\n\n# Download YouTube Video Details\n# Note: You can replace the following line with your own method that creates the\n#       dictionary with the video details. See YouTubeVideoDownloader class to\n#       understand the format in which we convert the comments and the transcript\n#       of the video before we pass them to the classifier\nvideo_details = ytDownloader.download_video(video_id='YOUTUBE_VIDEO_ID')\n\n# Make a prediction\nprediction, confidence_score = pseudoscienceClassifier.classify(video_details=video_details) \n```\n\n\n\n# Part 2: YouTube Recommendation Algorithm Audit Framework\n\n## 2.1. Framework Prerequisites\n\n### 2.1.1. Create MongoDB Database and Collections:\n\n1. Create a MongoDB database called: ```youtube_recommendation_audit``` either using Robo3T GUI or from the terminal.\n\n\n2. Create the following MongoDB collections under the ```youtube_recommendation_audit``` database that you just created:\n- ```audit_framework_videos```: All videos of the YouTube Audit framework will be stored in this collection.\n- ```audit_framework_youtube_homepage```: Holds the details of each repetition of the **YouTube Homepage** audit experiment.\n- ```audit_framework_youtube_search```: Holds the details of each repetition of the **YouTube Search Results** audit experiment.\n- ```audit_framework_youtube_video_recommendations```: Holds the details of each repetition of the **YouTube Video Recommendations** audit experiment.\n\n### 2.1.2. Download Google ChromeDriver\nYou can download the ChromeDriver that matches the one of your local Google Chrome version from \u003ca href=\"https://chromedriver.chromium.org/downloads\"\u003ehere\u003c/a\u003e. \nWe recommend the usage of Google ```ChromeDriver 83.0.4103.39```, however, if you download another one please make sure that you update ```youtubeauditframework/utils/YouTubeAuditFrameworkConfig.py```.\n\n- **Linux:**\n  ```bash\n  cd youtubeauditframework/utils/webdrivers\n  \n  wget https://chromedriver.storage.googleapis.com/83.0.4103.39/chromedriver_linux64.zip\n  \n  unzip chromedriver_linux64.zip \u0026\u0026 mv chromedriver_linux64 chromedriver\n  \n  rm chromedriver_linux64.zip\n  ```\n\n- **MacOS X:**\n  ```bash\n  cd youtubeauditframework/utils/webdrivers\n  \n  wget https://chromedriver.storage.googleapis.com/83.0.4103.39/chromedriver_mac64.zip\n  \n  unzip chromedriver_mac64.zip \u0026\u0026 mv chromedriver_mac64 chromedriver\n  \n  rm chromedriver_mac64.zip\n  ```\n\n\n## 2.2. User Profile Creation\n\n### 2.2.1. Create Google (YouTube) Accounts\nYou have to create your own Google Accounts for each YouTube user profile that you want to perform experiments with.\nAccording to our framework, the only aspect of each user profile that differs is the watch history, hence all the account information must \nbe the similar for all user profiles (e.g., similar age, gender, country, etc.) to avoid confounding effects by profile differences.\nHowever, feel free to extend our framework assessing more personalization factors (i.e., age) and create user profiles based on your use case.\n\nTo decrease the likelihood of Google automatically detecting your user profiles, please carefully craft each one assigning them a unique name and surname and perform standard phone verification.\nWhile crafting user profiles, make sure that you also update file ```youtubeauditframework/userprofiles/info/user_profiles_info.json``` with the information of each created YouTube User Profile. \n\n### 2.2.2. Initialize and Authenticate User Profile\nTo avoid Google banning or flagging the created user profiles, we perform manual authentication of each user profile before performing experiments using our framework.\nWe provide a script to perform user authentication and create all the necessary files for each crafted YouTube User profile before running any audit experiment. \nTo do this, perform the following for each created User Profile \n\n#### Step 1. Initialize User Profile Browser data directory \n```bash\ncd youtubeauditframework/userprofiles/helpers\n\npython initialize_authenticate_user_profile.py \u003cUSER_PROFILE_NICKNAME\u003e\n```\nMake sure that you run this for each user profile by providing the nickname of each user profile (as set in user profiles information file) in the beginning of the script. \nWhen running this script, a browser will automatically open and you will be able to perform manual Google authentication.\n\n#### Step 2. Manual User Profile Authentication\nPrevious step will open a browser and load the YouTube authentication page. Once this is done, proceed and authenticate the corresponding user manually.\n\n#### Step 3. Install Adblock Plus\nOnce the user is authenticated you MUST install Adblock Plus manually by visiting: https://adblockplus.org/\n\n#### Step 4. Close the browser and repeat all steps for each User Profile\nPlease ensure that you properly close the browser window before executing this script for another user profile, or before running any experiment.\n\n### 2.2.3. User Profile Training (Build User's Watch History)\nOnce you have created all the User Profiles that you want to use and you have also authenticated all users to YouTube, then you can use the following class to \nbuild the watch history of each user: ```youtubeauditframework/userprofiles/BuildUserWatchHistory.py```.\n\n#### Build User Watch History Example\n```python\nfrom youtubeauditframework.userprofiles.BuildUserWatchHistory import BuildUserWatchHistory\n\n# Set the User Profile's nickname for which you want to build the Watch History\nuser_profile = 'USER_PROFILE_NICKNAME'\n\n# Create an object of the helper class for building the User Profile's Watch History\nbuildUserWatchHistoryHelper = BuildUserWatchHistory(user_profile=user_profile)\n\n\"\"\" Build the User's Watch History \"\"\"\n# OPTION 1: Build the watch history of the user profile, but first create a file with the following naming convention \n#           \"\u003cUSER_PROFILE_NICKNAME\u003e_watch_history_videos.txt\", which includes all the YouTube Video IDs separated by\n#           breakline (Enter Key). Store this file inside the \"youtubeauditframework/userprofiles/info/\" directory.\nbuildUserWatchHistoryHelper.build_watch_history()\n\n# OPTION 2: Build the watch history of the user profile by providing a list of minimum 100 YouTube Video IDs.\n# watch_history_videos = ['\u003cVIDEO_ID\u003e', '\u003cVIDEO_ID\u003e', '\u003cVIDEO_ID\u003e']\n# buildUserWatchHistoryHelper.build_watch_history(watch_videos_list=watch_history_videos)\n\n# Ensure that the browser has closed\nbuildUserWatchHistoryHelper.close_selenium_browser()\n```\n\n\n### \u003cspan style=\"color:#F23E5C;\"\u003eImportant: Remember to set the date you built User Profiles Watch Histories before running the Framework\u003c/spam\u003e\n\nYouTube's \"Delete Watch History\" functionality allows you to only delete the watch history and the search history of a user \u003cins\u003e**before a specific date**\u003c/ins\u003e.\nDue to this, when using our framework, you first need to build the watch history of all the User Profiles that you want to use at one date, \nand start performing experiments using our framework the next date.\n\nAfter you have built the Watch History of all User Profiles, \nplease \u003cins\u003e**set the value of**\u003c/ins\u003e\u003c/span\u003e ```DELETE_WATCH_HISTORY_AFTER_DATE``` \u003cins\u003e**to the next date of that date in file:**\u003c/ins\u003e \n```youtubeauditframework/utils/YouTubeAuditFrameworkConfig.py```. \n\nFor example, if you build the watch histories of your User Profiles on 01-03-2021 then you should set ```DELETE_WATCH_HISTORY_AFTER_DATE='02-03-2021'``` and \u003cins\u003estart running experiments using our framework the next date on 02-03-2021\u003c/ins\u003e.\n\n\n## 2.3. Framework Usage (Running Audit Experiments)\nWe focus on three parts of the platform: 1) the homepage; 2) the search results page; and 3) the video recommendations section (recommendations when watching videos). \nWith our framework, we simulate logged-in and non-logged-in user's behavior with varying interests and measure how the watch history affects pseudoscientific content recommendation.\nBelow, we provide examples of how to run the difference experiments for each part of the YouTube platform.\n\n### 2.3.1. Running Audit Experiments\n\n#### - YouTube Homepage\n\nRun the following for each one of your User Profiles:\n\n```bash\ncd youtubeauditframework\n\npython perform_audit_youtube_homepage.py USER_PROFILE_NICKNAME\n```\n\n**Note:** We suggest that you simultaneously run this script for each user profile in your scenario to avoid variations in results due to time.\n\n#### - YouTube Search Results\n\nBefore auditing YouTube Search, you need to identify the **search terms** for which you will focus on your experiments. \nThen, run the following for each combination of search terms and User Profiles in your use case: \n\n```bash\ncd youtubeauditframework\n\n# Run for a specific Search Term and User Profile (Replace spaces in SEARCH_TERM with '_')\npython perform_audit_youtube_search.py USER_PROFILE_NICKNAME SEARCH_TERM\n\n# Run for a specific Search Term using the YouTube Data API\npython perform_audit_youtube_search.py YOUTUBE_DATA_API SEARCH_TERM\n```\n\n**Note:** We suggest that you simultaneously run this script for each Search Term for all the User Profiles in your scenario to avoid variations in results due to time.\n\n\n#### - YouTube Video Recommendations Section (Random Walks on YouTube's Recommendation Graph)\n\nBefore auditing YouTube's Video Recommendations Section, you need to identify the **search terms** for which you will focus on your experiments. \nThen, run the following for each combination of search terms and User Profiles in your use case:\n\n```bash\ncd youtubeauditframework\n\n# Run for a specific Search Term and User Profile (Replace spaces in SEARCH_TERM with '_')\npython perform_audit_video_recommendations.py USER_PROFILE_NICKNAME SEARCH_TERM\n\n# Run for a specific Search Term using the YouTube Data API\npython perform_audit_video_recommendations.py YOUTUBE_DATA_API SEARCH_TERM\n```\n\n**Note:** We suggest that you simultaneously run this script for each Search Term for all the User Profiles in your scenario to avoid variations in results due to time.\n\n\n\n### 2.3.2. Download and Annotate all Experiments' Videos\nWhen downloading YouTube videos while running the YouTube Recommendation Algorithm audit experiments, we only download and save the metadata of each video.\nHence, we provide a script that you can run after you finished running the audit experiments so that you also download the comments and the transcripts of each video, \nwhich are both required to annotate a video.\n\nExecute the following to download missing videos' information and annotate all videos that are not annotated, yet:\n```bash\ncd youtubeauditframework/helpers\n\npython download_annotate_experiment_videos.py\n```\n\n\n## 2.4. Analyzing Audit Experiments Results\n\nWe also provide some helper classes that allows you to analyze the results of the audit experiments, \nand generate similar plots as the plots included in the paper. \nNote that, you should analyze the results of the experiments only after you have performed all audit experiments for all User Profiles,\nand you have also [downloaded and annotated](#232-download-and-annotate-all-experiments-videos) all audit experiments' videos.\n\nBelow we provide some examples for how you can use these helper classes to analyze audit experiments results and generate plots for each different part of the YouTube platform:\n\n### - YouTube Homepage\n\n```python\nfrom youtubeauditframework.analysis.YouTubeHomepageAuditAnalysis import YouTubeHomepageAuditAnalysis\n\n# Set the mapping between the User Profiles used for the experiments and their Plot Legend labels\n# Below is an example from the one used in the paper\nplot_legend_labels = {\n  'SCIENCE_USER': 'Science Profile',\n  'PSEUDOSCIENCE_USER': 'Pseudoscience Profile',\n  'SCIENCE_PSEUDOSCIENCE_USER': 'Science/Pseudoscience Profile',\n  'NO_PERSONALIZATION': 'No Profile (Browser)',\n}\n\n# Create a YouTube Homepage Audit Experiment Analysis\nexperimentAnalysisObject = YouTubeHomepageAuditAnalysis(legend_labels_mapping=plot_legend_labels)\n\n# Perform audit experiments results analysis\nexperimentAnalysisObject.analyze_audit_experiments()\n\n# Generate Plot\nexperimentAnalysisObject.plot_results()\n```\n\n### - YouTube Search Results\n```python\nfrom youtubeauditframework.analysis.YouTubeSearchAuditAnalysis import YouTubeSearchAuditAnalysis\n\n# Set the considered Serach Terms to analyze and generate plots\n# Below is an example from the one used in the paper\nconsidered_search_terms = ['covid-19', 'coronavirus']\n\n# Set the mapping between the User Profiles used for the experiments and their Plot Legend labels\n# Below is an example from the one used in the paper\nplot_legend_labels = {\n  'SCIENCE_USER': 'Science Profile',\n  'PSEUDOSCIENCE_USER': 'Pseudoscience Profile',\n  'SCIENCE_PSEUDOSCIENCE_USER': 'Science/Pseudoscience Profile',\n  'NO_PERSONALIZATION': 'No Profile (Browser)',\n  'YOUTUBE_DATA_API': 'No Profile (YouTube Data API)'\n}\n\n# Create a YouTube Search Audit Experiment Analysis\nexperimentAnalysisObject = YouTubeSearchAuditAnalysis(search_terms=considered_search_terms, \n                                                      legend_labels_mapping=plot_legend_labels)\n\n# Perform audit experiments results analysis\nexperimentAnalysisObject.analyze_audit_experiments()\n\n# Generate Plot\nexperimentAnalysisObject.plot_results()\n```\n\n### - YouTube Video Recommendations Section (Random Walks)\n```python\nfrom youtubeauditframework.analysis.YouTubeVideoRecommendationsAuditAnalysis import YouTubeVideoRecommendationsAuditAnalysis\n\n# Set the considered Serach Terms to analyze and generate plots\n# Below is an example from the one used in the paper\nconsidered_search_terms = ['covid-19', 'coronavirus']\n\n# Set the mapping between the User Profiles used for the experiments and their Plot Legend labels\n# Below is an example from the one used in the paper\nplot_legend_labels = {\n  'SCIENCE_USER': 'Science Profile',\n  'PSEUDOSCIENCE_USER': 'Pseudoscience Profile',\n  'SCIENCE_PSEUDOSCIENCE_USER': 'Science/Pseudoscience Profile',\n  'NO_PERSONALIZATION': 'No Profile (Browser)',\n  'YOUTUBE_DATA_API': 'No Profile (YouTube Data API)'\n}\n\n# Create a YouTube Search Audit Experiment Analysis\nexperimentAnalysisObject = YouTubeVideoRecommendationsAuditAnalysis(search_terms=considered_search_terms, \n                                                                    legend_labels_mapping=plot_legend_labels)\n\n# Perform audit experiments (Random Walks) results analysis\nexperimentAnalysisObject.analyze_audit_experiments()\n\n\n# Generate Random Walks Plot\nexperimentAnalysisObject.plot_results()\n```\n\n**\u003cins\u003eNote:\u003c/ins\u003e** Also, note that the provided helper classes analyze the results considering Pseudoscientific videos found in the audit experiments and provide similar plots\nas those in the paper. If you wish to use our codebase considering another type(s) of inappropriate videos then you need to adapt both the codebase\nof our framework, as well as the helper classes for analyzing the results of the audit experiments.\n\n\n\n## 2.5. Framework Common Issues\nUnfortunately, due to regular Google Chrome Updates or other updates on the YouTube Website, our framework may not function properly from time to time. \nIn this case, we list below some of the most common issues that we faced to assist you with overcoming them when using our framework. \n\n#### - Google Chrome version and User-Agent of our crawler:\nIf you find problems running our framework (i.e., the browser is closing right after you start an experiment), then this probably due to a mismatch \nbetween the declared ChromeDriver downloaded, the User-Agent declared in ```youtubeauditframework/utils/YouTubeAuditFrameworkConfig.py``` and the current version of your \ninstalled Google Chrome. It is better if all these three match and you can start by finding the version of your installed browser in its \"About Google Chrome\" section.\nYou can download the ChromeDriver that matches your installed Google Chrome and Operating System from \u003ca href=\"https://chromedriver.chromium.org/downloads\"\u003ehere\u003c/a\u003e.\nLast, if you do not use the recommended ChromeDriver version then make sure that you update User-Agent string in ```youtubeauditframework/utils/YouTubeAuditFrameworkConfig.py```.\n\n#### - Regular updates of YouTube's HTML/CSS codebase:\nAutomated functionalities of framework, like getting the recommended videos of a given video, or deleting the watching history \nof a logged-in user may not work from time to time and this is mainly because YouTube regularly updates its HTML and CSS classes. \nHence, when you have such problems you may need to update the codebase of our framework with the latest XPaths of each element (i.e., button, video thumbnail) \nthat you can find by inspecting each element on the YouTube website (using Google Chrome inspect option).    \n\n#### - Ensure that all User Profiles are logged-in:\nBefore running an experiment with a given User Profile, ensure that this user is logged-in. \nYou can do this by running the helper script in ```youtubeauditframework/userprofiles/helpers/initialize_authenticate_user_profile.py```.\nYou can set the desired User Profile in the beginning of this script, and when running this helper script the corresponding browser (with the details and activity of this user) will open. \nThen you will be able to manually follow the authentication  flow and authenticate this user profile on YouTube. \nUpon successful authentication, please ensure that you properly close the opened browser window before running any experiments using this user profile.\n\n#### - Enable Third Party Access to all YouTube Accounts:\nIf you have trouble accessing your created YouTube accounts from the automated browsers, then ensure that \"Less secure app access\" is enabled for all accounts.\nYou can enable \"less secure app access\" to a Google Account in the following way:\n- Open a browser and login to Google using the credentials of the corresponding User Profile.\n- Visit \u003ca hred=\"https://myaccount.google.com/security\"\u003eGoogle Account Security\u003c/a\u003e settings.\n- Scroll down to \"Less secure app access\" section and click \"Turn on access\" or enable it directly from \u003ca href=\"https://myaccount.google.com/lesssecureapps\"\u003ehere\u003c/a\u003e.\n\n# Acknowledgements\nPlease see the \u003ca href=\"https://arxiv.org/abs/2010.11638\"\u003epaper\u003c/a\u003e for funding details and non-code related acknowledgements.\n\n# LICENSE\n\nMIT License\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkostantinos-papadamou%2Fpseudoscience-paper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkostantinos-papadamou%2Fpseudoscience-paper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkostantinos-papadamou%2Fpseudoscience-paper/lists"}