{"id":22447566,"url":"https://github.com/boddepallyvenkatesh06/clean-and-analyze-social-media-usage-data-with-python","last_synced_at":"2025-03-27T11:27:11.113Z","repository":{"id":195030569,"uuid":"692102943","full_name":"BoddepallyVenkatesh06/Clean-and-analyze-social-media-usage-data-with-Python","owner":"BoddepallyVenkatesh06","description":"Utilized Python for data cleaning and analysis of social media usage data, employing libraries like Pandas and Matplotlib. Improved insights for informed decision-making and business strategies.","archived":false,"fork":false,"pushed_at":"2023-09-15T15:15:47.000Z","size":683,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-01T16:11:17.582Z","etag":null,"topics":["pyhon","python-lambda","python-script","pythonframework","pythonsqlite"],"latest_commit_sha":null,"homepage":"https://secnhack.in/social-analyzer-analyzing-finding-a-person-profiles/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BoddepallyVenkatesh06.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-09-15T15:05:27.000Z","updated_at":"2024-07-11T18:49:58.000Z","dependencies_parsed_at":"2023-09-16T08:34:28.590Z","dependency_job_id":null,"html_url":"https://github.com/BoddepallyVenkatesh06/Clean-and-analyze-social-media-usage-data-with-Python","commit_stats":null,"previous_names":["boddepallyvenkatesh06/clean-and-analyze-social-media-usage-data-with-python"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BoddepallyVenkatesh06%2FClean-and-analyze-social-media-usage-data-with-Python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BoddepallyVenkatesh06%2FClean-and-analyze-social-media-usage-data-with-Python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BoddepallyVenkatesh06%2FClean-and-analyze-social-media-usage-data-with-Python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BoddepallyVenkatesh06%2FClean-and-analyze-social-media-usage-data-with-Python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BoddepallyVenkatesh06","download_url":"https://codeload.github.com/BoddepallyVenkatesh06/Clean-and-analyze-social-media-usage-data-with-Python/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245834995,"owners_count":20680123,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["pyhon","python-lambda","python-script","pythonframework","pythonsqlite"],"created_at":"2024-12-06T04:17:44.431Z","updated_at":"2025-03-27T11:27:11.096Z","avatar_url":"https://github.com/BoddepallyVenkatesh06.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Clean-and-analyze-social-media-usage-data-with-Python\n\n\u003cimg align=\"right\" alt=\"Twitter Sentiment Analysis\" width=\"1000\" height = \"500\" src=\"https://user-images.githubusercontent.com/106287208/187567503-2db262f6-c7f2-449d-9b11-01ff7e084022.png\"\u003e\n\n---\n\n\n# Table of Contents\n\n- [Problem Statement](https://github.com/globalsmile/Twitter-Sentiment-Analysis#Problem-Statement)\n- [Data Sourcing](https://github.com/globalsmile/Twitter-Sentiment-Analysis#Data-Sourcing)\n- [Data Preparation](https://github.com/globalsmile/Twitter-Sentiment-Analysis#Data-Preparation)\n- [Data Modeling](https://github.com/globalsmile/Twitter-Sentiment-Analysis#Data-Modeling)\n- [Data Visualization](https://github.com/globalsmile/Twitter-Sentiment-Analysis#Data-Visualization)\n- [Data Analysis](https://github.com/globalsmile/Twitter-Sentiment-Analysis#Data-Analysis)\n- [Insights](https://github.com/globalsmile/Twitter-Sentiment-Analysis#Insights)\n- [Shareable link](https://github.com/globalsmile/Twitter-Sentiment-Analysis#Shareable-Link)\n\n\n---\n\n# Problem Statement\n\nThe purpose of this analysis is to gain insights into the number of engagements the #NG30DaysofLearning has on twitter.\n\nFor this study we examined a variety of categories: the number of tweets, number of users, the most active users, the most mentioned tools e.t.c\n\n---\n\n# Data Sourcing\n\n- The dataset used for this analysis was scrapped from twitter jupyter notebook\n- The preview of the dataset and python code is shown below:\n\n\n```python\n!pip install snscrape\n```\n\n    Requirement already satisfied: snscrape in c:\\users\\user\\anaconda3\\lib\\site-packages (0.4.3.20220106)\n    Requirement already satisfied: lxml in c:\\users\\user\\anaconda3\\lib\\site-packages (from snscrape) (4.8.0)\n    Requirement already satisfied: beautifulsoup4 in c:\\users\\user\\anaconda3\\lib\\site-packages (from snscrape) (4.11.1)\n    Requirement already satisfied: requests[socks] in c:\\users\\user\\anaconda3\\lib\\site-packages (from snscrape) (2.27.1)\n    Requirement already satisfied: filelock in c:\\users\\user\\anaconda3\\lib\\site-packages (from snscrape) (3.6.0)\n    Requirement already satisfied: soupsieve\u003e1.2 in c:\\users\\user\\anaconda3\\lib\\site-packages (from beautifulsoup4-\u003esnscrape) (2.3.1)\n    Requirement already satisfied: idna\u003c4,\u003e=2.5 in c:\\users\\user\\anaconda3\\lib\\site-packages (from requests[socks]-\u003esnscrape) (3.3)\n    Requirement already satisfied: charset-normalizer~=2.0.0 in c:\\users\\user\\anaconda3\\lib\\site-packages (from requests[socks]-\u003esnscrape) (2.0.4)\n    Requirement already satisfied: urllib3\u003c1.27,\u003e=1.21.1 in c:\\users\\user\\anaconda3\\lib\\site-packages (from requests[socks]-\u003esnscrape) (1.26.9)\n    Requirement already satisfied: certifi\u003e=2017.4.17 in c:\\users\\user\\anaconda3\\lib\\site-packages (from requests[socks]-\u003esnscrape) (2021.10.8)\n    Requirement already satisfied: PySocks!=1.5.7,\u003e=1.5.6 in c:\\users\\user\\anaconda3\\lib\\site-packages (from requests[socks]-\u003esnscrape) (1.7.1)\n    \n\n\n```python\nimport pandas as pd\nimport snscrape.modules.twitter as sntwitter\n```\n\n\n```python\nquery = \"(#30DaysOfLearning OR #NG30DaysOfLearning) until:2022-06-26 since:2022-05-05\"\ntweets = []\nlimit = 30000\n\n\nfor tweet in sntwitter.TwitterHashtagScraper(query).get_items():\n    \n    if len(tweets) == limit:\n        break\n    else:\n        tweets.append([tweet.date, tweet.url, tweet.user.username, tweet.sourceLabel, tweet.user.location, tweet.content, tweet.likeCount, tweet.retweetCount,  tweet.quoteCount, tweet.replyCount])\n        \ndf = pd.DataFrame(tweets, columns=['Date', 'TweetURL','User', 'Source', 'Location', 'Tweet', 'Likes_Count','Retweet_Count', 'Quote_Count', 'Reply_Count'])\n\ndf.to_csv('30DLTweets.csv')\n```\n\n\n```python\ndf.head()\n```\n\n\n\n\n\u003cdiv\u003e\n\n\u003ctable border=\"1\" class=\"dataframe\"\u003e\n  \u003cthead\u003e\n    \u003ctr style=\"text-align: right;\"\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003eDate\u003c/th\u003e\n      \u003cth\u003eTweetURL\u003c/th\u003e\n      \u003cth\u003eUser\u003c/th\u003e\n      \u003cth\u003eSource\u003c/th\u003e\n      \u003cth\u003eLocation\u003c/th\u003e\n      \u003cth\u003eTweet\u003c/th\u003e\n      \u003cth\u003eLikes_Count\u003c/th\u003e\n      \u003cth\u003eRetweet_Count\u003c/th\u003e\n      \u003cth\u003eQuote_Count\u003c/th\u003e\n      \u003cth\u003eReply_Count\u003c/th\u003e\n    \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003cth\u003e0\u003c/th\u003e\n      \u003ctd\u003e2022-06-25 22:51:18+00:00\u003c/td\u003e\n      \u003ctd\u003ehttps://twitter.com/poetrineer/status/15408300...\u003c/td\u003e\n      \u003ctd\u003epoetrineer\u003c/td\u003e\n      \u003ctd\u003eTwitter for Android\u003c/td\u003e\n      \u003ctd\u003eOyo, Nigeria\u003c/td\u003e\n      \u003ctd\u003eSo as one of my commitment to document my lear...\u003c/td\u003e\n      \u003ctd\u003e0\u003c/td\u003e\n      \u003ctd\u003e0\u003c/td\u003e\n      \u003ctd\u003e0\u003c/td\u003e\n      \u003ctd\u003e1\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e1\u003c/th\u003e\n      \u003ctd\u003e2022-06-25 22:44:10+00:00\u003c/td\u003e\n      \u003ctd\u003ehttps://twitter.com/poetrineer/status/15408282...\u003c/td\u003e\n      \u003ctd\u003epoetrineer\u003c/td\u003e\n      \u003ctd\u003eTwitter for Android\u003c/td\u003e\n      \u003ctd\u003eOyo, Nigeria\u003c/td\u003e\n      \u003ctd\u003eFinally, here is my updated COVID-19 Data Anal...\u003c/td\u003e\n      \u003ctd\u003e1\u003c/td\u003e\n      \u003ctd\u003e0\u003c/td\u003e\n      \u003ctd\u003e0\u003c/td\u003e\n      \u003ctd\u003e0\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e2\u003c/th\u003e\n      \u003ctd\u003e2022-06-25 19:25:58+00:00\u003c/td\u003e\n      \u003ctd\u003ehttps://twitter.com/MichealOjuri/status/154077...\u003c/td\u003e\n      \u003ctd\u003eMichealOjuri\u003c/td\u003e\n      \u003ctd\u003eTwitter Web App\u003c/td\u003e\n      \u003ctd\u003eOyo, Nigeria\u003c/td\u003e\n      \u003ctd\u003e#30NGDaysOfLearning\\n#30daysoflearning \\n#micr...\u003c/td\u003e\n      \u003ctd\u003e0\u003c/td\u003e\n      \u003ctd\u003e0\u003c/td\u003e\n      \u003ctd\u003e0\u003c/td\u003e\n      \u003ctd\u003e0\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e3\u003c/th\u003e\n      \u003ctd\u003e2022-06-25 16:44:36+00:00\u003c/td\u003e\n      \u003ctd\u003ehttps://twitter.com/oye__aashu/status/15407377...\u003c/td\u003e\n      \u003ctd\u003eoye__aashu\u003c/td\u003e\n      \u003ctd\u003eTwitter for Android\u003c/td\u003e\n      \u003ctd\u003eNainital, India\u003c/td\u003e\n      \u003ctd\u003eDay 4/ #30daysoflearning learned all about arr...\u003c/td\u003e\n      \u003ctd\u003e3\u003c/td\u003e\n      \u003ctd\u003e0\u003c/td\u003e\n      \u003ctd\u003e0\u003c/td\u003e\n      \u003ctd\u003e1\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e4\u003c/th\u003e\n      \u003ctd\u003e2022-06-25 12:49:02+00:00\u003c/td\u003e\n      \u003ctd\u003ehttps://twitter.com/hsb_data/status/1540678455...\u003c/td\u003e\n      \u003ctd\u003ehsb_data\u003c/td\u003e\n      \u003ctd\u003eTwitter Web App\u003c/td\u003e\n      \u003ctd\u003eNew Jersey\u003c/td\u003e\n      \u003ctd\u003eLearning about sub queries on @DataCamp (SQL) ...\u003c/td\u003e\n      \u003ctd\u003e1\u003c/td\u003e\n      \u003ctd\u003e0\u003c/td\u003e\n      \u003ctd\u003e0\u003c/td\u003e\n      \u003ctd\u003e0\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\n\n\n\n```python\ndf.describe()\n```\n\n\n\n\n\u003cdiv\u003e\n\n\u003ctable border=\"1\" class=\"dataframe\"\u003e\n  \u003cthead\u003e\n    \u003ctr style=\"text-align: right;\"\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003eLikes_Count\u003c/th\u003e\n      \u003cth\u003eRetweet_Count\u003c/th\u003e\n      \u003cth\u003eQuote_Count\u003c/th\u003e\n      \u003cth\u003eReply_Count\u003c/th\u003e\n    \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003cth\u003ecount\u003c/th\u003e\n      \u003ctd\u003e683.000000\u003c/td\u003e\n      \u003ctd\u003e683.000000\u003c/td\u003e\n      \u003ctd\u003e683.000000\u003c/td\u003e\n      \u003ctd\u003e683.000000\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003emean\u003c/th\u003e\n      \u003ctd\u003e15.780381\u003c/td\u003e\n      \u003ctd\u003e3.812592\u003c/td\u003e\n      \u003ctd\u003e0.185944\u003c/td\u003e\n      \u003ctd\u003e1.166911\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003estd\u003c/th\u003e\n      \u003ctd\u003e41.164555\u003c/td\u003e\n      \u003ctd\u003e12.665561\u003c/td\u003e\n      \u003ctd\u003e0.737938\u003c/td\u003e\n      \u003ctd\u003e2.626554\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003emin\u003c/th\u003e\n      \u003ctd\u003e0.000000\u003c/td\u003e\n      \u003ctd\u003e0.000000\u003c/td\u003e\n      \u003ctd\u003e0.000000\u003c/td\u003e\n      \u003ctd\u003e0.000000\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e25%\u003c/th\u003e\n      \u003ctd\u003e1.000000\u003c/td\u003e\n      \u003ctd\u003e0.000000\u003c/td\u003e\n      \u003ctd\u003e0.000000\u003c/td\u003e\n      \u003ctd\u003e0.000000\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e50%\u003c/th\u003e\n      \u003ctd\u003e2.000000\u003c/td\u003e\n      \u003ctd\u003e0.000000\u003c/td\u003e\n      \u003ctd\u003e0.000000\u003c/td\u003e\n      \u003ctd\u003e0.000000\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e75%\u003c/th\u003e\n      \u003ctd\u003e8.000000\u003c/td\u003e\n      \u003ctd\u003e2.000000\u003c/td\u003e\n      \u003ctd\u003e0.000000\u003c/td\u003e\n      \u003ctd\u003e1.000000\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003emax\u003c/th\u003e\n      \u003ctd\u003e549.000000\u003c/td\u003e\n      \u003ctd\u003e248.000000\u003c/td\u003e\n      \u003ctd\u003e9.000000\u003c/td\u003e\n      \u003ctd\u003e29.000000\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\n\n\n\nThe dataset is also available at [30DLTweets](https://github.com/globalsmile/Twitter-Sentiment-Analysis/blob/main/30DLTweets.csv)\n\n---\n\n# Data Preparation\n\nData transformation was done in Power Query and the dataset was loaded into Microsoft Power BI Desktop for modeling.\n\nThe Twitter Sentiment dataset is given by a table named:\n\n- `30DLTweets` which has `10 columns and 680 rows` of observation\n\n\nThe tabulation below shows the `30DLTweets` table with its column names and their description:\n| Column Name | Description |\n| ----------- | ----------- |\n| Date | Represents the date and time of tweet |\n| TweetURL | Describes the tweet url |\n| User | Describes the username of the user |\n| Source | Descibes the device type of the user  |\n| Location | Describes the location of the user |\n| Tweet | Describes the content of the tweet |\n| Likes_Count | Represents the count of likes of the tweet |\n| Retweet_Count | Represents the count of retweets of the tweet |\n| Quote_Count | Represents the count of quote tweets on the tweet |\n| Reply_Count | Represents the count of reply on the tweet |\n\nData Cleaning for the dataset was done in power query as follows:\n\n- The `30DLTweets` table was split into a dimension and  2 fact tables respectively, hence called:\n1. `UserProfile`\n2. `TweetStats`\n3. `TweetProfile`\n\n- A calculated column `UserID` was created in each of the tables using the M-formula `UserID = [User] \u0026 \"_\" \u0026 [Source]`\n- Unnecessary columns were removed in each of the tables\n- Each of the columns in the tables were validated to have the correct data type\n\n\nTo ensure the accuracy of the dates in the `Date` column of `TweetStats` and `TweetProfile` tables, a date table was created for referencing using the M-formula:\n\n`{Number.From(List.Min(TweetProfile[Date]))..Number.From(List.Max(TweetProfile[Date]))}`\n\nHere is a breakdown of what the formula does:\n\nFor the dataset, we want the start date to reflect the earliest to latest date that we have in the data: May 9, 2022 - June 25, 2022.\n\n`Day Name` column was inserted into the date table and renamed to ` DayOfTheWeek` \n\nThe date table was named `Calender`.\n\n---\n\n# Data Modeling\n\nAfter the dataset was cleaned and transformed, it was ready to be modeled.\n\n- The `Calender` table was marked as the official date table in the dataset.\n- A `one-to-many (*:1) relationship` was created between the `TweetStats` and the `Calender` tables using the `date` column in each of the tables \n- A `one-to-many (*:1) relationship` was created between the `TweetProfile` and the `Calender` tables using the `date` column in each of the tables \n- A `one-to-many (*:1) relationship` was created between the `UserProfile` and the `TweetStats` tables using the `UserID` column in each of the tables \n- A `one-to-many (*:1) relationship` was created between the `UserProfile` and the `TweetProfile` tables using the `UserID` column in each of the tables \n- The realtioships formed in the data model is a `Star Schema` and is shown below:\n\n\u003cimg align=\"right\" alt=\"Data Model\" width=\"1000\" height = \"400\" src=\"https://user-images.githubusercontent.com/106287208/187106255-bd25a422-fd74-4f5e-a529-7a9052915252.png\"\u003e\n\n\n---\n\n# Data Visualization\n\nData visualization for the dataset was done in 3 folds using Microsoft Power BI Desktop:\n\n- The `Content Analysis`: Shows the tools by mention, word cloud, top active users, etc.\n-  The `Summary`: Shows the total number of tweets, total number of users, tweet by day of the week, etc.\n-  The `Dashboard`: Shows visualization from `Content Analysis` and `Summary` to provide answer to the [Problem Statement](https://github.com/globalsmile/Twitter-Sentiment-Analysis#Problem-Statement).\n\n\nFigure 1 shows visualizations from `Content Analysis` page\n\n| Figure 1 |\n| ----------- |\n| ![image](https://user-images.githubusercontent.com/106287208/187567316-46bc6332-7507-4f11-b3c7-a18a52ed8e14.png) |\n\nFigure 2 shows visualizations from `Summary` page\n\n| Figure 2 |\n| ----------- |\n| ![image](https://user-images.githubusercontent.com/106287208/187567503-2db262f6-c7f2-449d-9b11-01ff7e084022.png) |\n\nFigure 3 shows visualizations from `Dashboard` page\n\n| Figure 3 |\n| ----------- |\n| ![image](https://user-images.githubusercontent.com/106287208/187677655-9bbd0398-f0fe-4af0-870f-f9e090633f13.png) |\n\n---\n\n# Data Analysis\n\nMeasures used in visualization are:\n\n- Total No. of Tweets = `COUNT(TweetProfile[Tweet])`\n- Total No. of Users = `COUNT(UserProfile[user])`\n- Total No. of Sources = `COUNT(UserProfile[Source])`\n- Average Engagement Rate = `DIVIDE(Total No. of Users, Total No. of Tweets)`\n\n\nAs shown from [Data Visualization](https://github.com/globalsmile/Twitter-Sentiment-Analysis#Data-Visualization), It can be deduced that:\n\n- The were `680` tweets for the #NG30DaysOfLearning\n- The were about `234` users\n- The average engagement rate is `34%` \n\n---\n\n# Insights\n\nAs shown by [Data Visualization](https://github.com/globalsmile/Twitter-Sentiment-Analysis#Data-Visualization), It can be deduced that:\n\n- The most active user of the #NG30DaysOfLearning is [theoyinbooke](https://techcommunity.microsoft.com/t5/user/viewprofilepage/user-id/1379718) with 110 tweets\n- The most mentioned tool is Github\n\n---\n\n# Shareable Link\n\nYou can interact with the report here: \n\n[View Report](https://app.powerbi.com/view?r=eyJrIjoiZjMzMjk1ZDAtYzBjYy00OTZjLTk1YzQtMzI1MjE0NWFkOGYxIiwidCI6IjQ5ODY4YWYzLWNjNWYtNDIxNC04YjdmLTQwZjM3NDY0OWEwOSJ9)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fboddepallyvenkatesh06%2Fclean-and-analyze-social-media-usage-data-with-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fboddepallyvenkatesh06%2Fclean-and-analyze-social-media-usage-data-with-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fboddepallyvenkatesh06%2Fclean-and-analyze-social-media-usage-data-with-python/lists"}