{"id":16928334,"url":"https://github.com/maartengr/soan","last_synced_at":"2025-05-09T01:44:42.995Z","repository":{"id":40957188,"uuid":"153516999","full_name":"MaartenGr/soan","owner":"MaartenGr","description":"Social Analysis based on Whatsapp data","archived":false,"fork":false,"pushed_at":"2023-07-18T13:10:20.000Z","size":8957,"stargazers_count":143,"open_issues_count":9,"forks_count":67,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-31T20:47:34.777Z","etag":null,"topics":["nlp","sentiment-analysis","soan","tf-idf","whatsapp","whatsapp-analysis","whatsapp-statistics","word-cloud","wordcloud"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MaartenGr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-10-17T20:07:49.000Z","updated_at":"2025-03-21T19:34:57.000Z","dependencies_parsed_at":"2024-10-27T12:26:50.027Z","dependency_job_id":null,"html_url":"https://github.com/MaartenGr/soan","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaartenGr%2Fsoan","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaartenGr%2Fsoan/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaartenGr%2Fsoan/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaartenGr%2Fsoan/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MaartenGr","download_url":"https://codeload.github.com/MaartenGr/soan/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253176444,"owners_count":21866142,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["nlp","sentiment-analysis","soan","tf-idf","whatsapp","whatsapp-analysis","whatsapp-statistics","word-cloud","wordcloud"],"created_at":"2024-10-13T20:36:30.059Z","updated_at":"2025-05-09T01:44:42.954Z","avatar_url":"https://github.com/MaartenGr.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SoAn\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/MaartenGr/soan/raw/master/results/calendar.png\" height=\"400\"/\u003e\n\u003c/p\u003e\n\n\u003e Code for applying natural language processing methods on whatsapp conversations\n\n**SoAn** (Social Analysis) can be used to extract word frequency, word clouds, TF-IDF, sentiment analysis, and more\nfrom whatsapp conversations. The main application was initially used to analyze the messages between my wife and me, \nbut I extended so that it can be used for your own messages. \n\n\n\u003ca name=\"toc\"/\u003e\u003c/a\u003e\n## Table of Contents\n\n1. [Instructions](#instructions)\n\n2. [Output](#output)\n\n    a. [General Plots](#output-general)\n    \n    b. [TF-IDF](#output-tfidf)\n    \n    c. [Emoji](#output-emoji)\n    \n    d. [Sentiment](#output-sentiment)\n    \n    e. [Word Clouds](#output-wordclouds)\n    \n    f. [Topic Modeling](#output-topic)\n    \n\u003ca name=\"instructions\"/\u003e\u003c/a\u003e\n## 1. Instructions\n[Back to ToC](#toc)\n\nThere are several steps for using this repository:\n* **Download** or **fork** this repository\n* Install the requirements with `pip install -r requirements.txt`\n* Save your whatsapp.txt file in the data folder\n  * To download your whatsapp messages simply go open your whatsapp, go to a conversation, click the three vertical dots and export the file\n* Finally, from the commandline, run the following:\n  * `python soan.py --file whatsapp.txt --language english`\n* The results will be saved as images and text files in the results folder\n\nIn the notebooks folder, you will also find the **soan.ipynb** where you can run individual pieces of the code. \n\n\n\u003ca name=\"output\"/\u003e\u003c/a\u003e\n## 2. Output\n[Back to ToC](#toc)\n\n\n\u003ca name=\"output-general\"/\u003e\u003c/a\u003e\n#### 2.a General Plots\n\nThere are 4 types of plots to be generated:\n* Messages over time\n* Active days of each user\n  * Spider\n  * Histogram\n* Active hours of each user\n* Calendar plot\n\n* There are 2 types of stats that are generated:\n  * General statistics (text frequency, etc.)\n  * Timing\n  \nBelow are some examples of the plots above:\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/MaartenGr/soan/raw/master/results/spider_plot.png\" height=\"400\"/\u003e\n\u003cimg src=\"https://github.com/MaartenGr/soan/raw/master/results/Me_active_days.png\" height=\"200\"/\u003e\n\u003cimg src=\"https://github.com/MaartenGr/soan/raw/master/results/moments.png\" height=\"400\"/\u003e\n\u003c/p\u003e\n\nBelow are some examples of the text generated:\n\n##########################  \n  Number of Messages  \n##########################  \n  \n4444\tHer  \n3266\tMe  \n    \n#########################  \n  Messages per hour    \n#########################  \n  \nHer:\t0.1259887165820883  \nMe:\t0.09259206758710628  \n  \n\n\u003ca name=\"output-tfidf\"/\u003e\u003c/a\u003e\n#### 2.b TF-IDF\n\nUsing a class-based TF-IDF, I extract the most important words per person and plot them using a horizontal barchart with a mask as image.\nI created a horizontal bar chart with two bars stacked on top of each other both plotted on a background image. I started with a background image and plotted the actual values on the left and made it fully transparent with a white border to separate the bars. Then, on top of that I plotted which bars so that the right part of the image would get removed.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/MaartenGr/soan/raw/master/results/Me_tfidf.png\" height=\"400\"/\u003e\n\u003cimg src=\"https://github.com/MaartenGr/soan/blob/master/results/Her_tfidf.png\" height=\"400\"/\u003e\n\u003c/p\u003e\n\n**NOTE:** In the notebook, you will see more instructions on how to use your own image. \n\n\n\u003ca name=\"output-emoji\"/\u003e\u003c/a\u003e\n#### 2.c Emoji\n\nThese analysis are based on the Emojis used in each message. Below you can find the following:\n\n* Unique Emoji per user\n* Commonly used Emoji per user\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/MaartenGr/soan/raw/master/results/emoji_Me.png\" height=\"400\"/\u003e\n\u003c/p\u003e \n\n\u003ca name=\"output-sentiment\"/\u003e\u003c/a\u003e\n#### 2.d Sentiment Analysis\n\nThe sentiment from each sentence in the messages is extract per user using Vader and visualized as follows:\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/MaartenGr/soan/raw/master/results/sentiment.png\" height=\"400\"/\u003e\n\u003c/p\u003e \n\n\u003ca name=\"output-wordclouds\"/\u003e\u003c/a\u003e\n#### 2.e Sentiment Analysis\n\nFor each user, a word cloud will be made based on frequent and important words. Stopwords are removed\nif you have supplied the language:\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/MaartenGr/soan/raw/master/results/wordcloud_Me.png\" height=\"400\"/\u003e\n\u003c/p\u003e \n\n\u003ca name=\"output-topic\"/\u003e\u003c/a\u003e\n#### 2.f Topic Modeling\n\nFor each user, the most frequent topics using LDA and NMF are modeled and saved a .txt file:\n\n####  \n Me   \n####  \n  \nTopics in nmf model:  \nTopic #0: ga boodschappen nodig lieverd halen uurtje half  \nTopic #1: thuis wel goed haha lekker we morgen  \nTopic #2: lieverd dank hey fijn allerliefste plezier verwacht  \nTopic #3: gezellig jeey super jeeeey erg hartstikke samen  \nTopic #4: love you most more schattie much very  \n\n**Visualizations Wife**  \nBelow, you will find an overview of the visualizations I made for my wife, in part using this package:  \n\u003cimg src=\"https://github.com/MaartenGr/soan/blob/master/overview.png\"/\u003e\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaartengr%2Fsoan","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaartengr%2Fsoan","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaartengr%2Fsoan/lists"}