{"id":24075837,"url":"https://github.com/anurima-saha/topic_modelling_lda_hdbscan","last_synced_at":"2025-02-26T23:13:05.472Z","repository":{"id":251973526,"uuid":"839012715","full_name":"anurima-saha/Topic_Modelling_LDA_HDBSCAN","owner":"anurima-saha","description":"Using unsupervised learning to group reddit text and identify major conspiracy theories using NLP, LDA, spacy, SVD, SBert embedding and HDBSCAN.","archived":false,"fork":false,"pushed_at":"2024-08-06T20:30:02.000Z","size":4416,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-26T23:13:01.455Z","etag":null,"topics":["hdbscan","latent-dirichlet-allocation","natural-language-processing","sbert","spacy","topic-modeling","unsupervised-learning"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/anurima-saha.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-06T19:41:37.000Z","updated_at":"2024-12-18T03:02:16.000Z","dependencies_parsed_at":"2024-08-06T23:20:28.750Z","dependency_job_id":null,"html_url":"https://github.com/anurima-saha/Topic_Modelling_LDA_HDBSCAN","commit_stats":null,"previous_names":["anurima-saha/topic-modelling","anurima-saha/topic_modelling_lda_hdbscan"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anurima-saha%2FTopic_Modelling_LDA_HDBSCAN","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anurima-saha%2FTopic_Modelling_LDA_HDBSCAN/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anurima-saha%2FTopic_Modelling_LDA_HDBSCAN/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anurima-saha%2FTopic_Modelling_LDA_HDBSCAN/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/anurima-saha","download_url":"https://codeload.github.com/anurima-saha/Topic_Modelling_LDA_HDBSCAN/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240947673,"owners_count":19883031,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hdbscan","latent-dirichlet-allocation","natural-language-processing","sbert","spacy","topic-modeling","unsupervised-learning"],"created_at":"2025-01-09T19:29:40.050Z","updated_at":"2025-02-26T23:13:05.459Z","avatar_url":"https://github.com/anurima-saha.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Topic Modelling \n## Indentifying major conspiracy theories from reddit text\nThis project uses unsupervised learning to group reddit text and identify major conspiracy theories using NLP, LDA, spacy, SVD, SBert embedding and HDSCAN.\n\n## Data:\nShape - (17155, 2) \u003cbr\u003e\nColumns – date (in type datetime[ns]), text (in type object)\u003cbr\u003e\nDate Range - 01-01-2015 to 17-02-2023 \u003cbr\u003e\n\n## Process Flow:\n\n#### Data Cleaning:\n* Removing leading spaces\n  \u003e![image](https://github.com/user-attachments/assets/5fd5acac-90c4-4969-8956-047094962b1a)\n* Removing emojis and any other component that is not a word or a number\n  \u003e![image](https://github.com/user-attachments/assets/593a9198-069a-43fc-84c5-24c3e037c6cc)\n\n#### Term Extraction:\n**Spacy** model \"en_core_web_sm\" has been used for term extract along with the Matcher feature it provides.\nWe are trying to detect a pattern that begins with an adjective or a noun followed by singular/ plural common nouns or proper nouns along with hypen.\nFrom the extracted terms it is noticed that there are terms in text than do not contribute to our analysis like\n\u003e![image](https://github.com/user-attachments/assets/9ebc3d12-cbe7-4867-9e3e-bed3c4505236)\n\u003cbr\u003eSo, we invoke a second round of Data Cleaning that removes words like\n\u003e![image](https://github.com/user-attachments/assets/dbb611c5-5bcf-45ca-a9ec-5319a9c4fed9)\n\u003cbr\u003eThis is followed by term extraction using C values with theta = 100. The list of ten most common terms and 20 least common terms has been provided below\n\u003cbr\u003e![image](https://github.com/user-attachments/assets/5a0a3fd9-27cb-4f4b-8c4f-eff09bd2ec2a)\n#### Tokenization:\nWe create tokens from text data using Spacy pipeline incorporating the terms as created above\n\u003cbr\u003e![image](https://github.com/user-attachments/assets/3f91874e-bbe1-4025-8ee2-e02456a3bff5)\n\n## Topic Modelling:\nWe have used the LDA model from tomotopy with the following features\n\u003cbr\u003e![image](https://github.com/user-attachments/assets/2171ad25-90c6-47fc-b90a-50ab5d10ed87)\n\u003cbr\u003eThe twenty topics are plotted on a 2D space as below:\n\u003cbr\u003e![image](https://github.com/user-attachments/assets/3b3bb731-7c1a-48e4-b0fc-a523a71bb07d)\n\u003cbr\u003e Looking at top 50 words from each topic we label them as shown in the table below\n\u003e![image](https://github.com/user-attachments/assets/aae2159f-8c20-4c93-97fd-e7c9e0ebc6a6)\n![image](https://github.com/user-attachments/assets/217843fa-ad8d-44b0-8829-91d42a63d5d7)\n\n## Clustering:\n* The “text” data has been cleaned to remove emojis and unnecessary text and punctuation expect “.” as required for sentence tokenization.\n* We have removed all data that has less than 5 words.\n* We have removed data that begins with \"your post\" or “please contact” to remove reddit submission messages.\n\n#### Sentence Splitter:\nWe have used sentence splitter from Spacy to take a batch of 5 sentences at a time from each post\n![image](https://github.com/user-attachments/assets/f94c5567-f075-4eea-831d-8c394994b3a7)\n\n#### Embedding:\n###### SVD:\nWe have used TruncatedSVD to transform the data into a *20 dimensional vector*.\n![image](https://github.com/user-attachments/assets/205777a5-e3ce-4729-bb75-0fcdecbcc09c)\n\n###### Spacy:\nWe have used the \"en_core_web_lg\" from Spacy to transform the cleaned text into a *300-dimension vector*. Further dimension reduction has been done using TruncatedSVD to bring it down to 20 dimension.\n![image](https://github.com/user-attachments/assets/1dbfa992-4cf7-4e3b-997f-4d297da55b27)\n\n###### SBERT:\nDynamic embedding was done using SBERT sentence transformer from \"all-mpnet-base-v2\" to obtain a *768 dimension vector*.Further dimension reduction has been done using TruncatedSVD to bring it down to 20 dimensions.\n![image](https://github.com/user-attachments/assets/076c205a-a88f-439a-ace1-ab236b237bbf)\n\n###### Perplexity:\nAs seen above, with Perplexity=50, the SBERT model clearly produces better results as compared to LSA and Spacy. Thus, we tried to plot the vector projections with perplexity 100, 150 and 200, to compare results and select best model for clustering\n![image](https://github.com/user-attachments/assets/3a4c8f26-5c87-4cee-b27e-5db45e0c47a3) \n\u003cbr\u003ePerplexity of 200 seems to provide optimal results for the analysis\n\n#### HDBSCAN\nWe have considered a cluster size of 20 and fitted the SBERT vectors.\n![image](https://github.com/user-attachments/assets/42877263-0bb0-4982-8a8a-5cf362b927db)\nHighlighted terms in each cluster after incorporating c-values gives us the following topics as focus\n\u003cbr\u003e![image](https://github.com/user-attachments/assets/02e11cd6-effa-4063-b6ff-31c78b7c4e8b)\n\n## Results:\n\n#### Dynamic Bokeh Plot\n![image](https://github.com/user-attachments/assets/5b575bec-009d-4236-9749-a7b61a49fd4a)\n\nFor interactive feature and further details please refer to notebook and [project reoprt](https://github.com/anurima-saha/topic-modelling/blob/main/Project%20Report.pdf). A few examples have been highlighted below.\n\n1. Presidential Election Fraud\n   \u003cbr\u003e![image](https://github.com/user-attachments/assets/b2e55eaf-a7fb-403e-8ed5-13cbc2b6f891)\n   \n2. Evolution Theory vs Religious Beliefs\n  \u003cbr\u003e![image](https://github.com/user-attachments/assets/27531f4a-bda8-43bf-9e4a-f66a8b780bc7)\n\n3. Zionists – Israel and Palestine Crisis\n  \u003cbr\u003e![image](https://github.com/user-attachments/assets/d3f4b8ec-b35e-4e41-af87-39153f227579)\n\n4. Flat Earth\n   \u003cbr\u003e![image](https://github.com/user-attachments/assets/2804e4bd-ae42-4ab1-b083-18ec3e6fa7de)\n \n5. Ukraine War\n   \u003cbr\u003e![image](https://github.com/user-attachments/assets/ca8ffea1-e16e-406c-b55b-4a2f1ebadd9b)\n\n6. Anti – Maskers (COVID 19)\n   \u003cbr\u003e![image](https://github.com/user-attachments/assets/217dbd76-d953-4979-987e-9032670d74f3)\n\n   \n\n   \n\n\n\n\n\n\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanurima-saha%2Ftopic_modelling_lda_hdbscan","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanurima-saha%2Ftopic_modelling_lda_hdbscan","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanurima-saha%2Ftopic_modelling_lda_hdbscan/lists"}