{"id":13337901,"url":"https://github.com/giocoal/reddit-tldr-summarizer-and-topic-modeling","last_synced_at":"2025-03-11T08:32:06.520Z","repository":{"id":109121986,"uuid":"596131661","full_name":"giocoal/reddit-tldr-summarizer-and-topic-modeling","owner":"giocoal","description":"Extreme Extractive Text Summarization and Topic Modeling (using LSA and LDA techniques) over Reddit Posts from TLDRHQ dataset.","archived":false,"fork":false,"pushed_at":"2024-01-19T00:46:12.000Z","size":55026,"stargazers_count":6,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-10-23T20:11:20.718Z","etag":null,"topics":["extreme-summarization","latent-dirichlet-allocation","latent-semantic-analysis","lda","lda-model","lsa","lsa-model","nlp","part-of-speech-tagging","reddit","reddit-bot","reddit-dataset","social-media","summarization","text-analysis","text-preprocessing","text-summarization","tldr","tldr9","topic-modeling"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/giocoal.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-01T14:34:03.000Z","updated_at":"2024-03-04T13:18:05.000Z","dependencies_parsed_at":"2024-10-23T19:28:01.838Z","dependency_job_id":"c809c648-4110-4599-84f9-efb043ad8faf","html_url":"https://github.com/giocoal/reddit-tldr-summarizer-and-topic-modeling","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/giocoal%2Freddit-tldr-summarizer-and-topic-modeling","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/giocoal%2Freddit-tldr-summarizer-and-topic-modeling/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/giocoal%2Freddit-tldr-summarizer-and-topic-modeling/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/giocoal%2Freddit-tldr-summarizer-and-topic-modeling/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/giocoal","download_url":"https://codeload.github.com/giocoal/reddit-tldr-summarizer-and-topic-modeling/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243000834,"owners_count":20219751,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["extreme-summarization","latent-dirichlet-allocation","latent-semantic-analysis","lda","lda-model","lsa","lsa-model","nlp","part-of-speech-tagging","reddit","reddit-bot","reddit-dataset","social-media","summarization","text-analysis","text-preprocessing","text-summarization","tldr","tldr9","topic-modeling"],"created_at":"2024-07-29T19:15:10.600Z","updated_at":"2025-03-11T08:32:05.799Z","avatar_url":"https://github.com/giocoal.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Extreme Extractive Text Summarization and Topic Modeling over Reddit Posts from TLDRHQ dataset.\n\n[![Contributors][contributors-shield]][contributors-url]\n[![Forks][forks-shield]][forks-url]\n[![Stargazers][stars-shield]][stars-url]\n[![Issues][issues-shield]][issues-url]\n[![MIT License][license-shield]][license-url]\n[![LinkedIn][linkedin-shield]][linkedin-url]\n\n## Table of contents\n* [Abstract](#abstract)\n* [Paper](https://github.com/giocoal/reddit-tldr-summarizer-and-topic-modeling/blob/main/Paper/Paper%20-%20Text%20Summarization%20and%20Topic%20Modeling%20over%20Reddit%20Posts.pdf) and [Slides](https://github.com/giocoal/reddit-tldr-summarizer-and-topic-modeling/blob/main/Paper/Slides%20-%20Text%20Summarization%20and%20Topic%20Modeling%20over%20Reddit%20Posts.pdf)\n* [Requirements](#requirements)\n* [TLDRHQ: Data and Text Pre-processing](#tldrhq-data-and-text-pre-processing)\n* [Extreme Extractive Text Summarization](#extreme-extractive-summarization-task)\n* [Topic Modeling](#topic-modeling-task)\n* [Status](#status)\n* [Contact](#contact)\n* [License](#license)\n* [Contributing](#contributing)\n\n## Abstract\n\nReddit is a social news aggregation and discussion website where users post content (such as links, text posts,\nimages, and videos) on a wide range of topics in domain-specific boards called ”communities” or ”subreddits.”\nThe following project aims to implement text summarization and topic modelling pipelines on the textual\ncontent of Redditors’ posts. The dataset used to achieve these goals is the Reddit-based TL;DR summarization\ndataset TLDRHQ containing approximately 1.7 million Reddit posts (submissions as well as comments)\nobtained through scraping techniques. Each instance in the dataset includes a Reddit post and its TL;DR, which\nis an acronym for ”Too Long; Didn’t Read” and is an extremely short summary of the post’s content that is good\npractice for users to leave at the end of a post. While the Reddit usage has increased, the practice of write TL;DR\ndidn’t keep the pace. In this context, a system (such as a bot) capable to automatically generate the TL;DR of\na post could improve Reddit usability. However, the high abstractivity, heterogeneity and noisy of posts make\nthe text summarization task challenging. In this work a supervised extreme extractive summarization model\nis developed. Despite its lower complexity, results show that its performance are not so different with respect\nto the state of the art BertSumExt. Moreover the topic modeling analysis of the posts could be really useful in\nidentifying the hidden topics in a post and evaluate if it’s published in the right subreddit. In this project LSA and\nLDA techniques are used. On this dataset LSA outperformed LDA and identified 20 well defined topics providing\nthe respective document-topics and topic-terms matrices\n\n## Requirements\n\n- python 3.10.7\n- contractions==0.1.73\n- gensim==4.3.0\n- ipython==8.8.0\n- matplotlib==3.6.3\n- nltk==3.8.1\n- num2words==0.5.12\n- numpy==1.22.1\n- pandas==1.3.5\n- seaborn==0.12.2\n- simplemma==0.9.0\n- spacy==3.4.4\n- swifter==1.3.4\n- textblob==0.17.1\n- tqdm==4.64.1\n- scikit-learn==1.2.0\n- rouge-score==0.1.2\n- imbalanced-learn==0.10.1\n- wordcloud==1.8.2.2\n- pyLDAvis==3.3.1\n\n## TLDRHQ: Data and Text Pre-processing\n\n### Step 0. Prepare Folders\n\nFirst of all, create three empty folders: `./DatasetTLDRHQ`,`./ProcessedData` and `./Dataset_splitted`.\n\n### Step 1. Download and extract the dataset\n\nDownload annotations from the [official Google Drive Folder](https://drive.google.com/file/d/1jCi0Mn0k-pid5SSTafov11-e1A9LEZed/view?usp=sharing) and extract them in `./DatasetTLDRHQ`, resulting in a folder tree like this:\n\n```\nproject_folder\n└───Dataset_TLDRHQ\n    ├───dataset-m0\n    ├───dataset-m1\n    ├───dataset-m2\n    ├───dataset-m2021\n    ├───dataset-m3\n    ├───dataset-m4\n    └───dataset-m6\n```\n\n### Step 2. Perform data cleaning and splitting of the dataset\nRun the `0_cleaning.py` script which will perform data cleaning (removing duplicates), splits the dataset into training/validation and test sets (splitting the training set so that it is easier to manage) and then save it splitted into `.JSON` files in `./Dataset_splitted`. You get a directory tree like this:\n```\nproject_folder\n└───Dataset_splitted\n    ├───test.json\n    ├───train_1.json\n    ├───train_10.json\n    ├───train_2.json\n    ├───train_3.json\n    ├───train_4.json\n    ├───train_5.json\n    ├───train_6.json\n    ├───train_7.json\n    ├───train_8.json\n    ├───train_9.json\n    └───val.json\n```\n\n\n### Step 3. Perform text pre-processing on the dataset\nRun the `0_normalizing.py` script which will perform senteces splitting, text normalization, tokenization, stop-words removal, lemmatization and POS tagging on `document` variable, containing reddit posts. Then save it splitted into various `.JSON` files in `./ProcessedData`. You get a directory tree like this:\n```\nproject_folder\n└───ProcessedData\n    ├───test.json\n    ├───train_1.json\n    ├───train_10.json\n    ├───train_2.json\n    ├───train_3.json\n    ├───train_4.json\n    ├───train_5.json\n    ├───train_6.json\n    ├───train_7.json\n    ├───train_8.json\n    ├───train_9.json\n    └───val.json\n```\nThe text normalisation operations performed include, in order: Sentence Splitting, HTML tags and entities removal, Extra White spaces Removal, URLs Removal, Emoji Removal, User Age Processing (e.g. 25m becomes 25 male), Numbers Processing, Control Characters Removal, Case Folding, Repeated characters processing (e.g. reallllly becomes really), Fix and Expand English contradictions, Special Characters and Punctuation Removal, Tokenization (Uni-Grams), Stop-Words and 1-character tokens, Lemmatization and POS tagging.\n\n## Extreme Extractive Summarization task\n\n### Step 0. Split and clean 'ProcessedData' for easy management\nRun notebook `1_preprocessing4summarization.ipynb` in order to:\n- remove document without summary\n- remove document with a single sentence\n- split train dataset \n\n```\nproject_folder\n└───Processed Data For Summarization\n    ├───test_0.json\n    ├───test_1.json\n    ├───test_2.json\n    ├───train_1_0.json\n    ├───train_1_1.json\n    ├───train_1_2.json\n    ├───train_2_0.json\n    ├───train_2_1.json\n    ├───train_2_2.json\n    ├───  ...\n    ├───train_8_0.json\n    ├───train_8_1.json\n    ├───train_8_2.json\n    ├───train_9_0.json\n    ├───val_0.json\n    ├───val_1.json\n    └───val_2.json\n```\n\n### Step 1. Create a feature matrix for each of the JSON in 'Processed Data For Summarization'\nRun `1_featureMatrixGeneration.py` obtaining feature matrices (sentences x features). You get a directory tree like this:\n\n```\nproject_folder\n└───Feature Matrices\n    ├───test_0.csv\n    ├───test_1.csv\n    ├───test_2.csv\n    ├───train_1_0.csv\n    ├───train_1_1.csv\n    ├───train_1_2.csv\n    ├───train_2_0.csv\n    ├───train_2_1.csv\n    ├───train_2_2.csv\n    ├───  ...\n    ├───train_8_0.csv\n    ├───train_8_1.csv\n    ├───train_8_2.csv\n    ├───train_9_0.csv\n    ├───val_0.csv\n    ├───val_1.csv\n    └───val_2.csv\n```\n    \n Run the notebook `1_featureMatrixGeneration2.ipynb` to join train, val and test datasets. You get a directory tree like this:\n \n ```\n project_folder\n └───Feature Matrices\n    ├───test_0.csv\n    ├───test_1.csv\n    ├───test_2.csv\n    ├───train_1_0.csv\n    ├───train_1_1.csv\n    ├───train_1_2.csv\n    ├───train_2_0.csv\n    ├───train_2_1.csv\n    ├───train_2_2.csv\n    ├───  ...\n    ├───train_8_0.csv\n    ├───train_8_1.csv\n    ├───train_8_2.csv\n    ├───train_9_0.csv\n    ├───val_0.csv\n    ├───val_1.csv\n    ├───val_2.csv\n    ├───test.csv\n    ├───train.csv\n    └───val.csv\n```\n    \nFeatures generated at this step are the following:\n- sentence_relative_positions\n- sentence_similarity_score_1_gram\n- word_in_sentence_relative\n- NOUN_tag_ratio\n- VERB_tag_ratio\n- ADJ_tag_ratio\n- ADV_tag_ratio\n- TF_ISF\n \n    \n### Step 2. Perform CUR undersampling\nRun notebook `1_featureMatrixUndersampling.ipynb` in order to perform CUR undersampling on both train and validation data sets. You get a directory tree like this:\n\n```\nproject_folder\n└───Undersampled Data\n    ├───trainAndValMinorityClass.csv\n    └───trainAndValMajorityClassUndersampled.csv\n```\n\nMajority and minority class are splitted because CUR undersampling works only on the majority class\n\n### Step 3. Perform EditedNearestNeighbours(ENN) undersamplig\nRun notebook `1_featureMatrixAnalysis.ipynb` to perform EEN undersampling. You get a directory tree like this:\n\n```\nproject_folder\n└───Undersampled Data\n    ├───trainAndValUndersampledENN3.csv\n    ├───trainAndValMinorityClass.csv\n    └───trainAndValMajorityClassUndersampled.csv\n```\n\n### Step 4. Machine Learning model selection and evaluation\nRun notebook `1_featureMatrixAnalysis.ipynb` to perform a RandomizedSearcCV over the following models\n- RandomForestClassifier\n- LogisticRegression\n- HistGradientBoostingClassifier\n\nwith a few possible parameters configuration. \n\nThen, evaluate the resulting best model on the test set with respect to:\n- ROC curve\n- Recall\n- Precision\n- Accuracy\n\n### Step 5. Perform Maximal Marginal Relevance(MMR) selection\nRun notebook `1_featureMatrixAnalysis.ipynb` to perform MMR and obtain an extractive summary for each document in the test set.\n\n### Step 6. Summary Evaluation\nRun notebook `1_featureMatrixAnalysis.ipynb` to measure summaries quality by means of \n- Rouge1\n- Rouge2 \n- RougeL \n\n## Topic Modeling task\n### Step 0. Perform preprocessing\nRun the `2_preprocessing4topic_modeling.ipynb` script to process and extract only the useful data. The output is saved here:\n\n```\nproject_folder\n└───processed_dataset\n    ├───test.json\n```\n\n### Step 1. Perform topic modeling on the test set\nRun the `2_topic_modeling.ipynb` script which will perform LDA (with grid search of the best hyper-parameters) and LSA. The script saves 9 CSV files, 3 for LSA and 6 for LDA (UMass and CV coherence measures), containing: document-topic matrix, topic-term matrix and a table with topic insights.\n\n```\nproject_folder\n└───Results_topic_modeling\n    ├───lda_doc_topic.csv\n    ├───lda_doc_topic_CV.csv\n    ├───lda_top_terms.csv\n    ├───lda_top_terms_CV.csv\n    ├───lda_topic_term.csv\n    ├───lda_topic_term_cv.csv\n    ├───lsa_doc_topic.csv\n    ├───lsa_top_terms.csv\n    ├───lsa_topic_term.csv\n```\nSaves images regarding the number of words per document and wordcloud in\n\n```\nproject_folder\n└───Images\n```\n\nSaves hyperparameters grid search results for UMass and CV coherence in\n```\nproject_folder\n└───Hyperparameters\n    ├───tuning.csv\n    ├───tuning_CV.csv\n```\n\n## Status\n\n Project is: ![##c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)  _Done_\n\n\n## Contact\n\n[Giorgio Carbone](https://github.com/giocoal) - feel free to contact me!\n\n\n## License\n* \u003eYou can check out the full license [here](https://github.com/giocoal/reddit-tldr-summarizer-and-topic-modeling/blob/main/README.md)\n\nThis project is licensed under the terms of the **MIT** license.\n\n## Contributing\n\n1. Fork it (\u003chttps://github.com/giocoal/reddit-tldr-summarizer-and-topic-modeling.git\u003e)\n2. Create your feature branch (`git checkout -b feature/fooBar`)\n3. Commit your changes (`git commit -am 'Add some fooBar'`)\n4. Push to the branch (`git push origin feature/fooBar`)\n5. Create a new Pull Request\n\n\n# Contributors\n\n* [Giorgio Carbone](https://github.com/giocoal)\n* [Marco Scatassi](https://github.com/marco-scatassi)\n* [Gianluca Scuri](https://github.com/gianscuri)    \n\n\u003c!-- Project is: ![##c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)  _Done_\n Project is: ![##ff0000](https://via.placeholder.com/15/ff0000/000000?text=+)  _Under-Proccess_\n\n[![Build](https://github.com/SimonIT/spotifylyrics/workflows/Build/badge.svg)](https://github.com/SimonIT/spotifylyrics/actions?query=workflow%3ABuild)\n[![Current Release](https://img.shields.io/github/release/SimonIT/spotifylyrics.svg)](https://github.com/SimonIT/spotifylyrics/releases)\n[![License](https://img.shields.io/github/license/SimonIT/spotifylyrics.svg)](https://github.com/SimonIT/spotifylyrics/blob/master/LICENSE)\n[![GitHub All Releases](https://img.shields.io/github/downloads/SimonIT/spotifylyrics/total)](https://github.com/SimonIT/spotifylyrics/releases)\n\n\u003c!-- MARKDOWN LINKS \u0026 IMAGES --\u003e\n\u003c!-- https://www.markdownguide.org/basic-syntax/#reference-style-links --\u003e\n[contributors-shield]: https://img.shields.io/github/contributors/giocoal/reddit-tldr-summarizer-and-topic-modeling.svg?style=for-the-badge\n[contributors-url]: https://github.com/giocoal/reddit-tldr-summarizer-and-topic-modeling/graphs/contributors\n[forks-shield]: https://img.shields.io/github/forks/giocoal/reddit-tldr-summarizer-and-topic-modeling.svg?style=for-the-badge\n[forks-url]: https://github.com/giocoal/reddit-tldr-summarizer-and-topic-modeling/network/members\n[stars-shield]: https://img.shields.io/github/stars/giocoal/reddit-tldr-summarizer-and-topic-modeling.svg?style=for-the-badge\n[stars-url]: https://github.com/giocoal/reddit-tldr-summarizer-and-topic-modeling/stargazers\n[issues-shield]: https://img.shields.io/github/issues/giocoal/reddit-tldr-summarizer-and-topic-modeling.svg?style=for-the-badge\n[issues-url]: https://github.com/giocoal/reddit-tldr-summarizer-and-topic-modeling/issues\n[license-shield]: https://img.shields.io/github/license/giocoal/reddit-tldr-summarizer-and-topic-modeling.svg?style=for-the-badge\n[license-url]: https://github.com/giocoal/reddit-tldr-summarizer-and-topic-modeling/blob/master/LICENSE\n[linkedin-shield]: https://img.shields.io/badge/-LinkedIn-black.svg?style=for-the-badge\u0026logo=linkedin\u0026colorB=555\n[linkedin-url]: https://www.linkedin.com/in/giorgio-carbone-63154219b/\n[product-screenshot]: images/screenshot.png\n[Next.js]: https://img.shields.io/badge/next.js-000000?style=for-the-badge\u0026logo=nextdotjs\u0026logoColor=white\n[Next-url]: https://nextjs.org/\n[React.js]: https://img.shields.io/badge/React-20232A?style=for-the-badge\u0026logo=react\u0026logoColor=61DAFB\n[React-url]: https://reactjs.org/\n[Vue.js]: https://img.shields.io/badge/Vue.js-35495E?style=for-the-badge\u0026logo=vuedotjs\u0026logoColor=4FC08D\n[Vue-url]: https://vuejs.org/\n[Angular.io]: https://img.shields.io/badge/Angular-DD0031?style=for-the-badge\u0026logo=angular\u0026logoColor=white\n[Angular-url]: https://angular.io/\n[Svelte.dev]: https://img.shields.io/badge/Svelte-4A4A55?style=for-the-badge\u0026logo=svelte\u0026logoColor=FF3E00\n[Svelte-url]: https://svelte.dev/\n[Laravel.com]: https://img.shields.io/badge/Laravel-FF2D20?style=for-the-badge\u0026logo=laravel\u0026logoColor=white\n[Laravel-url]: https://laravel.com\n[Bootstrap.com]: https://img.shields.io/badge/Bootstrap-563D7C?style=for-the-badge\u0026logo=bootstrap\u0026logoColor=white\n[Bootstrap-url]: https://getbootstrap.com\n[JQuery.com]: https://img.shields.io/badge/jQuery-0769AD?style=for-the-badge\u0026logo=jquery\u0026logoColor=white\n[JQuery-url]: https://jquery.com\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgiocoal%2Freddit-tldr-summarizer-and-topic-modeling","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgiocoal%2Freddit-tldr-summarizer-and-topic-modeling","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgiocoal%2Freddit-tldr-summarizer-and-topic-modeling/lists"}