{"id":14982344,"url":"https://github.com/burhanahmed1/TwitSpark","last_synced_at":"2025-10-29T12:31:36.926Z","repository":{"id":247530157,"uuid":"826068805","full_name":"burhanahmed1/Twitter-Sentiment-Analysis-Using-PySpark","owner":"burhanahmed1","description":"This repository contains a project that demonstrates how to perform sentiment analysis on Twitter data using Apache Spark, including data preprocessing, feature engineering, model training, and evaluation.","archived":false,"fork":false,"pushed_at":"2024-07-09T04:48:43.000Z","size":1402,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-02T01:31:43.462Z","etag":null,"topics":["apache-spark","batch-gradient-descent","kmeans-clustering","knearest-neighbor-classification","machine-learning","matplotlib","nltk-python","numpy","pandas","pyspark","python","schotastic-gradient-descent","seaborn","sentiment-analysis","textblob-sentiment-analysis"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/burhanahmed1.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-09T03:53:23.000Z","updated_at":"2024-08-29T17:40:33.000Z","dependencies_parsed_at":"2024-07-09T08:17:10.854Z","dependency_job_id":"24abcc22-6f9a-4ffd-a961-d5a3fc3f57d2","html_url":"https://github.com/burhanahmed1/Twitter-Sentiment-Analysis-Using-PySpark","commit_stats":null,"previous_names":["burhanahmed1/twitter-sentiment-analysis-using-pyspark"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/burhanahmed1%2FTwitter-Sentiment-Analysis-Using-PySpark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/burhanahmed1%2FTwitter-Sentiment-Analysis-Using-PySpark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/burhanahmed1%2FTwitter-Sentiment-Analysis-Using-PySpark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/burhanahmed1%2FTwitter-Sentiment-Analysis-Using-PySpark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/burhanahmed1","download_url":"https://codeload.github.com/burhanahmed1/Twitter-Sentiment-Analysis-Using-PySpark/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238825708,"owners_count":19537112,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-spark","batch-gradient-descent","kmeans-clustering","knearest-neighbor-classification","machine-learning","matplotlib","nltk-python","numpy","pandas","pyspark","python","schotastic-gradient-descent","seaborn","sentiment-analysis","textblob-sentiment-analysis"],"created_at":"2024-09-24T14:05:14.283Z","updated_at":"2025-10-29T12:31:36.919Z","avatar_url":"https://github.com/burhanahmed1.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TwitSpark: Distributed Sentiment Analysis of Twitter Data Using Apache Spark\n\nTwitter Sentiment Analysis repository contains a project for performing sentiment analysis on Twitter data using Apache Spark.\n\n## Contents\n\n- `Sentiment_Analysis.ipynb`: Jupyter Notebook containing the code for the sentiment analysis.\n- `Sentiment.csv`: The dataset file containing the Twitter data and sentiment labels.\n\n## Project Overview\n\nThis project demonstrates how to use Apache Spark for sentiment analysis on Twitter data. The steps covered in the project include:\n\n1. **Data Loading**: Reading the dataset into Spark DataFrame.\n2. **Data Cleaning**: Preprocessing the data by handling missing values and performing necessary transformations.\n3. **Feature Engineering**: Extracting features from the text data for model training.\n4. **Model Training**: Training a machine learning model to classify the sentiment of tweets.\n5. **Evaluation**: Evaluating the model's performance using appropriate metrics.\n\n## Getting Started\n\n### Prerequisites\n\n- Apache Spark\n- Jupyter Notebook\n- Python\n- Required Python libraries: `pandas`, `numpy`, `nltk`, `pyspark`\n\n### Installation\n\n1. Clone the repository:\n\n   ```bash\n   git clone https://github.com/burhanahmed1/Twitter-Sentiment-Analysis-Using-PySpark.git\n   cd Twitter-Sentiment-Analysis-Using-PySpark\n   pip install -r requirements.txt\n   ```\n2. Install the required Python libraries:\n   ```bash\n   pip install pandas numpy nltk pyspark\n   ```\n3. Start Jupyter Notebook:\n   ```bash\n   jupyter notebook\n   ```\n\n4. Open `Sentiment_Analysis.ipynb` in Jupyter Notebook and run the cells to execute the project.\n\n### Results\nThe project demonstrates the effectiveness of using Apache Spark for sentiment analysis on large datasets. The final model achieves good accuracy in classifying the sentiment of tweets.\n- The accuracy of the sentiment model using **Logistic Regression** is `0.62`\n- Root Mean Squared Error (RMSE) and Explained Variance (R²) using **Linear Regression** are `0.7331581773635055` and `0.07966124788395001` respectively.\n- The accuracy of the sentiment model using **Batch Gradient Descent** is `0.73`\n- The accuracy of the sentiment model using **Schotastic Gradient Descent** is `0.75`\n\n### Visualizations \nData visualization techniques such as confusion matrices are used to evaluate the performance of the sentiment classification model and scatter plots are used to visualize the distribution and relationships of features in the dataset.\n\n\u003cdiv align=\"center\"\u003e\n  \n  \u003cimg src=\"sc/LR.png\" alt=\"Scatter PLot\" width=\"700\"/\u003e\n  \u003cimg src=\"sc/Kmeans.png\" alt=\"Scatter Plot\" width=\"700\"/\u003e\n  \u003cimg src=\"sc/KNN.png\" alt=\"Scatter Plot\" width=\"700\"/\u003e\n  \u003cimg src=\"sc/Confusion Matrix.png\" alt=\"Confusion Matrix\" width=\"700\"/\u003e\n\u003c/div\u003e\n\n## Contributing\nContributions are welcome! If you have any ideas, suggestions, or improvements, feel free to open an issue or submit a pull request.\n\n## License\nThis project is licensed under the MIT License.\n\n## Acknowledgments\nThanks to the open-source community for providing valuable tools and libraries.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fburhanahmed1%2FTwitSpark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fburhanahmed1%2FTwitSpark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fburhanahmed1%2FTwitSpark/lists"}