{"id":20839329,"url":"https://github.com/kennethkn/congresstweets-analysis","last_synced_at":"2026-04-25T19:31:47.961Z","repository":{"id":237652759,"uuid":"794970546","full_name":"kennethkn/congresstweets-analysis","owner":"kennethkn","description":"Data science analysis of the tweets of U.S. congress members, 2017-2023, using R, Python \u0026 PostgreSQL","archived":false,"fork":false,"pushed_at":"2024-05-02T19:02:28.000Z","size":9416,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-12T10:09:57.142Z","etag":null,"topics":["postgresql","python","rlanguage","sqlalchemy-python"],"latest_commit_sha":null,"homepage":"https://html-preview.github.io/?url=https://github.com/kennethkn/congresstweets-analysis/blob/main/analysis.html","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kennethkn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-02T10:24:28.000Z","updated_at":"2024-05-12T08:51:05.000Z","dependencies_parsed_at":"2024-05-02T23:08:58.205Z","dependency_job_id":"755186ec-0e50-43a7-9f83-53ce9e91de62","html_url":"https://github.com/kennethkn/congresstweets-analysis","commit_stats":null,"previous_names":["kennethkn/congresstweets-analysis"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/kennethkn/congresstweets-analysis","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kennethkn%2Fcongresstweets-analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kennethkn%2Fcongresstweets-analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kennethkn%2Fcongresstweets-analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kennethkn%2Fcongresstweets-analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kennethkn","download_url":"https://codeload.github.com/kennethkn/congresstweets-analysis/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kennethkn%2Fcongresstweets-analysis/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32274982,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-25T18:29:39.964Z","status":"ssl_error","status_checked_at":"2026-04-25T18:29:32.149Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["postgresql","python","rlanguage","sqlalchemy-python"],"created_at":"2024-11-18T01:13:20.457Z","updated_at":"2026-04-25T19:31:47.948Z","avatar_url":"https://github.com/kennethkn.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# congresstweets-analysis\n\nAnalysing the tweets of U.S. congress members (2017-2023) in relation to political affiliation and current issues, by considering the sentiment, frequency, and trends of related statements to understand the priorities and features of the two parties.\n\nLink to Analysis: https://html-preview.github.io/?url=https://github.com/kennethkn/congresstweets-analysis/blob/main/analysis.html\n\n## Project Description\n\nThis project aims to foster a boarder understanding of the bipartisan U.S. politics by analyzing the tweets of U.S. congress members, including democratic and republican senators and representatives. The importance of it lies in the potential to reveal patterns and trends in the political discourse of recent years. Understanding these patterns can provide insights into the priorities and strategies of the two parties. Additionally, analyzing the sentiment of tweets can reveal their stance on current issues.\n\n\u003c!-- , as well as their uniformity or diversity in opinion, the portion of radical and moderate members, and the potential for bipartisanship.\n\nThe main challenge in answering these questions is the processing and analysis of unstructured text data. NLP techniques will be required to extract meaningful information. Also, accurately determining the sentiment of a tweet can be challenging due to the nuances of human language. The use of machine learning models will contribute to a more in-depth analysis. However, the time and computational resources required, as well as my ability, can be a limitation.\n\nThere have been several online sources that have analyzed Twitter data to study political discourse. with a few involving the specific dataset I will be using. However, they either do not answer my questions or are no longer maintained, making my project unique in the use of up-to-date data and the specific investigation target of U.S. congress members. --\u003e\n\nThe dataset available for this project is from the GitHub repository [congresstweets](\u003chttps://github.com/alexlitel/congresstweets\u003e). It contains a comprehensive collection of tweets from U.S. congress members since 2017, making it a rich resource for a diverse analysis.\n\n\u003c!-- The heavy use of twitter/X by politicians to communicate with the public and express their opinions is seen in no other counties, by which I am fascinated. --\u003e\n\n## Questions to Answer\n\n1. What is the trend of the most common words/hashtags used by democratic and republican congress members in their tweets?\n2. What are the sentiments of tweets by democratic and republican congress members on significant issues such as COVID-19, climate change, abortion, gun control, and etc?\n\n## Methodology\n\nGiven the enormous size of the dataset (~4M entries), I have chosen a database approach to store and query the data. The database is hosted locally on my computer via PostgresSQL, but you can reproduce the database by executing Python scripts in the `scripts` folder, which holds scripts for database construction as well as text mining.\n\n1. Ready PostgreSQL server (`brew install postgresql \u0026\u0026 brew services start postgresql` if you are using macOS)\n2. Create a database named `congresstweets` (`createdb congresstweets`)\n3. Clone the repository\n4. Notice the empty `data/tweets/` folder. You need to download the tweets data from the [congresstweets repo](\u003chttps://github.com/alexlitel/congresstweets\u003e), as well as [here](https://archive.org/details/tweetsofcongress) for older 2017 data. Place the downloaded json files (eg `2020-03-24.json`) in the `data/tweets/` folder.\n5. Setup venv and activate it (`python -m venv venv \u0026\u0026 source venv/bin/activate`)\n6. Install the required packages (`pip install -r requirements.txt`)\n7. Open `.env` and replace `YOUR_USERNAME` with your PostgreSQL username. (`DATABASE_URL=postgresql://YOUR_USERNAME@localhost:5432/congresstweets`)\n8. Run `models.py` to create the tables.\n9. Run `db_insert_members.py` to populate the `members` table in the database.\n10. Run `db_insert_tweets.py` to populate the `tweets` table in the database.\n11. Run `text_mining.py` to populate columns pertaining to text mining results in the `tweets` table.\n12. Open `analysis.rmd` in RStudio and knit the file to generate the analysis.\n\n## Table of Contents\n\n1. Tweet Count by Party and Year\n2. Tweet Count by Chamber and Year\n3. Top Tweeters by Year\n4. Top Hashtags\n5. Top Hashtags by Party\n6. Top Hashtags by Party and Year\n7. Top Hashtags by Chamber\n8. Top Words\n9. Top Words by Party\n10. Top Words by Party and Year\n11. Top Words by Chamber\n12. Sentiment Analysis by Party and Year\n13. Sentiment Analysis by Chamber and Year\n14. Sentiment Analysis by Topic and Party\n15. Top Accounts Retweeted\n16. Top Accounts Retweeted by Party\n17. Top Accounts Quoted\n18. Top Accounts Quoted by Party\n19. Top Accounts Mentioned\n20. Top Accounts Mentioned by Party\n\n## Room for Improvement\n\n1. Use of BERT or GPT to infer topics from tweets.\n2. Even more categories, such as top words by chamber and year, sentiment of tweets by topic and chamber, etc.\n\n## Citation\n\nMajor credits to Alex Litel for providing the dataset. \u003chttps://github.com/alexlitel/congresstweets\u003e\n\u003c!-- \u003chttps://ucsd.libguides.com/congress_twitter/senators\u003e\n\u003chttps://ucsd.libguides.com/congress_twitter/reps\u003e --\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkennethkn%2Fcongresstweets-analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkennethkn%2Fcongresstweets-analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkennethkn%2Fcongresstweets-analysis/lists"}