{"id":20167310,"url":"https://github.com/sferez/twitter_toolbox","last_synced_at":"2025-04-10T01:21:26.980Z","repository":{"id":174619263,"uuid":"651140844","full_name":"sferez/Twitter_Toolbox","owner":"sferez","description":"Complete Toolbox for Scraping, Streaming, Interact with API, Cleaning, Preprocessing, Applying NLP on Twitter Data","archived":false,"fork":false,"pushed_at":"2023-06-24T07:44:43.000Z","size":30,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-24T03:03:12.914Z","etag":null,"topics":["data-collection","data-science","nlp","preprocessing","twitter","twitter-api","twitter-scraping","twitter-streaming-api"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sferez.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-06-08T15:27:27.000Z","updated_at":"2024-01-19T10:19:15.000Z","dependencies_parsed_at":"2023-07-30T03:30:12.566Z","dependency_job_id":null,"html_url":"https://github.com/sferez/Twitter_Toolbox","commit_stats":{"total_commits":7,"total_committers":1,"mean_commits":7.0,"dds":0.0,"last_synced_commit":"2617eb6ee083e73f2eeefe88e6b2aad618a0289b"},"previous_names":["sferez/twitter_toolbox"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sferez%2FTwitter_Toolbox","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sferez%2FTwitter_Toolbox/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sferez%2FTwitter_Toolbox/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sferez%2FTwitter_Toolbox/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sferez","download_url":"https://codeload.github.com/sferez/Twitter_Toolbox/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248138237,"owners_count":21053837,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-collection","data-science","nlp","preprocessing","twitter","twitter-api","twitter-scraping","twitter-streaming-api"],"created_at":"2024-11-14T00:49:51.857Z","updated_at":"2025-04-10T01:21:26.972Z","avatar_url":"https://github.com/sferez.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Twitter Toolbox\n\nWelcome to the Twitter Toolbox, a comprehensive suite designed to simplify data acquisition, preprocessing, and analysis\nfrom Twitter. This project is an up-to-date solution built in response to the recent changes in Twitter's API and front\nend. Given that several existing libraries are no longer maintained or updated, this Twitter Toolbox ensures a seamless\ndata extraction process for data analysts, researchers, marketers, and developers alike.\n\n## Table of Contents\n\n- [Twitter Toolbox](#twitter-toolbox)\n  - [Table of Contents](#table-of-contents)\n  - [Features](#features)\n  - [Articles](#articles)\n  - [Data Acquisition](#data-acquisition)\n  - [Preprocessing](#preprocessing)\n  - [NLP](#nlp)\n  - [Future Developments](#future-developments)\n  - [Contributions and Feedback](#contributions-and-feedback)\n  - [Disclaimer](#disclaimer)\n  - [Structure](#structure)\n\n## Features\n\nThe Twitter Toolbox offers a broad spectrum of functionalities, including:\n\n**Data Acquisition**: Our toolbox equips you with everything you need to extract a variety of data from Twitter, from\nstreaming and scraping real-time data to making API calls and hydrating or dehydrating tweets.\n\n**Preprocessing**: Our tools offer data cleaning, language filtering, data labeling, and group generation features to\nrefine\nyour dataset for accurate and reliable analyses.\n\n**Natural Language Processing** (NLP): The toolbox is equipped with sentiment analysis, emotion analysis, topic\nanalysis,\nand named entity recognition to provide you with meaningful insights from the content of tweets.\n\nEach of these capabilities is designed to help you make the most out of Twitter data, whether you're exploring public\nsentiment, detecting emotional trends, identifying key themes, or recognizing named entities such as organizations or\nindividuals.\n\n## Articles\n\nI have written a series of articles to explain how to use the Twitter Toolbox. You can find them here:\n\n- [Your Guide to Real-Time Tweet Streaming](https://medium.com/@simeon.ferez/ep1-twitter-toolbox-17436c8ba4e6)\n- [Your Ultimate Guide to Data Scraping](https://medium.com/python-in-plain-english/ep2-twitter-toolbox-your-ultimate-guide-to-data-scraping-fa9f7aa18b23)\n\n## Data Acquisition\n\nCollect data from Twitter using scraping, streaming and Twitter API.\n\nLearn more about the data\ncollection [here](https://github.com/sferez/Noisy_Entropy_Estimation/tree/main/src/dataAcquisition).\n\n## Preprocessing\n\nIn progress...\n\n## NLP\n\nIn progress...\n\n## Future Developments\nThe Twitter Toolbox is an evolving project. We plan to continue adding new features as they are developed. Stay tuned\nfor regular updates and improvements!\n\n## Contributions and Feedback\nThis toolbox is designed to grow with the contributions and feedback from the community. You are welcome to suggest new\nfeatures, report any issues, or even submit pull requests. Let's collaborate to create the most valuable Twitter Toolbox\npossible!\n\n## Disclaimer\nPlease note that the use of the Twitter API and all data retrieved through this toolbox should comply with the Twitter\nTerms of Service, Developer Agreement, and Developer Policy, including Twitter's privacy policy. This project includes a\ndehydration script to comply with Twitter's terms of service, allowing for sharing only the tweet_id. Always de-identify\nthe information and respect user privacy when sharing or publishing data.\n\n## Structure\n\nProject is structured as follows:\n\n```\n├── data (Data is not stored in the repository)\n├── src\n│   ├── dataAcquisition\n│   ├── preprocessing\n│   ├── nlp\n├── docs \n└──\n```\n\nData is stored in the following structure:\n\n```\n├── data\n│   ├── \u003cscraping\u003e (Scrape from user, hashtag or keyword)\n│   │   ├── \u003cuser\u003e\n│   │   │   ├── \u003cuser\u003e_\u003cstart\u003e_\u003cend\u003e.csv\n│   │   │   ├── \u003cuser\u003e_\u003cstart\u003e_\u003cend\u003e.csv\n│   │   │   └── ...\n│   │   ├── \u003cuser\u003e\n│   │   │   ├── \u003cuser\u003e_\u003cstart\u003e_\u003cend\u003e.csv\n│   │   │   ├── \u003cuser\u003e_\u003cstart\u003e_\u003cend\u003e.csv\n│   │   │   └── ...\n│   │   └── ...\n│   ├── \u003csample-stream\u003e (Stream 1% of tweets)\n│   │   ├── \u003cdate\u003e.csv\n│   │   ├── \u003cdate\u003e.csv\n│   │   └── ...\n│   ├── \u003ccovid-github\u003e (Scrape from Github and rehydrate)\n│   │   ├── \u003cdate\u003e.csv\n│   │   ├── \u003cdate\u003e.csv\n│   │   └── ...\n│   └──\n└──\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsferez%2Ftwitter_toolbox","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsferez%2Ftwitter_toolbox","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsferez%2Ftwitter_toolbox/lists"}