{"id":17835235,"url":"https://github.com/hsankesara/tweets-scrapper","last_synced_at":"2025-03-19T15:30:28.422Z","repository":{"id":87573201,"uuid":"214405226","full_name":"Hsankesara/Tweets-Scrapper","owner":"Hsankesara","description":"Get all the tweets from your favorite twitter accounts. ","archived":false,"fork":false,"pushed_at":"2019-10-12T04:47:29.000Z","size":27,"stargazers_count":19,"open_issues_count":0,"forks_count":6,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-28T19:27:04.124Z","etag":null,"topics":["csv","dataframe","scraping","script","tweets"],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Hsankesara.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"authors.txt","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-10-11T10:11:23.000Z","updated_at":"2024-10-25T14:48:16.000Z","dependencies_parsed_at":null,"dependency_job_id":"bf41cb71-319c-4e0e-bb4e-4a1658933481","html_url":"https://github.com/Hsankesara/Tweets-Scrapper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Hsankesara%2FTweets-Scrapper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Hsankesara%2FTweets-Scrapper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Hsankesara%2FTweets-Scrapper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Hsankesara%2FTweets-Scrapper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Hsankesara","download_url":"https://codeload.github.com/Hsankesara/Tweets-Scrapper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243999697,"owners_count":20381406,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv","dataframe","scraping","script","tweets"],"created_at":"2024-10-27T20:17:52.918Z","updated_at":"2025-03-19T15:30:28.409Z","avatar_url":"https://github.com/Hsankesara.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Tweets-Scrapper\n\nThis script has helped me to scrap more than 30K+ tweets from more that 40 authors. The script is written such that you only have to give it a list of Twitter handles and output csv file path and it'll download all the tweets, process them and save them to a csv file without any hassle. You can checkout the dataset [here](https://github.com/Hsankesara/The-Tweets-of-Wisdom/blob/master/A%20comprehensive%20study%20of%20wisdom.ipynb) on Github and [here](https://www.kaggle.com/hsankesara/the-tweets-of-wisdom) on Kaggle. Also, I have done a comprehensive data analysis which you can find [here](https://www.kaggle.com/hsankesara/a-comprehensive-study-of-wisdom). You can also checkout the jupyter notebook I have used to scrap 30K+ tweets [here](The_Wisdom_Scrapper.ipynb).\n\n## How the script works\n\nThe script will download tweets from all the authors whose Twitter handles are written in the _authors.txt_ file in the **newline seperated format**. The script will download direct tweets, retweets and retweets with a comment. In a retweeted tweet, I took all the information (name, handle, tweetcontent and creation time) of the orignal author and stored it. Furthermore, Every retweet with a comment contains \u0026lt;Q\u0026gt; and \u0026lt;/Q\u0026gt; tags. The author's comment is followed by \u0026lt;Q\u0026gt; tag and then the content of the retweet comes which is followed by \u0026lt;/Q\u0026gt;.\n\n## How to run it\n\n1. First clone the repository\n\n```bash\ngit clone https://github.com/Hsankesara/Tweets-Scrapper.git\n```\n\n2. Then download the python dependencies.\n\n```bash\ncd Tweets-Scrapper\npip3 install -r requirements.txt\n```\n\n3. Now, create `cred.json` file which is the copy of `cred.json.sample`,\n\n```bash\ncp cred.json.sample cred.json\n```\n\n4. Get Twitter credentials and write them in `cred.json` file. You can follow [this](https://www.slickremix.com/docs/how-to-get-api-keys-and-tokens-for-twitter/) to get your access tokens.\n   Now update the `cred.json` file with the tokens you've received from Twitter.\n\n5. Write the Twitter handle of the accounts you want to scrap in _authors.txt_ in **newline seperated format**.\n\n6. run the script\n\n```bash\npython3 scrap.py authors.txt tweets.csv\n```\n\n7. Wait for it! And you'll get all the tweets soon in the csv format.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhsankesara%2Ftweets-scrapper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhsankesara%2Ftweets-scrapper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhsankesara%2Ftweets-scrapper/lists"}