{"id":13631555,"url":"https://github.com/jonbakerfish/TweetScraper","last_synced_at":"2025-04-17T22:31:06.990Z","repository":{"id":38360531,"uuid":"48011933","full_name":"jonbakerfish/TweetScraper","owner":"jonbakerfish","description":"TweetScraper is a simple crawler/spider for Twitter Search without using API","archived":false,"fork":false,"pushed_at":"2021-04-03T17:33:40.000Z","size":60,"stargazers_count":1033,"open_issues_count":3,"forks_count":314,"subscribers_count":36,"default_branch":"master","last_synced_at":"2025-03-12T20:17:45.443Z","etag":null,"topics":["scrapy","tweets","twitter","twitter-search"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jonbakerfish.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"github":null,"patreon":null,"open_collective":"tweetscraper","ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2015-12-15T01:27:31.000Z","updated_at":"2025-03-10T18:58:54.000Z","dependencies_parsed_at":"2022-07-12T17:27:43.763Z","dependency_job_id":null,"html_url":"https://github.com/jonbakerfish/TweetScraper","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonbakerfish%2FTweetScraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonbakerfish%2FTweetScraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonbakerfish%2FTweetScraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonbakerfish%2FTweetScraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jonbakerfish","download_url":"https://codeload.github.com/jonbakerfish/TweetScraper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248726536,"owners_count":21151901,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["scrapy","tweets","twitter","twitter-search"],"created_at":"2024-08-01T22:02:29.834Z","updated_at":"2025-04-17T22:31:06.694Z","avatar_url":"https://github.com/jonbakerfish.png","language":"Python","funding_links":["https://opencollective.com/tweetscraper"],"categories":["Python"],"sub_categories":[],"readme":"# Introduction #\r\n`TweetScraper` can get tweets from [Twitter Search](https://twitter.com/explore). \r\nIt is built on [Scrapy](http://scrapy.org/) without using [Twitter's APIs](https://dev.twitter.com/rest/public).\r\nThe crawled data is not as *clean* as the one obtained by the APIs, but the benefits are you can get rid of the API's rate limits and restrictions. Ideally, you can get all the data from Twitter Search.\r\n\r\n**WARNING:** please be polite and follow the [crawler's politeness policy](https://en.wikipedia.org/wiki/Web_crawler#Politeness_policy).\r\n \r\n\r\n# Installation #\r\n1. Install `conda`, you can get it from [miniconda](https://docs.conda.io/en/latest/miniconda.html). The tested python version is `3.7`. \r\n\r\n2. Install selenium python bindings: https://selenium-python.readthedocs.io/installation.html. (Note: the `KeyError: 'driver'` is caused by wrong setup)\r\n\r\n3. For ubuntu or debian user, run:\r\n    \r\n    ```\r\n    $ bash install.sh\r\n    $ conda activate tweetscraper\r\n    $ scrapy list\r\n    $ #If the output is 'TweetScraper', then you are ready to go.\r\n    ```\r\n\r\n    the `install.sh` will create a new environment `tweetscraper` and install all the dependencies (e.g., `firefox-geckodriver` and `firefox`),\r\n\r\n# Usage #\r\n1. Change the `USER_AGENT` in `TweetScraper/settings.py` to identify who you are\r\n\t\r\n\t\tUSER_AGENT = 'your website/e-mail'\r\n\r\n2. In the root folder of this project, run command like: \r\n\r\n\t\tscrapy crawl TweetScraper -a query=\"foo,#bar\"\r\n\r\n\twhere `query` is a list of keywords seperated by comma and quoted by `\"`. The query can be any thing (keyword, hashtag, etc.) you want to search in [Twitter Search](https://twitter.com/search-home). `TweetScraper` will crawl the search results of the query and save the tweet content and user information. \r\n\r\n3. The tweets will be saved to disk in `./Data/tweet/` in default settings and `./Data/user/` is for user data. The file format is JSON. Change the `SAVE_TWEET_PATH` and `SAVE_USER_PATH` in `TweetScraper/settings.py` if you want another location.\r\n\r\n\r\n# Acknowledgement #\r\nKeeping the crawler up to date requires continuous efforts, please support our work via [opencollective.com/tweetscraper](https://opencollective.com/tweetscraper).\r\n\r\n\r\n# License #\r\nTweetScraper is released under the [GNU GENERAL PUBLIC LICENSE, Version 2](https://github.com/jonbakerfish/TweetScraper/blob/master/LICENSE)\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonbakerfish%2FTweetScraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjonbakerfish%2FTweetScraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonbakerfish%2FTweetScraper/lists"}