{"id":18019874,"url":"https://github.com/vpaliy/twitter-scraper","last_synced_at":"2025-03-26T20:31:32.189Z","repository":{"id":62585752,"uuid":"158108990","full_name":"vpaliy/twitter-scraper","owner":"vpaliy","description":"A bot that scrapes Twitter.","archived":false,"fork":false,"pushed_at":"2020-03-01T16:15:22.000Z","size":1610,"stargazers_count":13,"open_issues_count":0,"forks_count":3,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-03-19T11:05:26.576Z","etag":null,"topics":["bot","concurrency","scrapy-crawler","searcher","twitter"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vpaliy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-11-18T17:29:24.000Z","updated_at":"2022-10-26T09:20:20.000Z","dependencies_parsed_at":"2022-11-03T22:04:07.389Z","dependency_job_id":null,"html_url":"https://github.com/vpaliy/twitter-scraper","commit_stats":null,"previous_names":["thevpaliy/twitter-santa","vpaliy/twitter-santa"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vpaliy%2Ftwitter-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vpaliy%2Ftwitter-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vpaliy%2Ftwitter-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vpaliy%2Ftwitter-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vpaliy","download_url":"https://codeload.github.com/vpaliy/twitter-scraper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245731469,"owners_count":20663196,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bot","concurrency","scrapy-crawler","searcher","twitter"],"created_at":"2024-10-30T05:12:48.870Z","updated_at":"2025-03-26T20:31:31.255Z","avatar_url":"https://github.com/vpaliy.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# twitter-scraper\n\n\u003cimg src=\"https://github.com/vpaliy/twitter-santa/blob/master/art/final.jpg\" /\u003e\n\n## Why?\nThis [Def Con talk](https://www.youtube.com/watch?v=iAOOdYsK7MM) was my motivation behind creating this project. I built a more general version of the bot discussed in the talk;\n\nUse this bot for educational purposes only.\n\n## Basics\nBefore we get started, you need to know that this bot uses Python \u003e= 3 only.\n\nThis bot contains a small number of command line arguments that you can pass in.\nHere is the complete list of them:\n- `-a` or `--agents` (file containing user agents to be used while making HTTP requests to Twitter)\n- `-i` or `--invalidate` (erase all saved sessions; this includes cookies or any other data saved from previous web requests)\n- `-c` or `--config` (file containing the configurations for the bot; if not provided, we will used the default configurations in the config directory)\n- `-e` or `--executor-count` (specifies how many executors - explained below - we need to launch)\n\nSpeaking of interface design, I would define three important abstractions here:\n- Searchers (primary responsibility is to search for the desired tweets and pass them along to handlers)\n\n- Handlers (responsible for filtering out and parsing tweets to determine appropriate actions that should be applied to those tweets. When finished, pass the actions to executors)\n\n- Actions (many tweets request different actions to participate in a contest - retweet, like, follow, comment, etc. You can also customize it and add yours)\n\n- Executors (receive a list of actions from handlers and execute them. Plain and simple)\n\nLet's dive into the configuration of those objects. Here's a sample JSON file:\n\n```json\n{\n    \"searchers\": [{\n        \"count\": 1,\n        \"search-queries\": [\"rt to win\", \"#contest\"],\n        \"scan-time\": 560,\n        \"month-diff\": 1,\n        \"request-delay\": 5,\n        \"error-delay\": 5,\n        \"empty-request-delay\": 20,\n        \"error-tries\": 5,\n        \"empty-tries\": 5\n    },{\n        \"count\": 2,\n        \"search-queries\": [\"rt to donate\", \"#donation\", \"RT\", \"donation\"],\n        \"scan-time\": 560,\n        \"month-diff\": 1,\n        \"tweet-limit\": 500\n    }],\n\n    \"handlers\": [{\n        \"count\": 5,\n        \"keywords\": [\"win\", \"winner\", \"lucky\"],\n        \"avoid-usernames\": [\"bot\", \"bot spotter\", \"bot spotting\"]\n    },{\n        \"count\": 2,\n        \"keywords\": [\"charity\", \"donate\", \"donation\"],\n        \"avoid-usernames\": [\"bot\", \"bot spotter\", \"bot spotting\"]\n    }],\n\n    \"executors\": [{\n      \"count\": 2,\n      \"request-delay\": 5\n    }]\n}\n```\n\n**Searchers**:\n - `count` (defines how many searchers with the same configuration we need to create. **Optional**)\n\n - `search-queries` (an array of strings that should be used for searching tweets. **Required**)\n\n - `scan-time` (specifies for how long this searcher should be scrapping Twitter. **Optional**)\n\n - `month-diff` (defines the \"expiration date\" for tweets in terms of months. For instance, if 1 is specified, then we would only accept tweets that have been tweeted within last month. **Optional**)\n\n - `request-delay` (specifies for how long this searcher should wait before making an HTTP request. **Optional**)\n\n - `error-delay` (specifies for how long this searcher should wait before retrying an HTTP request after an error has occurred. **Optional**)\n\n - `empty-request-delay` (specifies for how long this searcher should wait before making another HTTP request after getting an empty list of tweets. **Optional**)\n\n - `error-tries` (specifies how many error tries are allowed before terminating the bot. **Optional**)\n\n - `empty-tries` (specifies how many \"empty\" tries are allowed before terminating the current searcher. **Optional**)\n\n - `pictures-only` (true/false values. Should every tweet contain an image? **Optional**)\n\n - `verified-accounts-only` (true/false values. Should we search for tweets from verified accounts only? **Optional**)\n\n\n**Handlers**:\n  - `count` (defines how many handlers with the same configuration we need to create. **Optional**)\n\n  - `keywords` (an array of strings that every tweet should contain in order to be processed. **Required**)\n\n  - `avoid-usernames` (an array of usernames that we should avoid - bot spotters as an example. **Optional**)\n\n  - `avoid-keywords` (an array of usernames that we should avoid - bot spotters as an example. **Optional**)\n\n\nMostly all of the optional configuration fields have a default value. You can look at the source code for more information.\n\n# Installation\n\n### Option 1: Install via git clone\n\nThis would be the easiest way to play around with this bot:\n```\n$ git clone https://github.com/vpaliy/twitter-santa.git\n$ cd twitter-santa/\n$ python3 tweebot/\n```\n\n### Option 2: Download with pip3\n Coming soon...\n\n# Disclaimer\nThis bot has been written only for educational purposes.\nI hold no liability for what you do with this bot or what happens to you by using this bot.\nAbusing this bot may result in a permanent ban from Twitter (your account or IP address).\n\n# License\n```\nMIT License\n\nCopyright (c) 2018 Vasyl Paliy\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvpaliy%2Ftwitter-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvpaliy%2Ftwitter-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvpaliy%2Ftwitter-scraper/lists"}