{"id":15303890,"url":"https://github.com/vladkens/twscrape","last_synced_at":"2025-05-13T21:08:18.631Z","repository":{"id":163153043,"uuid":"636864199","full_name":"vladkens/twscrape","owner":"vladkens","description":"2025! X / Twitter API scrapper with authorization support. Allows you to scrape search results, User's profiles (followers/following), Tweets (favoriters/retweeters) and more.","archived":false,"fork":false,"pushed_at":"2025-03-09T16:15:37.000Z","size":4029,"stargazers_count":1551,"open_issues_count":45,"forks_count":175,"subscribers_count":20,"default_branch":"main","last_synced_at":"2025-04-28T14:57:17.543Z","etag":null,"topics":["api","async","automation","elonmusk","httpx","python","scraper","snscrape","twitter","twitter-api","twitter-bot","twitter-scraper","x-api"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/twscrape/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vladkens.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"vladkens","buy_me_a_coffee":"vladkens"}},"created_at":"2023-05-05T20:39:28.000Z","updated_at":"2025-04-28T11:40:34.000Z","dependencies_parsed_at":null,"dependency_job_id":"89d153c2-72a0-436d-912a-8f8f41a573f6","html_url":"https://github.com/vladkens/twscrape","commit_stats":null,"previous_names":[],"tags_count":24,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vladkens%2Ftwscrape","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vladkens%2Ftwscrape/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vladkens%2Ftwscrape/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vladkens%2Ftwscrape/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vladkens","download_url":"https://codeload.github.com/vladkens/twscrape/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254028961,"owners_count":22002282,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","async","automation","elonmusk","httpx","python","scraper","snscrape","twitter","twitter-api","twitter-bot","twitter-scraper","x-api"],"created_at":"2024-10-01T07:01:08.758Z","updated_at":"2025-05-13T21:08:13.568Z","avatar_url":"https://github.com/vladkens.png","language":"Python","readme":"# twscrape\n\n\u003cdiv align=\"center\"\u003e\n\n[\u003cimg src=\"https://badges.ws/pypi/v/twscrape\" alt=\"version\" /\u003e](https://pypi.org/project/twscrape)\n[\u003cimg src=\"https://badges.ws/pypi/python/twscrape\" alt=\"py versions\" /\u003e](https://pypi.org/project/twscrape)\n[\u003cimg src=\"https://badges.ws/pypi/dm/twscrape\" alt=\"downloads\" /\u003e](https://pypi.org/project/twscrape)\n[\u003cimg src=\"https://badges.ws/github/license/vladkens/twscrape\" alt=\"license\" /\u003e](https://github.com/vladkens/twscrape/blob/main/LICENSE)\n[\u003cimg src=\"https://badges.ws/badge/-/buy%20me%20a%20coffee/ff813f?icon=buymeacoffee\u0026label\" alt=\"donate\" /\u003e](https://buymeacoffee.com/vladkens)\n\n\u003c/div\u003e\n\nTwitter GraphQL API implementation with [SNScrape](https://github.com/JustAnotherArchivist/snscrape) data models.\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\".github/example.png\" alt=\"example of cli usage\" height=\"400px\"\u003e\n\u003c/div\u003e\n\n## Install\n\n```bash\npip install twscrape\n```\nOr development version:\n```bash\npip install git+https://github.com/vladkens/twscrape.git\n```\n\n## Features\n- Support both Search \u0026 GraphQL Twitter API\n- Async/Await functions (can run multiple scrapers in parallel at the same time)\n- Login flow (with receiving verification code from email)\n- Saving/restoring account sessions\n- Raw Twitter API responses \u0026 SNScrape models\n- Automatic account switching to smooth Twitter API rate limits\n\n## Usage\n\nThis project requires authorized X/Twitter accounts to work with the API. You have two options:\n\n1. **Create Your Own Account**: While you can register a new account on X/Twitter yourself, it's can be difficult due to strict verification processes and high ban rates.\n\n2. **Use Ready Accounts**: For immediate access, you can get ready-to-use accounts with cookies from [our recommended provider](https://kutt.it/ueeM5f). Cookie-based accounts typically have fewer login issues.\n\nFor optimal performance and to avoid IP-based restrictions, we also recommend using proxies from [our provider](https://kutt.it/eb3rXk).\n\n**Disclaimer**: While X/Twitter's Terms of Service discourage using multiple accounts, this is a common practice for data collection and research purposes. Use responsibly and at your own discretion.\n\n```python\nimport asyncio\nfrom twscrape import API, gather\nfrom twscrape.logger import set_log_level\n\nasync def main():\n    api = API()  # or API(\"path-to.db\") – default is `accounts.db`\n\n    # ADD ACCOUNTS (for CLI usage see next readme section)\n\n    # Option 1. Adding account with cookies (more stable)\n    cookies = \"abc=12; ct0=xyz\"  # or '{\"abc\": \"12\", \"ct0\": \"xyz\"}'\n    await api.pool.add_account(\"user3\", \"pass3\", \"u3@mail.com\", \"mail_pass3\", cookies=cookies)\n\n    # Option2. Adding account with login / password (less stable)\n    # email login / password required to receive the verification code via IMAP protocol\n    # (not all email providers are supported, e.g. ProtonMail)\n    await api.pool.add_account(\"user1\", \"pass1\", \"u1@example.com\", \"mail_pass1\")\n    await api.pool.add_account(\"user2\", \"pass2\", \"u2@example.com\", \"mail_pass2\")\n    await api.pool.login_all() # try to login to receive account cookies\n\n    # API USAGE\n\n    # search (latest tab)\n    await gather(api.search(\"elon musk\", limit=20))  # list[Tweet]\n    # change search tab (product), can be: Top, Latest (default), Media\n    await gather(api.search(\"elon musk\", limit=20, kv={\"product\": \"Top\"}))\n\n    # tweet info\n    tweet_id = 20\n    await api.tweet_details(tweet_id)  # Tweet\n    await gather(api.retweeters(tweet_id, limit=20))  # list[User]\n\n    # Note: this method have small pagination from X side, like 5 tweets per query\n    await gather(api.tweet_replies(tweet_id, limit=20))  # list[Tweet]\n\n    # get user by login\n    user_login = \"xdevelopers\"\n    await api.user_by_login(user_login)  # User\n\n    # user info\n    user_id = 2244994945\n    await api.user_by_id(user_id)  # User\n    await gather(api.following(user_id, limit=20))  # list[User]\n    await gather(api.followers(user_id, limit=20))  # list[User]\n    await gather(api.verified_followers(user_id, limit=20))  # list[User]\n    await gather(api.subscriptions(user_id, limit=20))  # list[User]\n    await gather(api.user_tweets(user_id, limit=20))  # list[Tweet]\n    await gather(api.user_tweets_and_replies(user_id, limit=20))  # list[Tweet]\n    await gather(api.user_media(user_id, limit=20))  # list[Tweet]\n\n    # list info\n    await gather(api.list_timeline(list_id=123456789))\n\n    # trends\n    await gather(api.trends(\"news\"))  # list[Trend]\n    await gather(api.trends(\"sport\"))  # list[Trend]\n    await gather(api.trends(\"VGltZWxpbmU6DAC2CwABAAAACHRyZW5kaW5nAAA\"))  # list[Trend]\n\n    # NOTE 1: gather is a helper function to receive all data as list, FOR can be used as well:\n    async for tweet in api.search(\"elon musk\"):\n        print(tweet.id, tweet.user.username, tweet.rawContent)  # tweet is `Tweet` object\n\n    # NOTE 2: all methods have `raw` version (returns `httpx.Response` object):\n    async for rep in api.search_raw(\"elon musk\"):\n        print(rep.status_code, rep.json())  # rep is `httpx.Response` object\n\n    # change log level, default info\n    set_log_level(\"DEBUG\")\n\n    # Tweet \u0026 User model can be converted to regular dict or json, e.g.:\n    doc = await api.user_by_id(user_id)  # User\n    doc.dict()  # -\u003e python dict\n    doc.json()  # -\u003e json string\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```\n\n### Stoping iteration with break\n\nIn order to correctly release an account in case of `break` in loop, a special syntax must be used. Otherwise, Python's events loop will release lock on the account sometime in the future. See explanation [here](https://github.com/vladkens/twscrape/issues/27#issuecomment-1623395424).\n\n```python\nfrom contextlib import aclosing\n\nasync with aclosing(api.search(\"elon musk\")) as gen:\n    async for tweet in gen:\n        if tweet.id \u003c 200:\n            break\n```\n\n## CLI\n\n### Get help on CLI commands\n\n```sh\n# show all commands\ntwscrape\n\n# help on specific comand\ntwscrape search --help\n```\n\n### Add accounts\n\nTo add accounts use `add_accounts` command. Command syntax is:\n```sh\ntwscrape add_accounts \u003cfile_path\u003e \u003cline_format\u003e\n```\n\nWhere:\n`\u003cline_format\u003e` is format of line if accounts file splited by delimeter. Possible tokens:\n- `username` – required\n- `password` – required\n- `email` – required\n- `email_password` – to receive email code (you can use `--manual` mode to get code)\n- `cookies` – can be any parsable format (string, json, base64 string, etc)\n- `_` – skip column from parse\n\nTokens should be splited by delimeter, usually \"`:`\" used.\n\nExample:\n\nI have account files named `order-12345.txt` with format:\n```text\nusername:password:email:email password:user_agent:cookies\n```\n\nCommand to add accounts will be (user_agent column skiped with `_`):\n```sh\ntwscrape add_accounts ./order-12345.txt username:password:email:email_password:_:cookies\n```\n\n### Login accounts\n\n_Note:_ If you added accounts with cookies, login not required.\n\nRun:\n\n```sh\ntwscrape login_accounts\n```\n\n`twscrape` will start login flow for each new account. If X will ask to verify email and you provided `email_password` in `add_account`, then `twscrape` will try to receive verification code by IMAP protocol. After success login account cookies will be saved to db file for future use.\n\n#### Manual email verification\n\nIn case your email provider not support IMAP protocol (ProtonMail, Tutanota, etc) or IMAP is disabled in settings, you can enter email verification code manually. To do this run login command with `--manual` flag.\n\nExample:\n\n```sh\ntwscrape login_accounts --manual\ntwscrape relogin user1 user2 --manual\ntwscrape relogin_failed --manual\n```\n\n### Get list of accounts and their statuses\n\n```sh\ntwscrape accounts\n\n# Output:\n# username  logged_in  active  last_used            total_req  error_msg\n# user1     True       True    2023-05-20 03:20:40  100        None\n# user2     True       True    2023-05-20 03:25:45  120        None\n# user3     False      False   None                 120        Login error\n```\n\n### Re-login accounts\n\nIt is possible to re-login specific accounts:\n\n```sh\ntwscrape relogin user1 user2\n```\n\nOr retry login for all failed logins:\n\n```sh\ntwscrape relogin_failed\n```\n\n### Use different accounts file\n\nUseful if using a different set of accounts for different actions\n\n```\ntwscrape --db test-accounts.db \u003ccommand\u003e\n```\n\n### Search commands\n\n```sh\ntwscrape search \"QUERY\" --limit=20\ntwscrape tweet_details TWEET_ID\ntwscrape tweet_replies TWEET_ID --limit=20\ntwscrape retweeters TWEET_ID --limit=20\ntwscrape user_by_id USER_ID\ntwscrape user_by_login USERNAME\ntwscrape user_media USER_ID --limit=20\ntwscrape following USER_ID --limit=20\ntwscrape followers USER_ID --limit=20\ntwscrape verified_followers USER_ID --limit=20\ntwscrape subscriptions USER_ID --limit=20\ntwscrape user_tweets USER_ID --limit=20\ntwscrape user_tweets_and_replies USER_ID --limit=20\ntwscrape trends sport\n```\n\nThe default output is in the console (stdout), one document per line. So it can be redirected to the file.\n\n```sh\ntwscrape search \"elon mask lang:es\" --limit=20 \u003e data.txt\n```\n\nBy default, parsed data is returned. The original tweet responses can be retrieved with `--raw` flag.\n\n```sh\ntwscrape search \"elon mask lang:es\" --limit=20 --raw\n```\n\n### About `limit` param\n\nX API works through pagination, each API method can have different defaults for per page parameter (and this parameter can't be changed by caller). So `limit` param in `twscrape` is the desired number of objects (tweets or users, depending on the method). `twscrape` tries to return NO LESS objects than requested. If the X API returns less or more objects, `twscrape` will return whatever X gives.\n\n## Proxy\n\nThere are few options to use proxies.\n\n1. You can add proxy per account\n\n```py\nproxy = \"http://login:pass@example.com:8080\"\nawait api.pool.add_account(\"user4\", \"pass4\", \"u4@mail.com\", \"mail_pass4\", proxy=proxy)\n```\n\n2. You can use global proxy for all accounts\n\n```py\nproxy = \"http://login:pass@example.com:8080\"\napi = API(proxy=proxy)\ndoc = await api.user_by_login(\"elonmusk\")\n```\n\n3. Use can set proxy with environemt variable `TWS_RPOXY`:\n\n```sh\nTWS_PROXY=socks5://user:pass@127.0.0.1:1080 twscrape user_by_login elonmusk\n```\n\n4. You can change proxy any time like:\n\n```py\napi.proxy = \"socks5://user:pass@127.0.0.1:1080\"\ndoc = await api.user_by_login(\"elonmusk\")  # new proxy will be used\napi.proxy = None\ndoc = await api.user_by_login(\"elonmusk\")  # no proxy used\n```\n\n5. Proxy priorities\n\n- `api.proxy` have top priority\n- `env.proxy` will be used if `api.proxy` is None\n- `acc.proxy` have lowest priotity\n\nSo if you want to use proxy PER ACCOUNT, do NOT override proxy with env variable or by passing proxy param to API.\n\n_Note:_ If proxy not working, exception will be raised from API class.\n\n## Environment Variables\n\n- `TWS_PROXY` - global proxy for all accounts (e.g. `socks5://user:pass@127.0.0.1:1080`)\n- `TWS_WAIT_EMAIL_CODE` - timeout for email verification code during login (default: `30`, in seconds)\n- `TWS_RAISE_WHEN_NO_ACCOUNT` - raise `NoAccountError` exception when no available accounts, instead of waiting (default: `false`, values: `false`/`0`/`true`/`1`)\n\n## Limitations\n\nX/Twitter regularly [updates](https://x.com/elonmusk/status/1675187969420828672) their rate limits. Current basic behavior:\n- Request limits reset every 15 minutes for each endpoint individually\n- Each account has separate limits for different operations (search, profile views, etc.)\n\nAPI data limitations:\n- `user_tweets` \u0026 `user_tweets_and_replies` - can return ~3200 tweets maximum\n- Rate limits may vary based on account age and status\n\n## Articles\n- [How to still scrape millions of tweets in 2023](https://medium.com/@vladkens/how-to-still-scrape-millions-of-tweets-in-2023-using-twscrape-97f5d3881434)\n- [_(Add Article)_](https://github.com/vladkens/twscrape/edit/main/readme.md)\n\n## See also\n- [twitter-advanced-search](https://github.com/igorbrigadir/twitter-advanced-search) – guide on search filters\n- [twitter-api-client](https://github.com/trevorhobenshield/twitter-api-client) – Implementation of Twitter's v1, v2, and GraphQL APIs\n- [snscrape](https://github.com/JustAnotherArchivist/snscrape) – is a scraper for social networking services (SNS)\n- [twint](https://github.com/twintproject/twint) – Twitter Intelligence Tool\n","funding_links":["https://github.com/sponsors/vladkens","https://buymeacoffee.com/vladkens"],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvladkens%2Ftwscrape","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvladkens%2Ftwscrape","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvladkens%2Ftwscrape/lists"}