{"id":13414090,"url":"https://github.com/n0madic/twitter-scraper","last_synced_at":"2025-03-14T20:31:07.330Z","repository":{"id":38773959,"uuid":"159687941","full_name":"n0madic/twitter-scraper","owner":"n0madic","description":"Scrape the Twitter frontend API without authentication with Golang.","archived":true,"fork":false,"pushed_at":"2023-11-04T22:39:41.000Z","size":159,"stargazers_count":878,"open_issues_count":27,"forks_count":176,"subscribers_count":15,"default_branch":"master","last_synced_at":"2024-07-31T20:53:21.910Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/n0madic.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-11-29T15:31:50.000Z","updated_at":"2024-07-30T02:43:47.000Z","dependencies_parsed_at":"2023-02-09T11:00:25.881Z","dependency_job_id":"71551b46-6202-4530-b8e5-f2ab4a557db3","html_url":"https://github.com/n0madic/twitter-scraper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/n0madic%2Ftwitter-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/n0madic%2Ftwitter-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/n0madic%2Ftwitter-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/n0madic%2Ftwitter-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/n0madic","download_url":"https://codeload.github.com/n0madic/twitter-scraper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243642086,"owners_count":20323954,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-30T20:01:57.412Z","updated_at":"2025-03-14T20:31:07.026Z","avatar_url":"https://github.com/n0madic.png","language":"Go","funding_links":[],"categories":["Third-party APIs","第三方API`第三方API 汇总`","Go","第三方api","第三方API","Utility"],"sub_categories":["Utility/Miscellaneous","查询语","实用程序/Miscellaneous","HTTP Clients","Fail injection"],"readme":"# Twitter Scraper\n\n[![Go Reference](https://pkg.go.dev/badge/github.com/n0madic/twitter-scraper.svg)](https://pkg.go.dev/github.com/n0madic/twitter-scraper)\n\nTwitter's API is annoying to work with, and has lots of limitations —\nluckily their frontend (JavaScript) has it's own API, which I reverse-engineered.\nNo API rate limits. No tokens needed. No restrictions. Extremely fast.\n\nYou can use this library to get the text of any user's Tweets trivially.\n\n## Installation\n\n```shell\ngo get -u github.com/n0madic/twitter-scraper\n```\n\n## Usage\n\n### Authentication\n\nNow all methods require authentication!\n\n#### Login\n\n```golang\nerr := scraper.Login(\"username\", \"password\")\n```\n\nUse username to login, not email!\nBut if you have email confirmation, use email address in addition:\n\n```golang\nerr := scraper.Login(\"username\", \"password\", \"email\")\n```\n\nIf you have two-factor authentication, use code:\n\n```golang\nerr := scraper.Login(\"username\", \"password\", \"code\")\n```\n\nStatus of login can be checked with:\n\n```golang\nscraper.IsLoggedIn()\n```\n\nLogout (clear session):\n\n```golang\nscraper.Logout()\n```\n\nIf you want save session between restarts, you can save cookies with `scraper.GetCookies()` and restore with `scraper.SetCookies()`.\n\nFor example, save cookies:\n\n```golang\ncookies := scraper.GetCookies()\n// serialize to JSON\njs, _ := json.Marshal(cookies)\n// save to file\nf, _ = os.Create(\"cookies.json\")\nf.Write(js)\n```\n\nand load cookies:\n\n```golang\nf, _ := os.Open(\"cookies.json\")\n// deserialize from JSON\nvar cookies []*http.Cookie\njson.NewDecoder(f).Decode(\u0026cookies)\n// load cookies\nscraper.SetCookies(cookies)\n// check login status\nscraper.IsLoggedIn()\n```\n\n#### Open account\n\nIf you don't want to use your account, you can try login as a Twitter app:\n\n```golang\nerr := scraper.LoginOpenAccount()\n```\n\n### Get user tweets\n\n```golang\npackage main\n\nimport (\n    \"context\"\n    \"fmt\"\n    twitterscraper \"github.com/n0madic/twitter-scraper\"\n)\n\nfunc main() {\n    scraper := twitterscraper.New()\n    err := scraper.LoginOpenAccount()\n    if err != nil {\n        panic(err)\n    }\n    for tweet := range scraper.GetTweets(context.Background(), \"Twitter\", 50) {\n        if tweet.Error != nil {\n            panic(tweet.Error)\n        }\n        fmt.Println(tweet.Text)\n    }\n}\n```\n\nIt appears you can ask for up to 50 tweets.\n\n### Get single tweet\n\n```golang\npackage main\n\nimport (\n    \"fmt\"\n\n    twitterscraper \"github.com/n0madic/twitter-scraper\"\n)\n\nfunc main() {\n    scraper := twitterscraper.New()\n    err := scraper.Login(username, password)\n    if err != nil {\n        panic(err)\n    }\n    tweet, err := scraper.GetTweet(\"1328684389388185600\")\n    if err != nil {\n        panic(err)\n    }\n    fmt.Println(tweet.Text)\n}\n```\n\n### Search tweets by query standard operators\n\nNow the search only works for authenticated users!\n\nTweets containing “twitter” and “scraper” and “data“, filtering out retweets:\n\n```golang\npackage main\n\nimport (\n    \"context\"\n    \"fmt\"\n    twitterscraper \"github.com/n0madic/twitter-scraper\"\n)\n\nfunc main() {\n    scraper := twitterscraper.New()\n    err := scraper.Login(username, password)\n    if err != nil {\n        panic(err)\n    }\n    for tweet := range scraper.SearchTweets(context.Background(),\n        \"twitter scraper data -filter:retweets\", 50) {\n        if tweet.Error != nil {\n            panic(tweet.Error)\n        }\n        fmt.Println(tweet.Text)\n    }\n}\n```\n\nThe search ends if we have 50 tweets.\n\nSee [Rules and filtering](https://developer.twitter.com/en/docs/tweets/rules-and-filtering/overview/standard-operators) for build standard queries.\n\n\n#### Set search mode\n\n```golang\nscraper.SetSearchMode(twitterscraper.SearchLatest)\n```\n\nOptions:\n\n* `twitterscraper.SearchTop` - default mode\n* `twitterscraper.SearchLatest` - live mode\n* `twitterscraper.SearchPhotos` - image mode\n* `twitterscraper.SearchVideos` - video mode\n* `twitterscraper.SearchUsers` - user mode\n\n### Get profile\n\n```golang\npackage main\n\nimport (\n    \"fmt\"\n    twitterscraper \"github.com/n0madic/twitter-scraper\"\n)\n\nfunc main() {\n    scraper := twitterscraper.New()\n    scraper.LoginOpenAccount()\n    profile, err := scraper.GetProfile(\"Twitter\")\n    if err != nil {\n        panic(err)\n    }\n    fmt.Printf(\"%+v\\n\", profile)\n}\n```\n\n### Search profiles by query\n\n```golang\npackage main\n\nimport (\n    \"context\"\n    \"fmt\"\n    twitterscraper \"github.com/n0madic/twitter-scraper\"\n)\n\nfunc main() {\n    scraper := twitterscraper.New().SetSearchMode(twitterscraper.SearchUsers)\n    err := scraper.Login(username, password)\n    if err != nil {\n        panic(err)\n    }\n    for profile := range scraper.SearchProfiles(context.Background(), \"Twitter\", 50) {\n        if profile.Error != nil {\n            panic(profile.Error)\n        }\n        fmt.Println(profile.Name)\n    }\n}\n```\n\n### Get trends\n\n```golang\npackage main\n\nimport (\n    \"fmt\"\n    twitterscraper \"github.com/n0madic/twitter-scraper\"\n)\n\nfunc main() {\n    scraper := twitterscraper.New()\n    trends, err := scraper.GetTrends()\n    if err != nil {\n        panic(err)\n    }\n    fmt.Println(trends)\n}\n```\n\n### Use Proxy\n\nSupport HTTP(s) and SOCKS5 proxy\n\n#### with HTTP\n\n```golang\nerr := scraper.SetProxy(\"http://localhost:3128\")\nif err != nil {\n    panic(err)\n}\n```\n\n#### with SOCKS5\n\n```golang\nerr := scraper.SetProxy(\"socks5://localhost:1080\")\nif err != nil {\n    panic(err)\n}\n```\n\n### Delay requests\n\nAdd delay between API requests (in seconds)\n\n```golang\nscraper.WithDelay(5)\n```\n\n### Load timeline with tweet replies\n\n```golang\nscraper.WithReplies(true)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fn0madic%2Ftwitter-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fn0madic%2Ftwitter-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fn0madic%2Ftwitter-scraper/lists"}