Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/n0madic/twitter-scraper
Scrape the Twitter frontend API without authentication with Golang.
https://github.com/n0madic/twitter-scraper
Last synced: 3 months ago
JSON representation
Scrape the Twitter frontend API without authentication with Golang.
- Host: GitHub
- URL: https://github.com/n0madic/twitter-scraper
- Owner: n0madic
- License: mit
- Archived: true
- Created: 2018-11-29T15:31:50.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2023-11-04T22:39:41.000Z (about 1 year ago)
- Last Synced: 2024-07-31T20:53:21.910Z (5 months ago)
- Language: Go
- Homepage:
- Size: 155 KB
- Stars: 878
- Watchers: 15
- Forks: 176
- Open Issues: 27
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-go - twitter-scraper - Scrape the Twitter Frontend API without authentication and limits. (Third-party APIs / Utility/Miscellaneous)
- my-awesome - n0madic/twitter-scraper - 11 star:0.9k fork:0.2k Scrape the Twitter frontend API without authentication with Golang. (Go)
- awesome-go-extra - twitter-scraper - 11-29T15:31:50Z|2022-06-16T08:22:32Z| (Third-party APIs / Fail injection)
README
# Twitter Scraper
[](https://pkg.go.dev/github.com/n0madic/twitter-scraper)
Twitter's API is annoying to work with, and has lots of limitations —
luckily their frontend (JavaScript) has it's own API, which I reverse-engineered.
No API rate limits. No tokens needed. No restrictions. Extremely fast.
You can use this library to get the text of any user's Tweets trivially.
## Installation
```shell
go get -u github.com/n0madic/twitter-scraper
```
## Usage
### Authentication
Now all methods require authentication!
#### Login
```golang
err := scraper.Login("username", "password")
```
Use username to login, not email!
But if you have email confirmation, use email address in addition:
```golang
err := scraper.Login("username", "password", "email")
```
If you have two-factor authentication, use code:
```golang
err := scraper.Login("username", "password", "code")
```
Status of login can be checked with:
```golang
scraper.IsLoggedIn()
```
Logout (clear session):
```golang
scraper.Logout()
```
If you want save session between restarts, you can save cookies with `scraper.GetCookies()` and restore with `scraper.SetCookies()`.
For example, save cookies:
```golang
cookies := scraper.GetCookies()
// serialize to JSON
js, _ := json.Marshal(cookies)
// save to file
f, _ = os.Create("cookies.json")
f.Write(js)
```
and load cookies:
```golang
f, _ := os.Open("cookies.json")
// deserialize from JSON
var cookies []*http.Cookie
json.NewDecoder(f).Decode(&cookies)
// load cookies
scraper.SetCookies(cookies)
// check login status
scraper.IsLoggedIn()
```
#### Open account
If you don't want to use your account, you can try login as a Twitter app:
```golang
err := scraper.LoginOpenAccount()
```
### Get user tweets
```golang
package main
import (
"context"
"fmt"
twitterscraper "github.com/n0madic/twitter-scraper"
)
func main() {
scraper := twitterscraper.New()
err := scraper.LoginOpenAccount()
if err != nil {
panic(err)
}
for tweet := range scraper.GetTweets(context.Background(), "Twitter", 50) {
if tweet.Error != nil {
panic(tweet.Error)
}
fmt.Println(tweet.Text)
}
}
```
It appears you can ask for up to 50 tweets.
### Get single tweet
```golang
package main
import (
"fmt"
twitterscraper "github.com/n0madic/twitter-scraper"
)
func main() {
scraper := twitterscraper.New()
err := scraper.Login(username, password)
if err != nil {
panic(err)
}
tweet, err := scraper.GetTweet("1328684389388185600")
if err != nil {
panic(err)
}
fmt.Println(tweet.Text)
}
```
### Search tweets by query standard operators
Now the search only works for authenticated users!
Tweets containing “twitter” and “scraper” and “data“, filtering out retweets:
```golang
package main
import (
"context"
"fmt"
twitterscraper "github.com/n0madic/twitter-scraper"
)
func main() {
scraper := twitterscraper.New()
err := scraper.Login(username, password)
if err != nil {
panic(err)
}
for tweet := range scraper.SearchTweets(context.Background(),
"twitter scraper data -filter:retweets", 50) {
if tweet.Error != nil {
panic(tweet.Error)
}
fmt.Println(tweet.Text)
}
}
```
The search ends if we have 50 tweets.
See [Rules and filtering](https://developer.twitter.com/en/docs/tweets/rules-and-filtering/overview/standard-operators) for build standard queries.
#### Set search mode
```golang
scraper.SetSearchMode(twitterscraper.SearchLatest)
```
Options:
* `twitterscraper.SearchTop` - default mode
* `twitterscraper.SearchLatest` - live mode
* `twitterscraper.SearchPhotos` - image mode
* `twitterscraper.SearchVideos` - video mode
* `twitterscraper.SearchUsers` - user mode
### Get profile
```golang
package main
import (
"fmt"
twitterscraper "github.com/n0madic/twitter-scraper"
)
func main() {
scraper := twitterscraper.New()
scraper.LoginOpenAccount()
profile, err := scraper.GetProfile("Twitter")
if err != nil {
panic(err)
}
fmt.Printf("%+v\n", profile)
}
```
### Search profiles by query
```golang
package main
import (
"context"
"fmt"
twitterscraper "github.com/n0madic/twitter-scraper"
)
func main() {
scraper := twitterscraper.New().SetSearchMode(twitterscraper.SearchUsers)
err := scraper.Login(username, password)
if err != nil {
panic(err)
}
for profile := range scraper.SearchProfiles(context.Background(), "Twitter", 50) {
if profile.Error != nil {
panic(profile.Error)
}
fmt.Println(profile.Name)
}
}
```
### Get trends
```golang
package main
import (
"fmt"
twitterscraper "github.com/n0madic/twitter-scraper"
)
func main() {
scraper := twitterscraper.New()
trends, err := scraper.GetTrends()
if err != nil {
panic(err)
}
fmt.Println(trends)
}
```
### Use Proxy
Support HTTP(s) and SOCKS5 proxy
#### with HTTP
```golang
err := scraper.SetProxy("http://localhost:3128")
if err != nil {
panic(err)
}
```
#### with SOCKS5
```golang
err := scraper.SetProxy("socks5://localhost:1080")
if err != nil {
panic(err)
}
```
### Delay requests
Add delay between API requests (in seconds)
```golang
scraper.WithDelay(5)
```
### Load timeline with tweet replies
```golang
scraper.WithReplies(true)
```