Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/Fallenstedt/twitter-stream
A Go wrapper for Twitter's V2 Filtered Stream API
https://github.com/Fallenstedt/twitter-stream
golang streaming-tweets twitter twitter-api twitter-streaming-api
Last synced: about 2 months ago
JSON representation
A Go wrapper for Twitter's V2 Filtered Stream API
- Host: GitHub
- URL: https://github.com/Fallenstedt/twitter-stream
- Owner: Fallenstedt
- License: mit
- Archived: true
- Created: 2020-12-21T22:21:12.000Z (over 3 years ago)
- Default Branch: master
- Last Pushed: 2022-11-05T03:09:43.000Z (over 1 year ago)
- Last Synced: 2024-04-25T00:42:02.367Z (2 months ago)
- Topics: golang, streaming-tweets, twitter, twitter-api, twitter-streaming-api
- Language: Go
- Homepage: https://pkg.go.dev/github.com/fallenstedt/twitter-stream
- Size: 8.65 MB
- Stars: 29
- Watchers: 6
- Forks: 9
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Lists
- awesome-modern-twitter-api - twitter-stream - A Go wrapper for Twitter's V2 Filtered Stream API. (Libraries / Go)
README
# TwitterStream
[](https://developer.twitter.com/en/docs/twitter-api)
[](https://goreportcard.com/report/github.com/fallenstedt/twitter-stream)
[](https://pkg.go.dev/github.com/fallenstedt/twitter-stream)
> ⚠️ This repo is read-only and is no longer worked on. Twitter was a wonderful platform, built by a wonderful team. With its recent changes, its time to re-evaluate what we invest in and the world we want to build. To the projects using this library, I'll leave the repo read-only so you can migrate. on your own time
TwitterStream is a Go library for creating streaming rules and streaming tweets with [Twitter's v2 Filtered Streaming API](https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/introduction).
- See [my blog post](https://www.fallenstedt.com/blog/twitter-stream/) for a tutorial on Twitter's Filtered Stream endpoint.
- See [examples](https://github.com/fallenstedt/twitter-stream/tree/master/example) to start adding your own rules and start streaming.
## Installation
`go get github.com/fallenstedt/twitter-stream`
## Examples
See [examples](https://github.com/fallenstedt/twitter-stream/tree/master/example), or follow the guide below.
#### Starting a stream
##### Obtain an Access Token using your Twitter Access Key and Secret.
You need an access token to do any streaming. `twitterstream` provides an easy way to fetch an access token. Use your
API key and secret API key from twitter to request an access token.
```go
tok, err := twitterstream.NewTokenGenerator().SetApiKeyAndSecret("key", "secret").RequestBearerToken()
```
##### Create a streaming api
Create a twitterstream instance with your access token from above.
```go
api := twitterstream.NewTwitterStream(tok.AccessToken)
```
##### Create rules
We need to create [twitter streaming rules](https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/build-a-rule) so we can get tweets that we want.
The filtered stream endpoints deliver filtered Tweets to you in real-time that match on a set of rules that are applied to the stream. Rules are made up of operators that are used to match on a variety of Tweet attributes.
Below we create three rules. One for puppy tweets with images, another for cat tweets with images, and the other of unique English golang job postings. Each rule is
associated with their own tag.
```go
rules := twitterstream.NewRuleBuilder().
AddRule("cat has:images", "cat tweets with images").
AddRule("puppy has:images", "puppy tweets with images").
AddRule("lang:en -is:retweet -is:quote (#golangjobs OR #gojobs)", "golang jobs").
Build()
// Create will create twitter rules
// dryRun is set to false. Set to true to test out your request
res, err := api.Rules.Create(rules, false)
// Get will get your current rules
res, err := api.Rules.Get()
// Delete will delete your rules by their id
// dryRun is set to false. Set to true to test out your request
res, err := api.Rules.Delete(rules.NewDeleteRulesRequest(1468427075727945728, 1468427075727945729), false)
```
##### Set your unmarshal hook
It is encouraged you set an unmarshal hook for thread-safety. Go's `bytes.Buffer` is not thread safe. Sharing a `bytes.Buffer`
across multiple goroutines introduces risk of panics when decoding json.
To avoid panics, it's encouraged to unmarshal json in the same goroutine where the `bytes.Buffer` exists. Use `SetUnmarshalHook` to set a function that unmarshals json.
By default, twitterstream's unmarshal hook will return `[]byte` if you want to live dangerously.
```go
type StreamDataExample struct {
Data struct {
Text string `json:"text"`
ID string `json:"id"`
CreatedAt time.Time `json:"created_at"`
AuthorID string `json:"author_id"`
} `json:"data"`
Includes struct {
Users []struct {
ID string `json:"id"`
Name string `json:"name"`
Username string `json:"username"`
} `json:"users"`
} `json:"includes"`
MatchingRules []struct {
ID string `json:"id"`
Tag string `json:"tag"`
} `json:"matching_rules"`
}
api.SetUnmarshalHook(func(bytes []byte) (interface{}, error) {
data := StreamDataExample{}
if err := json.Unmarshal(bytes, &data); err != nil {
fmt.Printf("failed to unmarshal bytes: %v", err)
}
return data, err
})
```
##### Start Stream
Start your stream. This is a long-running HTTP GET request.
You can request additional tweet data by adding [query params](https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream).
Use the `twitterstream.NewStreamQueryParamsBuilder()` to start a stream with the data you want.
```go
// Steps from above, Placed into a single function
// This assumes you have at least one streaming rule configured.
// returns a configured instance of twitterstream
func fetchTweets() stream.IStream {
tok, err := twitterstream.NewTokenGenerator().SetApiKeyAndSecret(KEY, SECRET).RequestBearerToken()
if err != nil {
panic(err)
}
api := twitterstream.NewTwitterStream(tok).Stream
api.SetUnmarshalHook(func(bytes []byte) (interface{}, error) {
data := StreamDataExample{}
if err := json.Unmarshal(bytes, &data); err != nil {
fmt.Printf("failed to unmarshal bytes: %v", err)
}
return data, err
})
// https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream
streamExpansions := twitterstream.NewStreamQueryParamsBuilder().
AddExpansion("author_id").
AddTweetField("created_at").
Build()
// StartStream will start the stream
err = api.StartStream(streamExpansions)
if err != nil {
panic(err)
}
return api
}
// This will run forever
func initiateStream() {
fmt.Println("Starting Stream")
// Start the stream
// And return the library's api
api := fetchTweets()
// When the loop below ends, restart the stream
defer initiateStream()
// Start processing data from twitter after starting the stream
for tweet := range api.GetMessages() {
// Handle disconnections from twitter
// https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/integrate/handling-disconnections
if tweet.Err != nil {
fmt.Printf("got error from twitter: %v", tweet.Err)
// Notice we "StopStream" and then "continue" the loop instead of breaking.
// StopStream will close the long running GET request to Twitter's v2 Streaming endpoint by
// closing the `GetMessages` channel. Once it's closed, it's safe to perform a new network request
// with `StartStream`
api.StopStream()
continue
}
result := tweet.Data.(StreamDataExample)
// Here I am printing out the text.
// You can send this off to a queue for processing.
// Or do your processing here in the loop
fmt.Println(result.Data.Text)
}
fmt.Println("Stopped Stream")
}
```
## Contributing
Pull requests and feature requests are always welcome.
Please accompany a pull request with tests.