Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/sandjelkovic/shokker
Social media trend listener. Pipeline from Twitter via Kafka to Cassandra storage. The basis for data analysis.
https://github.com/sandjelkovic/shokker
cassandra kafka reactive reactor-kafka spring-boot spring-cloud-stream spring-kafka twitter twitter-producer twitter-streaming-api
Last synced: about 1 month ago
JSON representation
Social media trend listener. Pipeline from Twitter via Kafka to Cassandra storage. The basis for data analysis.
- Host: GitHub
- URL: https://github.com/sandjelkovic/shokker
- Owner: sandjelkovic
- Created: 2020-04-18T11:45:58.000Z (almost 5 years ago)
- Default Branch: master
- Last Pushed: 2021-11-05T19:45:26.000Z (about 3 years ago)
- Last Synced: 2024-11-07T12:52:34.013Z (3 months ago)
- Topics: cassandra, kafka, reactive, reactor-kafka, spring-boot, spring-cloud-stream, spring-kafka, twitter, twitter-producer, twitter-streaming-api
- Language: Kotlin
- Homepage:
- Size: 149 KB
- Stars: 1
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Shokker
## Services
- Twitter Producer
- Tweet Mapper
- Post Persister
- Backing services
- Kafka
- Zookeper
- Cassandra
### Twitter producer
Simple Kotlin application. Run with arguments.
Example config for IntelliJ's run configuration:
```
CONSUMER_KEY=TWITTER_CONSUMER_KEY CONSUMER_SECRET=TWITTER_CONSUMER_SECRET ACCESS_TOKEN=TWITTER_ACCESS_TOKEN TOKEN_SECRET=TWITTER_TOKEN_SECRET HASHTAGS=#bigdata,#cassandra,#kafka
```
Breakdown of required arguments:
Twitter API secrets:
```
CONSUMER_KEY
CONSUMER_SECRET
ACCESS_TOKEN
TOKEN_SECRET
```
List of hashtags to listen to. Comma-separated values, with `#` included
```
HASHTAGS
```
### Twitter Producer
Spring boot application. Can be run from IDE or terminal with `./gradlew bootRun`
### Post Persister
Spring boot application. Can be run from IDE or terminal with `./gradlew bootRun`
### Backing services
All of these services can be easily run via `docker-compose`.
Start with `docker-compose up -d` and stop with `docker-compose stop`