https://github.com/iamtodor/kafka-twitter-project
https://github.com/iamtodor/kafka-twitter-project
Last synced: about 2 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/iamtodor/kafka-twitter-project
- Owner: iamtodor
- License: mit
- Created: 2020-08-25T09:37:10.000Z (over 4 years ago)
- Default Branch: master
- Last Pushed: 2021-03-21T14:46:53.000Z (about 4 years ago)
- Last Synced: 2025-01-27T23:47:16.336Z (4 months ago)
- Language: Java
- Size: 892 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
## This repo is a pet project.
### The purpose of the project is to prove Java, Kafka, Prometheus and Grafana knowledge.
### The main components are:
- Twitter
- Kafka
- ElasticSearch
- Prometheus
- Grafana
### Overview:
- For the twitter, I used [Hosebird Client](https://github.com/twitter/hbc)
- Kafka 2.5.0 is running locally
- For ElasticSearch I took PaaS https://bonsai.io/
- Prometheus 2.20.1 is running locally
- Grafana 7.1.5 is running locally
### It has the following modules:
- vanilla twitter producer
- kafka connect twitter
- vanilla elastic search consumer
- kafka connect sink elasticsearch
- kstreams:
- it filters data by user_followers
- it counts words in every tweet
- it counts total tweets by terms from a certain author
### Vanilla producer and consumer are tested.
### I left twitter API keys deliberately to easy clone and reproduce. To make it work follow HOWTO.md file
### Notes:
* to run es_consumer_connector be sure to add guava.jar to classpath
### TODO:
* add how to page