https://github.com/aparo/kafka-integration-patterns

Kafka integration patterns
https://github.com/aparo/kafka-integration-patterns

alpakka docker elasticsearch integration kafka scala

Last synced: 3 months ago
JSON representation

Kafka integration patterns

• Host: GitHub
• URL: https://github.com/aparo/kafka-integration-patterns
• Owner: aparo
• License: apache-2.0
• Created: 2018-09-29T16:35:54.000Z (almost 8 years ago)
• Default Branch: master
• Last Pushed: 2018-09-29T16:38:02.000Z (almost 8 years ago)
• Last Synced: 2025-10-07T14:01:10.280Z (10 months ago)
• Topics: alpakka, docker, elasticsearch, integration, kafka, scala
• Language: Scala
• Size: 138 KB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
This is a little demo project to show how a combination of a [akka-streams](http://www.lightbend.com/activator/template/akka-stream-scala),

 [Reactive Kafka](https://github.com/akka/reactive-kafka), [Kafka](https://kafka.apache.org/) and ElasticSearch can be used as the ingestion part

 of a fast data system.

#Setup

Docker configuration: add in share:

    /opt/data

    /opt/dockerdata

    

Download the geoname sample:

    ./fetch_geoname.sh

Setup Prometheus/Grafana and family:

    cd docker/dockprom

    docker-compose up -d

    

Setup ElasticSearch/Kibana/kafka:

    cd docker/kafka_elk/

    docker-compose up -d

Build the apps:

    sbt

    project csvToKafka

    docker:publishLocal

    project kafkaToES

    docker:publishLocal

    

Add the new deployed apps:

    docker-compose up -d

Enjoy:

- Prometheus: http://0.0.0.0:9090 (admin:admin)

- Grafana: http://0.0.0.0:3000 (admin:admin)

- Kafka-Manager: http://0.0.0.0:9000

- Kibana: http://0.0.0.0:5601 

- Cerebro: http://0.0.0.0:9001