https://github.com/sloppycoder/awslab
scripts and playbook for setting up Kafka on AWS for testing
https://github.com/sloppycoder/awslab
aws aws-sdk cloud kafka ruby streaming
Last synced: 23 days ago
JSON representation
scripts and playbook for setting up Kafka on AWS for testing
- Host: GitHub
- URL: https://github.com/sloppycoder/awslab
- Owner: sloppycoder
- Created: 2018-12-14T22:01:26.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2019-03-31T09:27:38.000Z (about 7 years ago)
- Last Synced: 2025-01-11T08:14:08.687Z (over 1 year ago)
- Topics: aws, aws-sdk, cloud, kafka, ruby, streaming
- Language: Ruby
- Size: 2.38 MB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Setup an AWS environment to test Apache Kafka
This repo contains instructions and scripts to setup a Kafka installation on AWS for development and testing.
| folder | content |
|----| --- |
| infra | Instance runs InfluxDB, Grafana, used as monitoring hosts for the Kafka cluster. |
| kafka | setup a Kafka cluster for testing. The instances are launched into a private VPC that does not have public accessible IP addresses. |
## Prerequisites
* Ruby - 2.2 or higher will work. I use 2.5
* Ansible 2.7
## Usage
```shell
# clone this repo
git submodule update –-init
# get GEMS used in the code
bundle
# setup influxd db monitoring host
cd infra
ruby infra.rb
ansible-playbook -i "m.t.co," influxdb.yml --extra-vars "fqdn=b.vino9.net"
# manually create new influx database called kafka
# install grafana plugin
# grafana-cli plugins install grafana-piechart-panel
# create instances for kafka cluster
cd ../kafka
ruby kafka_cluster.rb
ansible -i hosts.yml all -m ping
# install confluent platform
cd cp-ansible
ansible-playbook -i ../hosts.yml all.yml
# configure kafka storage and monitoring
cd ..
ansible-playbook -i hosts.yml setup_kafka.yml
# check status
ansible -i hosts.yml broker -a "sudo systemctl status confluent-kafka "
ansible -i hosts.yml broker -a "df "
```