{"id":19818826,"url":"https://github.com/sloppycoder/awslab","last_synced_at":"2026-05-09T23:18:42.775Z","repository":{"id":90083370,"uuid":"161845025","full_name":"sloppycoder/awslab","owner":"sloppycoder","description":"scripts and playbook for setting up Kafka on AWS for testing","archived":false,"fork":false,"pushed_at":"2019-03-31T09:27:38.000Z","size":2491,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-11T08:14:08.687Z","etag":null,"topics":["aws","aws-sdk","cloud","kafka","ruby","streaming"],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sloppycoder.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-12-14T22:01:26.000Z","updated_at":"2019-03-31T09:27:39.000Z","dependencies_parsed_at":"2023-03-13T18:00:01.047Z","dependency_job_id":null,"html_url":"https://github.com/sloppycoder/awslab","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sloppycoder%2Fawslab","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sloppycoder%2Fawslab/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sloppycoder%2Fawslab/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sloppycoder%2Fawslab/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sloppycoder","download_url":"https://codeload.github.com/sloppycoder/awslab/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241174956,"owners_count":19922417,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-sdk","cloud","kafka","ruby","streaming"],"created_at":"2024-11-12T10:17:00.553Z","updated_at":"2026-05-09T23:18:37.727Z","avatar_url":"https://github.com/sloppycoder.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Setup an AWS environment to test Apache Kafka\n\nThis repo contains instructions and scripts to setup a Kafka installation on AWS for development and testing.\n\n\n| folder | content |\n|----| --- |\n| infra | Instance runs InfluxDB, Grafana, used as monitoring hosts for the Kafka cluster. |\n| kafka | setup a Kafka cluster for testing. The instances are launched into a private VPC that does not have public accessible IP addresses. |\n\n\n## Prerequisites\n* Ruby - 2.2 or higher will work. I use 2.5\n* Ansible 2.7\n\n## Usage\n```shell\n\n# clone this repo\ngit submodule update –-init\n\n# get GEMS used in the code\nbundle\n\n# setup influxd db monitoring host\ncd infra\nruby infra.rb\nansible-playbook -i \"m.t.co,\" influxdb.yml --extra-vars \"fqdn=b.vino9.net\"\n# manually create new influx database called kafka\n# install grafana plugin\n# grafana-cli plugins install grafana-piechart-panel\n\n# create instances for kafka cluster\ncd ../kafka\nruby kafka_cluster.rb\nansible -i hosts.yml all -m ping\n\n# install confluent platform\ncd cp-ansible\nansible-playbook -i ../hosts.yml all.yml\n\n# configure kafka storage and monitoring\ncd ..\nansible-playbook -i hosts.yml setup_kafka.yml\n\n# check status\nansible -i hosts.yml broker -a \"sudo systemctl status confluent-kafka \"\nansible -i hosts.yml broker -a \"df  \"\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsloppycoder%2Fawslab","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsloppycoder%2Fawslab","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsloppycoder%2Fawslab/lists"}