https://github.com/funkygao/gafka

A full ecosystem that is built around kafka powered by golang
https://github.com/funkygao/gafka

haproxy kafka kafka-rest-proxy messaging monitoring pubsub redis zookeeper

Last synced: about 1 month ago
JSON representation

A full ecosystem that is built around kafka powered by golang

• Host: GitHub
• URL: https://github.com/funkygao/gafka
• Owner: funkygao
• License: apache-2.0
• Created: 2015-11-09T14:08:26.000Z (over 10 years ago)
• Default Branch: master
• Last Pushed: 2018-09-19T08:59:56.000Z (over 7 years ago)
• Last Synced: 2024-11-22T03:06:49.666Z (over 1 year ago)
• Topics: haproxy, kafka, kafka-rest-proxy, messaging, monitoring, pubsub, redis, zookeeper
• Language: Go
• Homepage:
• Size: 79.1 MB
• Stars: 100
• Watchers: 15
• Forks: 21
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • Changelog: Changelog.md
  • License: LICENSE.md
  • Code of conduct: CODE_OF_CONDUCT.md

Awesome Lists containing this project

README

          
# gafka [![Build Status](https://travis-ci.org/funkygao/gafka.png?branch=dev)](https://travis-ci.org/funkygao/gafka)

    

                       __   _            

                      / _| | |           

       __ _    __ _  | |_  | | __   __ _ 

      / _` |  / _` | |  _| | |/ /  / _` |

     | (_| | | (_| | | |   |   <  | (_| |

      \__, |  \__,_| |_|   |_|\_\  \__,_|

       __/ |                             

      |___/                              

A full ecosystem for kafka/redis/PubSub/ElasticSearch/Zookeeper/haproxy.

Hope it can help you.

### Components

- [ehaproxy](https://github.com/funkygao/gafka/tree/master/cmd/ehaproxy)

  Elastic haproxy that sits in front of kateway.

- [kateway](https://github.com/funkygao/gafka/tree/master/cmd/kateway)

  A fully-managed real-time secure and reliable RESTful Cloud Pub/Sub streaming message/job service.

- [actord](https://github.com/funkygao/gafka/tree/master/cmd/actord)

  kateway job scheduler and webhook dispatcher.

- [gk](https://github.com/funkygao/gafka/tree/master/cmd/gk)

 

  Unified multi-datacenter multi-cluster kafka swiss-knife management console.

- [zk](https://github.com/funkygao/gafka/tree/master/cmd/zk)

  A handy zookeeper CLI that supports recursive operation without any dependency.

- [es](https://github.com/funkygao/gafka/tree/master/cmd/es)

  ElasticSearch console.

- [kguard](https://github.com/funkygao/gafka/tree/master/cmd/kguard)

  Kafka clusters body guard that emits health info to InfluxDB and exports key warnings to zabbix for alerting.

### Install

    export PATH=$PATH:$GOPATH/bin

    #========================================

    # install go-bindata and go annotations

    #========================================

    go install github.com/jteeuwen/go-bindata/go-bindata

    go install github.com/funkygao/goannotation

    #========================================

    # install gafka

    #========================================

    go get github.com/funkygao/gafka

    cd $GOPATH/src/github.com/funkygao/gafka

    ./build.sh -a # build all components

    #========================================

    # try the gafka command 'gk'

    #========================================

    gk -h

### Status

Currently gafka manages:

- 4 data centers 

- 50+ kafka clusters

- 100+ kafka brokers

- 500+ kafka topics

- 2000+ kafka partitions

- 10Billion messages per day

- peak load

  - 1Million message per second

  - 8TB transfered per hour

- 5TB redis