https://github.com/krallistic/druid-kubernetes

https://github.com/krallistic/druid-kubernetes

Last synced: 2 months ago
JSON representation

• Host: GitHub
• URL: https://github.com/krallistic/druid-kubernetes
• Owner: krallistic
• Created: 2016-10-31T20:21:57.000Z (about 8 years ago)
• Default Branch: master
• Last Pushed: 2016-12-02T09:08:36.000Z (about 8 years ago)
• Last Synced: 2024-03-17T16:20:35.190Z (10 months ago)
• Language: Smarty
• Size: 44.9 KB
• Stars: 35
• Watchers: 7
• Forks: 17
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
Work in Progress. 

# druid-kubernetes

Setup Druid on a Kubernetes Cluster with Helm. 

The Setup of zookeeper and postgres (as metadata storage is handled by subcharts), and therefore zK run with a PetSet in a 3 Node HA.

All Druid Nodes run in their own pods, which are created by deployments. Should be somewhat scaleable.

SegmentCache is persistentVolume with `HostPath` so historicals on the same node could be using the same segment cache. SegmentStorage is currently set to NFS.

Emmit metrics point to a tranquility node which indexes it into the cluster back again.

#Run it (currently only tested on minikube)

- checkout repo

- build docker images, see `build_images.sh`

- `helm init` (assumes working kubectl, for example with `minikube start`)

- `helm dep update` Pull in deps

- `helm upgrade --debug development druid/ --install` Start Cluster, takes a while till zookeeper is fully functional (~5min)

- Access the pivot UI `minikube service druid-pivot-service` and you should see the Metric Datasource which are druids internal metrics.

- Delete the Cluster with `helm delete development --purge ` PersistentStorage from the zK must be deleted manually.

#Assorted Notes: 

#Install Mini cube

curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.12.2/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

#Install kubectl

curl -Lo kubectl http://storage.googleapis.com/kubernetes-release/release/v1.3.0/bin/linux/amd64/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin/

#Completion

brew install bash-completion

#Install Helm-client

brew cask install helm

minikube start

eval $(minikube docker-env)

#Deployments, not used anymore

kubectl create -f local-zk/Deployment.yml

kubectl create -f metadata-mysql/Deployment.yml

kubectl create -f coordinator/Deployment.yml

kubectl create -f overlord/Deployment.yml

kubectl create -f middleManager/Deployment.yml

kubectl create -f historical/Deployment.yml

kubectl create -f broker/Deployment.yml

kubectl create -f pivot/Deployment.yml

minikube service druid-quickstart-service

#Jenkins

helm install stable/jenkins

minikube service hoping-kitten-jenkins --url

printf $(printf '\%o' `kubectl get secret --namespace default hoping-kitten-jenkins -o jsonpath="{.data.jenkins-admin-password[*]}"`);echo