{"id":15677115,"url":"https://github.com/graykode/kubernetes-glusterfs-aws","last_synced_at":"2025-05-07T00:44:37.527Z","repository":{"id":110173071,"uuid":"179011382","full_name":"graykode/kubernetes-glusterfs-aws","owner":"graykode","description":"file system clustering as glusterfs in kubernetes environment on aws platform","archived":false,"fork":false,"pushed_at":"2019-04-12T16:07:19.000Z","size":153,"stargazers_count":13,"open_issues_count":0,"forks_count":4,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-31T04:41:10.329Z","etag":null,"topics":["ansible","aws","gluster-kubernetes","glusterfs","glusterfs-cluster","heketi","kubernetes","terraform"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/graykode.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-04-02T06:16:43.000Z","updated_at":"2022-11-16T18:44:00.000Z","dependencies_parsed_at":"2023-04-13T21:32:21.214Z","dependency_job_id":null,"html_url":"https://github.com/graykode/kubernetes-glusterfs-aws","commit_stats":{"total_commits":4,"total_committers":1,"mean_commits":4.0,"dds":0.0,"last_synced_commit":"b377f731c7e732a1d0c4d64fb067b6ed602a6761"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/graykode%2Fkubernetes-glusterfs-aws","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/graykode%2Fkubernetes-glusterfs-aws/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/graykode%2Fkubernetes-glusterfs-aws/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/graykode%2Fkubernetes-glusterfs-aws/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/graykode","download_url":"https://codeload.github.com/graykode/kubernetes-glusterfs-aws/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252793562,"owners_count":21805053,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansible","aws","gluster-kubernetes","glusterfs","glusterfs-cluster","heketi","kubernetes","terraform"],"created_at":"2024-10-03T16:08:34.319Z","updated_at":"2025-05-07T00:44:37.517Z","avatar_url":"https://github.com/graykode.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Kubernetes-glusterfs-aws\n\nThis is simple tutorial about file system clustering using `glusterfs` with `heketi` in `kubernetes` environment on `aws` platform.\n\n\u003cp align=\"center\"\u003e\u003cimg width=\"100\" src=\"https://upload.wikimedia.org/wikipedia/commons/9/93/Amazon_Web_Services_Logo.svg\" /\u003e\n\u003cimg width=\"100\" src=\"https://upload.wikimedia.org/wikipedia/en/0/00/Kubernetes_%28container_engine%29.png\" /\u003e\n\u003cimg width=\"100\" src=\"http://www.hostdime.in/blog/wp-content/uploads/2017/06/glusterfs-square-logo.png\" /\u003e\n\u003cimg width=\"100\" src=\"https://avatars1.githubusercontent.com/u/12890374?s=200\u0026v=4\" /\u003e\n\u003c/p\u003e\n\n\n\n## 0. How to Configure Kubernetes in aws platform?\n\nI already configured `kubernetes` on `aws` platform using [alicek106/aws-terraform-kubernetes](https://github.com/alicek106/aws-terraform-kubernetes).\n\nYou can more easily Set up combination with `Terraform`, `ansible`, `kubespray`!. They just do it all!.\n\n\u003cimg src=\"https://github.com/alicek106/aws-terraform-kubernetes/blob/master/pictures/kube.png?raw=true\"\u003e\n\n#### Note : when Set Variables in variables.tf, I recommend you like this.\n\n```\nnumber_of_controller = 0\n\nnumber_of_etcd = 0\n\nnumber_of_controller_etcd = 1\n```\n\nThen, You can see finished aws setting.\n\n![](aws.png)\n\n\n\n## 1. Overall Network System\n\n\u003cp align=\"center\"\u003e\u003cimg width=\"600\" src=\"overall.jpg\" /\u003e\u003c/p\u003e\n\nI referenced code from [psyhomb/heketi](https://github.com/psyhomb/heketi).\n\ncopy your `.ppk` key `docker` to `master node(10.43.0.40)`\n\n```shell\nroot@terraform-aws-kube:/aws-terraform-kubernetes/ansible$scp -i ../keys/tf-kube ../keys/tf-kube ubuntu@\u003cyour master node's public address\u003e:/home/ubuntu\n```\n\nplease connect `docker` to `master node(10.43.0.40)`\n\n```shell\nroot@terraform-aws-kube:/aws-terraform-kubernetes/ansible$ssh -i ../keys/tf-kube ubuntu@\u003cyour master node's public address\u003e\n```\n\n#### Tips.\n\nIn `ec2`, you can access the **root** directly through `ssh` or `scp` by setting as shown below.\n\n```shell\nroot@ip-10-43-0-40:/home/ubuntu$ vim /root/.ssh/authorized_keys\n```\n\nThen, remove this line\n\n```\nno-port-forwarding,no-agent-forwarding,no-X11-forwarding,command=\"echo 'Please login as the user \\\"ubuntu\\\" rather than the user \\\"root\\\".';echo;sleep 10\"\n```\n\n\n\n#### 1.1 Install and setting gluster server in all Node 0,1,2.(Attention!except Master node!)\n\nPlease install gluster server in all your Woker Node. \n\n##### Note  Part of (1.1 Install and Setting) apply all node in the same(ip-10-43-0-30, ip-10-43-0-31, ip-10-43-0-32)\n\nI suffered Trouble shooting when i had used old version gluster(3.1.0) 😥😥, but after fixed 4.1 version, It is worked!!\n\n```shell\nroot@ip-10-43-0-30:~$ apt install software-properties-common\nroot@ip-10-43-0-30:~$ add-apt-repository ppa:gluster/glusterfs-4.1\nroot@ip-10-43-0-30:~$ apt-get update \u0026\u0026 apt-get upgrade\nroot@ip-10-43-0-30:~$ apt install thin-provisioning-tools\nroot@ip-10-43-0-30:~$ apt install glusterfs-server glusterfs-client\n```\n\n\n\ncheck `glusterfs-server` status\n\n```shell\nroot@ip-10-43-0-30:~$ service glusterd start\nroot@ip-10-43-0-30:~$ service glusterd status\n```\n\n**Do this(1.1 chapter) apply to other nodes in the same**\n\n\n\n#### 1.2 Configure file\n\nplease return to `ip-10-43-0-40(kubernetes master)`\n\nInstall `kubectl` in [Install kubectl binary using curl](https://kubernetes.io/docs/tasks/tools/install-kubectl/).\n\n```shell\nroot@ip-10-43-0-40:/home/ubuntu$ curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl\nroot@ip-10-43-0-40:/home/ubuntu$ chmod +x ./kubectl\nroot@ip-10-43-0-40:/home/ubuntu$ mv ./kubectl /usr/local/bin/kubectl\n```\n\n\n\nCheck your node using `kubectl get nodes`.\n\n```shell\nroot@ip-10-43-0-40:/home/ubuntu$ kubectl get nodes\nNAME                                            STATUS   ROLES    AGE   VERSION\nip-10-43-0-30.ap-northeast-2.compute.internal   Ready    node     61m   v1.12.3\nip-10-43-0-31.ap-northeast-2.compute.internal   Ready    node     61m   v1.12.3\nip-10-43-0-32.ap-northeast-2.compute.internal   Ready    node     61m   v1.12.3\nip-10-43-0-40.ap-northeast-2.compute.internal   Ready    master   63m   v1.12.3\n```\n\n\n\nSet Linux Shortcut Settings\n\n```shell\nroot@ip-10-43-0-40:/home/ubuntu$ node1='ip-10-43-0-30.ap-northeast-2.compute.internal'\nroot@ip-10-43-0-40:/home/ubuntu$ node2='ip-10-43-0-31.ap-northeast-2.compute.internal'\nroot@ip-10-43-0-40:/home/ubuntu$ node3='ip-10-43-0-32.ap-northeast-2.compute.internal'\n```\n\n\n\nCreate required directories.\n\n```shell\nroot@ip-10-43-0-40:~$ mkdir -p /data/heketi/{db,.ssh} \u0026\u0026 chmod 700 /data/heketi/.ssh\n```\n\n\n\nCreate required directories to all nodes\n\n```shell\nroot@ip-10-43-0-40:~$ for NODE in $node1 $node2 $node3; do ssh -i tf-kube root@${NODE} 'mkdir -p /data/heketi/{db,.ssh} \u0026\u0026 chmod 700 /data/heketi/.ssh'; done\n```\n\n\n\nGenerate heketi ssh keys that will be used by heketi api for password-less login to glusterfs servers\n\n```shell\nroot@ip-10-43-0-40:/home/ubuntu$ ssh-keygen -t rsa -b 2048 -f /data/heketi/.ssh/id_rsa\n```\n\n##### Note When blank space about passpharse, just press Enter as skipping\n\n\n\nCopy `.ssh` dir to all glusterfs servers using `scp`.\n\n```shell\nroot@ip-10-43-0-40:/home/ubuntu$ for NODE in $node1 $node2 $node3; do scp -i tf-kube -r /data/heketi/.ssh root@${NODE}:/data/heketi; done\n```\n\n\n\nImport `ssh` public key to all glusterfs servers (node1, node2 and node3 in our example)\n\n```shell\nroot@ip-10-43-0-40:/home/ubuntu$ for NODE in $node1 $node2 $node3; do cat /data/heketi/.ssh/id_rsa.pub | ssh -i tf-kube root@${NODE} \"cat \u003e\u003e /root/.ssh/authorized_keys\"; done\n```\n\n\n\n#### 1.3 Configure Heketi Cli and kubernetes\n\n```shell\nroot@ip-10-43-0-40:/home/ubuntu$ git clone https://github.com/graykode/kubernetes-glusterfs-aws \u0026\u0026 cd kubernetes-glusterfs-aws\n```\n\n\n\n##### In now, We assume that we are at /home/ubuntu/kubernets-glusterfs-aws and use the ip-10-43-0-40 node.\n\nYou have to use `HEKETI_VERSION` `8.0.0`\n\n```shell\n$ chmod +x heketi.sh\n$ ./heketi.sh\n$ heketi-cli -v\nheketi-cli v8.0.0\n```\n\n\n\nAdd label on all Kubernetes nodes with GlusterFS\n\n```shell\n$ kubectl label node $node1 storagenode=glusterfs\n$ kubectl label node $node2 storagenode=glusterfs\n$ kubectl label node $node3 storagenode=glusterfs\n```\n\n\n\nIn `kubernetes/heketi-secret.yaml`, key do role about password. You can change key or not. this is optional point. If you want to set password(key) as `12345678`, `echo -n '12345678' | base64` then write `MTIzNDU2Nzg=` in `kubernetes/heketi-secret.yaml` key\n\n\n\nCreate Secret Object in kubernetes. \n\n```shell\n$ kubectl apply -f kubernetes/heketi-secret.yaml\n```\n\n\n\nCreate heketi api deployment and service objects\n\n```shell\n$ kubectl apply -f kubernetes/heketi-deployment.json\n```\n\n\n\n##### You can get a number of api server port.\n\nIn below this, `30257` is a number of your api server port.\n\n```shell\n$ kubectl get svc -l glusterfs=heketi-service\nNAME     TYPE       CLUSTER-IP      EXTERNAL-IP       PORT(S)          AGE\nheketi   NodePort   X.X.X.X          \u003cnone\u003e        8080:30257/TCP      16s\n```\n\n\n\nCheck your api server is fine.\n\n```shell\n$ curl -s $node1:30257/hello\n```\n\n\n\n#### 1.4 Attach EBS in your each EC2\n\nIt is recommended that you create an EBS volume of 10GB or more. After Creating, Attach to your each ec2 node(**just only woker node 1,2,3, not master**)\n\nThen, run this to make topology network clustering. (Assume that the password is `password`)\n\n```shell\n$ heketi-cli --user admin --secret password --server http://$node1:30257 topology load --json heketi-topology.json\n```\n\n\n\nThen, run this to see topology information.\n\n```shell\n$ heketi-cli --user admin --secret password --server http://$node1:30257 topology info\n```\n\n\n\n##### Tips.\n\nIf you took some mistake when making topology, you can delete object(topology, node, volume or device) like this.\n\n```shell\n$ heketi-cli --user admin --secret password --server http://$node1:30257 topology node delete \u003cnodeid\u003e # delete node which has nodeid\n$ heketi-cli --user admin --secret password --server http://$node1:30257 topology device delete \u003cdeviceid\u003e # delete node which has deviceid\n$ heketi-cli --user admin --secret password --server http://$node1:30257 topology cluster delete \u003cclusterid\u003e # delete node which has clusterid\n```\n\n\n\n#### 1.5 Heketi DB\n\nCreate glusterfs shared volume\n\n```shell\n$ heketi-cli --user admin --secret password --server http://$node1:30257 setup-openshift-heketi-storage --listfile heketi-storage.json\n```\n\n\n\nMigrate Heketi database to previously created glusterfs volume\\\n\n```shell\n$ kubectl apply -f heketi-storage.json\n```\n\n\n\nCheck status\n\n```shell\n$ kubectl get job -o wide\n```\n\n\n\nThen, heketi database migrated to `heketidbstorage` glusterfs replicated volume and remove local.\n\n```shell\n$ for NODE in $node1 $node2 $node3; do ssh -i ../tf-kube root@${NODE} \"rm -f /data/heketi/db/heketi.db \u0026\u0026 mount.gluster\nfs ${NODE}:/heketidbstorage /data/heketi/db\"; done\n```\n\n\n\n##### Tips.\n\nIf you want to make mount persistent, Please refer to [here](https://github.com/psyhomb/heketi#heketi-db).\n\n\n\n#### 1.6 Kubernetes StorageClass object\n\nSet your cluster id in `heketi-storageclass.yaml` by `$ heketi-cli --user admin --secret password --server http://$node1:30257 topology info`. After it, Create storageclass Object in Kubernetes.\n\n```shell\n$ kubectl apply -f kubernetes/heketi-storageclass.yaml\n$ kubectl get storageclass\n```\n\n\n\n#### 1.7 Kubernetes PersistentVolumeClaim\n\n```shell\n$ kubectl apply -f kubernetes/heketi-pvc.yaml\n$ kubectl get pvc test-claim\n$ kubectl get pv\n```\n\n\n\nCheck your pvc obejct fine\n\n```shell\n$ kubectl describe pvc test-claim\n# Successfully provisioned volume pvc\n```\n\n\n\n## To Next Job\n\nbinding pvc with flask docker deployment.\n\n\n\n## License\n\nThis Project license from TaeHwan Jung(Who modify more easily) and Miloš Bunčić(Orignal repository).\n\n\n\n## Author\n\n- Tae Hwan Jung(@graykode)\n- My Email : nlkey2022@gmail.com","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgraykode%2Fkubernetes-glusterfs-aws","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgraykode%2Fkubernetes-glusterfs-aws","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgraykode%2Fkubernetes-glusterfs-aws/lists"}