{"id":13700616,"url":"https://github.com/deeppavlov/stand_kubernetes_cluster","last_synced_at":"2026-07-02T09:30:15.805Z","repository":{"id":83451122,"uuid":"123584961","full_name":"deeppavlov/stand_kubernetes_cluster","owner":"deeppavlov","description":null,"archived":false,"fork":false,"pushed_at":"2023-08-05T04:58:13.000Z","size":978,"stargazers_count":5,"open_issues_count":2,"forks_count":8,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-05T19:21:34.677Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Dockerfile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deeppavlov.png","metadata":{"files":{"readme":"README.MD","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-03-02T13:52:17.000Z","updated_at":"2023-08-25T02:20:07.000Z","dependencies_parsed_at":null,"dependency_job_id":"f2b2015b-ba68-49a4-9011-68f218e3ddf6","html_url":"https://github.com/deeppavlov/stand_kubernetes_cluster","commit_stats":null,"previous_names":["deepmipt/stand_kubernetes_cluster"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeppavlov%2Fstand_kubernetes_cluster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeppavlov%2Fstand_kubernetes_cluster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeppavlov%2Fstand_kubernetes_cluster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeppavlov%2Fstand_kubernetes_cluster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deeppavlov","download_url":"https://codeload.github.com/deeppavlov/stand_kubernetes_cluster/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240392542,"owners_count":19794112,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T20:00:59.450Z","updated_at":"2026-07-02T09:30:15.722Z","avatar_url":"https://github.com/deeppavlov.png","language":"Dockerfile","funding_links":[],"categories":["Dockerfile"],"sub_categories":[],"readme":"# Demo stand. Kubernetes cluster\n\n## Components deployment\n\nDemo stand Kubernetes cluster is based on following components:\n\n1. **Docker** with **NVIDIA Container Runtime for Docker**\n2. Kubernetes packages: **kubeadm**, **kubelet**, **kubectl**, **kubernetes-cni**\n3. **Docker registry** \n\nDocker and all Kubernetes packages should be deployed both on master and each worker node.\nNVIDIA Container Runtime for Docker should be deployed on each worker node with GPUs.\n\nThis docs are tested for following versions of demo stand components:\n* Docker `19.03.1`\n* Kubelet `v1.15.3`\n* Kubeadm: `v1.15.3`\n\n### Docker installation\n1. Install Docker:\n    ```\n    sudo apt-get update\n    sudo apt-get install docker-ce\n    ```\n\n2. Install NVIDIA Container Runtime for Docker:\n\n    1. Add package repositories:\n        ```\n        curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | \\\n        sudo apt-key add -\n        \n        distribution=$(. /etc/os-release;echo $ID$VERSION_ID)\n        \n        curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \\\n        sudo tee /etc/apt/sources.list.d/nvidia-docker.list\n        \n        sudo apt-get update\n        ```\n    2. Install nvidia-docker2 and reload the Docker daemon configuration:\n        ```\n        sudo apt-get install -y nvidia-docker2\n        sudo pkill -SIGHUP dockerd\n        ```\n\nReference resources:\n1. Docker installation: https://docs.docker.com/install/linux/docker-ce/ubuntu/#install-docker-ce-1\n2. NVIDIA Docker installation: https://github.com/NVIDIA/nvidia-docker\n    \n### Kubernetes installation\n1. Kubernetes can't operate with swap enabled on host machine. Disable swap on master and each slave node:\n    ```\n    sudo swapoff -a\n    ```\n2. Install Kubernetes:\n\n    1. Add package repositories:\n        ```\n        sudo apt-get update \u0026\u0026 sudo apt-get install -y apt-transport-https\n        curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -\n        echo \"deb https://apt.kubernetes.io/ kubernetes-xenial main\" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list\n        ```\n    2. Install Kubernetes packages:\n        ```\n        sudo apt-get update\n        sudo apt-get install -y kubelet kubeadm kubectl kubernetes-cni\n        ```\n\nReference resources:\n1. Kubernetes docs, kubeadm installation: https://kubernetes.io/docs/setup/independent/install-kubeadm/\n \n### Docker registry deployment\nFor now plain insecure HTTP registry is used. \n\n1. If required, install Docker on Docker registry host machine:\n    ```\n    sudo apt-get update\n    sudo apt-get install docker-ce\n    ```\n2. Deploy Docker registry:\n    ```\n    docker run -d -p \u003cregistry port, default=5000\u003e:5000 --restart=always --name registry registry:2\n    ```\n3. To enable insecure access to registry, follow these steps on master and each slave node (as well as each host that wants to access the registry):\n    \n    1. Edit or create `/etc/docker/daemon.json` file with updating it with following contents:\n        ```\n        {\n            \"insecure-registries\" : [\"\u003cregistry domain or IP\u003e:\u003cregistry port\u003e\"]\n        }\n        ```\n    2. Restart Docker for the changes to take effect:\n        ```\n        sudo systemctl daemon-reload\n        sudo systemctl restart docker\n        ```\n    \nReference resources:\n1. Docker docs, docker registry: https://docs.docker.com/registry/\n2. Docker docs, insecure registry deployment: https://docs.docker.com/registry/insecure/\n3. Docker docs, secure registry deployment: https://docs.docker.com/registry/deploying/\n\n## Cluster deployment\n\n### Master node deployment:\nAll commands are assumed to be executed on the master node. We use **Flannel** as pod network driver.\n\n1. Clone the repo and `cd` to the `kube_scripts` dir:\n    ```\n    git clone https://github.com/deepmipt/stand_kubernetes_cluster.git\n    cd stand_kubernetes_cluster/tools/kube_scripts\n    ``` \n2. Initiate cluster master node:\n    ```\n    sudo sysctl net.bridge.bridge-nf-call-iptables=1\n    sudo kubeadm init --pod-network-cidr=10.244.0.0/16\n    ```\n   If succeeded, `kubeadm join` will be printed at the end of init process:\n    ```\n    kubeadm join --token \u003ctoken\u003e \u003cmaster-ip\u003e:\u003cmaster-port\u003e --discovery-token-ca-cert-hash sha256:\u003chash\u003e\n    ```\n    Save it to run while nodes joining.\n3. Make **cubectl** work for your user:\n    ```\n    mkdir -p $HOME/.kube\n    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config\n    sudo chown $(id -u):$(id -g) $HOME/.kube/config\n    ```\n4. Deploy Flannel network driver:\n    ```\n    kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/62e44c867a2846fefb68bd5f178daf4da3095ccb/Documentation/kube-flannel.yml\n    ```\n5. Or you can simply run `kubeadm_init_flannel.sh` script from `stand_kubernetes_cluster/tools/kube_scripts` instead of steps 2-4:\n    ```\n    sudo sh kubeadm_init_flannel.sh\n    ```\n6. Deploy NVIDIA device plugin for Kubernetes with running `deploy_nvidia_plugin.sh` script from `stand_kubernetes_cluster/tools/kube_scripts`:\n    ```\n    sudo sh deploy_nvidia_plugin.sh\n    ```\n\nReference resources:\n1. NVIDIA device plugin for Kubernetes: https://github.com/NVIDIA/k8s-device-plugin\n\n### Worker nodes deployment:\n1. To add worker nodes to the cluster run with `--ignore-preflight-errors=all` option saved `kubeadm join` as sudo on each worker node and restart kubelet:\n    ```\n    sudo kubeadm join \u003cmaster-ip\u003e:\u003cmaster-port\u003e --token \u003ctoken\u003e --discovery-token-ca-cert-hash sha256:\u003chash\u003e\n    ```\n    \n### Cluster operations:\n1. Check all system pods are running:\n    ```\n    kubectl get all --namespace=kube-system\n    ```\n2. List all nodes to check their availability:\n    ```\n    kubectl -n kube-system get nodes\n    ```\n3. To get `kubeadm join` run on master node:\n    ```\n    sudo kubeadm token create --print-join-command\n    ```\n4. To tear down the node:\n    1. Run on master:\n        ```\n        kubectl drain \u003cnode name\u003e --delete-local-data --force --ignore-daemonsets\n        kubectl delete node \u003cnode name\u003e\n        ```\n    2. Run on node `reset_node.sh` script from `stand_kubernetes_cluster/tools/kube_scripts`:\n        ```\n        sudo sh reset_node.sh\n        ```\n5. To totally dismantle cluster first tear down all worker nodes then tear down master node as written above.\n\nReference resources:\n1. Kubernetes docs, crating cluster: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/\n2. GitHub, Kube-router deployment: https://github.com/cloudnativelabs/kube-router/blob/master/Documentation/kubeadm.md\n3. Habrahabr, Kubernetes deployment: https://habrahabr.ru/company/southbridge/blog/334846/\n4. Habrahabr, Kubernetes deployment: https://habr.com/post/348688/\n\n## Docker images deployment\nBuild Docker images and push them to registry. Here is the basic reference:\n1. Docker docs, push/pull operations: https://docs.docker.com/registry/\n2. Docker docs, image naming: https://docs.docker.com/registry/introduction/#understanding-image-naming\n\n## Cluster payload deployment and management\n\n### Common stand components deployment\nFor now demo stand solution requires some Kubernetes objects like **Namespaces**, **PersistentVolumes** and so on to be created before stand skills and services (payload) are deployed.\nYaml configs for these objects are located in `stand_kubernetes_cluster/kuber_configs/common`.\n\nTo create these objects:\n\n1. `cd` to `stand_kubernetes_cluster/kuber_configs/common`\n2. We run demo stand payload in `stand-demo` **Namespace**. To create in run:\n    ```\n    kubectl create -f namespaces/stand_demo_ns.yaml\n    ```   \n3. Create hostpath volumes for stand component and logs:\n    ```\n    kubectl create -f volumes/logs_hostpath\n    kubectl create -f volumes/components_hostpath\n    kubectl create -f volumes/db_hostpath\n    kubectl create -f volumes/rb_hostpath\n    ``` \n    \nFor Kubernetes storaging and **Volumes** you can reference:\n1. Kubernetes docs, **Volumes**: https://kubernetes.io/docs/concepts/storage/volumes/\n2. Kubernetes docs, **PersistentVolumes** manual: https://kubernetes.io/docs/tasks/configure-pod-container/configure-persistent-volume-storage/\n\n### Stand payload deployment\nSo far, payload deployment includes following steps:\n1. Building of **Docker image** and pushing it to the registry\n2. **Deployment** definition and launching\n3. **Service** definition and launching\n\nYaml configs for payload **Deployments** and **Services** are located in `stand_kubernetes_cluster/kuber_configs/models`.\n\nTo deploy stand payload:\n\n1. Push  payload **Docker image** to the cluster registry (Russian NER example):\n    ```\n    sudo docker push kubeadm.ipavlov.mipt.ru:5000/stand/ner_ru\n    ```\n2. `cd` to `stand_kubernetes_cluster/kuber_configs/models`\n3. Create payload **Deployment** (Russian NER example):\n    ```\n    kubectl create -f stand_ner_ru/stand_ner_ru_dp.yaml\n    ``` \n4. Create payload **Service** (Russian NER example):\n    ```\n    kubectl create -f stand_ner_ru/stand_ner_ru_lb.yaml\n    ```\n5. Or you can simply run following instead of steps 3-4:\n    ```\n    kubectl create -f stand_ner_ru\n    ``` \n\nFor **Deployments** and **Services** definition you can reference:\n1. Kubernetes docs, deployments: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/\n2. Kubernetes docs, deployments API: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.9/#deployment-v1-apps\n3. Kubernetes docs, services: https://kubernetes.io/docs/concepts/services-networking/service/\n4. Kubernetes docs, services API: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.9/#service-v1-core\n\n### Stand payload operations\n\nWe run demo stand payload in `stand-demo` **Namespace**. So we should use `-n stand-demo` flag in all `kubectl` operations with demo stand objects.\n\nDemo stand objects listing:\n1. List **Services**:\n    ```\n    kubectl -n stand-demo get services\n    ```\n2. List **Deployments**:\n    ```\n    kubectl -n stand-demo get deployments\n    ```\n3. List **Pods**:\n    ```\n    kubectl -n stand-demo get pods\n    ```\n\nGet detailed information about object:\n1. **Service**:\n    ```\n    kubectl -n stand-demo describe service \u003cservice_name\u003e\n    ```\n2. **Deployment**:\n    ```\n    kubectl -n stand-demo describe deployment \u003cdeployment_name\u003e\n    ```\n3. **Pod**:\n    ```\n    kubectl -n stand-demo describe pod \u003cpod_name\u003e\n    ```\n    \nDelete object:\n1. **Service**:\n    ```\n    kubectl -n stand-demo delete service \u003cservice_name\u003e\n    ```\n    Or:\n    ```\n    kubectl delete -f \u003cservice_config.yaml\u003e\n    ```\n2. **Deployment**:\n    ```\n    kubectl -n stand-demo delete deployment \u003cdeployment_name\u003e\n    ```\n    Or:\n    ```\n    kubectl delete -f \u003cdeployment_config.yaml\u003e\n    ```\n3. **Pod** (new pod will be instantly created instead of deleted):\n    ```\n    kubectl -n stand-demo delete pod \u003cpod_name\u003e\n    ```\n\n# License\n\nApache 2.0 - licensed.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeeppavlov%2Fstand_kubernetes_cluster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeeppavlov%2Fstand_kubernetes_cluster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeeppavlov%2Fstand_kubernetes_cluster/lists"}