{"id":21881195,"url":"https://github.com/wtsi-hgi/hgi-k8s","last_synced_at":"2026-04-19T19:04:37.126Z","repository":{"id":146402471,"uuid":"167384507","full_name":"wtsi-hgi/hgi-k8s","owner":"wtsi-hgi","description":null,"archived":false,"fork":false,"pushed_at":"2019-02-20T17:02:23.000Z","size":58,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-22T00:44:40.518Z","etag":null,"topics":["jupyterhub","kubernetes","kubespray"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wtsi-hgi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-01-24T14:52:51.000Z","updated_at":"2019-02-20T17:02:25.000Z","dependencies_parsed_at":null,"dependency_job_id":"9d7dd265-7332-4585-af18-0b0679b59cfc","html_url":"https://github.com/wtsi-hgi/hgi-k8s","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/wtsi-hgi/hgi-k8s","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wtsi-hgi%2Fhgi-k8s","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wtsi-hgi%2Fhgi-k8s/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wtsi-hgi%2Fhgi-k8s/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wtsi-hgi%2Fhgi-k8s/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wtsi-hgi","download_url":"https://codeload.github.com/wtsi-hgi/hgi-k8s/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wtsi-hgi%2Fhgi-k8s/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32018765,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-18T20:23:30.271Z","status":"online","status_checked_at":"2026-04-19T02:00:07.110Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["jupyterhub","kubernetes","kubespray"],"created_at":"2024-11-28T09:18:06.452Z","updated_at":"2026-04-19T19:04:37.118Z","avatar_url":"https://github.com/wtsi-hgi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HGI Kubernetes\n\nThis repo provides everything necessary to provision a Kubernetes\ncluster on Sanger's OpenStack and deploy the following services:\n\n* JupyterHub\n\nK8s deployment is done using a modified version of\n[Kubespray](https://github.com/kubernetes-sigs/kubespray) to run on\nSanger's infrastructure, from our sister team in [Cellular\nGenetics](https://github.com/cellgeni). Their repo, itself a fork of\nupstream Kubespray, is checked in here. HGI's implementation doesn't\nnecessarily follow best practices, in a software engineer or devops\nsense, but it works. For now.\n\nThis documentation serves as a step-by-step guide, following that\nprovided by Cellgeni, to get everything up-and-running.\n\n## Provisioning the Cluster\n\n### Getting Started\n\nYou will need an OpenStack project/tenancy to deploy your cluster into.\nFor deployment, you will need a host that can speak to OpenStack, with\nTerraform, Ansible and the OpenStack client installed. It's definitely\nworth setting your environment up in a `tmux` session, to avoid having\nto go through the rigmarole each time!\n\nClone this repository:\n\n    git clone --recurse-submodule https://github.com/wtsi-hgi/hgi-k8s.git\n\nThe Cellgeni Kubespray module is set to track upstream master. If this\nhas become out-dated, it can be updated with:\n\n    git submodule update --remote\n\nThe first thing you'll need to do is source `activate.rc`, which sets\nthe appropriate environment variables for communicating with our\nOpenStack project. This works by retrieving HGI's secrets from GitLab,\nprovisioned by [`hgi-systems`](https://github.com/wtsi-hgi/hgi-systems).\nFor this to run successfully -- presuming `hgi-systems` is still a thing\n-- the `GITLAB_URL` and `GITLAB_TOKEN` environment variables must be\nset; these values can be found in the usual place.\n\nNote that the automatic fetching of credentials is a nicety; they can be\nhardcoded into `activate.rc` -- along with the other things that are\nhardcoded and will need modifying if you're deploying into a different\nOpenStack project -- providing they're not checked in!\n\n*TODO* `activate.rc` also sources some Cellgeni dependencies that are\nnot currently checked in. This is mostly concerned with setting up and\ninitialising a Conda environment with the necessary bits-and-pieces to\nsetup K8s (specifically Terraform and Ansible).\n\nIf you don't have an `id_rsa` SSH key, you will also need to generate\none and add it to your SSH agent:\n\n```bash\nssh-keygen -t rsa\neval $(ssh-agent -s)\nssh-add ~/.ssh/id_rsa\n```\n\nThis could potentially be done differently, such that there is a special\nK8s key. Currently, the K8s cluster is using `mercury`'s key.\n\n### Deploying the Infrastructure\n\nThe Terraform inventories used to create the infrastructure in our\nOpenStack project is defined in `inventory`. Specifically, for the `dev`\ncluster, the configuration is defined in `inventory/dev/cluster.tf`.\nThe important settings here are the number and flavour of K8s master and\nworker nodes, which ought to be set to fit into your OpenStack project's\nquota.\n\nNote that instance flavours are given by their ID, rather than their\nname (e.g., `o1.medium`). These can be looked up with the OpenStack\nclient:\n\n    openstack flavor list\n\nNote that Kubespray uses etcd nodes; standalone and/or cohabiting the\nK8s master nodes. etcd's consensus algorithm prefers an odd number of\netcd nodes and this is enforced in Kubespray's Ansible playbook;\ntherefore, ensure the number of these nodes is odd. Our development\ncluster uses one master K8s node and no standalone; increasing the\nnumber of K8s masters increases resilience, but this parity constraint\nmust be maintained.\n\nNote that we have chosen to use a bastion node to route all traffic --\ni.e., it gets a floating IP -- therefore all K8s master and worker nodes\nare only available within the private network.\n\nThe other settings in the Terraform configuration are either\nSanger-specific (i.e., probably won't need changing) or aren't\nused/artefacts from ~stealing~ lifting this from Cellgeni (e.g.,\nGlusterFS).\n\nFirst, to initialise Terraform, run the following from your cluster's\ninventory directory:\n\n    terraform init ../../cellgeni-kubespray/contrib/terraform/openstack\n\n(As a convenience, you might want to symlink\n`../../cellgeni-kubespray/contrib/terraform/openstack`, as you'll be\ntyping this a lot!)\n\nThen, to create the cluster, run:\n\n    terraform apply -var-file=cluster.tf ../../cellgeni-kubespray/contrib/terraform/openstack\n\nNote that `provision.sh` notionally performs this operation, but is not\ntested. It provides a useful reference in the meantime.\n\n#### Destroying the Infrastructure\n\n`terraform destroy` can be used to bring down the infrastructure, but\nnote that if you've configured any instances with Ansible, or run any\nservices in K8s, then you'll have to do some manual clean up first. If\nyou don't, OpenStack objects -- particularly networking components and\nvolumes -- can get orphaned, which then become very difficult to remove.\n\n### Configuring the Infrastructure\n\nOnce Terraform has created its infrastructure, Ansible can be used to\nconfigure the instances as a K8s cluster. We must first make some\nchanges to our cluster's inventory:\n\n* We have observed that Terraform consistently wipes the contents of the\n  `no-floating.yml` group variables, for your cluster's inventory. It\n  doesn't delete the file, just empties the contents. It's weird. You\n  will need to restore this with Git, say:\n\n      git checkout -- inventory/dev/group_vars/no-floating.yml\n\n* `no-floating.yml`, once restored, defines the SSH arguments that\n  Ansible will use to route to the cluster nodes via the bastion host.\n  The floating IP of the bastion host is hardcoded into this file and\n  must be changed to the actual floating IP of the bastion host.\n\n* The `all.yml` group variables will also need to be edited:\n  * `openstack_lbaas_subnet_id` needs to be set to that logged in\n    Terraform's state file (i.e., `terraform show terraform.tfstate | grep ' subnet_id'`)\n  * `bin_dir` should be `/usr/local/bin`, if it's not already\n  * `cloud_provider` should be `openstack`, if it's not already\n\n* The `k8s-cluster.yml` group variables will also need to be edited:\n  * `kube_network_plugin` should be `calico`, if it's not already\n  * `resolvconf_mode` should be `docker_dns`, if it's not already\n\nTo use Calico networking, your OpenStack project's network's ports need\nto be configured, where `$CLUSTER` is defined appropriately (e.g.,\n`dev`):\n\n```bash\njoin -t\" \" -o \"1.2\" \u003c(openstack port list -f value -c device_id -c id | sort) \\\n                    \u003c(openstack server list --name \"${CLUSTER}-.*\" -f value -c ID | sort) \\\n| xargs -n1 openstack port set --allowed_address ip-address=10.233.0.0/18 \\\n                               --allowed_address ip-address=10.233.64.0/18\n```\n\nFinally, before Ansible can be run, the `TERRAFORM_STATE_ROOT`\nenvironment variable should be set to the cluster's inventory root, say:\n\n```bash\nexport TERRAFORM_STATE_ROOT=\"$(pwd)/inventory/${CLUSTER}\"\n```\n\nTo test everything is working and that Ansible can contact all your\ncluster's hosts, run the following from the `cellgeni-kubespray`\ndirectory:\n\n    ansible -i ../inventory/${CLUSTER}/hosts -m ping all\n\nProviding all is well, the playbook can then be run to install K8s:\n\n    ansible-playbook --become -i ../inventory/${CLUSTER}/hosts cluster.yml\n\nThis will take a bit of time...\n\n## Accessing Kubernetes\n\nTo access and administer K8s from your machine, you will need to\n[install `kubectl`](https://kubernetes.io/docs/tasks/tools/install-kubectl/)\nand set up networking and SSH on your local machine.\n\n* Copy the K8s cluster's SSH key to your local machine and `chmod` it\n  appropriately:\n\n  ```bash\n  scp user@host:/path/to/k8s/ssh/key ~/.ssh/k8s.key\n  chmod 600 ~/.ssh/k8s.key\n  ```\n\n* Create a K8s SSH configuration file and `include` it in your master\n  SSH configuration:\n\n  ```ssh\n  Host 10.0.0.*\n    ProxyCommand ssh -W %h:%p k8s-bastion\n    User ubuntu\n    IdentityFile ~/.ssh/k8s.key\n    ForwardX11 yes\n    ForwardAgent yes\n    ForwardX11Trusted yes\n\n  Host k8s-bastion\n    Hostname BASTION_IP\n    User ubuntu\n    IdentityFile ~/.ssh/k8s.key\n    ForwardX11 yes\n    ForwardAgent yes\n    ForwardX11Trusted yes\n\n  Host k8s-master\n    Hostname MASTER_IP\n    ProxyCommand ssh -W %h:%p k8s-bastion\n    User ubuntu\n    IdentityFile ~/.ssh/k8s.key\n\n  Host k8s-tunnel\n    Hostname BASTION_IP\n    LocalForward 16443 MASTER_IP:6443\n    IdentityFile ~/.ssh/k8s.key\n    ServerAliveInterval 5\n    ServerAliveCountMax 1\n    User ubuntu\n  ```\n\n  Where `BASTION_IP` is the floating IP address of your bastion host and\n  `MASTER_IP` is the IP address of any K8s master node within your\n  subnet.\n\n* Add a route to your K8s subnet via the bastion host, from your local\n  machine. For example, on macOS:\n\n  ```bash\n  sudo route add -net 10.0.0.0/24 ${BASTION_IP}\n  ```\n\n  *Note: You probably won't need to do this. This step is included for\n  completeness' sake.*\n\n* Get K8s certificates and keys:\n\n  ```bash\n  # List keys\n  ssh k8s-master sudo ls /etc/kubernetes/ssl\n\n  # Get admin keys; change remote filename appropriately\n  ssh k8s-master sudo cat /etc/kubernetes/ssl/admin-kube-master-1-key.pem \u003e admin-key.pem\n  ssh k8s-master sudo cat /etc/kubernetes/ssl/admin-kube-master-1.pem \u003e admin.pem\n  ssh k8s-master sudo cat /etc/kubernetes/ssl/ca.pem \u003e ca.pem\n  ```\n\n* Configure `kubectl`:\n\n  ```bash\n  kubectl config set-cluster default-cluster --server=https://127.0.0.1:16443 --certificate-authority=ca.pem\n  kubectl config set-credentials default-admin --certificate-authority=ca.pem --client-key=admin-key.pem --client-certificate=admin.pem\n  kubectl config set-context default-system --cluster=default-cluster --user=default-admin\n  kubectl config use-context default-system\n  ```\n\n  Ensure that `clusters.cluster.server`, in `~/.kube/config`, points to\n  `https://127.0.0.1:16443`, so it can be accessed via an SSH tunnel.\n\n* Start the SSH tunnel:\n\n  ```bash\n  ssh -fN k8s-tunnel\n  ```\n\n  You may need to kick this periodically.\n\n`kubectl` should now be operational. In future, only the SSH tunnel will\nneed to be started to use `kubectl`.\n\n### Using the Kubernetes Dashboard\n\nProviding the SSH tunnel from the previous section is open, we can also\nuse the K8s dashboard, using:\n\n    kubectl proxy\n\nWhich starts a proxy to the dashboard at\nhttp://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/\n\nHowever, to be able to use this, you will first need to create an admin\nuser and acquire an access token, by following [these\ninstructions](https://github.com/kubernetes/dashboard/wiki/Creating-sample-user).\n\nSee also: [Kubernetes Dashboard Overview](https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/#accessing-the-dashboard-ui)\n\n## Deploying Services\n\n### JupyterHub\n\nOur JupyterHub is deployed via [Helm](https://helm.sh) using the [Zero\nto JupyterHub](https://zero-to-jupyterhub.readthedocs.io/en/stable/)\nchart. The aforementioned website gives a complete overview of how to\ninstall and configure JupyterHub; herein follows a summary.\n\n* Install Helm on a machine that can access the K8s cluster (e.g., the\n  same one you installed `kubectl` on to). For example, on macOS, you\n  can use Homebrew:\n\n  ```bash\n  brew install kubernetes-helm\n  ```\n\n* Install Helm on to the K8s cluster:\n\n  ```bash\n  kubectl --namespace kube-system create serviceaccount tiller\n  kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller\n  helm init --service-account tiller\n  kubectl patch deploy --namespace kube-system tiller-deploy -p '{\"spec\":{\"template\":{\"spec\":{\"serviceAccount\":\"tiller\"}}}}'\n  ```\n\n  Note that if Helm is already installed on the cluster, you will just\n  need to run `helm init --client-only` on the client machine.\n\n* Add JupyterHub to Helm:\n\n  ```bash\n  helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/\n  helm repo update\n  ```\n\n* JupyterHub needs its own persistent storage, to manage its state. To\n  this end, we create a new cinder storage class, per the definition in\n  Cellgeni's Kubespray repo:\n\n  ```bash\n  kubectl create -f cellgeni-kubespray/sanger/storage/sc-rw-once.yaml\n  ```\n\n* Start or upgrade JupyterHub, in the `jpt` namespace, with the\n  following:\n\n  ```bash\n  helm upgrade --install jpt jupyterhub/jupyterhub --namespace jpt --version 0.7.0 --values jupyter/jupyter-config.yaml\n  ```\n\nThe configuration in `jupyter/jupyter-config` defines, amongst other\nthings (see the aforementioned documentation for details):\n\n* Authentication: We have chosen to use LDAP authentication. Note that,\n  in spite of the documentation, a number of fields must be explicitly\n  set for the service to start. Specifically:\n\n  * `ldap.server.address`\n  * `ldap.server.port`\n  * `ldap.server.ssl`\n  * `ldap.dn.lookup`\n  * `ldap.dn.user.escape`\n  * `ldap.dn.user.validRegex`\n  * `ldap.dn.templates`\n\n  This\n  [mapping](https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/a6824b7db1e4c42d7944f08510b7b1ac18b6de1a/images/hub/jupyterhub_config.py#L305-L318)\n  of these values to the [underlying authentication\n  module](https://github.com/jupyterhub/ldapauthenticator) is used, for\n  future reference.\n\n* Notebook Image: We have chosen to use our own notebook image, which is\n  derived from the official Docker image stacks for SciPy and R. The\n  repo for this can be found on GitHub as\n  [`hgi-jupyter-notebook`](https://github.com/wtsi-hgi/hgi-jupyter-notebook)\n  and is available for deployment from [Docker\n  Hub](https://hub.docker.com/r/mercury/hgi-jupyter-notebook).\n\n  This provides, as of writing, Python 3.6 and R 3.5 kernels, as well as\n  common packages. This image will probably develop with time to suite\n  our users' needs.\n\n* Resources:\n\n  * `singleuser.storage.capacity`, which defines the size of the\n    persistent volume claim used by each users' pod;\n  * `singleuser.memory.{limit,guarantee}`, which define the available\n    memory to each users' pod;\n  * `singleuser.cpu.{limit,guarantee}`, which define the available\n    compute to each users' pod.\n\n  We can use these values to easily calculate the maximum capacity of\n  our cluster, in terms of concurrent users, by dividing them into the\n  respective resources provided by the K8s worker nodes. The maximum\n  capacity, in terms of total users, can likewise be found by dividing\n  the total cluster volume quota by the prescribed PVC size.\n\nWhen JupyterHub starts, it initiates a `proxy-public` service in the\n`jpt` namespace, in K8s. This is provisioned with a floating IP, for\nexternal endpoints, and allows access to the service over HTTP.\n\n## To Do...\n\n* [ ] Ingress controller, with TLS\n* [ ] DNS setup (A record)\n* [ ] Firewall setup, to allow access through VPN\n* [ ] Internal network name resolution within K8s pods\n* [ ] Reverse proxy K8s dashboard for convenience? (n.b., Security risk)\n* [ ] Move to vanilla Kubespray and patch as neccessary, to make\n      upstream updates easier to manage\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwtsi-hgi%2Fhgi-k8s","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwtsi-hgi%2Fhgi-k8s","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwtsi-hgi%2Fhgi-k8s/lists"}