{"id":20037882,"url":"https://github.com/openedx/openedx-k8s-harmony","last_synced_at":"2025-03-02T06:24:08.017Z","repository":{"id":64842309,"uuid":"570649331","full_name":"openedx/openedx-k8s-harmony","owner":"openedx","description":"A Prototype Helm Chart for deploying multiple Open edX instances (via Tutor) onto a cluster.","archived":false,"fork":false,"pushed_at":"2024-04-05T05:08:54.000Z","size":1568,"stargazers_count":10,"open_issues_count":21,"forks_count":10,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-04-09T23:10:51.893Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openedx.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-11-25T18:03:47.000Z","updated_at":"2024-03-29T19:10:24.000Z","dependencies_parsed_at":"2022-12-16T05:21:37.321Z","dependency_job_id":"c0d06897-c1d2-445e-ba5e-76a330c9ca65","html_url":"https://github.com/openedx/openedx-k8s-harmony","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openedx%2Fopenedx-k8s-harmony","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openedx%2Fopenedx-k8s-harmony/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openedx%2Fopenedx-k8s-harmony/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openedx%2Fopenedx-k8s-harmony/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openedx","download_url":"https://codeload.github.com/openedx/openedx-k8s-harmony/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241467842,"owners_count":19967737,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T10:23:34.560Z","updated_at":"2025-03-02T06:24:07.975Z","avatar_url":"https://github.com/openedx.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Harmony Project - Open edX on Kubernetes\n\nThis project is focused on making it easy to set up a standardized, scalable, secure Kubernetes environment that can host **multiple instances** of [Open edX](https://www.openedx.org). See [Motivation](#motivation) below.\n\nSpecifically, this repository contains:\n\n* A Helm Chart that can install necessary shared resources into your cluster (a load balancer / ingress controller, autoscaling infrastructure, monitoring tools, databases, etc.)\n* A [Tutor](https://docs.tutor.overhang.io/) plugin that configures Tutor to build images that will use the shared resources deployed by the Helm chart.\n\nThis repository does not set up the Kubernetes cluster or provision individual Open edX instances.\n\nSee [technology stack and architecture](#technology-stack-and-architecture) below for more details.\n\n## Motivation\n\nMany Open edX providers and users have a need to deploy multiple instances of Open edX onto Kubernetes, but there is currently no standardized way to do so and each provider must build their own tooling to manage that. This project aims to provide an easy and standardized approach that incorporates industry best practices and lessons learned.\n\nIn particular, this project aims to provide the following benefits to Open edX operators:\n\n* **Ease of use** and **rapid deployment**: This project aims to provide an Open edX hosting environment that just works out of the box, that can be easily upgraded, and that follows best practices for monitoring, security, etc.\n* **Lower costs** by sharing resources where it makes sense. For example, by default Tutor's k8s feature will deploy a separate load balancer and ingress controller for each Open edX instance, instead of a shared ingress controller for all the instances in the cluster. Likewise for MySQL, MongoDB, ElasticSearch, and other resources. By using shared resources by default, costs can be dramatically reduced and operational monitoring and maintenance is greatly simplified.\n  * For setups with many small instances, this shared approach provides a huge cost savings with virtually no decrease in performance.\n  * For larger instances on the cluster that need dedicated resources, they can easily be configured to do so.\n* **Scalable hosting** for instances of any size. This means for example that the default configuration includes autoscaling of LMS pods to handle increased traffic.\n* **Flexibility**: this project aims to be \"batteries included\" and to support setting up all the resources that you need, with useful default configurations, but it is carefully designed so that operators can configure, replace, or disable any components as needed.\n\n## Technology stack and architecture\n\n1. At the base is a Kubernetes cluster, which you must provide (e.g. using OpenTofu to provision Amazon EKS).\n   * Any cloud provider such as AWS or Digital Ocean should work. There are OpenTofu examples in the `infra-examples` folder but it is just a starting point and not recommended for production use.\n2. On top of that, this project's helm chart will install the shared resources you need - an ingress controller, monitoring, database clusters, etc. The following are included but can be disabled/replaced if you prefer an alternative:\n   * Ingress controller: [ingress-nginx](https://kubernetes.github.io/ingress-nginx/)\n   * Automatic HTTPS cert provisioning: [cert-manager](https://cert-manager.io/)\n   * Autoscaling: `metrics-server` and `vertical-pod-autoscaler`\n   * Search index: ElasticSearch or OpenSearch\n   * Monitoring: TODO\n   * Database clusters: TODO (for now we recommend provisioning managed MySQL/MongoDB database clusters from your cloud provider using OpenTofu or a tool like [Grove](https://grove.opencraft.com/).)\n   * Where possible, we try to configure these systems to **auto-detect** newly deployed Open edX instances and adapt to them automatically; where that isn't possible, Tutor plugins are used so that the instances self-register or self-provision the shared resources as needed.\n3. [Tutor](https://docs.tutor.overhang.io/) is used to build the container images that will be deployed onto the cluster.\n   * This project's Tutor plugin is required to make the images compatible with the shared resources deployed by the Helm chart.\n   * The\n[pod-autoscaling plugin](https://github.com/eduNEXT/tutor-contrib-pod-autoscaling) is required for autoscaling.\n   * You can use the `tutor k8s` commands directly (as documented below) or you can use a CI/CD tool like [Grove](https://grove.opencraft.com/) to deploy instances/images.\n\n## FAQ\n\n### Is this ready to use in production?\n\nWe are tracking that in [issue 26](https://github.com/openedx/openedx-k8s-harmony/issues/26), so check that issue for the current status.\n\n### This project aims to support many small/medium instances deployed onto a cluster; is it also suitable for deploying one really high traffic instance?\n\nSupporting one really large instance is not a core design goal, but it should work well and we may consider including this as a goal in the future. Please reach out to us or get involved with this project if you have this requirement.\n\n### What are the gotchas related to cert-manager?\n\nThis helm chart uses [ingress-nginx](https://kubernetes.github.io/ingress-nginx/) as a load balancer alongside [cert-manager](https://cert-manager.io/) to provide automatic SSL certificates. Because of how Helm works, the cert-manager sub-chart will be installed into the same namespace as the parent harmony chart. But if you already have cert-manager on your cluster, this will create a conflict. You should take special care not to install cert-manager twice due to it installing several non-namespaced resources. If you already installed cert-manager by different means, make sure set `cert-manager.enabled: false` for this chart.\n\nIn addition, [the cert-manager Helm charts do not install the required CRDs used by cert-manager](https://cert-manager.io/docs/installation/upgrading/#crds-managed-separately), so you will need to manually install and upgrade them to the correct version as described in the instructions below. This is due to the some limitations in the management of CRDs by Helm.\n\n### How the autoscaling capabilities are implemented in this project?\n\nTutor does not offer an autoscaling mechanism by default. This is a critical feature when your application starts to\nreceive more and more traffic. Kubernetes offers two main autoscaling methods:\n\n* **Pod-based scaling**: This mechanism consists of the creation and adjustment of new pods to cover growing workloads.\nHere we can mention tools like\n[**Horizontal Pod autoscaler (HPA)**](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)\nand [**Vertical pod autoscaler (VPA)**](https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler).\nThe first consist of automatically increasing or decreasing the number of pods in response to a workload's metric\nconsumption (generally CPU and memory), and the second one aims to stabilize the consumption and resources of every pod\nby providing suggestions on the best configuration for a workload based on historical resource usage measurements. Both\nof them are meant to be applied over Kubernetes Deployment instances.\n\n* **Node-based scaling:** This mechanism allows the addition of new NODES to the Kubernetes cluster so compute resources\nare guaranteed to schedule new incoming workloads. Tools worth mentioning in this category are\n[**cluster-autoscaler (CA)**](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler) and\n[Karpenter](https://karpenter.sh/).\n\nFor the scope of this project, the focus will be in the **pod-based scaling** mechanisms since Node-based scaling tools\nrequire configuration which is external to the cluster and this is out of the scope for this Helm chart for now.\n\nThe approach will be to use pod autoscaling on each environment separately (assuming there are installations on different\nnamespaces) following the steps below:\n\n1. **Install the global dependencies**: this Helm chart offers the option of installing the dependencies required to\nget HPA and VPA working (they are included as subcharts). These are the Helm charts for **metrics-server** and **VPA**.\nBy default these dependencies are not installed, however you can enable them via the Helm chart values if they aren't\nstill present in your cluster.\n2. **Enable the pod-autoscaling plugin per environment**: the\n[pod-autoscaling plugin](https://github.com/eduNEXT/tutor-contrib-pod-autoscaling) enables the implementation of HPA and\nVPA to start scaling an installation workloads. Variables for the plugin configuration are documented there.\n\n#### Node-autoscaling with Karpenter in EKS Clusters\n\nThis section provides a guide on how to install and configure [Karpenter](https://karpenter.sh/) in a EKS cluster. We'll use\ninfrastructure examples included in this repo for such purposes.\n\n\u003e Prerequisites:\n\n* An aws account id\n* Kubectl 1.27\n* OpenTofu 1.6.x or higher\n* Helm\n\n1. Clone this repository and navigate to `./infra-examples/aws`. You'll find OpenTofu modules for `vpc` and `k8s-cluster`\nresources. Proceed creating the `vpc` resources first, followed by the `k8s-cluster` resources. Make sure to have the target\nAWS account ID available, and then execute the following commands on every folder:\n\n   ```sh\n   tofu init\n   tofu plan\n   tofu apply -auto-approve\n   ```\n\n   It will create an EKS cluster in the new VPC. Required Karpenter resources will also be created.\n\n2. Once the `k8s-cluster` is created, run the `tofu output` command on that module and copy the following output variables:\n\n   * cluster_name\n   * karpenter_irsa_role_arn\n   * karpenter_instance_profile_name\n\n   These variables will be required in the next steps.\n\n3. Karpenter is a dependency of the harmony chart that can be enabled or disabled. To include Karpenter in the Harmony Chart,\n**it is crucial** to configure these variables in your `values.yaml` file:\n\n   * `karpenter.enabled`: true\n   * `karpenter.serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn`: \"\u003c`karpenter_irsa_role_arn` value from module\u003e\"\n   * `karpenter.settings.aws.defaultInstanceProfile`: \"\u003c`karpenter_instance_profile_name` value from module\u003e\"\n   * `karpenter.settings.aws.clusterName`:  \"\u003c`cluster_name` value from module\u003e\"\n\n   Find below an example of the Karpenter section in the `values.yaml` file:\n\n   ```yaml\n   karpenter:\n      enabled: true\n      serviceAccount:\n         annotations:\n            eks.amazonaws.com/role-arn: \"\u003ckarpenter_irsa_role_arn\u003e\"\n      settings:\n         aws:\n            # -- Cluster name.\n            clusterName: \"\u003ccluster_name\"\n            # -- Cluster endpoint. If not set, will be discovered during startup (EKS only)\n            # From version 0.25.0, Karpenter helm chart allows the discovery of the cluster endpoint. More details in\n            # https://github.com/aws/karpenter/blob/main/website/content/en/docs/upgrade-guide.md#upgrading-to-v0250\n            # clusterEndpoint: \"https://XYZ.eks.amazonaws.com\"\n            # -- The default instance profile name to use when launching nodes\n            defaultInstanceProfile: \"\u003ckarpenter_instance_profile_name\u003e\"\n   ```\n\n4. Now, install the Harmony Chart in the new EKS cluster using [these instructions](#usage-instructions). This will provide a\nvery basic Karpenter configuration with one [provisioner](https://karpenter.sh/docs/concepts/provisioners/) and one\n[node template](https://karpenter.sh/docs/concepts/node-templates/). Please refer to the official documentation to\nget further details.\n\n\u003e **NOTE:**\n\u003e This Karpenter installation does not support multiple provisioners or node templates for now.\n\n5. To test Karpenter, you can proceed with the instructions included in the\n[official documentation](https://karpenter.sh/docs/getting-started/getting-started-with-karpenter/#first-use).\n\n\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e\n\n## Usage Instructions\n\n### Step 1: Use Helm to provision a kubernetes cluster using this chart\n\n#### Option 1a: Setting up Harmony Chart on a cloud-hosted Kubernetes Cluster (recommended)\n\nFor this recommended approach, you need to have a Kubernetes cluster in the cloud **with at least 12GB of usable\nmemory** (that's enough to test 2 Open edX instances).\n\n1. Make sure you can access the cluster from your machine: run `kubectl cluster-info` and make sure it displays some\n   information about the cluster (e.g. two URLs).\n2. Copy `values-example.yaml` to a new `values.yaml` file and edit it to put in your email address and customize\n   other settings. The email address is required for Lets Encrypt to issue HTTPS certificates. It is not shared\n   with anyone else. For a full configuration reference, see the `charts/harmony-chart/values.yaml` file.\n3. Install [Helm](https://helm.sh/) if you don't have it already.\n4. Add the Harmony Helm repository:\n\n   ```shell\n   helm repo add openedx-harmony https://openedx.github.io/openedx-k8s-harmony\n   helm repo update\n   ```\n\n5. Install the cert-manager CRDs if using cert-manager:\n\n   ```shell\n   kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.10.1/cert-manager.crds.yaml --namespace=harmony\n   ```\n\n   You can check the version of cert-manager that is going to be installed by the chart by checking the corresponding\n   line in the `charts/harmony-chart/Chart.yaml` file.\n6. Install the Harmony chart by running:\n\n   ```shell\n   helm install harmony --namespace harmony --create-namespace -f values.yaml openedx-harmony/harmony-chart\n   ```\n\nNote: in the future, if you apply changes to `values.yaml`, please run this command to update the deployment of the chart:\n\n```shell\nhelm upgrade harmony --namespace harmony -f values.yaml openedx-harmony/harmony-chart\n```\n\n#### Option 1b: Setting up Harmony Chart locally on Minikube\n\n*Note: if possible, it's preferred to use a cloud-hosted cluster instead (see previous section). But if you don't have a\ncluster available in the cloud, you can use minikube to try this out locally. The minikube version does not support\nHTTPS and is more complicated due to the need to use tunnelling.*\n\n1. First, [install `minikube`](https://minikube.sigs.k8s.io/docs/start/) if you don't have it already.\n2. Run `minikube start` (you can also use `minikube dashboard` to access the Kubernetes dashboard).\n3. Add the Helm repository and install the Harmony chart using the `values-minikube.yaml` file as configuration:\n\n   ```shell\n   helm repo add openedx-harmony https://openedx.github.io/openedx-k8s-harmony\n   helm repo update\n   helm install harmony --namespace harmony --create-namespace -f values-minikube.yaml openedx-harmony/harmony-chart\n   ```\n\n4. Run `minikube tunnel` (you may need to enter a password), and then you should be able to access the cluster (see\n   \"External IP\" below). If this approach is not working, an alternative is to run\\\n   `minikube service harmony-ingress-nginx-controller -n harmony`\\\n   and then go to the URL it says, e.g. `http://127.0.0.1:52806` plus `/cluster-echo-test`\n   (e.g. `http://127.0.0.1:52806/cluster-echo-test`)\n5. In this case, skip step 2 (\"Get the external IP\") and use `127.0.0.1` as the external IP. You will need to remember\n   to include the port numbers shown above when accessing the instances.\n\n### Step 2: Get the external IP\n\nThe [ingress NGINX Controller](https://kubernetes.github.io/ingress-nginx/) is used to automatically set up an HTTPS\nreverse proxy for each Open edX instance as it gets deployed onto the cluster. There is just one load balancer with a\nsingle external IP for all the instances on the cluster. To get its IP, use:\n\n```shell\nkubectl get svc -n harmony harmony-ingress-nginx-controller\n```\n\nTo test that your load balancer is working, go to `http://\u003cthe external ip\u003e/cluster-echo-test` .\nYou may need to ignore the HTTPS warnings, but then you should see a response with some basic JSON output.\n\n### Step 3: Deploying an Open edX instance using Tutor\n\nImportant: First, get the load balancer's IP (see \"external IP\" above), and set the DNS records for the instance you\nwant to create to be pointing to the load balancer (Usually if you want the LMS at `lms.example.com`, you'll need to set\ntwo A records for `lms.example.com` and `*.lms.example.com`, pointing to the external IP from the load balancer).\n\nYou also will need to have the tutor-contrib-harmony-plugin installed into Tutor:\n\n```shell\npip install -e 'git+https://github.com/openedx/openedx-k8s-harmony.git#egg=tutor-contrib-harmony-plugin\u0026subdirectory=tutor-contrib-harmony-plugin'\n```\n\nNext, create a Tutor config directory unique to this instance, and configure it:\n\n```shell\nexport INSTANCE_ID=openedx-01\nexport TUTOR_ROOT=~/deployments/tutor-k8s/$INSTANCE_ID\ntutor plugins enable k8s_harmony\ntutor config save -i --set K8S_NAMESPACE=$INSTANCE_ID\n```\n\nThen deploy it:\n\n```shell\ntutor k8s start\ntutor k8s init\n```\n\nNote that the `init` command may take quite a long time to complete. Use the commands that Tutor says (\"To view the logs\nfrom this job, run:\") in a separate terminal in order to monitor the status.\n\n**You can repeat step 3 many times to install multiple instances onto the cluster.**\n\n\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e\n\n## Configuration Reference\n\n### Multi-tenant Elasticsearch\n\nTutor creates an Elasticsearch pod as part of the Kubernetes deployment. Depending on the number of instances, memory\nand CPU use can be lowered by running a central ES cluster instead of an ES pod for every instance.\n\n**Please note that this will only work for \"Redwood\" version and later. The OpenSearch implementation is not yet confirmed to work as expected.**\n\nTo enable set `elasticsearch.enabled=true` in your `values.yaml` and deploy the chart.\n\nFor each instance you would like to enable this on, set the configuration values in the respective `config.yml`:\n\n```yaml\nRUN_ELASTICSEARCH: false\nK8S_HARMONY_ENABLE_SHARED_SEARCH_CLUSTER: true\nK8S_HARMONY_SEARCH_CLUSTER_HTTP_AUTH: instance-name:desired-password\nK8S_HARMONY_SEARCH_CLUSTER_INDEX_PREFIX: prefix-\n```\n\nIf the `K8S_HARMONY_SEARCH_CLUSTER_HTTP_AUTH` or `K8S_HARMONY_SEARCH_CLUSTER_INDEX_PREFIX` is not set, the settings are\npopulated with random value to ensure uniqueness.\n\n* Create the user on the cluster with `tutor harmony create-elasticsearch-user`.\n* Copy the Elasticsearch CA certificate to the instance's namespace where `$INSTANCE_NAMESPACE` is where the instance is installed in. The `$HARMONY_NAMESPACE` should be set to the namespace where the Harmony is installed to.\n  ```shell\n  kubectl get secret \"search-cluster-certificates-elasticsearch\" -n \"$HARMONY_NAMESPACE\" -o \"yaml\" | \\\n      grep -v '^\\s*namespace:\\s' | \\\n      sed s/-elasticsearch//g |\\\n      kubectl apply -n \"$INSTANCE_NAMESPACE\" --force -f -\n  ```\n* Rebuild your Open edX image `tutor images build openedx`.\n* Finally, redeploy your changes: `tutor k8s start \u0026\u0026 tutor k8s init`.\n\n#### Caveats\n\nIn order for SSL to work without warnings the CA certificate needs to be mounted in the relevant pods. This is not yet implemented as due to an [outstanding issue in tutor](https://github.com/overhangio/tutor/issues/791) that had not yet been completed at the time of writing.\n\n## Extended Documentation\n\n### How to uninstall this chart\n\nJust run `helm uninstall --namespace harmony harmony` to uninstall this.\n\n### How to create a cluster for testing on DigitalOcean\n\nIf you use DigitalOcean, you can use OpenTofu to quickly spin up a cluster, try this out, then shut it down again.\n\nHere's how. First, put the following into `infra-examples/digitalocean/secrets.auto.tfvars` including a valid DigitalOcean access token:\n\n```conf\ncluster_name = \"harmony-test\"\ndo_token = \"digital-ocean-token\"\n```\n\nThen run:\n\n```sh\ncd infra-examples/digitalocean\ntofu init\ntofu apply\ncd ..\nexport KUBECONFIG=`pwd`/infra-examples/digitalocean/kubeconfig\n```\n\nThen follow steps 1-4 above. When you're done, run `tofu destroy` to clean\nup everything.\n\n## Appendix C: how to create a cluster for testing on AWS\n\nSimilarly, if you use AWS, you can use OpenTofu to spin up a cluster, try this out, then shut it down again.\nHere's how. First, put the following into `infra-examples/aws/vpc/secrets.auto.tfvars` and `infra-examples/aws/k8s-cluster/secrets.auto.tfvars`:\n\n   ```terraform\n   account_id  = \"012345678912\"\n   aws_region  = \"us-east-1\"\n   name        = \"tutor-multi-test\"\n   ```\n\nThen run:\n\n   ```bash\n   aws sts get-caller-identity   # to verify that awscli is properly configured\n   cd infra-examples/aws/vpc\n   tofu init\n   tofu apply               # run time is approximately 1 minute\n   cd ../k8s-cluster\n   tofu init\n   tofu apply               # run time is approximately 30 minutes\n\n   # to configure kubectl\n   aws eks --region us-east-1 update-kubeconfig --name tutor-multi-test --alias tutor-multi-test\n   ```\n\nThen follow steps 1-4 above. When you're done, run `tofu destroy` in both the `aws` and `k8s-cluster` modules to clean up everything.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenedx%2Fopenedx-k8s-harmony","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenedx%2Fopenedx-k8s-harmony","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenedx%2Fopenedx-k8s-harmony/lists"}