{"id":13846251,"url":"https://github.com/uswitch/yggdrasil","last_synced_at":"2025-10-05T03:02:07.052Z","repository":{"id":33521060,"uuid":"137380084","full_name":"uswitch/yggdrasil","owner":"uswitch","description":"Envoy Control Plane for Kubernetes Multi-cluster Ingress","archived":false,"fork":false,"pushed_at":"2025-02-25T16:34:12.000Z","size":5558,"stargazers_count":194,"open_issues_count":5,"forks_count":17,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-04-04T04:35:30.345Z","etag":null,"topics":["envoy","kubernetes","loadbalancing"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/uswitch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-06-14T15:59:22.000Z","updated_at":"2025-02-25T16:22:46.000Z","dependencies_parsed_at":"2024-06-18T21:34:58.617Z","dependency_job_id":"bd208ce0-46e5-449d-b5f9-6285d053150b","html_url":"https://github.com/uswitch/yggdrasil","commit_stats":null,"previous_names":[],"tags_count":27,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uswitch%2Fyggdrasil","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uswitch%2Fyggdrasil/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uswitch%2Fyggdrasil/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uswitch%2Fyggdrasil/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/uswitch","download_url":"https://codeload.github.com/uswitch/yggdrasil/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248655030,"owners_count":21140416,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["envoy","kubernetes","loadbalancing"],"created_at":"2024-08-04T18:00:21.093Z","updated_at":"2025-10-05T03:02:07.042Z","avatar_url":"https://github.com/uswitch.png","language":"Go","funding_links":[],"categories":["Load balancing","Operators vs Controllers","Go"],"sub_categories":["[Jenkins](#jenkins)","Ingress"],"readme":"# Yggdrasil\n\n\u003cimg src=\"./logo/logo.jpg\" width=\"150\"\u003e\n\n----\nYggdrasil is an Envoy control plane that configures listeners and clusters based off Kubernetes ingresses from multiple Kube Clusters. This allows you to have an envoy cluster acting as a mutli-cluster loadbalancer for Kubernetes. This was something we needed as we wanted our apps to be highly available in the event of a cluster outage but did not want the solution to live inside of Kubernetes itself.\n\n`Note:` Currently we support versions 1.20.x to 1.29.x of Envoy.\u003c/br\u003e\n`Note:` Yggdrasil now uses [Go modules](https://github.com/golang/go/wiki/Modules) to handle dependencies.\n\n## Usage\nYggdrasil will watch all Ingresses in each Kubernetes Cluster that you give it via the Kubeconfig flag. Any ingresses that match any of the ingress classes that you have specified will have a listener and cluster created that listens on the same Host as the Host defined in the Ingress object. If you have multiple clusters Yggdrasil will create a cluster address for each Kubernetes cluster your Ingress is in, the address is the address of the ingress loadbalancer.\n\n[Joseph Irving](https://github.com/Joseph-Irving) has published a [blog post](https://medium.com/uswitch-labs/multi-cluster-kubernetes-load-balancing-in-aws-with-yggdrasil-c1583ea7d78f) which describes our need for and use of Yggdrasil at Uswitch.\n\n## Setup\nPlease see the [Getting Started](/docs/GETTINGSTARTED.md) guide for a walkthrough of setting up a simple HTTP service with Yggdrasil and envoy.\n\nThe basic setup is to have a cluster of envoy nodes which connect to Yggdrasil via GRPC and get given dynamic listeners and clusters from it. Yggdrasil is set up to talk to each Kubernetes api where it will watch the ingresses for any that are using the ingress class it's watching for.\n\n![Yggdrasil Diagram](/img/yggdrasil.png)\n\nYour envoy nodes only need a very minimal config where they are simply set up to get dynamic clusters and listeners from Yggdrasil.\nExample envoy config:\n```yaml\nadmin:\n  access_log_path: /tmp/admin_access.log\n  address:\n    socket_address: { address: 0.0.0.0, port_value: 9901 }\n\ndynamic_resources:\n  lds_config:\n    resource_api_version: V3\n    api_config_source:\n      transport_api_version: V3\n      api_type: GRPC\n      grpc_services:\n      - envoy_grpc:\n          cluster_name: xds_cluster\n  cds_config:\n    resource_api_version: V3\n    api_config_source:\n      transport_api_version: V3\n      api_type: GRPC\n      grpc_services:\n      - envoy_grpc:\n          cluster_name: xds_cluster\n\nstatic_resources:\n  clusters:\n  - name: xds_cluster\n    connect_timeout: 0.25s\n    type: STATIC\n    lb_policy: ROUND_ROBIN\n    http2_protocol_options: {}\n    load_assignment:\n      cluster_name: xds_cluster\n      endpoints:\n      - lb_endpoints:\n        - endpoint:\n            address:\n              socket_address:\n                address: yggdrasil\n                port_value: 8080\n```\n\nYour ingress set up then looks like this:\n\n![Envoy Diagram](/img/envoysetup.png)\n\nWhere the envoy nodes are loadbalancing between each cluster for a given ingress.\n\n### Health Check\nYggdrasil always configures a path on your Envoy nodes at `/yggdrasil/status`, this can be used to health check your envoy nodes, it will only return 200 if your nodes have started and been configured by Yggdrasil.\n\n## Annotations\nYggdrasil allows for some customisation of the route and cluster config per Ingress through the annotations below.\n\n| Name                                                         | type     |\n|--------------------------------------------------------------|----------|\n| [yggdrasil.uswitch.com/healthcheck-path](#health-check-path) | string   |\n| [yggdrasil.uswitch.com/timeout](#timeouts)                   | duration |\n| [yggdrasil.uswitch.com/cluster-timeout](#timeouts)           | duration |\n| [yggdrasil.uswitch.com/route-timeout](#timeouts)             | duration |\n| [yggdrasil.uswitch.com/per-try-timeout](#timeouts)           | duration |\n| [yggdrasil.uswitch.com/weight](#weight)                      | uint32   |\n| [yggdrasil.uswitch.com/retry-on](#retries)                   | string   |\n\n### Health Check Path\nSpecifies a path to configure a [HTTP health check](https://www.envoyproxy.io/docs/envoy/v1.19.0/api-v3/config/core/v3/health_check.proto#config-core-v3-healthcheck-httphealthcheck) to. Envoy will not route to clusters that fail health checks.\n\n* [config.core.v3.HealthCheck.HttpHealthCheck.Path](https://www.envoyproxy.io/docs/envoy/v1.19.0/api-v3/config/core/v3/health_check.proto#envoy-v3-api-field-config-core-v3-healthcheck-httphealthcheck-path)\n\n### Timeouts\nAllows for adjusting the timeout in envoy.\n\nThe `yggdrasil.uswitch.com/cluster-timeout` annotation will set the [config.cluster.v3.Cluster.ConnectTimeout](https://www.envoyproxy.io/docs/envoy/v1.19.0/api-v3/config/cluster/v3/cluster.proto#envoy-v3-api-field-config-cluster-v3-cluster-connect-timeout)\n\nThe `yggdrasil.uswitch.com/route-timeout` annotation will set the [config.route.v3.RouteAction.Timeout](https://www.envoyproxy.io/docs/envoy/v1.19.0/api-v3/config/route/v3/route_components.proto#envoy-v3-api-field-config-route-v3-routeaction-timeout)\n\nthe `yggdrasil.uswitch.com/per-try-timeout` annotation will set the [config.route.v3.RetryPolicy.PerTryTimeout](https://www.envoyproxy.io/docs/envoy/v1.19.0/api-v3/config/route/v3/route_components.proto#envoy-v3-api-field-config-route-v3-retrypolicy-per-try-timeout)\n\nThe `yggdrasil.uswitch.com/timeout` annotation will set all of the above with the same value. This annotation has the lowest priority, if set with one of the other TO annotations, the specific one will override the general annotation.\n\n### Weight\nAllows for adjusting the [load balancer weights](https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/endpoint/v3/endpoint_components.proto#config-endpoint-v3-lbendpoint) in envoy.\n\n### Retries\nAllows overwriting the default retry policy's [config.route.v3.RetryPolicy.RetryOn](https://www.envoyproxy.io/docs/envoy/v1.19.0/api-v3/config/route/v3/route_components.proto#envoy-v3-api-field-config-route-v3-retrypolicy-retry-on) set by the `--retry-on` flag (default 5xx). Accepts a comma-separated list of retry-on policies.\n\n### Example\nBelow is an example of an ingress with some of the annotations specified\n\n```yaml\napiVersion: extensions/v1beta1\nkind: Ingress\nmetadata:\n  name: example-com\n  namespace: default\n  annotations:\n    yggdrasil.uswitch.com/healthcheck-path: /healthz\n    yggdrasil.uswitch.com/timeout: 30s\n    yggdrasil.uswitch.com/weight: \"12\"\n    yggdrasil.uswitch.com/retry-on: gateway-error,connect-failure\nspec:\n  rules:\n  - host: example.com\n    http:\n      paths:\n      - backend:\n          serviceName: example\n          servicePort: 80\n```\n\n## Dynamic TLS certificates synchronization from Kubernetes secrets\n\nDownstream TLS certificates can be dynamically fetched and updated from Kubernetes secrets configured under ingresses' `spec.tls` by setting `syncSecrets` true in Yggdrasil configuration (false by default).\n\nIn this mode, only a single `certificate` may be specified in Yggdrasil configuration. It will be used for hosts with misconfigured or invalid secret.\n\n**Note**: ECDSA \u003e256 keys are not supported by envoy and will be discarded. See https://github.com/envoyproxy/envoy/issues/10855\n\n## Configuration\nYggdrasil can be configured using a config file e.g:\n```json\n{\n  \"nodeName\": \"foo\",\n  \"ingressClasses\": [\"multi-cluster\", \"multi-cluster-staging\"],\n  \"syncSecrets\": false,\n  \"certificates\": [\n    {\n      \"hosts\": [\"*.api.com\"],\n      \"cert\": \"path/to/cert\",\n      \"key\": \"path/to/key\"\n    }\n  ],\n  \"clusters\": [\n    {\n      \"token\": \"xxxxxxxxxxxxxxxx\",\n      \"apiServer\": \"https://cluster1.api.com\",\n      \"ca\": \"pathto/cluster1/ca\"\n    },\n    {\n      \"tokenPath\": \"/path/to/a/token\",\n      \"apiServer\": \"https://cluster2.api.com\",\n      \"ca\": \"pathto/cluster2/ca\"\n    }\n  ]\n}\n```\n\nThe list of certificates will be loaded by Yggdrasil and served to the Envoy nodes by inlining the key pairs. These will then be used to group the ingress into different filter chains, split using hosts.\n\n`nodeName` is the same `node-name` that you start your envoy nodes with.\nThe `ingressClasses` is a list of ingress classes that yggdrasil will watch for.\nEach cluster represents a different Kubernetes cluster with the token being a service account token for that cluster. `ca` is the Path to the ca certificate for that cluster.\n\n## Metrics\nYggdrasil has a number of Go, gRPC, Prometheus, and Yggdrasil-specific metrics built in which can be reached by cURLing the `/metrics` path at the health API address/port (default: 8081). See [Flags](#Flags) for more information on configuring the health API address/port.\n\nThe Yggdrasil-specific metrics which are available from the API are:\n\n| Name                        | Description                                    | Type     |\n|-----------------------------|------------------------------------------------|----------|\n| yggdrasil_cluster_updates   | Number of times the clusters have been updated | counter  |\n| yggdrasil_clusters          | Total number of clusters generated             | gauge    |\n| yggdrasil_ingresses         | Total number of matching ingress objects       | gauge    |\n| yggdrasil_listener_updates  | Number of times the listener has been updated  | counter  |\n| yggdrasil_virtual_hosts     | Total number of virtual hosts generated        | gauge    |\n\n## Flags\n```\n--address string                              yggdrasil envoy control plane listen address (default \"0.0.0.0:8080\")\n--ca string                                   trustedCA\n--cert string                                 certfile\n--config string                               config file\n--config-dump                                 Enable config dump endpoint at /configdump on the health-address HTTP server\n--debug                                       Log at debug level\n--envoy-listener-ipv4-address string          IPv4 address by the envoy proxy to accept incoming connections (default \"0.0.0.0\")\n--envoy-port uint32                           port by the envoy proxy to accept incoming connections (default 10000)\n--health-address string                       yggdrasil health API listen address (default \"0.0.0.0:8081\")\n-h, --help                                        help for yggdrasil\n--host-selection-retry-attempts int           Number of host selection retry attempts. Set to value \u003e=0 to enable (default -1)\n--http-ext-authz-allow-partial-message        When this field is true, Envoy will buffer the message until max_request_bytes is reached (default true)\n--http-ext-authz-cluster string               The name of the upstream gRPC cluster\n--http-ext-authz-failure-mode-allow           Changes filters behaviour on errors (default true)\n--http-ext-authz-max-request-bytes uint32     Sets the maximum size of a message body that the filter will hold in memory (default 8192)\n--http-ext-authz-pack-as-bytes                When this field is true, Envoy will send the body as raw bytes.\n--http-ext-authz-timeout duration             The timeout for the gRPC request. This is the timeout for a specific request. (default 200ms)\n--http-grpc-logger-cluster string             The name of the upstream gRPC cluster\n--http-grpc-logger-name string                Name of the access log\n--http-grpc-logger-request-headers strings    access logs request headers\n--http-grpc-logger-response-headers strings   access logs response headers\n--http-grpc-logger-timeout duration           The timeout for the gRPC request (default 200ms)\n--ingress-classes strings                     Ingress classes to watch\n--key string                                  keyfile\n--kube-config stringArray                     Path to kube config\n--max-ejection-percentage int32               maximal percentage of hosts ejected via outlier detection. Set to \u003e=0 to activate outlier detection in envoy. (default -1)\n--node-name string                            envoy node name\n--retry-on string                             default comma-separated list of retry policies (default \"5xx\")\n--tracing-provider                            name of HTTP Connection Manager tracing provider to include - currently only zipkin config is supported\n--upstream-healthcheck-healthy uint32         number of successful healthchecks before the backend is considered healthy (default 3)\n--upstream-healthcheck-interval duration      duration of the upstream health check interval (default 10s)\n--upstream-healthcheck-timeout duration       timeout of the upstream healthchecks (default 5s)\n--upstream-healthcheck-unhealthy uint32       number of failed healthchecks before the backend is considered unhealthy (default 3)\n--upstream-port uint32                        port used to connect to the upstream ingresses (default 443)\n--use-remote-address                          populates the X-Forwarded-For header with the client address. Set to true when used as edge proxy\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuswitch%2Fyggdrasil","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fuswitch%2Fyggdrasil","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuswitch%2Fyggdrasil/lists"}