{"id":16846908,"url":"https://github.com/mumoshu/crossover","last_synced_at":"2025-07-23T23:39:06.323Z","repository":{"id":64297605,"uuid":"210658042","full_name":"mumoshu/crossover","owner":"mumoshu","description":"Minimal sufficient Envoy xDS for Kubernetes that knows https://smi-spec.io/ ","archived":false,"fork":false,"pushed_at":"2023-08-30T13:53:32.000Z","size":182,"stargazers_count":73,"open_issues_count":3,"forks_count":3,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-10-14T13:05:51.628Z","etag":null,"topics":["canary-deployment","envoy","helm","kubernetes","kubernetes-configmaps","service-mesh-interface","sidecar-containers","xds"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mumoshu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-24T17:16:28.000Z","updated_at":"2024-07-25T08:48:07.000Z","dependencies_parsed_at":"2024-06-20T05:42:11.190Z","dependency_job_id":"d91c3f67-9fc2-4664-bd1a-20cc0721e3bf","html_url":"https://github.com/mumoshu/crossover","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mumoshu%2Fcrossover","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mumoshu%2Fcrossover/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mumoshu%2Fcrossover/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mumoshu%2Fcrossover/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mumoshu","download_url":"https://codeload.github.com/mumoshu/crossover/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243846976,"owners_count":20357294,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["canary-deployment","envoy","helm","kubernetes","kubernetes-configmaps","service-mesh-interface","sidecar-containers","xds"],"created_at":"2024-10-13T13:05:50.228Z","updated_at":"2025-03-17T05:32:04.780Z","avatar_url":"https://github.com/mumoshu.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Crossover\n\n[![CircleCI](https://circleci.com/gh/mumoshu/crossover.svg?style=svg)](https://circleci.com/gh/mumoshu/crossover)\n\n[![dockeri.co](https://dockeri.co/image/mumoshu/crossover)](https://hub.docker.com/r/mumoshu/crossover)\n\n`Crossover` is **the minimal and sufficient xDS for [Envoy](https://www.envoyproxy.io/)** that helps Canary Deployment, without sacrificing Envoy's rich feature set.\n\n## Features\n\n- **Minimal Dependencies**\n- **Gradual Migration**\n- **Understandable**\n- **Feature Complete**: Access to every feature Envoy provides. `crossover` makes no leaky abstraction on top.\n\n### Minimal Dependencies\n\nDepends only on the Go standard library and [go-yaml](https://github.com/go-yaml/yaml).\n\nNo need for `kubectl`, `client-go`, `apimachineary` or even Envoy's `go-control-plane`.\n\n### Gradual Migration\n\nYou always start with a standard Envoy without `crossover`. If you're happy with that, keep using it and don't bother with adding additional moving part to the mix.\n\nCanary deployments with [Flagger](https://github.com/weaveworks/flagger)?\n\nReconfigure Envoy without time-consuming and unreliable reloads?\n\nEnter `crossover`.\n\nAdd `crossover` as a sidecar container to your Envoy. Update configmaps and trafficsplist via kubectl or helm to load changes into Envoy in near-realtime. That's all you need to get started, really!\n\n### Understandable\n\nNo gRPC, REST server or serious K8s controller to maintain and debug.\n\n`crossover` is a simple golang program that feeds only necessary parts of the config to Envoy via xDS.\n\nIt fetches well-known K8s `ConfigMap` and SMI `TrafficSplit` resources via Kubernetes' REST API, write config files for Envoy, rename the files so that Envoy can atomically reconfigure itself.\n\nFrom Envoy's perspective, there's just xDS data stored at `/srv/runtime/current/*.yaml` visible from within Envoy, that are read from Envoy's standard `local file` config-source.\n\n### Feature Complete\n\nAccess to every feature Envoy provides. `crossover` makes no leaky abstraction on top of Envoy.\n\nYou write regular Envoy config files. `crossover` just merges and syncs it to Envoy for you. \n\n## Design\n\n```\n\n[ Envoy ]--reads--\u003e[ xDS YAML file ]\u003c--writes--[ Crossover ]--reads--\u003e[ ConfigMap, TrafficSplit ]\u003c--write--[ Kubectl, Helm, etc. ]\n```\n\n[Envoy](https://www.envoyproxy.io/) is able to reconfigure itself at runtime by reading local files or by querying one or more management servers called xDS servers.\n\n`crossover` is an implementation of xDS that translates configs stored in Kubernetes to Envoy primitives.\n\nIt is an init container and a sidecar container for Envoy, deployed to your Kubernetes cluster along with Envoy, serving xDS.\n\nIt continuously reads Kubernetes ConfigMap and [SMI TrafficSplit](https://github.com/deislabs/smi-spec/blob/master/traffic-split.md) resources, to generate and feed xDS resources to Envoy via the local filesystem.\n\nAs it relies solely on the local filesystem to communicate with Envoy, it is transparent to the operator. You don't need any experience in gRPC, which is typically required for managing famous xDS implementations. Just `kubectl exec` into the Envoy pod and `cat` standard Envoy config files generated by the loader to debug. \n\nThanks to Envoy's inotify support and Kubernetes' Watch API, any changes made via ConfigMap and TrafficSplit are still recongnizable by Envoy in near real-time.\n\n## Why not...\n\n[Why not use configmap volumes?](https://github.com/mumoshu/crossover#why-not-use-configmap-volumes) Because it doesn't apply changes to the local disk in [a way Envoy wants](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/runtime.html).\n\nWhy not write a gRPC server that translates SMI resources to xDS resources? Because I don't want to introduce a leaky abstraction.\n\n## Use-cases\n\n- Ingress Gateway\n- Canary Releases\n- In-Cluster Router/Load-Balancer\n\n### Ingress Gateway\n\nTurn [stable/envoy](https://github.com/helm/charts/tree/master/stable/envoy) chart into a dynamically configurable API Gateway, Ingress Gateway or Front Proxy\n\n### Canary Releases\n\nDo weighted load-balancing and canary deployments with zero Envoy restart, redeployment and CRD. [Just distribute RDS(Route Discovery Service) files via configmaps!](https://www.envoyproxy.io/learn/incremental-deploys#weighted-load-balancing).\n\n### In-Cluster Router/Load-Balancer\n\nWanna add retries, circuit-breakers, tracing, metrics to your traffic? Deploy Envoy paired with `crossover` in front of your apps. No need for service meshes from day 1.\n\n## What's this?\n\n`crossover` is an init-container AND a sidecar for your Envoy proxy to use K8s ConfigMaps as xDS backend.\n\nThis works by loading kvs defined within specified configmap(s) and writing files assuming the key is the filename and the value is the content.\n\nYou then point your Envoy to read xDS from the directory `/srv/runtime/*.yaml`.\n\n`crossover` writes files read from configmap(s) into the directory, triggers [symlink swap](https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#updating-runtime-values-via-symbolic-link-swap)\n so that Envoy finally detects and applies changes. \n \n ## Why not use configmap volumes?\n \nYou may [already know that K8s supports mounting configmaps as container volumes out of the box](https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#add-configmap-data-to-a-volume).\n\nThe downside of using that feature to feed Envoy xDS files is that it takes 1 minute(default, configurable via kubelet `--sync-interval`) a change is reflected to the volume.\n\nAnd more importantly, Envoy is unable to detect changes made in configmap volumes due to that it relies on `inotify` `MOVE` events to occur, where configmap volume changes only trigger the below events:\n\n```\nroot@envoy-675dc8d98b-tvw9b:/# inotifywait -m /xds/rds.yaml\nSetting up watches.\nWatches established.\n/xds/rds.yaml OPEN\n/xds/rds.yaml ACCESS\n/xds/rds.yaml CLOSE_NOWRITE,CLOSE\n/xds/rds.yaml ATTRIB\n/xds/rds.yaml DELETE_SELF\n```\n\nSo in nutshell, `crossover` is the minimal and sufficient companion to actually distribute xDS data via configmaps, without using more advanced CRD-based solutions like Istio and VMWare Contour.\n\n## Usage\n\n```console\n$ crossover -h\nUsage of ./crossover:\n  -apiserver string\n    \tK8s api endpoint (default \"https://kubernetes\")\n  -configmap value\n    \tthe configmap to process.\n  -dry-run\n    \tprint processed configmaps and secrets and do not submit them to the cluster.\n  -insecure\n    \tdisable tls server verification\n  -namespace string\n    \tthe namespace to process.\n  -onetime\n    \trun one time and exit.\n  -output-dir string\n    \tDirectory to putput xDS configs so that Envoy can read\n  -smi\n    \tEnable SMI integration\n  -sync-interval duration\n    \tthe time duration between template processing. (default 1m0s)\n  -token-file string\n    \tpath to serviceaccount token file (default \"/var/run/secrets/kubernetes.io/serviceaccount/token\")\n  -trafficsplit value\n    \tthe trafficsplit to be watched and merged into the configmap\n  -watch\n    \tuse watch api to detect changes near realtime\n```\n\n## Getting Started\n\n### ConfigMap-Only Mode\n\nTry weighted load-balancing using `crossover`!\n\nDeploy Envoy along with the loader using the `stable/envoy` chart:\n\n```\nhelm repo add stable https://kubernetes-charts.storage.googleapis.com\nhelm upgrade --install envoy stable/envoy -f example/values.yaml -f example/values.services.yaml\n```\n\nOr by using `crossover/envoy` chart:\n\n```\nhelm repo add crossover https://mumoshu.github.io/crossover\nhelm upgrade --install envoy crossover/envoy -f example/values.upstreams.yaml\n```\n\nThen install backends - we use @stefanprodan's awesome [podinfo](https://github.com/stefanprodan/podinfo):\n\n```\nhelm repo add flagger https://flagger.app\nhelm upgrade --install bold-olm flagger/podinfo --set canary.enabled=false\nhelm upgrade --install eerie-octopus flagger/podinfo --set canary.enabled=false\n```\n\nIn another terminal, run the tester pod to watch traffic shifts:\n\n```\nkubectl run -it --rm --image alpine:3.9 tester sh\n\napk add --update curl\nwatch curl http://envoy:10000\n```\n\nFinally, try changing load-balancing weights instantly and without restarting Envoy at all:\n\n```\n# 100% bold-olm\nhelm upgrade --install envoy stable/envoy -f example/values.yaml -f example/values.services.yaml \\\n  --set services.podinfo.backends.eerie-octopus-podinfo.weight=0 \\\n  --set services.podinfo.backends.bold-olm-podinfo.weight=100\n\n# 100% eerie-octopus\nhelm upgrade --install envoy stable/envoy -f example/values.yaml -f example/values.services.yaml \\\n  --set services.podinfo.backends.eerie-octopus-podinfo.weight=100 \\\n  --set services.podinfo.backends.bold-olm-podinfo.weight=0\n```\n\nSee [example/values.yaml](example/values.yaml) for more details on the configuration.\n\n### ConfigMap + SMI TrafficSplit Mode\n\nThe setup is mostly similar to that for CofigMap-only mode.\n\nJust add `services.podinfo.smi.enabled=true` while installing Envoy:\n\n```\nhelm upgrade --install envoy stable/envoy \\\n  -f example/values.yaml -f example/values.services.yaml \\\n  --set services.podinfo.smi.enabled=true\n```\n\nOr with the `crossover/envoy` chart:\n\n```\nhelm repo add crossover https://mumoshu.github.com/crossover\nhelm upgrade --install envoy crossover/envoy -f example/values.services.yaml \\\n  --set services.podinfo.smi.enabled=true\n```\n\nOr to be extra sure, you can do it wihout `example/values.services.yaml`:\n\n```\nhelm upgrade --install envoy stable/envoy   --namespace test   -f example/values.yaml   -f \u003c(cat \u003c\u003cEOF\nservices:\n  podinfo:\n    smi:\n      enabled: true\n    backends:\n      eerie-octopus-podinfo:\n        port: 9898\n        weight: 50\n      bold-olm-podinfo:\n        port: 9898\n        weight: 50\nEOF\n)\n```\n\nNow you are ready to change weights by creating and modifying TrafficSplit like this:\n\n```\napiVersion: split.smi-spec.io/v1alpha2\nkind: TrafficSplit\nmetadata:\n  name: podinfo\nspec:\n  # The root service that clients use to connect to the destination application.\n  service: podinfo\n  # Services inside the namespace with their own selectors, endpoints and configuration.\n  backends:\n  - service: eerie-octopus-podinfo\n    weight: 0\n  - service: bold-olm-podinfo\n    weight: 100\n```\n\nUnder the hood, `crossover` reads `podinfo` trafficsplit and `envoy-xds` configmap, merges the trafficsplit into the configmap to produce the final configmap `envoy-xds-gen`. It is `envoy-xds-gen` which is loaded into `envoy`. \n\nFor convenience, there are several manifest files each with different set of weights:\n\n```\n# 100% bold-olm\nkubectl apply -f podinfo-v0.trafficsplit.yaml\n\n# 75% bold-olm 25% eerie-octopus\nkubectl apply -f podinfo-v1.trafficsplit.yaml\n\n#...\n\n# 0% bold-olm 100% eerie-octupus\nkubectl apply -f podinfo-v4.trafficsplit.yaml\n```\n\n## Developing\n\nBring your own K8s cluster, move to the project root, and run the following commands to give it a ride:\n\n```\nsudo mkdir /srv/runtime\nsudo chmod -R 777 /srv/runtime\nk get secret -o json $(k get secret | grep default-token | awk '{print $1 }') | jq -r .data.token | base64 -D \u003e mytoken\nexport APISERVER=$(k config view --minify -o json | jq -r .clusters[0].cluster.server)\nmake build \u0026\u0026 ./crossover --namespace default --token-file ./mytoken --configmap incendiary-shark-envoy-xds --onetime --insecure --apiserver \"http://127.0.0.1:8001\"\n```\n\n## FAQ\n\n### Why is the init container needed?\n\n`crossover` needs to be included in your pod not only as a sidecar but also as an init container.\n\nThat's because Envoy checks for the existence of xDS-managed config files on startup. That is, if any of xDS-managed config files that are\nreferenced from Envoy's bootstrap config is missing on startup, Envoy fails starting.\n\nOnce Kubernetes adds [the first-class support for sidecar containers](https://github.com/kubernetes/enhancements/issues/753), the requisite of the init container is likely to go away.\n\nAnyways, `crossover` is very resource-efficient in terms of image size and memory, it shouldn't be a huge problem.\n\n## References\n\n### Technical information to use Envoy's dynamic runtime config via local files\n\n- [File Based Dynamic Configuration of Routes in Envoy Proxy](https://medium.com/grensesnittet/file-based-dynamic-configuration-of-routes-in-envoy-proxy-6234dae968d2)\n- [\"How does one atomically change a symlink to a directory in busybox?\"](https://unix.stackexchange.com/questions/5093/how-does-one-atomically-change-a-symlink-to-a-directory-in-busybox)\n- [Runtime configuration — envoy](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/runtime.html)\n\n### Other Envoy xDS implementations\n\nKubernetes CRD and gRPC server based implementations\n\n- [Istio](https://istio.io/)\n- [Contour](https://github.com/projectcontour/contour)\n- [Ambassador](https://github.com/datawire/ambassador)\n- [Gloo](https://github.com/solo-io/gloo)\n- [flagger-appmesh-gateway](https://github.com/stefanprodan/flagger-appmesh-gateway)\n\nConsul and gRPC server based implementations\n\n- [gojek/consul-envoy-xds](https://github.com/gojek/consul-envoy-xds)\n- [tak2siva/Envoy-Pilot](https://github.com/tak2siva/Envoy-Pilot)\n\nKubernetes Services and Endpoints with gRPC server based implementations\n\n- [jrockway/ekglue](https://github.com/jrockway/ekglue)\n\nConfig file and gRPC server based implementations\n\n- [QubitProducts/iris](https://github.com/QubitProducts/iris)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmumoshu%2Fcrossover","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmumoshu%2Fcrossover","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmumoshu%2Fcrossover/lists"}