{"id":13646000,"url":"https://github.com/monzo/egress-operator","last_synced_at":"2025-04-12T15:35:14.461Z","repository":{"id":52433929,"uuid":"227243365","full_name":"monzo/egress-operator","owner":"monzo","description":"A Kubernetes operator to produce egress gateway Envoy pods and control access to them with network policies","archived":false,"fork":false,"pushed_at":"2025-01-10T16:14:12.000Z","size":16383,"stargazers_count":252,"open_issues_count":8,"forks_count":27,"subscribers_count":69,"default_branch":"master","last_synced_at":"2025-04-03T15:09:32.967Z","etag":null,"topics":["egress","envoy","kubernetes","networking","operator","security"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/monzo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-12-11T00:43:19.000Z","updated_at":"2025-04-02T14:04:42.000Z","dependencies_parsed_at":"2024-01-04T10:31:04.252Z","dependency_job_id":"7711825f-32b4-44c0-9ab4-d8256bdfa8ea","html_url":"https://github.com/monzo/egress-operator","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monzo%2Fegress-operator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monzo%2Fegress-operator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monzo%2Fegress-operator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monzo%2Fegress-operator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/monzo","download_url":"https://codeload.github.com/monzo/egress-operator/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248590111,"owners_count":21129751,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["egress","envoy","kubernetes","networking","operator","security"],"created_at":"2024-08-02T01:02:46.603Z","updated_at":"2025-04-12T15:35:14.441Z","avatar_url":"https://github.com/monzo.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# egress-operator\nAn operator to produce egress gateway pods and control access to them with network policies, and a coredns plugin to route egress traffic to these pods.\n\nThe idea is that instead of authorizing egress traffic with protocol inspection,\nyou instead create a internal clusterIP for every external service you use, lock\nit down to only a few pods via a network policy, and then set up your dns server\nto resolve the external service to that clusterIP.\n\nBuilt with kubebuilder: https://book.kubebuilder.io/\n\nThe operator accepts ExternalService objects, which aren't namespaced, which define a dns name and ports for an external service.\nIn the `egress-operator-system` namespace, it creates:\n- An envoy configmap for a TCP/UDP proxy to that service (UDP not working until the next envoy release that enables it)\n- A deployment for some envoy pods with that config\n- A horizontal pod autoscaler to keep the deployment correctly sized\n- A service for that deployment\n- A network policy only allowing pods in other namespaces with the label `egress.monzo.com/allowed-\u003cyourservice\u003e: true`\n\n## Pre-requisites\n\n1. You need to have a private container repository for hosting the egress-operator image, such as an AWS Elastic Container Repository (ECR) or a GCP Container Registry (GCR), which needs to be accessible from your cluster. This will be referred to as `yourrepo` in the instructions below.\n2. Your local system must have a recent version of `golang` for building the code, which you can install by following instructions [here](https://golang.org/doc/install).\n3. Your local system must have Kubebuilder for code generation, which you can install by following instructions [here](https://book.kubebuilder.io/quick-start.html).\n4. Your local system must have Kustomize for building the Kubernetes manifests, which you can install by following instructions [here](https://kubernetes-sigs.github.io/kustomize/installation/).\n5. Your cluster must be running CoreDNS instead of kube-dns, which may not be the case if you are using a managed Kubernetes service. [This article](https://medium.com/google-cloud/using-coredns-on-gke-3973598ab561) provides some help for GCP Kubernetes Engine, and guidance for AWS Elastic Kubernetes Service can be found [here](https://docs.aws.amazon.com/eks/latest/userguide/coredns.html).\n\n## Installing\n\n### Testing locally against a remote cluster\n\n```bash\nmake run\n```\nThis creates an ExternalService object to see the controller-manager creating managed resources in the remote cluster.\n\n### Setting up CoreDNS plugin\n\nThe CoreDNS plugin rewrites responses for external service hostnames managed by egress-operator.\n\nBuild a CoreDNS image which contains the plugin:\n```bash\ncd coredns-plugin\nmake docker-build docker-push IMG=yourrepo/egress-operator-coredns:latest\n```\n\nYou'll need to swap out the image of your coredns kubedns Deployment for `yourrepo/egress-operator-coredns:latest`:\n```bash\nkubectl edit deploy coredns -n kube-system   # Your Deployment name may vary\n```\n\nAnd edit the coredns Corefile in ConfigMap to put in `egressoperator egress-operator-system cluster.local`:\n```bash\nkubectl edit configmap coredns-config -n kube-system   # Your ConfigMap name may vary\n```\n\nExample CoreDNS config:\n\n```Caddy\n.:53 {\n    egressoperator egress-operator-system cluster.local\n    kubernetes cluster.local\n    forward . /etc/resolv.conf\n}\n```\n\n### Set up the controller manager and its `CustomResourceDefinition` in the cluster\n\n```\nmake docker-build docker-push install IMG=yourrepo/egress-operator:v0.1\nmake deploy IMG=yourrepo/egress-operator:v0.1\n```\n\n## Usage\n\nOnce the controller and dns server are running, create ExternalService objects which denote what dns name you want\nto capture traffic for. Dns queries for this name will be rewritten to point to gateway pods.\n\nBy default, your client pods need a label `egress.monzo.com/allowed-gateway: nameofgateway` to be able to reach\nthe destination, but you can always write an additional NetworkPolicy selecting gateway pods and allowing all traffic,\nfor testing purposes.\n\nAn example ExternalService:\n\n```yaml\napiVersion: egress.monzo.com/v1\nkind: ExternalService\nmetadata:\n  name: google\nspec:\n  dnsName: google.com\n  # optional, defaults to false, instructs dns server to rewrite queries for dnsName\n  hijackDns: true\n  ports:\n  - port: 443\n    # optional, defaults to TCP\n    protocol: TCP\n  # optional, defaults to 3\n  minReplicas: 5\n  # optional, defaults to 12\n  maxReplicas: 10\n  # optional, defaults to 50\n  targetCPUUtilizationPercentage: 30\n  # optional, if not provided then defaults to 100m, 50Mi, 2, 1Gi\n  resources:\n    requests:\n      cpu: 1\n      memory: 100Mi\n    limits:\n      cpu: 2\n      memory: 200Mi\n```\n\n### Blocking non-gateway traffic\n\nThis operator won't block any traffic for you, it simply sets up some permitted routes for traffic through the egress\ngateways. You'll need a default-deny policy to block traffic that doesn't go through gateways. To do that, you probably\nneed a policy like this in every namespace that you want to control:\n\n```yaml\napiVersion: networking.k8s.io/v1\nkind: NetworkPolicy\nmetadata:\n  name: default-deny-external-egress\n  namespace: your-application-namespace\nspec:\n  podSelector: {}\n  policyTypes:\n  - Egress\n  egress:\n  - to:\n    - ipBlock:\n        # ensure your internal IP range is allowed here\n        # traffic to external IPs will not be allowed from this namespace.\n        # therefore, pods will have to use egress gateways\n        cidr: 10.0.0.0/8\n```\n\nIf you already have a default deny egress policy, the above won't be needed. You'll instead want to explicitly allow\negress from your pods to all gateway pods. The ingress policies on gateway pods will ensure that only correct traffic is\nallowed.\n\n### Configuration\n\nGlobal configuration of the operator is set using environment variables.\n\nNode Selectors and Taint tolerations can be added to gateway pods to ensure pods\nrun on nodes that are permitted to access the internet. Example:\n\n```yaml\nenv:\n- name: NODE_SELECTOR_KEY\n  value: role\n- name: NODE_SELECTOR_VALUE\n  value: egress-pods\n- name: TAINT_TOLERATION_KEY\n  value: egress-pods\n- name: TAINT_TOLERATION_VALUE\n  value: \"true\"\n```\n\nResults in this gateway pod configuration:\n\n```yaml\nspec:\n  nodeSelector:\n    role: egress-pods\n  tolerations:\n  - effect: NoSchedule\n    key: egress-pods\n    value: \"true\"\n  ...\n```\n\n**You can also configure the [pod topology spread constraint](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/)**\n\n```yaml\nenv:\n- name: ENABLE_POD_TOPOLOGY_SPREAD\n  value: \"true\"\n- name: POD_TOPOLOGY_ZONE_MAX_SKEW\n  value: 1\n- name: POD_TOPOLOGY_HOSTNAME_MAX_SKEW\n  value: 1\n```\n\nThis will inject a topology spread constraint into the gateway pods, which will\nensure that pods are spread across zones and hosts.\n\n```yaml\nspec:\n  topologySpreadConstraints:\n    - labelSelector:\n        matchLabels:\n          app: egress-gateway\n          egress.monzo.com/gateway: egress-gateway-name\n      maxSkew: 1\n      topologyKey: topology.kubernetes.io/zone\n      whenUnsatisfiable: ScheduleAnyway\n    - labelSelector:\n        matchLabels:\n          app: egress-gateway\n          egress.monzo.com/gateway: egress-gateway-name\n```\n\nYou can also setup services to have their endpoints aware of network topologies with [topology aware routing](https://kubernetes.io/docs/concepts/services-networking/topology-aware-routing/).\n\nIf you set the environment variable `ENABLE_SERVICE_TOPOLOGY_MODE=\"true\"` this will enable setting the annotations `service.kubernetes.io/topology-mode=\"auto\"`.\n\nIf you turn this on you can also set `spec.serviceTopologyMode` to any value if you want to disable this for particular egress'. For example\nif you set `spec.serviceTopologyMode: \"None\"` this will disable topology aware routing for your ExternalService.\n\n| Variable name                      | Default                                   | Description                                               |\n|------------------------------------|-------------------------------------------|-----------------------------------------------------------|\n| ENVOY_IMAGE                        | `envoyproxy/envoy-alpine:v1.16.5`         | Name of the Envoy Proxy image to use                      |\n| TAINT_TOLERATION_KEY               | Empty, no tolerations applied             | Toleration key to apply to gateway pods                   |\n| TAINT_TOLERATION_VALUE             | Empty, no tolerations applied             | Toleration value to apply to gateway pods                 |\n| NODE_SELECTOR_KEY                  | Empty, no node selector added             | Node selector label key to apply to gateway pods          |\n| NODE_SELECTOR_VALUE                | Empty, no node selector added             | Node selector label value to apply to gateway pods        |\n| POD_TOPOLOGY_ZONE_MAX_SKEW_KEY     | `topology.kubernetes.io/zone`             | Topology key for the zone constraint                      |\n| POD_TOPOLOGY_ZONE_MAX_SKEW         | Empty, won't inject a zone constraint     | Value of maxSkew for the zone constraint                  |\n| POD_TOPOLOGY_HOSTNAME_MAX_SKEW_KEY | `kubernetes.io/hostname`                  | Topology key for the hostname constraint                  |\n| POD_TOPOLOGY_HOSTNAME_MAX_SKEW     | Empty, won't inject a hostname constraint | Value of maxSkew for the hostname constraint              |\n| ENABLE_SERVICE_TOPOLOGY_MODE       | Empty, won't add the annotation           | Set to 'true' to add the topology mode service annotation |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmonzo%2Fegress-operator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmonzo%2Fegress-operator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmonzo%2Fegress-operator/lists"}