{"id":13750492,"url":"https://github.com/salrashid123/envoy_control","last_synced_at":"2025-04-09T13:08:30.339Z","repository":{"id":49198638,"uuid":"131372602","full_name":"salrashid123/envoy_control","owner":"salrashid123","description":"Sample Envoy control plane 'hello world'","archived":false,"fork":false,"pushed_at":"2024-11-20T14:17:33.000Z","size":228,"stargazers_count":66,"open_issues_count":6,"forks_count":23,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-02T11:08:10.703Z","etag":null,"topics":["envoy","envoyproxy"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/salrashid123.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-04-28T04:26:50.000Z","updated_at":"2025-02-10T06:45:44.000Z","dependencies_parsed_at":"2024-06-18T22:32:28.581Z","dependency_job_id":"993fcad6-5207-4cc1-8472-d99ada6a9f80","html_url":"https://github.com/salrashid123/envoy_control","commit_stats":{"total_commits":14,"total_committers":1,"mean_commits":14.0,"dds":0.0,"last_synced_commit":"86f4eaad9f2acd46bac019cb0620dfbb66940820"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salrashid123%2Fenvoy_control","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salrashid123%2Fenvoy_control/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salrashid123%2Fenvoy_control/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salrashid123%2Fenvoy_control/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/salrashid123","download_url":"https://codeload.github.com/salrashid123/envoy_control/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248045233,"owners_count":21038553,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["envoy","envoyproxy"],"created_at":"2024-08-03T08:00:36.708Z","updated_at":"2025-04-09T13:08:30.289Z","avatar_url":"https://github.com/salrashid123.png","language":"Go","funding_links":[],"categories":["Documentation"],"sub_categories":[],"readme":"# Envoy control plane \"hello world\"\n\n\nA couple of weeks ago  (update:  `11/20/20` _years ago_!) i wanted to program and understand how the control plane for [Envoy Proxy](https://www.envoyproxy.io/) works.\nI know its used in various comprehensive control systems like [Istio](https://www.istio.io/) and ofcourse at Lyft.\n\nThis repo/article describes a sample golang control plane for an Envoy Proxy.  It demonstrates its dynamic configuration by \ngetting a specific pre-detetermined setting set push to each proxy at runtime.  \n\nFor reference, the basic equivalent envoy `yaml` file that this xds server emulates is contained in `bbc.yaml`\n\nWe will be using \n\n* [Envoy v3(https://www.envoyproxy.io/docs/envoy/latest/api/api_supported_versions) \n* [Aggregate Discovery Service](https://www.envoyproxy.io/docs/envoy/latest/api-v3/service/discovery/v3/ads.proto)\n  ADS will return the Cluster, Listener and Secrets all in one Snapshot\n\nThat is, once Envoy is started, it reads in an empty configuration which only tells it where the control plane gRPC server exists.\n\nAfter connecting to the control plane, it receives configuration information to setup an upstream cluster, listener and secret set.  The \nspecific listener and cluster is trivial:  it merely proxies a request for `https://www.bbc.com/robots.txt`, then after a minute, it will update the envoy config to proxy for `www.yahoo.com/robots`  then finally after another minute, `blog.salrashid.me`.\n\nThe Secrets are distributed are basically just a TLS certificate for envoy:\n\n`client`-\u003e TLS -\u003e `envoy` -\u003e `upstream` in three ways:\n\n* TLS certs are provided in three ways\n\n1. ADS server will directly return embedded TLS configuration with inline certificates\n\n2. ADS server will return the reference value for the `Secret` to use.  The secret value is a static local reference to the certificates\n\n3. ADS server will return the `Secret` reference as well as the `Secret` itself.\n\n\n\n\u003e Note: much of the code and config i got here is taken from the Envoy [integration test suite](https://github.com/envoyproxy/go-control-plane/tree/master/pkg/test/main)\n\n\n## Additional Reading\n\n- [Matt Klien's blog](https://blog.envoyproxy.io/the-universal-data-plane-api-d15cec7a)\n- [Envoy Configuration Guide](https://www.envoyproxy.io/docs/envoy/latest/configuration/overview/v2_overview#)\n- [Envoy xDS data plane API](https://github.com/envoyproxy/data-plane-api/blob/master/XDS_PROTOCOL.md)\n- [Envoy golang control plane](https://github.com/envoyproxy/go-control-plane)\n- [Envoy java control plane](https://github.com/envoyproxy/java-control-plane)\n- [Control Plane Sample](https://github.com/uswitch/yggdrasil/blob/master/pkg/envoy/boilerplate.go)\n\n![images/envoy.png](images/envoy.png)\n\n---\n\n## Setup\n\n\n### Start Control Plane\n\ndo just this:\n\n```bash\n$ go run src/main.go \n\nINFO[0000] Starting control plane                       \nINFO[0000] management server listening                   port=18000\n```\n\nThe code is almost entirely contained in [src/main.go](src/main.go) which launches the control plane and proceeds to setup a static config to proxy to a set of `/robots.txt` files from three sites:\n\n```golang\n[]string{\"www.bbc.com\", \"www.yahoo.com\", \"blog.salrashid.me\"}\n```\n\nEvery 60 seconds, the host will rotate over which means for the first 60 seconds, you'll see the robots.txt file from bbc, then yahoo then google.\n\nNote, we increment the snapshot version number and the host as well after envoy node connects:\n\n```log\n$ go run  src/main.go \n\nINFO[0000] Starting control plane                       \nINFO[0000] management server listening                   port=18000\nINFO[0001] OnStreamOpen 1 open for                      \nINFO[0001] OnStreamRequest type.googleapis.com/envoy.config.cluster.v3.Cluster \nINFO[0001] \u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e creating cluster, remoteHost, nodeID service_bbc,  www.bbc.com, test-id \nINFO[0001] \u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e creating listener listener_0 \nINFO[0001] \u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e creating Secret server_cert \nINFO[0001] \u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e creating snapshot Version 1 \nINFO[0001] OnStreamResponse...                          \nINFO[0001] cb.Report()  callbacks                        fetches=0 requests=1\nINFO[0001] OnStreamRequest type.googleapis.com/envoy.config.cluster.v3.Cluster \nINFO[0001] OnStreamRequest type.googleapis.com/envoy.config.listener.v3.Listener \nINFO[0001] OnStreamResponse...                          \nINFO[0001] cb.Report()  callbacks                        fetches=0 requests=3\nINFO[0001] OnStreamRequest type.googleapis.com/envoy.config.listener.v3.Listener \nINFO[0061] \u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e creating cluster, remoteHost, nodeID service_bbc,  www.yahoo.com, test-id \nINFO[0061] \u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e creating listener listener_0 \nINFO[0061] \u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e creating Secret server_cert \nINFO[0061] \u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e creating snapshot Version 2 \nINFO[0061] OnStreamResponse...                          \nINFO[0061] cb.Report()  callbacks                        fetches=0 requests=4\n```\n\nYou can review the code to see how the structure is nested and initialized.\n\nIf you just set the value to bbc and not iterate, the code will behave as if [bbc.yaml](bbc.yaml) config file was passed to envoy:\n\n### Run Envoy\n\nTo run envoy, just download a local envoy\n\n\n``` bash\n   docker cp `docker create envoyproxy/envoy-dev:latest`:/usr/local/bin/envoy .\n```\n\nThen invoke\n\n```\n./envoy -c baseline.yaml -l debug\n```\n\n### Access proxy\n\n```bash\ncurl  -H \"Host: http.domain.com\" \\\n   --resolve  http.domain.com:10000:127.0.0.1 \\\n   --cacert certs/root-ca.crt https://http.domain.com:10000/\n```\n\n---\n\n\n#### Create Cluster\n\n```golang\n\t\tvar clusterName = \"service_bbc\"\n\t\tvar remoteHost = v\n\n\t\tlog.Infof(\"\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e creating cluster, remoteHost, nodeID %s,  %s, %s\", clusterName, v, nodeId)\n\n\t\thst := \u0026core.Address{Address: \u0026core.Address_SocketAddress{\n\t\t\tSocketAddress: \u0026core.SocketAddress{\n\t\t\t\tAddress:  remoteHost,\n\t\t\t\tProtocol: core.SocketAddress_TCP,\n\t\t\t\tPortSpecifier: \u0026core.SocketAddress_PortValue{\n\t\t\t\t\tPortValue: uint32(443),\n\t\t\t\t},\n\t\t\t},\n\t\t}}\n\t\tuctx := \u0026envoy_api_v2_auth.UpstreamTlsContext{}\n\t\ttctx, err := ptypes.MarshalAny(uctx)\n\t\tif err != nil {\n\t\t\tlog.Fatal(err)\n\t\t}\n\n\t\tc := []types.Resource{\n\t\t\t\u0026cluster.Cluster{\n\t\t\t\tName:                 clusterName,\n\t\t\t\tConnectTimeout:       ptypes.DurationProto(2 * time.Second),\n\t\t\t\tClusterDiscoveryType: \u0026cluster.Cluster_Type{Type: cluster.Cluster_LOGICAL_DNS},\n\t\t\t\tDnsLookupFamily:      cluster.Cluster_V4_ONLY,\n\t\t\t\tLbPolicy:             cluster.Cluster_ROUND_ROBIN,\n\t\t\t\tLoadAssignment: \u0026endpoint.ClusterLoadAssignment{\n\t\t\t\t\tClusterName: clusterName,\n\t\t\t\t\tEndpoints: []*endpoint.LocalityLbEndpoints{{\n\t\t\t\t\t\tLbEndpoints: []*endpoint.LbEndpoint{\n\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\tHostIdentifier: \u0026endpoint.LbEndpoint_Endpoint{\n\t\t\t\t\t\t\t\t\tEndpoint: \u0026endpoint.Endpoint{\n\t\t\t\t\t\t\t\t\t\tAddress: hst,\n\t\t\t\t\t\t\t\t\t}},\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t},\n\t\t\t\t\t}},\n\t\t\t\t},\n\t\t\t\tTransportSocket: \u0026core.TransportSocket{\n\t\t\t\t\tName: \"envoy.transport_sockets.tls\",\n\t\t\t\t\tConfigType: \u0026core.TransportSocket_TypedConfig{\n\t\t\t\t\t\tTypedConfig: tctx,\n\t\t\t\t\t},\n\t\t\t\t},\n\t\t\t},\n\t\t}\n```\n\n#### Create Listener\n\nCreate a basic listener on port `10000`\n\n```golang\n\t\tvar listenerName = \"listener_0\"\n\t\tvar targetHost = v\n\t\tvar targetPrefix = \"/\"\n\t\tvar virtualHostName = \"local_service\"\n\t\tvar routeConfigName = \"local_route\"\n\n\t\tlog.Infof(\"\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e creating listener \" + listenerName)\n\n\t\trte := \u0026route.RouteConfiguration{\n\t\t\tName: routeConfigName,\n\t\t\tVirtualHosts: []*route.VirtualHost{{\n\t\t\t\tName:    virtualHostName,\n\t\t\t\tDomains: []string{\"*\"},\n\t\t\t\tRoutes: []*route.Route{{\n\t\t\t\t\tMatch: \u0026route.RouteMatch{\n\t\t\t\t\t\tPathSpecifier: \u0026route.RouteMatch_Prefix{\n\t\t\t\t\t\t\tPrefix: targetPrefix,\n\t\t\t\t\t\t},\n\t\t\t\t\t},\n\t\t\t\t\tAction: \u0026route.Route_Route{\n\t\t\t\t\t\tRoute: \u0026route.RouteAction{\n\t\t\t\t\t\t\tClusterSpecifier: \u0026route.RouteAction_Cluster{\n\t\t\t\t\t\t\t\tCluster: clusterName,\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\tPrefixRewrite: \"/robots.txt\",\n\t\t\t\t\t\t\tHostRewriteSpecifier: \u0026route.RouteAction_HostRewriteLiteral{\n\t\t\t\t\t\t\t\tHostRewriteLiteral: targetHost,\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t},\n\t\t\t\t\t},\n\t\t\t\t}},\n\t\t\t}},\n\t\t}\n\n\t\tmanager := \u0026hcm.HttpConnectionManager{\n\t\t\tCodecType:  hcm.HttpConnectionManager_AUTO,\n\t\t\tStatPrefix: \"ingress_http\",\n\t\t\tRouteSpecifier: \u0026hcm.HttpConnectionManager_RouteConfig{\n\t\t\t\tRouteConfig: rte,\n\t\t\t},\n\t\t\tHttpFilters: []*hcm.HttpFilter{{\n\t\t\t\tName: wellknown.Router,\n\t\t\t}},\n\t\t}\n```\n\n\n#### 1. TLS Static \n\nIn the first TLS option, we will create a static TLS certs to beam down:\n\n```golang\n\t\tpriv, err := ioutil.ReadFile(\"certs/server.key\")\n\t\tif err != nil {\n\t\t\tlog.Fatal(err)\n\t\t}\n\t\tpub, err := ioutil.ReadFile(\"certs/server.crt\")\n\t\tif err != nil {\n\t\t\tlog.Fatal(err)\n\t\t}\n\n\t\tsdsTls := \u0026envoy_api_v3_auth.DownstreamTlsContext{\n\t\t\tCommonTlsContext: \u0026envoy_api_v3_auth.CommonTlsContext{\n\t\t\t\tTlsCertificates: []*envoy_api_v3_auth.TlsCertificate{{\n\t\t\t\t\tCertificateChain: \u0026core.DataSource{\n\t\t\t\t\t\tSpecifier: \u0026core.DataSource_InlineBytes{InlineBytes: []byte(pub)},\n\t\t\t\t\t},\n\t\t\t\t\tPrivateKey: \u0026core.DataSource{\n\t\t\t\t\t\tSpecifier: \u0026core.DataSource_InlineBytes{InlineBytes: []byte(priv)},\n\t\t\t\t\t},\n\t\t\t\t}},\n\t\t\t},\n\t\t}\n\n\t\t...\n\t\t...\n\n\n\t\tscfg, err := ptypes.MarshalAny(sdsTls)\n\t\tif err != nil {\n\t\t\tlog.Fatal(err)\n\t\t}\n\n\t\tvar l = []types.Resource{\n\t\t\t\u0026listener.Listener{\n\t\t\t\tName: listenerName,\n\t\t\t\tAddress: \u0026core.Address{\n\t\t\t\t\tAddress: \u0026core.Address_SocketAddress{\n\t\t\t\t\t\tSocketAddress: \u0026core.SocketAddress{\n\t\t\t\t\t\t\tProtocol: core.SocketAddress_TCP,\n\t\t\t\t\t\t\tAddress:  localhost,\n\t\t\t\t\t\t\tPortSpecifier: \u0026core.SocketAddress_PortValue{\n\t\t\t\t\t\t\t\tPortValue: 10000,\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t},\n\t\t\t\t\t},\n\t\t\t\t},\n\t\t\t\tFilterChains: []*listener.FilterChain{{\n\t\t\t\t\tFilters: []*listener.Filter{{\n\t\t\t\t\t\tName: wellknown.HTTPConnectionManager,\n\t\t\t\t\t\tConfigType: \u0026listener.Filter_TypedConfig{\n\t\t\t\t\t\t\tTypedConfig: pbst,\n\t\t\t\t\t\t},\n\t\t\t\t\t}},\n\t\t\t\t\tTransportSocket: \u0026core.TransportSocket{\n\t\t\t\t\t\tName: \"envoy.transport_sockets.tls\",\n\t\t\t\t\t\tConfigType: \u0026core.TransportSocket_TypedConfig{\n\t\t\t\t\t\t\tTypedConfig: scfg,\n\t\t\t\t\t\t},\n\t\t\t\t\t},\n\t\t\t\t}},\n\t\t\t}}\n\n```\n\n#### 2. SDS Static \n\nIn the second option, you need to enable the SDS Static reference\n\nEdit `baseline.yaml`, and uncomment\n\n```yaml\n\n  secrets:\n  - name: server_cert\n    tls_certificate:\n      certificate_chain:\n        filename: certs/server.crt\n      private_key:\n        filename: certs/server.key\n```\n\nthen in `main.go`, \n\n```golang\n\t\tsdsTls := \u0026envoy_api_v3_auth.DownstreamTlsContext{\n\t\t\tCommonTlsContext: \u0026envoy_api_v2_auth.CommonTlsContext{\n\t\t\t\tTlsCertificateSdsSecretConfigs: []*envoy_api_v2_auth.SdsSecretConfig{{\n\t\t\t\t\tName: \"server_cert\",\n\t\t\t\t}},\n\t\t\t},\n\t\t}\n```\n\nWhat the above setting will do is instruct envoy to look for a local static secret (you know,the one we just uncommented)\n\n\n#### 3. SDS Dynamic\n\nThe following Secrets config will provide a reference *and* beam down the Secret itself\n\n```golang\n\t\tsdsTls := \u0026envoy_api_v3_auth.DownstreamTlsContext{\n\t\t\tCommonTlsContext: \u0026envoy_api_v3_auth.CommonTlsContext{\n\t\t\t\tTlsCertificateSdsSecretConfigs: []*envoy_api_v3_auth.SdsSecretConfig{{\n\t\t\t\t\tName: \"server_cert\",\n\t\t\t\t\tSdsConfig: \u0026core.ConfigSource{\n\t\t\t\t\t\tConfigSourceSpecifier: \u0026core.ConfigSource_Ads{\n\t\t\t\t\t\t\tAds: \u0026core.AggregatedConfigSource{},\n\t\t\t\t\t\t},\n\t\t\t\t\t\tResourceApiVersion: core.ApiVersion_V3,\n\t\t\t\t\t},\n\t\t\t\t}},\n\t\t\t},\n\t\t}\n```\n\n#### Commit Snapshot\n\nFinal step is to commit a snapshot of the full config including the cluster, listener and secret\n```golang\n\t\tsnap := cachev3.NewSnapshot(fmt.Sprint(version), nil, c, nil, l, nil, s)\n\t\tconfig.SetSnapshot(nodeId, snap)\n```\n\n\nYou can verify the cluster was dynamically added in by viewing the envoy admin console at ```http://localhost:9000```.  A sample output of that console:\n\n![images/admin_clusters.png](images/admin_clusters.png)\n\n\n### Running static configuration\n\nIf you would rather just run the static configuration (i.,e no ADS server), just run\n\n\n```bash\n./envoy -c bbc.yaml\n```\n\n---\n\n## Conclusion\n\nI wrote this primarily just to understand how envoy works..As this is the first time i've configured and worked through the structures within Envoy,\nits very likely i've missed some construct or concept.  If you see anything amiss, please drop me a line and I'll correct it.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalrashid123%2Fenvoy_control","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsalrashid123%2Fenvoy_control","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalrashid123%2Fenvoy_control/lists"}