{"id":16434573,"url":"https://github.com/halfrost/sentinel-go-envoy-proxy-wasm","last_synced_at":"2025-10-27T02:30:55.856Z","repository":{"id":44389964,"uuid":"511696541","full_name":"halfrost/sentinel-go-envoy-proxy-wasm","owner":"halfrost","description":"Sentinel Go Proxy-WASM extension","archived":false,"fork":false,"pushed_at":"2022-10-31T17:16:53.000Z","size":811,"stargazers_count":7,"open_issues_count":1,"forks_count":1,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-02-01T01:33:08.756Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/halfrost.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-07-07T22:57:13.000Z","updated_at":"2024-07-08T02:14:14.000Z","dependencies_parsed_at":"2023-01-20T15:15:22.341Z","dependency_job_id":null,"html_url":"https://github.com/halfrost/sentinel-go-envoy-proxy-wasm","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/halfrost%2Fsentinel-go-envoy-proxy-wasm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/halfrost%2Fsentinel-go-envoy-proxy-wasm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/halfrost%2Fsentinel-go-envoy-proxy-wasm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/halfrost%2Fsentinel-go-envoy-proxy-wasm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/halfrost","download_url":"https://codeload.github.com/halfrost/sentinel-go-envoy-proxy-wasm/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238422838,"owners_count":19469630,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T08:49:34.355Z","updated_at":"2025-10-27T02:30:50.525Z","avatar_url":"https://github.com/halfrost.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# sentinel-go-envoy-proxy-wasm\nSentinel Go Proxy-WASM extension\n\n\n# Introduction\n\nDue to the limitations of the sentinel token server for Envoy Global Rate Limiting Service model, scenarios and performance are limited. Considering the rapid development of cloud native in recent years, the voice of sentinel wasm extension in the community is getting higher and higher. Using the Envoy wasm extension and the native implementation of sentinel go, comprehensive traffic governance capabilities and standard coverage can be implemented.\n\n# Environment and Tools\n\n- tinygo 0.24.0 linux/amd64 (using go version go1.18.4 and LLVM version 14.0.0)\n- func-e 1.1.3\n- kubernetes/kubectl\n- Istio\n- Envoy\n\n# How it works\n\nThe sentinel wasm extension implemented is based on the sentinel-go. So it uses proxy-wasm-go-sdk, which is an encapsulation of the Proxy-Wasm ABI, providing a series of APIs in the Go language. Using this SDK, it is possible to generate Wasm binaries that are compliant with the Proxy-Wasm.\n\nBefore discussing the wasm architecture design, let’s talk about the Wasm Virtual Machine (Wasm VM). Wasm VM is the environment for loading plugins. In Envoy, VMs are created in each thread and isolated from each other. So the wasm extension program we created will be copied into the thread created by Envoy and loaded on each VM. The Proxy-Wasm allows multiple plugins in a single VM. In other words, a VM can be used by multiple plugins.\n\n\n![](./static/wasm_vm.png)\n\nFrom the vm loading timing, the thread first loads the vm, then the vm creates the context, and then loads the plugin context. The sentinel plugin is loaded by the vm. At this time, we can load some of the rule CRD specification yaml files we specified to perform the sentinel initialization operation. After the initial operation of sentinel plugin is completed, it starts to monitor http connection.\n\n![](./static/plugin.png)\n\nFor each tcp connection, the plugin will intercept each upstream and downstream data packets. At this time, there are 5 hooks that can be executed. The entry is initialized when OnNewConnection is started, and the go statistical coroutine is started. Call the exit function at OnStreamDone and output a BlockError error message.\n\nIf there are operations such as current limiting and fusing in the process, a BlockError will be catched in OnUpstreamData, and this error message will be returned to the client. Each connection will trigger entry funciton. If it exits normally, it will trigger exit function, and if the limit traffic rule is triggered, it will return BlockError.\n\n# Example\n\n## How to run test client\n\nThe test client is used to control qps. That is, how many requests are requested per second.\n\n\n```go\n$ cd ./example\n$ go build\n$ ./client -q 100 -u some-test -p 8080\n```\n\nThe client can receive 3 parameters, `q` is qps, the unit is per second, `u` is url path. `p` is port.\n\n\n## How to run this Envoy Wasm Extensions \n\nWe will be using [TinyGo](https://tinygo.org), [proxy-wasm-go-sdk](https://github.com/tetratelabs/proxy-wasm-go-sdk) and [func-e CLI](https://func-e.io) to build and test an Envoy Wasm extension. Then we'll show a way to configure the Wasm module using the EnvoyFilter resource and deploy it to Envoy sidecars in a Kubernetes cluster.\n\n### Installing func-e CLI\n\nLet's get started by downloading func-e CLI and installing it to `/usr/local/bin`:\n\n```sh\ncurl https://func-e.io/install.sh | sudo bash -s -- -b /usr/local/bin\n```\n\nOnce downloaded, let's run it to make sure all is good:\n\n```sh\n$ func-e --version\nfunc-e version 1.1.3\n```\n\n### Installing TinyGo\n\nTinyGo powers the SDK we'll be using as Wasm doesn't support the official Go compiler. \n\nLet's download and install the TinyGo:\n\n```sh\nwget https://github.com/tinygo-org/tinygo/releases/download/v0.24.0/tinygo_0.24.0_amd64.deb\nsudo dpkg -i tinygo_0.24.0_amd64.deb\n```\n\nYou can run `tinygo version` to check the installation is successful:\n\n```sh\n$ tinygo version\ntinygo version 0.24.0 linux/amd64 (using go version go1.18.4 and LLVM version 14.0.0)\n```\n\n### Scaffolding the Wasm module\n\nWe'll start by creating a new folder for our extension, initializing the Go module, and downloading the SDK dependency:\n\n```sh\n$ mkdir header-filter \u0026\u0026 cd header-filter\n$ go mod init header-filter\n$ go mod edit -require=github.com/tetratelabs/proxy-wasm-go-sdk@main\n$ go mod download github.com/tetratelabs/proxy-wasm-go-sdk\n```\n\nNext, let's create the `main.go` file where the code for our WASM extension will like this: [main.go](https://github.com/halfrost/sentinel-go-envoy-proxy-wasm/blob/sentinel-go/main.go)\n\nSave the above contents to a file called `main.go`.\n\nLet's build the filter to check everything is good:\n\n```sh\ntinygo build -o main.wasm -scheduler=none -target=wasi main.go\n```\n\nThe build command should run successfully, and it should generate a file called `main.wasm`.\n\nWe'll use `func-e` to run a local Envoy instance to test the extension we've built.\n\nFirst, we need an Envoy config that will configure the extension:\n\n```yaml\nstatic_resources:\n  listeners:\n    - name: tcp\n      address:\n        socket_address:\n          address: 0.0.0.0\n          port_value: 18000\n      filter_chains:\n        - filters:\n            - name: envoy.filters.network.wasm\n              typed_config:\n                \"@type\": type.googleapis.com/envoy.extensions.filters.network.wasm.v3.Wasm\n                config:\n                  # Instruct that the PluginContext should behave as a Tcp filter.\n                  configuration:\n                    \"@type\": type.googleapis.com/google.protobuf.StringValue\n                    value: |\n                      {\n                        \"config_path\": \"./sentinel.yml\",\n                        \"resource_name\": \"some-test\"\n                      }\n                  # Use the same vm_config as above, so we can reuse the same VM for multiple queues.\n                  vm_config:\n                    vm_id: \"sender\"\n                    runtime: \"envoy.wasm.runtime.v8\"\n                    code:\n                      local:\n                        filename: \"main.wasm\"\n\n            - name: envoy.tcp_proxy\n              typed_config:\n                \"@type\": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy\n                stat_prefix: ingress\n                cluster: web_service\n\n    - name: http\n      address:\n        socket_address:\n          address: 127.0.0.1\n          port_value: 8099\n      filter_chains:\n        - filters:\n            - name: envoy.http_connection_manager\n              typed_config:\n                \"@type\": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager\n                stat_prefix: ingress_http\n                codec_type: auto\n                route_config:\n                  name: local_route\n                  virtual_hosts:\n                    - name: local_service\n                      domains:\n                        - \"*\"\n                      routes:\n                        - match:\n                            prefix: \"/\"\n                          direct_response:\n                            status: 200\n                            body:\n                              inline_string: \"example body\\n\"\n                http_filters:\n                  - name: envoy.filters.http.router\n                    typed_config:\n                      \"@type\": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router\n\n  clusters:\n    - name: web_service\n      connect_timeout: 0.25s\n      type: STATIC\n      lb_policy: ROUND_ROBIN\n      load_assignment:\n        cluster_name: mock_service\n        endpoints:\n          - lb_endpoints:\n              - endpoint:\n                  address:\n                    socket_address:\n                      address: 127.0.0.1\n                      port_value: 8099\n\nadmin:\n  access_log_path: \"/dev/null\"\n  address:\n    socket_address:\n      address: 0.0.0.0\n      port_value: 8001\n```\n\nSave the above to `envoy.yaml` file.\n\nThe Envoy configuration sets up a single listener on port 18000 that returns a direct response (HTTP 200) with body `example body`. Inside the `http_filters` section, we're configuring the `envoy.filters.http.wasm` filter and referencing the local WASM file (`main.wasm`) we've built earlier.\n\nLet's run the Envoy with this configuration in the background:\n\n```sh\nfunc-e run -c envoy.yaml \u0026\n```\n\nEnvoy instance should start without any issues. Once it's started, we can send a request to the port Envoy is listening on (`18000`):\n\n```sh\n$ curl localhost:10000\n[2021-06-22 16:39:31.491][5314][info][wasm] [external/envoy/source/extensions/common/wasm/context.cc:1218] wasm log: OnHttpRequestHeaders\n[2021-06-22 16:39:31.491][5314][info][wasm] [external/envoy/source/extensions/common/wasm/context.cc:1218] wasm log: OnHttpResponseHeaders\n[2021-06-22 16:39:31.492][5314][info][wasm] [external/envoy/source/extensions/common/wasm/context.cc:1218] wasm log: 2 finished\nexample body\n```\n\nThe output shows the two log entries - one from the OnHttpRequestHeaders handler and the second one from the OnHttpResponseHeaders handler. The last line is the example response returned by the direct response configuration in the filter.\n\nYou can stop the proxy by bringing the process to the foreground with `fg` and pressing CTRL+C to stop it.\n\n\n### Deploying Wasm module to Istio using EnvoyFilter\n\nThe resource we can use to deploy a Wasm module to Istio is called the EnvoyFilter. EnvoyFilter gives us the ability to customize the Envoy configuration. It allows us to modify values, configure new listeners or clusters, and add filters.\n\nIn the previous example, there was no need to push or publish the `main.wasm` file anywhere, as it was accessible by the Envoy proxy because everything was running locally. However, now that we want to run the Wasm module in Envoy proxies that are part of the Istio service mesh, we need to make the `main.wasm` file available to all those proxies so they can load and run it.\n\nSince Envoy can be extended using filters, we can use the Envoy HTTP Wasm filter to implement an HTTP filter with a Wasm module. This filter allows us to configure the Wasm module and load the module file.\n\nHere's a snippet that shows how to load a Wasm module using the Envoy HTTP Wasm filter:\n\n```yaml\nname: envoy.filters.http.wasm\ntyped_config:\n  \"@type\": type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm\n  config:\n    config:\n      name: \"my_plugin\"\n      vm_config:\n        runtime: \"envoy.wasm.runtime.v8\"\n        code:\n          local:\n            filename: \"/etc/envoy_filter_http_wasm_example.wasm\"\n        allow_precompiled: true\n    configuration:\n       '@type': type.googleapis.com/google.protobuf.StringValue\n       value: |\n         {}\n```\n\nThis particular snippet is reading the Wasm file from the local path. Note that \"local\" in this case refers to the container the Envoy proxy is running in.\n\nOne way we could bring the Wasm module to that container is to use a persistent volume, for example. We'd then copy the Wasm file to the persistent disk and use the following annotations to mount the volume into the Envoy proxy sidecars:\n\n```yaml\nsidecar.istio.io/userMount: '[{\"name\": \"wasmfilters\", \"mountPath\": \"/wasmfilters\"}]'\nsidecar.istio.io/userVolume: '[{\"name\": \"wasmfilters\", \"gcePersistentDisk\": { \"pdName\": \"my-data-disk\", \"fsType\": \"ext4\" }}]'\n```\n\nNote that the above snippet assumes a persistent disk running in GCP. The disk could be any other persistent volume as well. We'd then have to patch the existing Kubernetes deployments and 'inject' the above annotations.\n\nLuckily for us, there is another option. Remember the local field from the Envoy HTTP Wasm filter configuration? Well, there's also a remote field we can use to load the Wasm module from a remote location, a URL. The remote field simplifies things a lot! We can upload the .wasm file to remote storage, get the public URL to the module, and then use it.\n\nIn this example, we'll upload the module to a GCP storage account and made the file publicly accessible.\n\nThe updated configuration would now look like this:\n\n```yaml\nvm_config:\n  runtime: envoy.wasm.runtime.v8\n  code:\n    remote:\n      http_uri:\n        uri: [PUBLIC-URL]/extension.wasm\n        sha256: \"[sha]\"\n```\n\nYou can get the SHA by running sha256sum command. If you're using Istio 1.9 or newer, you don't have to provide the sha256 checksum, as Istio will fill that automatically. However, if you're using Istio 1.8 or older, the sha256 checksum is required, and it prevents the Wasm module from being downloaded each time.\n\nLet's create a new storage bucket first (use your name/alias instead of the `wasm-bucket` value), using the `gsutil` command (the command is available in the GCP cloud shell): \n\n```sh\ngsutil mb gs://wasm-bucket\nCreating gs://wasm-bucket/...\n```\n\n\u003eNote that you can use any other storage service to host the Wasm extension.\n\nNext, we use the commands below to copy the built extension to the Google Cloud Storage and make it publicly accessible:\n\n```sh\nBUCKET_NAME=\"wasm-bucket\"\n\n# Copy the extension to the storage bucket\ngsutil cp main.wasm gs://$BUCKET_NAME\n\n# Make the extension readable to all users\ngsutil acl ch -u AllUsers:R gs://$BUCKET_NAME/main.wasm\n```\n\nThe URL where the uploaded file is available is: `http://BUCKET_NAME.storage.googleapis.com/OBJECT_NAME`. For example, `http://wasm-bucket.storage.googleapis.com/main.wasm`.\n\nWe can now create the EnvoyFilter resource that tells Envoy where to download the extension as well as where to inject it (make sure you update the `uri` field with your bucket URI):\n\n```yaml\napiVersion: networking.istio.io/v1alpha3\nkind: EnvoyFilter\nmetadata:\n  name: headers-extension\nspec:\n  configPatches:\n  - applyTo: EXTENSION_CONFIG\n    patch:\n      operation: ADD\n      value:\n        name: headers-extension\n        typed_config:\n          \"@type\": type.googleapis.com/udpa.type.v1.TypedStruct\n          type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm\n          value:\n            config:\n              vm_config:\n                vm_id: headers-extension-vm\n                runtime: envoy.wasm.runtime.v8\n                code:\n                  remote:\n                    http_uri:\n                      uri: http://istio-weekly-ep7.storage.googleapis.com/main.wasm\n              configuration:\n                \"@type\": type.googleapis.com/google.protobuf.StringValue\n                value: |\n                  header_1=somevalue\n                  header_2=secondvalue\n  - applyTo: HTTP_FILTER\n    match:\n      context: SIDECAR_INBOUND\n      listener:\n        filterChain:\n          filter:\n            name: envoy.filters.network.http_connection_manager\n    patch:\n      operation: INSERT_BEFORE\n      value:\n        name: headers-extension\n        config_discovery:\n          config_source:\n            ads: {}\n            initial_fetch_timeout: 0s # wait indefinitely to prevent bad Wasm fetch\n          type_urls: [ \"type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm\"]\n```\n\nNote that we're deploying the EnvoyFilters to the default namespace. We could also deploy them to a root namespace (e.g. `istio-system`) if we wanted to apply the filter to all workloads in the mesh. Additionally, we could specify the selectors to pick the workloads to which we want to apply the filter.\n\nSave the above YAML to `envoyfilter.yaml` file and create it:\n\n```sh\n$ kubectl apply -f envoyfilter.yaml\nenvoyfilter.networking.istio.io/headers-extension created\n```\n\nTo try out the module, you can deploy a sample workload. \n\nI am using this httpbin example:\n\n```yaml\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: httpbin\n---\napiVersion: v1\nkind: Service\nmetadata:\n  name: httpbin\n  labels:\n    app: httpbin\n    service: httpbin\nspec:\n  ports:\n  - name: http\n    port: 8000\n    targetPort: 80\n  selector:\n    app: httpbin\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: httpbin\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      app: httpbin\n      version: v1\n  template:\n    metadata:\n      labels:\n        app: httpbin\n        version: v1\n    spec:\n      serviceAccountName: httpbin\n      containers:\n      - image: docker.io/kennethreitz/httpbin\n        imagePullPolicy: IfNotPresent\n        name: httpbin\n        ports:\n        - containerPort: 80\n```\n\nSave the above file to `httpbin.yaml` and deploy it using `kubectl apply -f httpbin.yaml`. \n\nBefore continuing, check that the httpbin Pod is up and running:\n\n```sh\n$ kubectl get po\nNAME                       READY   STATUS        RESTARTS   AGE\nhttpbin-66cdbdb6c5-4pv44   2/2     Running       1          11m\n```\n\nTo see if something went wrong with downloading the Wasm module, you can look at the istiod logs.\n\nLet's try out the deployed Wasm module!\n\nWe will create a single Pod inside the cluster, and from there, we will send a request to `http://httpbin:8000/get`\n\n```sh\n$ kubectl run curl --image=curlimages/curl -it --rm -- /bin/sh\nDefaulted container \"curl\" out of: curl, istio-proxy, istio-init (init)\nIf you don't see a command prompt, try pressing enter.\n/ $\n```\n\nOnce you get the prompt to the curl container, send a request to the `httpbin` service:\n\n```sh\n/ $ curl -v http://httpbin:8000/headers\n\u003e GET /headers HTTP/1.1\n\u003e User-Agent: curl/7.35.0\n\u003e Host: httpbin:8000\n\u003e Accept: */*\n\u003e\n\u003c HTTP/1.1 200 OK\n\u003c server: envoy\n\u003c date: Mon, 22 Jun 2021 18:52:17 GMT\n\u003c content-type: application/json\n\u003c content-length: 525\n\u003c access-control-allow-origin: *\n\u003c access-control-allow-credentials: true\n\u003c x-envoy-upstream-service-time: 3\n...\n```\n\nThen start the client with some configuration, we can begin testing.\n\n### Cleanup\n\nTo delete all created resources from your cluster, run the following:\n\n```sh\nkubectl delete envoyfilter headers-extension\nkubectl delete deployment httpbin\nkubectl delete svc httpbin\nkubectl delete sa httpbin\n```\n\n# To be solved\n\n## 1. Tinygo has limitations\n\nGiven the limitations of tinygo, cgo is not perfectly supported. This means that I cannot use etcd and prometheus related functions. Because both of them will depend on xxhash. xxhash requires cgo compile dependencies. So I commented out etcd and prometheus and their associated dependencies. After commenting out these code, recompiling can indeed pass. But sentinel will have bugs. After debugging, it is found that in the line of exporter/metric/prometheus/exporter.go:20:2, if without prometheus client dependency package, the metric is 0, which will cause part of the underlying statistical count of sentinel to be 0. The flow control logic will not be triggered.\n\nIn the wasm plugin, common statistical counts are easy to implement, but it involves some low-level metrics in prometheus, which are currently limited by tinygo, resulting in compilation errors. I also looked for solutions on Stack Overflow and github, and saw some more complicated wasm plugins. The tinygo version has not yet written functions related to prometheus. Regarding the issue of cgo, an issue has been raised to the tinygo team. They are currently working on a fix, the issue is here: [https://github.com/tinygo-org/tinygo/issues/3044](https://github.com/tinygo-org/tinygo/issues/3044)\n\n## 2. Problems with reflect\n\nApart from the cgo issue, there are some issues with reflect. You may be curious, where is reflect used? In fact, it will be used when reading yaml files. Use map to pass some values ​​to plugin in yaml file, such as resource name, crd path. These are wrapped in a json. When tinygo parses it, it doesn't know what fields are in it, it parses it with interface{} type. This part will use reflect package. In the current implementation code, the code for reading crd yaml is implemented. The bug is the same as this issue: [https://github.com/tinygo-org/tinygo/issues/2660](https://github.com/tinygo-org/tinygo/issues/2660). There is a workaround for this problem, that is, don't pass a map of mutable structures by value. In this way, tinygo will not use interface to serialize when parsing, and it will not trigger the reflect bug. But this bug is really common. If you pass a json data, it will panic. json is a map of mutable structures.\n\n\n# Next\n\n- Solved cgo compile problem.\n- Combine opensergo to refactor some module.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhalfrost%2Fsentinel-go-envoy-proxy-wasm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhalfrost%2Fsentinel-go-envoy-proxy-wasm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhalfrost%2Fsentinel-go-envoy-proxy-wasm/lists"}