https://github.com/derekargueta/envoy-rot-cipher-filter
An example of an isolated third-party Envoy filter
https://github.com/derekargueta/envoy-rot-cipher-filter
bazel cpp cpp11 envoy envoy-development envoy-filter filter-development http protobuf proxy
Last synced: 3 months ago
JSON representation
An example of an isolated third-party Envoy filter
- Host: GitHub
- URL: https://github.com/derekargueta/envoy-rot-cipher-filter
- Owner: derekargueta
- Created: 2017-12-22T04:18:40.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2021-04-29T01:30:20.000Z (about 4 years ago)
- Last Synced: 2025-01-16T01:34:45.406Z (5 months ago)
- Topics: bazel, cpp, cpp11, envoy, envoy-development, envoy-filter, filter-development, http, protobuf, proxy
- Language: Starlark
- Homepage:
- Size: 36.1 KB
- Stars: 1
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
Envoy Rot Cipher Filter
===========================
A simple filter that probably (hopefully) isn't of much production value, but demonstrates how to write a third-party [Envoy](https://github.com/envoyproxy/envoy) filter that is compatible with both the v1 and v2 API. I hope that this repository serves as a nice guide for people who maybe don't want to dive into the internals of Envoy too deeply, but want to be able to build custom Envoy filters _outside_ of the main source code.
I wanted to write this because the [current defacto example](https://github.com/envoyproxy/envoy-filter-example) doesn't use any configuration data, which is one of the parts I wrestled with the most in building this filter. As a young whippersnapper new-grad, I thought that this "outside perspective" might be helpful for others getting started with C++/Envoy development.
FYI The code here is not the greatest and isn't meant to be, but I do plan on coming back here and cleaning up somewhat.
If you don't understand the explanations here, please open an issue/PR.
Also, the filter's source code should be well-documented (actually over-documented) to provide further explanations.
## Compiling & Running
```bash
git submodule update --init # we need the envoy source to compile/test
make compile
```
The `compile` target should call the `compile/proto` target to generate the protobuf classes in the `gen/` directory.
I've included a basic `app.py` web server that just prints the headers it receives. To run the whole thing:
terminal pane 1:
```bash
./bazel-bin/envoy -c examples/example.v2.yaml --v2-config-only
```
terminal pane 2:
```
python examples/app.py
```
terminal pane 3:
```
curl -H "x-rot: asdf" localhost:8000
```
and in pane 2 you should see something like
```
user-agent: curl/7.54.0
accept: */*
x-forwarded-proto: http
x-request-id: 5a16b07d-d811-4fdf-b9bc-790efdd1d97b
x-rot: nopq
content-length: 0
```
where you can see the letters of `x-rot` have been rotated.
## Usage
This filter accepts two configuration options:
- `rot_value`: an integer describing how much to rotate each letter by. For example, 13 would be the classic Caesar Cipher
- `rot_header`: a string that names which header to apply the cipher to. If the header is not found, no action is taken and the filter continues.
v2 Example
```
http_filters:
- name: rot_cipher
config: { rot_value: 13, rot_header: x-rot }
- name: envoy.router
```
Full example configurations are in the `examples/` directory.
## Filter Development
Every filter has 2 main components:
- The config, which for HTTP filters should be a subclass of [`NamedHttpFilterConfigFactory`](https://github.com/envoyproxy/envoy/blob/b01b13865ac86c6b5716e128137afd29ef3147d0/include/envoy/server/filter_config.h#L196)
- The filter implementation, which for HTTP filters would be a subclass of either [`StreamDecoderFilter`](https://github.com/envoyproxy/envoy/blob/3599784052e75ac1b18b5342aa711d987a852bf0/include/envoy/http/filter.h#L289), [`StreamEncoderFilter`](https://github.com/envoyproxy/envoy/blob/3599784052e75ac1b18b5342aa711d987a852bf0/include/envoy/http/filter.h#L402), or [`StreamFilter`](https://github.com/envoyproxy/envoy/blob/3599784052e75ac1b18b5342aa711d987a852bf0/include/envoy/http/filter.h#L438) for filters that do encoding and decoding.
To support the [v2 API](https://www.envoyproxy.io/docs/envoy/latest/api-v2/api), you'll additionally need a `.proto` file (if your filter uses configuration data. If it doesn't then there's no need for a `.proto` file).
Simply put, if your config is v1, then Envoy will call `createFilterFactory` with a [`Json::Object`](https://github.com/envoyproxy/envoy/blob/master/include/envoy/json/json_object.h) which you can then pick apart and validate.
If your config is v2, then it will call `createFilterFactoryFromProto` with a [`Protobuf::Message`](https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.message), which you can just cast to your protobuf-generated class.
The details of how/why this procedure works is worthy of a separate write-up, but I believe it has a lot to do with how the v2 API can use [gRPC](https://grpc.io/) streaming.
(More on Envoy's use of gRPC in the [Envoy docs](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/grpc.html))
If you look at filters in the main Envoy source code they have a lot more pieces due to the complexity, for example they have intermediate `config` classes that the Json and protobuf data get converted to so that the filters only need 1 constructor. But if you strip it all down, these are the bare minimums.
I started with `rot_cipher.proto` to define what the configuration would look like.
That file then produces `rot_cipher.pb.(cc|h)` in `gen/`.
This is a pretty ad-hoc procedure that I constructed, and quite frankly the bazel `BUILD` file could be improved to use the native Bazel support for protobufs instead of calling the protobuf compiler in the `Makefile`.
_But is using a `.proto` necessary?_
_For the v2 API, yes. You might be able to hack something weird using [Protobuf Reflection](https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.message#Reflection) but that's a scary path._
After wrapping my head around how the protobufs integrate into Envoy filter configs (first-time protobuf user here), the rest was pretty straight forward to achieve based on the [envoy-filter-example](https://github.com/envoyproxy/envoy-filter-example). Some important classes to look at are [HeaderMap](https://github.com/envoyproxy/envoy/blob/master/include/envoy/http/header_map.h) for manipulating headers and [Object](https://github.com/envoyproxy/envoy/blob/master/include/envoy/json/json_object.h) for manipulating Json Objects. Additionally, [json_loader.cc](https://github.com/envoyproxy/envoy/blob/master/source/common/json/json_loader.cc), [config/utility.h](https://github.com/envoyproxy/envoy/blob/master/source/common/config/utility.h), and [protobuf/utility.h](https://github.com/envoyproxy/envoy/blob/master/source/common/protobuf/utility.h) were pretty helpful.
As an aside, before starting this filter I went through the [Protobuf C++ Tutorial](https://developers.google.com/protocol-buffers/docs/cpptutorial), substituting [proto3](https://developers.google.com/protocol-buffers/docs/proto3) syntax for [proto2](https://developers.google.com/protocol-buffers/docs/proto).
## Problems I ran Into
(this part is still kind of mumble-y and under revision)
I had an issue with the v2 API config where each time I tried to start Envoy I got the error below:
```
[2017-12-22 13:55:03.124][4915417][debug][config] external/envoy/source/server/listener_manager_impl.cc:29] filter #0:
[2017-12-22 13:55:03.124][4915417][debug][config] external/envoy/source/server/listener_manager_impl.cc:30] name: envoy.http_connection_manager
[2017-12-22 13:55:03.195][4915417][debug][config] external/envoy/source/server/config/network/http_connection_manager.cc:226] filter #0
[2017-12-22 13:55:03.195][4915417][debug][config] external/envoy/source/server/config/network/http_connection_manager.cc:227] name: rot_cipher
[2017-12-22 13:55:03.195][4915417][critical][main] external/envoy/source/server/server.cc:72] error initializing configuration 'examples/example.v2.yaml': Unable to parse JSON as proto (INVALID_ARGUMENT:: Cannot find field.): {"rot_value":13,"rot_header":"x-rot"}
[2017-12-22 13:55:03.195][4915417][debug][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:523] shutting down thread local cluster manager
```
I had managed to narrow it down to this line in my code:
```cpp
const auto& typed_config = dynamic_cast(proto_config);
```
All generated protobuf messages subclass `Protobuf::Message` (which is the type of `proto_config`) so this dynamic cast shouldn't be a problem. I actually found this snippet in the [Envoy source](https://github.com/envoyproxy/envoy/blob/bbe6618b90d87500e7ed4304996881c4be4abb09/source/common/protobuf/utility.h#L149).
I had gone over the configuration multiple times and knew it was correct based on playing with other filters.
At the end my wits, I started throwing print statements into the Envoy source code, thinking maybe there was a bug somewhere in the JSON parsing?
I started with the `HttpConnectionManagerConfig` right after it logs what filter it's processing, since the debug log kindly outputs what line it is at :)
This is at the [end of the constructor](https://github.com/envoyproxy/envoy/blob/9ed62923a8ff6745407046c4451ce757348d966f/source/server/config/network/http_connection_manager.cc#L221) which has the following code.
```cpp
const auto& filters = config.http_filters();
for (int32_t i = 0; i < filters.size(); i++) {
const ProtobufTypes::String& string_name = filters[i].name();
const auto& proto_config = filters[i];
ENVOY_LOG(debug, " filter #{}", i);
ENVOY_LOG(debug, " name: {}", string_name);
const Json::ObjectSharedPtr filter_config =
MessageUtil::getJsonObjectFromMessage(proto_config.config());
// Now see if there is a factory that will accept the config.
auto& factory = Config::Utility::getAndCheckFactory(string_name);
HttpFilterFactoryCb callback;
if (filter_config->getBoolean("deprecated_v1", false)) {
callback = factory.createFilterFactory(*filter_config->getObject("value", true),
stats_prefix_, context);
} else {
ProtobufTypes::MessagePtr message =
Config::Utility::translateToFactoryConfig(proto_config, factory);
callback = factory.createFilterFactoryFromProto(*message, stats_prefix_, context);
}
filter_factories_.push_back(callback);
}
```
I inspected the `filter_config` JSON object and it had the `rot_value` and `rot_header` as expected, so then I looked inside the call to [`Config::Utility::translateToFactoryConfig(...)`](https://github.com/envoyproxy/envoy/blob/6b2823da5006e92bc4b365e9e8804a4f6a2eba37/source/common/config/utility.h#L190) which contains the following code.
```cpp
template
static ProtobufTypes::MessagePtr translateToFactoryConfig(const ProtoMessage& enclosing_message,
Factory& factory) {
ProtobufTypes::MessagePtr config = factory.createEmptyConfigProto();
if (config == nullptr) {
throw EnvoyException(fmt::format(
"{} factory returned nullptr instead of empty config message.", factory.name()));
}
if (enclosing_message.has_config()) {
MessageUtil::jsonConvert(enclosing_message.config(), *config);
}
return config;
}
```
I wasn't getting the nullptr error message so the config wasn't coming back null... WAIT. `createEmptyConfigProto`? That function is required by the abstract class and I remember pasting in some code for it... I hadn't thought much of that function and just pasted what the `envoy-example-filter` code had, which is `return ProtobufTypes::MessagePtr{new Envoy::ProtobufWkt::Empty()}`. If the empty message is _not_ of type `example::RotCipher`, then yeah there would be issues in casting it to a `Protobuf::Message` and then to an `example::RotCipher`. I checked some built-in Envoy filters that had non-empty configurations and sure enough, they return an empty instance of the generated protobuf config.
RateLimit config
```cpp
ProtobufTypes::MessagePtr createEmptyConfigProto() override {
return ProtobufTypes::MessagePtr{new envoy::api::v2::filter::http::RateLimit()};
}
```
So once I changed the RotCipher config's `createEmtpyConfigProto` to
```cpp
ProtobufTypes::MessagePtr createEmptyConfigProto() override {
return ProtobufTypes::MessagePtr{new example::RotCipher()};
}
```
all was well! The error message was a little cryptic (yet correct), but sure enough it was a user-error.
## Thoughts/Reflections
This was mostly an exercise in gaining familiarity with Envoy filter development so here's some thoughts on how the whole thing went.
- There is very little internal documentation of how the mechanics work. This is no secret, but the documentation from Envoy could use some love and help from the community (low-hanging fruit for new Envoy contributors!)
- The organization of the Envoy code made it a little difficult to piece this project together. For example, _all_ the Envoy filter configs are in one directoy ([`server/config/http/`](https://github.com/envoyproxy/envoy/tree/master/source/server/config/http)), and _all_ the Envoy filter implementations are in a different directory ([`source/common/`](https://github.com/envoyproxy/envoy/tree/master/source/common)), and there's multiple `config` directories for [non-filter config stuff](https://github.com/envoyproxy/envoy/tree/master/source/common/config), etc. so it felt like a scavenger hunt to get the complete image of what all 1 filter needs. This organization makes sense for the size of Envoy's project, but again makes it difficult to piece together everything that, for example, the buffer filter does.
- Despite the above 2 "complaints", Envoy does provide a very clean API to work with. I found this to be a much easier exercise than implementing an Nginx module, which took a day-long workshop and I'm still not confident in my Nginx module development.
- I noticed there's some scripts in `envoy/tools/` such as `stack_decode.py` that may have been helpful in debugging but I couldn't figure out how to get them to work.
- The bazel build procedure is pretty slick. Props to the Envoy team!
- If you want to write Envoy filters, you can't shy away from reading a lot of C++ :)