https://github.com/wasmedge/wasm-log-flex
A log processing framework in Rust. Ingest from database and log files, transform, and output to Kafka / ElasticSearch. Runs side-by-side with containers in k8s pods in WasmEdge.
https://github.com/wasmedge/wasm-log-flex
Last synced: 9 months ago
JSON representation
A log processing framework in Rust. Ingest from database and log files, transform, and output to Kafka / ElasticSearch. Runs side-by-side with containers in k8s pods in WasmEdge.
- Host: GitHub
- URL: https://github.com/wasmedge/wasm-log-flex
- Owner: WasmEdge
- Created: 2023-06-07T05:11:08.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2023-08-25T00:37:45.000Z (almost 3 years ago)
- Last Synced: 2024-03-02T18:40:01.888Z (over 2 years ago)
- Language: Rust
- Homepage:
- Size: 406 KB
- Stars: 6
- Watchers: 2
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Wasm Log Flex
> The ultimate log processing framework for the WebAssembly platform.
Wasm Log Flex(`wlf`) is a log processing frame created for the WebAssembly platform. Like many other popular frameworks, our log processing framework will consist of three main `component`s:
- `Collector`: collecting logs from various services (e.g., MySQL Binlog, Docker, Kubernetes, and File).
- `Transformer`: manipulate logs emitted from collectors or other transformers.
- `Dispatcher`: dispatch processed logs to their final destinations (e.g., Kafka, Redis, ElasticSearch).
Components communicate with each other using custom `Event`. They can be chained together to form a pipeline, like this:
Currently, we have the MySQL Binlog collector, the kafka, elasticsearch and redis dispatchers, the binlog filter and event replicator transformers.
Developers can easily create their own components by implementing the `ComponentApi` trait:
```Rust
#[async_trait]
pub trait ComponentApi: 'static + Send + Sync {
// Returns the unique id of the component.
fn id(&self) -> &str;
// Return the component kind(collector, transformer, or dispatcher)
fn kind(&self) -> ComponentKind;
// Run the component. Use the `router` to recv/send events from/to other components
async fn run(&self, router: Arc) -> Result<(), Box>;
}
```
In additional to use the library, users who just want a quick setup can use the `wlf-aio`(wasm-log-flex-all-in-one) binary, which packs all the components and can be easily configured.
## Quick Start(using `wlf-aio`)
1. To use the `wlf`, use docker to initiate a local environment `docker compose -f examples/binlog_to_kafka_redis_es.docker-compose.yaml up -d`, which brings up a kafka, a mysql, a elasticsearch, and a redis.
2. Build the `wlf-aio` binary: `cargo build --target=wasm32-wasi -p wlf-aio -r`.
3. Install the Wasmedge WebAssembly runtime: https://wasmedge.org/docs/start/install.
4. Run the example: `wasmedge --dir /configs:examples/configs target/wasm32-wasi/debug/wlf-aio.wasm --config /configs/binlog_to_kafka_redis_es.yaml`
The example uses the following configuration:
```yaml
collectors:
- id: binlog_collector
type: Binlog
destination: filter
user: root
password: password
transformers:
- id: filter
type: BinlogFilter
destination: replicator
rules:
- exclude:
database: d1
table: "*"
- include:
database: d1
table: t1
- id: replicator
type: EventReplicator
destinations:
- redis
- kafka
- elasticsearch
dispatchers:
- id: kafka
type: Kafka
topic: logFlex.%{/database}.%{/table}
bootstrap_brokers: ["127.0.0.1:9092"]
- id: redis
type: Redis
mode:
type: Pub
channel: logFlex.%{/database}.%{/table}
- id: elasticsearch
type: Elasticsearch
url: http://localhost:9200
index: wlf-%{/database}-%{/table}
```
The example collects `Binlog` events from Mysql Binlog, filters and replicates them, and then forward them to both kafka, redis, and elasticsearch.
`wlf-aio` also supports reading maxwell configuration directly, just use a `*.properties` file as the config argument then it will automatcially convert the maxwell config to ours.