{"id":19888341,"url":"https://github.com/wasmedge/wasm-log-flex","last_synced_at":"2025-10-13T23:02:16.321Z","repository":{"id":173604330,"uuid":"650448498","full_name":"WasmEdge/wasm-log-flex","owner":"WasmEdge","description":"A log processing framework in Rust. Ingest from database and log files, transform, and output to Kafka / ElasticSearch. Runs side-by-side with containers in k8s pods in WasmEdge.","archived":false,"fork":false,"pushed_at":"2023-08-25T00:37:45.000Z","size":416,"stargazers_count":6,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-03-02T18:40:01.888Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/WasmEdge.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-06-07T05:11:08.000Z","updated_at":"2023-11-20T13:41:14.000Z","dependencies_parsed_at":null,"dependency_job_id":"390cf30d-6402-49e1-93b7-4ceb3cd6bf8a","html_url":"https://github.com/WasmEdge/wasm-log-flex","commit_stats":null,"previous_names":["markcty/wasm-log-flex"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WasmEdge%2Fwasm-log-flex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WasmEdge%2Fwasm-log-flex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WasmEdge%2Fwasm-log-flex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WasmEdge%2Fwasm-log-flex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/WasmEdge","download_url":"https://codeload.github.com/WasmEdge/wasm-log-flex/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224324497,"owners_count":17292521,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T18:06:58.338Z","updated_at":"2025-10-13T23:02:16.261Z","avatar_url":"https://github.com/WasmEdge.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Wasm Log Flex\n\n\u003e The ultimate log processing framework for the WebAssembly platform.\n\nWasm Log Flex(`wlf`) is a log processing frame created for the WebAssembly platform. Like many other popular frameworks, our log processing framework will consist of three main `component`s:\n\n- `Collector`: collecting logs from various services (e.g., MySQL Binlog, Docker, Kubernetes, and File).\n- `Transformer`: manipulate logs emitted from collectors or other transformers.\n- `Dispatcher`: dispatch processed logs to their final destinations (e.g., Kafka, Redis, ElasticSearch).\n\nComponents communicate with each other using custom `Event`. They can be chained together to form a pipeline, like this:\n\n![Architecture](assets/Architecture.png)\n\nCurrently, we have the MySQL Binlog collector, the kafka, elasticsearch and redis dispatchers, the binlog filter and event replicator transformers. \n\nDevelopers can easily create their own components by implementing the `ComponentApi` trait:\n\n```Rust\n#[async_trait]\npub trait ComponentApi: 'static + Send + Sync {\n    // Returns the unique id of the component.\n    fn id(\u0026self) -\u003e \u0026str;\n    // Return the component kind(collector, transformer, or dispatcher)\n    fn kind(\u0026self) -\u003e ComponentKind;\n    // Run the component. Use the `router` to recv/send events from/to other components\n    async fn run(\u0026self, router: Arc\u003cEventRouter\u003e) -\u003e Result\u003c(), Box\u003cdyn Error\u003e\u003e;\n}\n```\n\nIn additional to use the library, users who just want a quick setup can use the `wlf-aio`(wasm-log-flex-all-in-one) binary, which packs all the components and can be easily configured.\n\n## Quick Start(using `wlf-aio`)\n\n1. To use the `wlf`, use docker to initiate a local environment `docker compose -f examples/binlog_to_kafka_redis_es.docker-compose.yaml up -d`, which brings up a kafka, a mysql, a elasticsearch, and a redis.\n\n2. Build the `wlf-aio` binary: `cargo build --target=wasm32-wasi -p wlf-aio -r`.\n\n3. Install the Wasmedge WebAssembly runtime: https://wasmedge.org/docs/start/install.\n\n4. Run the example: `wasmedge --dir /configs:examples/configs target/wasm32-wasi/debug/wlf-aio.wasm --config /configs/binlog_to_kafka_redis_es.yaml`\n\nThe example uses the following configuration:\n```yaml\ncollectors:\n  - id: binlog_collector\n    type: Binlog\n    destination: filter\n    user: root\n    password: password\ntransformers:\n  - id: filter\n    type: BinlogFilter\n    destination: replicator\n    rules:\n      - exclude:\n          database: d1\n          table: \"*\"\n      - include:\n          database: d1\n          table: t1\n  - id: replicator\n    type: EventReplicator\n    destinations:\n      - redis\n      - kafka\n      - elasticsearch\ndispatchers:\n  - id: kafka\n    type: Kafka\n    topic: logFlex.%{/database}.%{/table}\n    bootstrap_brokers: [\"127.0.0.1:9092\"]\n  - id: redis\n    type: Redis\n    mode:\n      type: Pub\n      channel: logFlex.%{/database}.%{/table}\n  - id: elasticsearch\n    type: Elasticsearch\n    url: http://localhost:9200\n    index: wlf-%{/database}-%{/table}\n```\nThe example collects `Binlog` events from Mysql Binlog, filters and replicates them, and then forward them to both kafka, redis, and elasticsearch.\n\n`wlf-aio` also supports reading maxwell configuration directly, just use a `*.properties` file as the config argument then it will automatcially convert the maxwell config to ours.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwasmedge%2Fwasm-log-flex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwasmedge%2Fwasm-log-flex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwasmedge%2Fwasm-log-flex/lists"}