https://github.com/systemcraftsman/redpanda-connect-dynamic-plugin-demo
https://github.com/systemcraftsman/redpanda-connect-dynamic-plugin-demo
Last synced: 1 day ago
JSON representation
- Host: GitHub
- URL: https://github.com/systemcraftsman/redpanda-connect-dynamic-plugin-demo
- Owner: SystemCraftsman
- Created: 2026-05-28T19:45:54.000Z (4 days ago)
- Default Branch: main
- Last Pushed: 2026-05-28T19:54:43.000Z (4 days ago)
- Last Synced: 2026-05-28T21:22:54.246Z (4 days ago)
- Language: Python
- Size: 20.5 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Redpanda Connect Dynamic Plugins
Working code examples for building [Redpanda Connect](https://docs.redpanda.com/redpanda-connect/home/) dynamic plugins using the Python SDK. The examples walk through writing a processor plugin from scratch, declaring its manifest, wiring it into a pipeline, and making it configurable.
## What This Project Does
The code demonstrates how to extend Redpanda Connect with custom Python logic at runtime using the RPC plugin mechanism. You write a Python function decorated with `@redpanda_connect.processor`, drop a `plugin.yaml` manifest alongside it, and run `rpk connect run --rpc-plugins=plugin.yaml` to load it without recompiling anything.
## Prerequisites
- [Redpanda Connect](https://docs.redpanda.com/redpanda-connect/home/) (`rpk` CLI, v4.56.0 or later)
- Python 3.12 or later
- [uv](https://docs.astral.sh/uv/) (Python package manager)
## Project Structure
```
.
├── plugins/
│ ├── yell-processor/ # Simple plugin (uppercases messages)
│ │ ├── yell_processor.py
│ │ └── plugin.yaml
│ └── yell-processor-configurable/ # Extended version with runtime config
│ ├── yell_processor.py
│ └── plugin.yaml
├── connect.yaml # Pipeline config for the simple plugin
├── connect-configurable.yaml # Pipeline config for the configurable plugin
└── Dockerfile.reviewer # Containerized review environment
```
## Quick Start
Initialize the plugin project and install the SDK:
```bash
cd plugins/yell-processor
uv init --no-readme
uv add redpanda_connect
```
Run the simple pipeline from the repo root:
```bash
rpk connect run --rpc-plugins=plugins/yell-processor/plugin.yaml connect.yaml
```
Expected output (5 lines, one per event):
```
USER SIGNED UP FROM MOBILE-APP
PAYMENT PROCESSED FOR ORDER 1042
DASHBOARD EXPORT REQUESTED
SESSION TIMEOUT ON WEB-CLIENT
INVENTORY SYNC COMPLETED
```
## Running the Configurable Plugin
Initialize and run the configurable version:
```bash
cd plugins/yell-processor-configurable
uv init --no-readme
uv add redpanda_connect
cd ../..
rpk connect run --rpc-plugins=plugins/yell-processor-configurable/plugin.yaml connect-configurable.yaml
```
Expected output (5 lines, prefix ">> " applied, repeated twice):
```
>> USER SIGNED UP FROM MOBILE-APP>> USER SIGNED UP FROM MOBILE-APP
>> PAYMENT PROCESSED FOR ORDER 1042>> PAYMENT PROCESSED FOR ORDER 1042
>> DASHBOARD EXPORT REQUESTED>> DASHBOARD EXPORT REQUESTED
>> SESSION TIMEOUT ON WEB-CLIENT>> SESSION TIMEOUT ON WEB-CLIENT
>> INVENTORY SYNC COMPLETED>> INVENTORY SYNC COMPLETED
```
## Review Environment
Build and run the Docker container with all prerequisites installed:
```bash
docker build -f Dockerfile.reviewer -t rpconnect-reviewer .
docker run -it --rm rpconnect-reviewer bash
```
Follow the tutorial steps inside the container to verify everything works from scratch.