https://github.com/rpcpool/yellowstone-grpc-kafka
Yellowstone gRPC Kafka consumer/dedup/producer
https://github.com/rpcpool/yellowstone-grpc-kafka
Last synced: 10 months ago
JSON representation
Yellowstone gRPC Kafka consumer/dedup/producer
- Host: GitHub
- URL: https://github.com/rpcpool/yellowstone-grpc-kafka
- Owner: rpcpool
- Created: 2024-10-03T10:20:57.000Z (over 1 year ago)
- Default Branch: master
- Last Pushed: 2025-03-18T19:34:05.000Z (about 1 year ago)
- Last Synced: 2025-05-03T01:22:55.769Z (11 months ago)
- Language: Rust
- Size: 57.6 KB
- Stars: 32
- Watchers: 6
- Forks: 13
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
Awesome Lists containing this project
README
# Yellowstone gRPC Kafka Tool
Forward gRPC stream to Kafka, dedup, read stream from Kafka with gRPC server.
Modes:
- `grpc2kafka` — connect to gRPC with specified filter and sent all incoming messages to the Kafka
- `dedup` — consume messages from Kafka and sent deduplicated messages to another topic (right now only support `memory` as deduplication backend)
- `kafka2grpc` — provide gRPC endpoint with sending messages from Kafka
```bash
$ cargo run --bin grpc-kafka -- --help
Yellowstone gRPC Kafka Tool
Usage: grpc-kafka [OPTIONS] --config
Commands:
dedup Receive data from Kafka, deduplicate and send them back to Kafka
grpc2kafka Receive data from gRPC and send them to the Kafka
kafka2grpc Receive data from Kafka and send them over gRPC
help Print this message or the help of the given subcommand(s)
Options:
-c, --config Path to config file
--prometheus Prometheus listen address
-h, --help Print help
-V, --version Print version
```
##### Development
```bash
# run kafka locally
docker-compose -f docker-kafka.yml up
# create topic
kafka_2.13-3.5.0/bin/kafka-topics.sh --bootstrap-server localhost:29092 --create --topic grpc1
# send messages from gRPC to Kafka
cargo run --bin grpc-kafka -- --config config-kafka.json grpc2kafka
# read messages from Kafka
kafka_2.13-3.5.0/bin/kafka-console-consumer.sh --bootstrap-server localhost:29092 --topic grpc1
```