{"id":28070043,"url":"https://github.com/rpcpool/yellowstone-grpc-kafka","last_synced_at":"2025-05-12T19:35:55.589Z","repository":{"id":258937394,"uuid":"867028645","full_name":"rpcpool/yellowstone-grpc-kafka","owner":"rpcpool","description":"Yellowstone gRPC Kafka consumer/dedup/producer","archived":false,"fork":false,"pushed_at":"2025-03-18T19:34:05.000Z","size":59,"stargazers_count":32,"open_issues_count":6,"forks_count":13,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-05-03T01:22:55.769Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rpcpool.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-03T10:20:57.000Z","updated_at":"2025-04-24T19:54:01.000Z","dependencies_parsed_at":"2024-11-11T03:01:45.009Z","dependency_job_id":null,"html_url":"https://github.com/rpcpool/yellowstone-grpc-kafka","commit_stats":null,"previous_names":["rpcpool/yellowstone-grpc-kafka"],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rpcpool%2Fyellowstone-grpc-kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rpcpool%2Fyellowstone-grpc-kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rpcpool%2Fyellowstone-grpc-kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rpcpool%2Fyellowstone-grpc-kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rpcpool","download_url":"https://codeload.github.com/rpcpool/yellowstone-grpc-kafka/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253808714,"owners_count":21967586,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-12T19:35:55.097Z","updated_at":"2025-05-12T19:35:55.576Z","avatar_url":"https://github.com/rpcpool.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Yellowstone gRPC Kafka Tool\n\nForward gRPC stream to Kafka, dedup, read stream from Kafka with gRPC server.\n\nModes:\n\n- `grpc2kafka` — connect to gRPC with specified filter and sent all incoming messages to the Kafka\n- `dedup` — consume messages from Kafka and sent deduplicated messages to another topic (right now only support `memory` as deduplication backend)\n- `kafka2grpc` — provide gRPC endpoint with sending messages from Kafka\n\n```bash\n$ cargo run --bin grpc-kafka -- --help\nYellowstone gRPC Kafka Tool\n\nUsage: grpc-kafka [OPTIONS] --config \u003cCONFIG\u003e \u003cCOMMAND\u003e\n\nCommands:\n  dedup       Receive data from Kafka, deduplicate and send them back to Kafka\n  grpc2kafka  Receive data from gRPC and send them to the Kafka\n  kafka2grpc  Receive data from Kafka and send them over gRPC\n  help        Print this message or the help of the given subcommand(s)\n\nOptions:\n  -c, --config \u003cCONFIG\u003e          Path to config file\n      --prometheus \u003cPROMETHEUS\u003e  Prometheus listen address\n  -h, --help                     Print help\n  -V, --version                  Print version\n```\n\n##### Development\n\n```bash\n# run kafka locally\ndocker-compose -f docker-kafka.yml up\n# create topic\nkafka_2.13-3.5.0/bin/kafka-topics.sh --bootstrap-server localhost:29092 --create --topic grpc1\n# send messages from gRPC to Kafka\ncargo run --bin grpc-kafka -- --config config-kafka.json grpc2kafka\n# read messages from Kafka\nkafka_2.13-3.5.0/bin/kafka-console-consumer.sh --bootstrap-server localhost:29092 --topic grpc1\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frpcpool%2Fyellowstone-grpc-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frpcpool%2Fyellowstone-grpc-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frpcpool%2Fyellowstone-grpc-kafka/lists"}