{"id":28070047,"url":"https://github.com/rpcpool/yellowstone-grpc-gamma","last_synced_at":"2025-05-12T19:36:01.810Z","repository":{"id":217191815,"uuid":"735374262","full_name":"rpcpool/yellowstone-grpc-gamma","owner":"rpcpool","description":"Custom gRPC plugin for Gamma Capital (with timestamp field)","archived":false,"fork":false,"pushed_at":"2024-11-15T14:38:01.000Z","size":1535,"stargazers_count":2,"open_issues_count":0,"forks_count":3,"subscribers_count":4,"default_branch":"v1.18-gamma","last_synced_at":"2025-04-01T11:49:12.375Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rpcpool.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-24T17:22:17.000Z","updated_at":"2024-08-08T15:54:32.000Z","dependencies_parsed_at":"2024-01-15T00:52:31.551Z","dependency_job_id":"2c200b02-ee8a-4a6d-8a62-2cb072a4bc0e","html_url":"https://github.com/rpcpool/yellowstone-grpc-gamma","commit_stats":null,"previous_names":["rpcpool/yellowstone-grpc-gamma"],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rpcpool%2Fyellowstone-grpc-gamma","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rpcpool%2Fyellowstone-grpc-gamma/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rpcpool%2Fyellowstone-grpc-gamma/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rpcpool%2Fyellowstone-grpc-gamma/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rpcpool","download_url":"https://codeload.github.com/rpcpool/yellowstone-grpc-gamma/tar.gz/refs/heads/v1.18-gamma","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253808714,"owners_count":21967586,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-12T19:35:56.959Z","updated_at":"2025-05-12T19:36:01.796Z","avatar_url":"https://github.com/rpcpool.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Yellowstone Dragon's Mouth - a Geyser based gRPC interface for Solana\n\nThis repo contains a fully functional gRPC interface for Solana. It is built around Solana's Geyser interface. In this repo we have the plugin as well as sample clients for multiple languages.\n\nIt provides the ability to get slots, blocks, transactions, and account update notifications over a standardised path. \n\nFor additional documentation,  please see: https://docs.triton.one/rpc-pool/grpc-subscriptions\n\n#### Known bugs\n\nBlock reconstruction inside gRPC plugin based on information provided by BlockMeta, unfortunately number of entries for blocks generated on validators always equal to zero. These blocks always will have zero entries. See issue on GitHub: https://github.com/solana-labs/solana/issues/33823\n\n### Validator\n\n```bash\n$ solana-validator --geyser-plugin-config yellowstone-grpc-geyser/config.json\n```\n\n### Plugin config check\n\n```\ncargo-fmt \u0026\u0026 cargo run --bin config-check -- --config yellowstone-grpc-geyser/config.json\n```\n\n### Block reconstruction\n\nGeyser interface on block update do not provide detailed information about transactions and accounts updates. To provide this information with block message we need to collect all messages and expect specified order. By default if we failed to reconstruct full block we log error message and increase `invalid_full_blocks_total` counter in prometheus metrics. If you want to panic on invalid reconstruction you can change option `block_fail_action` in config to `panic` (default value is `log`).\n\n### Filters for streamed data\n\nPlease check [yellowstone-grpc-proto/proto/geyser.proto](yellowstone-grpc-proto/proto/geyser.proto) for details.\n\n   - `commitment` — commitment level: `processed` / `confirmed` / `finalized`\n   - `accounts_data_slice` — array of objects `{ offset: uint64, length: uint64 }`, allow to receive only required data from accounts\n   - `ping` — optional boolean field. Some cloud providers (like Cloudflare, Fly.io) close the stream if client doesn't send anything during some time. As workaroud you can send same filter every N seconds, but this would be not optimal since you need to keep this filter. Instead, you can send subscribe request with `ping` field set to `true` and ignore rest of the fields in the request. Since we sent `Ping` message every 15s from the server, you can send subscribe request with `ping` as reply and receive `Pong` message.\n\n#### Slots\n\n   - `filter_by_commitment` — by default slots sent for all commitment levels, but with this filter you can receive only selected commitment level\n\n#### Account\n\nAccounts can be filtered by:\n\n   - `account` — acount Pubkey, match to any Pubkey from the array\n   - `owner` — account owner Pubkey, match to any Pubkey from the array\n   - `filters` — same as `getProgramAccounts` filters, array of `dataSize` or `Memcmp` (bytes, base58, base64 are supported)\n\nIf all fields are empty then all accounts are broadcasted. Otherwise fields works as logical `AND` and values in arrays as logical `OR` (except values in `filters` that works as logical `AND`).\n\n#### Transactions\n\n   - `vote` — enable/disable broadcast `vote` transactions\n   - `failed` — enable/disable broadcast `failed` transactions\n   - `signature` — match only specified transaction\n   - `account_include` — filter transactions that use any account from the list\n   - `account_exclude` — opposite to `account_include`\n   - `account_required` — require all accounts from the list to be used in transaction\n\nIf all fields are empty then all transactions are broadcasted. Otherwise fields works as logical `AND` and values in arrays as logical `OR`.\n\n#### Entries\n\nCurrently we do not have filters for the entries, all entries broadcasted.\n\n#### Blocks\n\n   - `account_include` — filter transactions and accounts that use any account from the list\n   - `include_transactions` — include all transactions\n   - `include_accounts` — include all accounts updates\n   - `include_entries` — include all entries\n\n#### Blocks meta\n\nSame as `Blocks` but without `transactions`, `accounts` and entries. Currently we do not have filters for block meta, all messages are broadcasted.\n\n### Limit filters\n\nIt's possible to add limits for filters in the config. If `filters` field is omitted then filters doesn't have any limits.\n\n```json\n\"grpc\": {\n   \"filters\": {\n      \"accounts\": {\n         \"max\": 1,\n         \"any\": false,\n         \"account_max\": 10,\n         \"account_reject\": [\"TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA\"],\n         \"owner_max\": 10,\n         \"owner_reject\": [\"11111111111111111111111111111111\"]\n      },\n      \"slots\": {\n         \"max\": 1\n      },\n      \"transactions\": {\n         \"max\": 1,\n         \"any\": false,\n         \"account_include_max\": 10,\n         \"account_include_reject\": [\"TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA\"],\n         \"account_exclude_max\": 10,\n         \"account_required_max\": 10\n      },\n      \"blocks\": {\n         \"max\": 1,\n         \"account_include_max\": 10,\n         \"account_include_any\": false,\n         \"account_include_reject\": [\"TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA\"],\n         \"include_transactions\": true,\n         \"include_accounts\" : false,\n         \"include_entries\" : false\n      },\n      \"blocks_meta\": {\n         \"max\": 1\n      },\n      \"entry\": {\n         \"max\": 1\n      }\n   }\n}\n```\n\n### Unary gRPC methods\n\n#### Ping\n\n#### GetLatestBlockhash\n\n#### GetBlockHeight\n\n#### GetSlot\n\n#### IsBlockhashValid\n\n#### GetVersion\n\n### Examples\n\n   - [Go](examples/golang)\n   - [Rust](examples/rust)\n   - [TypeScript](examples/typescript)\n\n**NOTE**: Some load balancers will terminate gRPC connections if there are no messages sent from the client for a period of time.\nIn order to mitigate this you need to send a message periodically. The `ping` field in the SubscribeRequest is used for this purpose.\nThe gRPC server already sends pings to the client, so you can simply reply with a ping and your connection will remain open.\nYou can see in the rust example how to reply to the ping from the server with the client.\n\n### gRPC Tools\n\n#### Google Pub/Sub\n\n```bash\n$ cargo run --bin grpc-google-pubsub -- --help\nYellowstone gRPC Google Pub/Sub Tool\n\nUsage: grpc-google-pubsub [OPTIONS] --config \u003cCONFIG\u003e \u003cCOMMAND\u003e\n\nCommands:\n  grpc2pubsub        Receive data from gRPC and send them to the Pub/Sub\n  pubsub2stdout      Dev: subscribe to message from Pub/Sub and print them to Stdout\n  pubsubTopicCreate  Dev: create Pub/Sub topic\n  pubsubTopicDelete  Dev: delete Pub/Sub topic\n  help               Print this message or the help of the given subcommand(s)\n\nOptions:\n  -c, --config \u003cCONFIG\u003e          Path to config file\n      --prometheus \u003cPROMETHEUS\u003e  Prometheus listen address\n  -h, --help                     Print help\n  -V, --version                  Print version\n```\n\n##### Development\n\n```bash\n# export creds\nexport GOOGLE_APPLICATION_CREDENTIALS=/path/to/google/project/creds.json\n# send messages from gRPC to Google Pub/Sub\ncargo run --bin grpc-google-pubsub -- --config yellowstone-grpc-tools/config-google-pubsub.json grpc2pubsub\n```\n\nwith emulator:\n\n```bash\n# retrive `USER_CONFIG_DIR`\n$ gcloud info --format='get(config.paths.global_config_dir)'\n# run emulator, data dir by default: `\u003cUSER_CONFIG_DIR\u003e/emulators/pubsub`\n$ gcloud beta emulators pubsub start\n...\n# send serialized gRPC messages to Google Pub/Sub with PUBSUB_EMULATOR_HOST\n$ PUBSUB_EMULATOR_HOST=localhost:8085 cargo run --bin grpc-google-pubsub -- --config yellowstone-grpc-tools/config-google-pubsub.json grpc2pubsub\n# print type of messages from Google Pub/Sub with PUBSUB_EMULATOR_HOST\n$ PUBSUB_EMULATOR_HOST=localhost:8085 cargo run --bin grpc-google-pubsub -- --config yellowstone-grpc-tools/config-google-pubsub.json --prometheus 1 pubsub2stdout\n```\n\n#### Kafka\n\nIn addition to gRPC Geyser Plugin we provide Kafka tool. This tool can works in 3 modes:\n\n- `grpc2kafka` — connect to gRPC with specified filter and sent all incoming messages to the Kafka\n- `dedup` — consume messages from Kafka and sent deduplicated messages to another topic (right now only support `memory` as deduplication backend)\n- `kafka2grpc` — provide gRPC endpoint with sending messages from Kafka\n\n```bash\n$ cargo run --bin grpc-kafka -- --help\nYellowstone gRPC Kafka Tool\n\nUsage: grpc-kafka [OPTIONS] --config \u003cCONFIG\u003e \u003cCOMMAND\u003e\n\nCommands:\n  dedup       Receive data from Kafka, deduplicate and send them back to Kafka\n  grpc2kafka  Receive data from gRPC and send them to the Kafka\n  kafka2grpc  Receive data from Kafka and send them over gRPC\n  help        Print this message or the help of the given subcommand(s)\n\nOptions:\n  -c, --config \u003cCONFIG\u003e          Path to config file\n      --prometheus \u003cPROMETHEUS\u003e  Prometheus listen address\n  -h, --help                     Print help\n  -V, --version                  Print version\n```\n\n##### Development\n\n```bash\n# run kafka locally\ndocker-compose -f ./yellowstone-grpc-tools/docker-kafka.yml up\n# create topic\nkafka_2.13-3.5.0/bin/kafka-topics.sh --bootstrap-server localhost:29092 --create --topic grpc1\n# send messages from gRPC to Kafka\ncargo run --bin grpc-kafka -- --config yellowstone-grpc-tools/config-kafka.json grpc2kafka\n# read messages from Kafka\nkafka_2.13-3.5.0/bin/kafka-console-consumer.sh --bootstrap-server localhost:29092 --topic grpc1\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frpcpool%2Fyellowstone-grpc-gamma","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frpcpool%2Fyellowstone-grpc-gamma","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frpcpool%2Fyellowstone-grpc-gamma/lists"}