{"id":22852264,"url":"https://github.com/ironforge-cloud/geyser-kafka","last_synced_at":"2025-10-30T23:07:20.345Z","repository":{"id":168664111,"uuid":"643938933","full_name":"ironforge-cloud/geyser-kafka","owner":"ironforge-cloud","description":"Solana geyser plugin implementing a Kafka publisher","archived":false,"fork":false,"pushed_at":"2024-01-11T02:42:49.000Z","size":475,"stargazers_count":7,"open_issues_count":2,"forks_count":2,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-03-27T22:51:08.399Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ironforge-cloud.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-22T13:20:34.000Z","updated_at":"2025-02-13T09:22:11.000Z","dependencies_parsed_at":"2025-02-06T11:42:58.540Z","dependency_job_id":null,"html_url":"https://github.com/ironforge-cloud/geyser-kafka","commit_stats":null,"previous_names":["ironforge-cloud/geyser-kafka"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ironforge-cloud%2Fgeyser-kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ironforge-cloud%2Fgeyser-kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ironforge-cloud%2Fgeyser-kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ironforge-cloud%2Fgeyser-kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ironforge-cloud","download_url":"https://codeload.github.com/ironforge-cloud/geyser-kafka/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248854186,"owners_count":21172310,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-13T06:07:42.516Z","updated_at":"2025-10-23T20:50:57.046Z","avatar_url":"https://github.com/ironforge-cloud.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Solana AccountsDB Plugin for Kafka\n\n\u003e This codebase was forked from the [Blockdaemon/solana-accountsdb-plugin-kafka](https://github.com/Blockdaemon/solana-accountsdb-plugin-kafka) repository\n\nKafka publisher for use with Solana's [plugin framework](https://docs.solana.com/developing/plugins/geyser-plugins).\n\n## Installation\n\n### Binary releases\n\nFind binary releases at: https://github.com/ironforge/geyser-kafka/releases\n\n### Building from source\n\n#### Prerequisites\n\nYou will need version 3.12 or later of the protobuf compiler `protoc` installed.\n\n#### Build\n\n```shell\ncargo build --release\n```\n\n- Linux: `./target/release/libsolana_accountsdb_plugin_kafka.so`\n- macOS: `./target/release/libsolana_accountsdb_plugin_kafka.dylib`\n\n**Important:** Solana's plugin interface requires the build environment of the Solana validator and this plugin to be **identical**.\n\nThis includes the Solana version and Rust compiler version.\nLoading a plugin targeting wrong versions will result in memory corruption and crashes.\n\n## Config\n\nThe config is specified via the plugin's JSON config file. It contains settings that apply to all\nenvironments and some that are environment specific.\n\n### Environment Config Values\n\nThis config needs to be added per environment. Thus using different allow list settings for\ndifferent Kafka clusters is possible. Each item in the `environments` array is an `EnvConfig`\nand follows the below schema.\n\nAn environment config for local development and testing also can be provided, please see \n_Local Environment Config Values_ below.\n\n* **name** (`String`)\n    * Name of the environment\n* **kafka** (`HashMap\u003cString, String\u003e`)\n    * Kafka [`librdkafka` config options](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md).\n* **program_allowlist** (`Vec\u003cString\u003e`)\n    * Allowlist of programs to publish.\n        * If empty, all accounts are published.\n        * If not empty, only accounts owned by programs in this list are published.\n* **program_allowlist_url** (`String`)\n    * URL to fetch allowlist updates from\n    * The file must be json, and with the following schema:\n      ```json\n      {\n        \"result\": [\n            \"11111111111111111111111111111111\",\n            \"22222222222222222222222222222222\"\n        ]\n      }\n      ```\n* **program_allowlist_auth** (`String`)\n    * Allowlist Authorization header value.\n        * If provided the request to the program_allowlist_url will add an\n            'Authorization: \u003cvalue\u003e' header.\n        * A sample auth header value would be 'Bearer my_long_secret_token'.\n* **program_allowlist_slot_interval** (u64)\n    * Slots interval which determines how many slots to wait before updating the allowlist.\n\n### Global Config Values\n\nThe below config values are global and apply to all environments. Environment specific settings\nare added to the `environments` array.\n\n* **cluster** (`String`)\n    * The cluster value to include in messages such that Kafka event receivers have context\n      regarding updates (implemented for account updates only for now).\n    * possible values `mainnet|devnet|testnet` (default: `mainnet`)\n* **shutdown_timeout_ms** (`u64`)\n    * Time the plugin is given to flush out all messages to Kafka\n        * and gracefully shutdown upon exit request.\n* **update_account_topic** (`String`)\n    * Kafka topic to send account updates to.\n        * Omit to disable.\n* **slot_status_topic** (`String`)\n    * Kafka topic to send slot status updates to.\n        * Omit to disable.\n* **transaction_topic** (`String`)\n    * Kafka topic to send transaction updates to.\n        * Omit to disable.\n* **update_account_topic_overrides** (`HashMap\u003cString, HashSet\u003cString\u003e\u003e`)\n    * Kafka topic overrides to send specific account updates to. \n      * Omit to disable. \n      * The keys are the alternate topics and the value is a collection of program addresses.\n        If an account's owner matches one of those addresses its updates are sent to the\n        alternative topic instead of `[update_account_topic]`.\n* **publish_all_accounts** (`bool`)\n    * Publish all accounts on startup.\n        * Omit to disable.\n* **publish_accounts_without_signature** (`bool`)\n    * Publishes account updates even if the txn_signature is not present.\n        * This will include account updates that occur without a corresponding\n            * transaction, i.e. caused by validator book-keeping.\n        * Omit to disable.\n* **publish_account_deletions** (bool)\n   * Detects account deletions from transactions where lamports went to `0` for an account\n     and publishes them for each other account, that is part\n     of that transaction (assuming it could be the owner of the deleted account).\n   *  **NOTE** that this will enable transaction notifications. \n   * Omit to disable. **Default:** `false`\n* **wrap_messages** (`bool`)\n    * Wrap all messages in a unified wrapper object.\n        * Omit to disable.\n* **environments** (`Vec\u003cEnvConfig\u003e`)\n    * Kafka cluster and allow list configs for different environments.\n        * See [EnvConfig].\n\n* **prometheus**: (`String`)\n  * _Optional_ Prometheus endpoint, if provided metrics will be sent there\n\n### Example Config\n\n```json\n{\n  \"libpath\": \"/home/solana/geyser-kafka/target/release/libsolana_accountsdb_plugin_kafka.so\",\n  \"cluster\": \"mainnet\",\n  \"shutdown_timeout_ms\": 30000,\n  \"update_account_topic\": \"geyser.mainnet.account_update\",\n  \"update_slot_topic\": \"geyser.mainnet.slot_update\",\n  \"update_transaction_topic\": \"geyser.mainnet.transaction_update\",\n  \"update_account_topic_overrides\": {\n    \"geyser.mainnet.spl.account_update\": [\n      \"TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA\",\n      \"TokenzQdBNbLqP5VEhdkAS6EPFLC1PHnBqCXEpPxuEb\"\n    ]\n  },\n  \"publish_all_accounts\": false,\n  \"publish_accounts_without_signature\": false,\n  \"publish_account_deletions\": true,\n  \"wrap_messages\": false,\n  \"environments\": [\n    {\n      \"name\": \"dev\",\n      \"program_allowlist_url\": \"https://example.com/supported-programs\",\n      \"program_allowlist_auth\": \"Bearer \u003cdev secret bearer token\u003e\",\n      \"program_allowlist_slot_interval\": 150,\n      \"kafka\": {\n        \"bootstrap.servers\": \"dev.bootstrap-server:9092\",\n        \"sasl.username\": \"\u003cusername\u003e\",\n        \"sasl.password\": \"\u003cbase64 encoded password\u003e\",\n        \"sasl.mechanism\": \"SCRAM-SHA-256\",\n        \"security.protocol\": \"SASL_SSL\",\n        \"request.required.acks\": \"1\",\n        \"message.timeout.ms\": \"30000\",\n        \"compression.type\": \"lz4\",\n        \"partitioner\": \"random\"\n      }\n    },\n    {\n      \"name\": \"stage\",\n      \"program_allowlist_url\": \"https://example.com/supported-programs\",\n      \"program_allowlist_auth\": \"Bearer \u003cstage secret bearer token\u003e\",\n      \"program_allowlist_slot_interval\": 90,\n      \"kafka\": {\n        \"bootstrap.servers\": \"stage.bootstrap-server:9092\",\n        \"sasl.username\": \"\u003cusername\u003e\",\n        \"sasl.password\": \"\u003cbase64 encoded password\u003e\",\n        \"sasl.mechanism\": \"SCRAM-SHA-256\",\n        \"security.protocol\": \"SASL_SSL\",\n        \"request.required.acks\": \"1\",\n        \"message.timeout.ms\": \"30000\",\n        \"compression.type\": \"lz4\",\n        \"partitioner\": \"random\"\n      }\n    },\n    {\n      \"name\": \"local\",\n      \"url\": \"http://localhost:9999\",\n      \"include_system_accounts\": false\n    }\n  ]\n}\n```\n### Local Environment Config Values\n\nIn order to facilitate local development and testing the plugin itself a `local` environment\ncan be configured. Mainly it switches out how events are published. Instead of publishing to\nKafka they are published to the configured URL using the respective topic as the path appended\nto that URL.\n\n**NOTE**: this should only be used when running the Geyser plugin with a local solana test\nvalidator as it publishes events over HTTP synchronously which will cause performance issues\nwhen used with a live cluster.\n\nIn the above example account updates would be published to\n`http://localhost:9999/geyser.mainnet.account_update`.\n\nTools like [geyser-store](https://github.com/ironforge-cloud/geyser-store) can be used to to\ntrace and store events locally for further analysis or to allow checking them in tests.\n\n* **name**: The name of the environment.\n* **program_allowlist**: A list of programs whose accounts should be published. If empty, all\n  accounts are published including system program accounts unless `include_system_accounts` is\n  `false`\n* **url**: The URL to publish to.\n* **include_system_accounts**: If `true`, then all system accounts are included when no `program_allowlist` is set. Otherwise, the following accounts are ignored:\n    * System Program: `11111111111111111111111111111111`\n    * BPF Loader: `BPFLoaderUpgradeab1e1111111111111111111111`\n    * Vote Program: `Vote111111111111111111111111111111111111111`\n    * Config Program: `Config1111111111111111111111111111111111111`\n\n### Message Keys\n\nThe message types are keyed as follows:\n- **Account update:** account address (public key)\n- **Slot status:** slot number\n- **Transaction notification:** transaction signature\n\n### Filtering\n\n~~If `program_ignores` are specified, then these addresses will be filtered out of the account updates\nand transaction notifications.  More specifically, account update messages for these accounts will not be emitted,\nand transaction notifications for any transaction involving these accounts will not be\nemitted.~~\n`program_ignores` were removed and only `program_allowlist` is supported now.\n\n### Message Wrapping\n\nIn some cases it may be desirable to send multiple types of messages to the same topic,\nfor instance to preserve relative order.  In this case it is helpful if all messages conform to a single schema.\nSetting `wrap_messages` to true will wrap all three message types in a uniform wrapper object so that they\nconform to a single schema.\n\nNote that if `wrap_messages` is true, in order to avoid key collision, the message keys are prefixed with a single byte,\nwhich is dependent on the type of the message being wrapped.  Account update message keys are prefixed with\n65 (A), slot status keys with 83 (S), and transaction keys with 84 (T).\n\n## Buffering\n\nThe Kafka producer acts strictly non-blocking to allow the Solana validator to sync without much induced lag.\nThis means incoming events from the Solana validator will get buffered and published asynchronously.\n\nWhen the publishing buffer is exhausted any additional events will get dropped.\nThis can happen when Kafka brokers are too slow or the connection to Kafka fails.\nTherefor it is crucial to choose a sufficiently large buffer.\n\nThe buffer size can be controlled using `librdkafka` config options, including:\n- `queue.buffering.max.messages`: Maximum number of messages allowed on the producer queue.\n- `queue.buffering.max.kbytes`: Maximum total message size sum allowed on the producer queue.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fironforge-cloud%2Fgeyser-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fironforge-cloud%2Fgeyser-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fironforge-cloud%2Fgeyser-kafka/lists"}