{"id":14968481,"url":"https://github.com/blockdaemon/solana-accountsdb-plugin-kafka","last_synced_at":"2025-05-16T04:03:40.739Z","repository":{"id":37747719,"uuid":"456379404","full_name":"Blockdaemon/solana-accountsdb-plugin-kafka","owner":"Blockdaemon","description":"Solana geyser plugin implementing a Kafka publisher","archived":false,"fork":false,"pushed_at":"2025-05-07T17:41:49.000Z","size":593,"stargazers_count":114,"open_issues_count":4,"forks_count":43,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-05-16T04:03:18.166Z","etag":null,"topics":["kafka","solana"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Blockdaemon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-02-07T06:08:25.000Z","updated_at":"2025-05-07T17:41:45.000Z","dependencies_parsed_at":"2023-12-21T07:52:51.908Z","dependency_job_id":"43c6e20a-0406-4e71-afa0-d47144f7ceff","html_url":"https://github.com/Blockdaemon/solana-accountsdb-plugin-kafka","commit_stats":{"total_commits":267,"total_committers":10,"mean_commits":26.7,"dds":"0.32209737827715357","last_synced_commit":"83eeaff13529ff0587f6bab7b58cd2d652f724ec"},"previous_names":[],"tags_count":38,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Blockdaemon%2Fsolana-accountsdb-plugin-kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Blockdaemon%2Fsolana-accountsdb-plugin-kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Blockdaemon%2Fsolana-accountsdb-plugin-kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Blockdaemon%2Fsolana-accountsdb-plugin-kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Blockdaemon","download_url":"https://codeload.github.com/Blockdaemon/solana-accountsdb-plugin-kafka/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254464891,"owners_count":22075570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","solana"],"created_at":"2024-09-24T13:39:59.919Z","updated_at":"2025-05-16T04:03:40.733Z","avatar_url":"https://github.com/Blockdaemon.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Solana AccountsDB Plugin for Kafka\n\nKafka publisher for use with Solana's [plugin framework](https://docs.solana.com/developing/plugins/geyser-plugins).\n\n## Installation\n\n### Binary releases\n\nFind binary releases [here](https://github.com/Blockdaemon/solana-accountsdb-plugin-kafka/releases).\n\n### Building from source\n\n#### Prerequisites\n\nYou will need version 3.15 or later of the protobuf compiler `protoc` installed, since it is required for the `--experimental_allow_proto3_optional` option.\n\nNote that as of this writing, ubuntu 22.04 still has an obsolete of `protoc`.\n\nFor ubuntu, CI imports one from debian:\n\n```shell\necho 'deb http://ftp.debian.org/debian stable main' | sudo tee -a /etc/apt/sources.list.d/debian.list\nsudo apt-get update\nsudo apt-get satisfy -f -y \"protobuf-compiler (\u003e=3.15)\"\n```\n\nYou may need the appropriate debian keys:\n\n```shell\nsudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 648ACFD622F3D138\nsudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 0E98404D386FA1D9\n```\n\n#### Build\n\n```shell\ncargo build --release\n```\n\n- Linux: `./target/release/libsolana_accountsdb_plugin_kafka.so`\n- macOS: `./target/release/libsolana_accountsdb_plugin_kafka.dylib`\n\n**Important:** Solana's plugin interface requires the build environment of the Solana validator and this plugin to be **identical**.\n\nThis includes the Solana version and Rust compiler version.\nLoading a plugin targeting wrong versions will result in memory corruption and crashes.\n\n## Config\n\nConfig is specified via the plugin's JSON config file.\n\n### Example Config\n\n```json\n{\n  \"libpath\": \"target/release/libsolana_accountsdb_plugin_kafka.so\",\n  \"kafka\": {\n    \"bootstrap.servers\": \"localhost:9092\",\n    \"request.required.acks\": \"1\",\n    \"message.timeout.ms\": \"30000\",\n    \"compression.type\": \"lz4\",\n    \"partitioner\": \"murmur2_random\",\n    \"statistics.interval.ms\": \"1000\"\n  },\n  \"shutdown_timeout_ms\": 30000,\n  \"filters\": [{\n    \"update_account_topic\": \"solana.testnet.account_updates\",\n    \"slot_status_topic\": \"solana.testnet.slot_status\",\n    \"transaction_topic\": \"solana.testnet.transactions\",\n    \"program_ignores\": [\n      \"Sysvar1111111111111111111111111111111111111\",\n      \"Vote111111111111111111111111111111111111111\"\n    ],\n    \"publish_all_accounts\": false,\n    \"wrap_messages\": false\n  }]\n}\n```\n\n### Reference\n\n- `libpath`: Path to Kafka plugin\n- `kafka`: [`librdkafka` config options](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md).\n- `shutdown_timeout_ms`: Time the plugin is given to flush out all messages to Kafka upon exit request.\n- `prometheus`: Optional port to provide metrics in Prometheus format.\n- `filters`: Vec of filters with next fields:\n  - `update_account_topic`: Topic name of account updates. Omit to disable.\n  - `slot_status_topic`: Topic name of slot status update. Omit to disable.\n  - `transaction_topic`: Topic name of transaction update. Omit to disable.\n  - `program_ignores`: Account addresses to ignore (see Filtering below).\n  - `program_filters`: Solana program IDs to include.\n  - `account_filters`: Solana accounts to include.\n  - `publish_all_accounts`: Publish all accounts on startup. Omit to disable.\n  - `include_vote_transactions`: Include Vote transactions.\n  - `include_failed_transactions`: Include failed transactions.\n  - `wrap_messages`: Wrap all messages in a unified wrapper object. Omit to disable (see Message Wrapping below).\n\n### Message Keys\n\nThe message types are keyed as follows:\n\n- **Account update:** account address (public key)\n- **Slot status:** slot number\n- **Transaction notification:** transaction signature\n\n### Filtering\n\nIf `program_ignores` are specified, then these addresses will be filtered out of the account updates\nand transaction notifications.  More specifically, account update messages for these accounts will not be emitted,\nand transaction notifications for any transaction involving these accounts will not be emitted.\n\n### Message Wrapping\n\nIn some cases it may be desirable to send multiple types of messages to the same topic,\nfor instance to preserve relative order.  In this case it is helpful if all messages conform to a single schema.\nSetting `wrap_messages` to true will wrap all three message types in a uniform wrapper object so that they\nconform to a single schema.\n\nNote that if `wrap_messages` is true, in order to avoid key collision, the message keys are prefixed with a single byte,\nwhich is dependent on the type of the message being wrapped.  Account update message keys are prefixed with\n65 (A), slot status keys with 83 (S), and transaction keys with 84 (T).\n\n## Buffering\n\nThe Kafka producer acts strictly non-blocking to allow the Solana validator to sync without much induced lag.\nThis means incoming events from the Solana validator will get buffered and published asynchronously.\n\nWhen the publishing buffer is exhausted any additional events will get dropped.\nThis can happen when Kafka brokers are too slow or the connection to Kafka fails.\nTherefore it is crucial to choose a sufficiently large buffer.\n\nThe buffer size can be controlled using `librdkafka` config options, including:\n\n- `queue.buffering.max.messages`: Maximum number of messages allowed on the producer queue.\n- `queue.buffering.max.kbytes`: Maximum total message size sum allowed on the producer queue.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblockdaemon%2Fsolana-accountsdb-plugin-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblockdaemon%2Fsolana-accountsdb-plugin-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblockdaemon%2Fsolana-accountsdb-plugin-kafka/lists"}