{"id":28070041,"url":"https://github.com/rpcpool/yellowstone-fumarole","last_synced_at":"2025-05-12T19:35:55.771Z","repository":{"id":278634693,"uuid":"907518620","full_name":"rpcpool/yellowstone-fumarole","owner":"rpcpool","description":"Public repo for Yellowstone-Fumarole","archived":false,"fork":false,"pushed_at":"2025-05-05T19:42:04.000Z","size":356,"stargazers_count":20,"open_issues_count":3,"forks_count":4,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-05-05T19:52:31.676Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rpcpool.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-12-23T19:10:48.000Z","updated_at":"2025-05-03T18:21:19.000Z","dependencies_parsed_at":null,"dependency_job_id":"6f23b1c2-6fa8-40e4-bfb2-ad438c591307","html_url":"https://github.com/rpcpool/yellowstone-fumarole","commit_stats":null,"previous_names":["rpcpool/yellowstone-fumarole"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rpcpool%2Fyellowstone-fumarole","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rpcpool%2Fyellowstone-fumarole/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rpcpool%2Fyellowstone-fumarole/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rpcpool%2Fyellowstone-fumarole/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rpcpool","download_url":"https://codeload.github.com/rpcpool/yellowstone-fumarole/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253808714,"owners_count":21967586,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-12T19:35:50.018Z","updated_at":"2025-05-12T19:35:55.756Z","avatar_url":"https://github.com/rpcpool.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# yellowstone-fumarole\n\nPublic repo for Yellowstone-Fumarole\n\n## Fume CLI\n\nSee more details in fume [README](fume/README.md)\n\n## Rust client\n\nWe offer a simple fumarole Rust client crate in `crates/yellowstone-fumarole-client`.\n\nAn example can be found in `examples/rust/client.rs`.\nSee rust example [README](examples/rust/README.md) for more details.\n\n\n## How Fumarole Works\n\nFumarole collects and stores geyser events from multiple RPC nodes, creating a unified log where each geyser event is stored with a unique ever increasing offset.\n\nTo enhance scalability for both reads and writes, Fumarole distributes data across multiple partitions. \n\nEvery shard has its private sequence generator that assigned unique offset to\neach geyser event.\n\n### Consumer groups\n\nFumarole supports consumer groups, designed to scale reads similarly to Kafka consumer groups. However, Fumarole specifically implements static consumer groups.\n\nWhen creating a consumer group, you can define the number of parallel readers, with a maximum limit of six. In a group with six members, each member is assigned a subset of blockchain event partitions. Every member maintains its own offset tracker, which records its position in the Fumarole log.\n\nTo ensure proper operation, all members must remain active. If any member becomes stale, the entire consumer group is considered stale.\n\nConsumer group are bound to a commitment level: you must decide if you want to listen on \n`PROCESSED`, `CONFIRMED` or `FINALIZED` data.\n\n### Parallel streaming\n\nConsumer group with size greater than `1` allows you to streaming geyser event in paralle on two distinct TCP connection.\n\nEach TCP connections returns different geyser event.\n\nEach member of your consumer group gets exclusive access on a equal amount of partitions in Fumarole.\n\n**Note**: If you create a consumer group of size \u003e `1` you need to make sure\nto consume from every member, otherwise you will eventually make your consumer group stale.\n\n### Stale consumer group\n\nFumarole has a retention policy of two (`2`) hours.\n\nIf you create a consumer group and one of the member still points to stale offset, the whole consumer group will become stale.\n\nOnce a consumer group is stale you cannot use it anymore and must delete it using [fume](fume).\n\n### Consumer group offset commitment\n\nFumarole uses automatic offset commitment and stores your offset every `500ms`.\n\nLater version of Fumarole will support manual offset commit which better precision and\nremoves the risk of skipping data during transmission failure.\n\n### Creating a consumer group\n\nWhen you crate a consumer group you must provide the following information:\n\n1. The name of the group\n2. The size of the group: maximum of `6`\n3. What you want to listen to: `account`, `tx` or `both`\n4. Initial offset:\n    - LATEST: start at the peek of the log\n    - EARLIEST: start at the beginning of the log\n    - SLOT: You provide a slot number where you want to start at. Fumarole will clip you closest to that slot.\n\nAs of right now, every customer account is limited to 15 consumer groups.\n\nTo create a consumer group, use [fume create-cg](fume) command:\n\n```sh\nfume --config \u003cPATH/TO/CONFIG.YAML\u003e create-cg --help\nUsage: fume create-cg [OPTIONS]\n\n  Creates a consumer group\n\nOptions:\n  --name TEXT                     Consumer group name to subscribe to, if none\n                                  provided a random name will be generated\n                                  following the pattern\n                                  'fume-\u003crandom-6-character\u003e'.\n  --size INTEGER                  Size of the consumer group\n  --commitment [processed|confirmed|finalized]\n                                  Commitment level  [default: confirmed]\n  --include [all|account|tx]      Include option  [default: all]\n  --seek [earliest|latest|slot]   Seek option  [default: latest]\n  --help                          Show this message and exit.\n```\n\n### Consumer group size recommandation\n\nYou don't have to over provisionned your consumer group.\nBigger consumer group can be more complex to manager and higher risk of having staleness.\n\nWhatever you want to consume always start with a consumer group of size `1`\nand increase as you need.\n\nThe most important criteria to consume data is still your bandwidth capacity and network latency with our datacenters.\n\n\n### Consumer groups limitations\n\n- Maximum group size : 6\n- Number of consumer groups per customer account: 15\n- Event you can subscribe too: account updates and transactions\n- Consumer group can not change commitment level once created.\n- If one member of the consumer group become stale, the entire consumer group become stale.\n- Stale consumer group cannot be recuperate\n- Time before stale : TBD\n- One TCP connection per member\n- Because of partitionning, streaming geyser event are not sorted by slot.\n- Fumarole deliver at-least one semantics\n\n## Dragonsmouth vs Fumarole\n\n|| gRPC | Persisted | Redundant | \n|-------|------|-----------|-----------|\n| Fumarole | ✅ | ✅ | ✅ |\n| Dragonsmouth | ✅ | ❌ | ❌ |  \n\n\n**Persisted** : If you drop your connection with Fumarole and reconnect within a reasonnable amount of\ntime, you won't loose any data. You restart right where you left off.\n\n**Redundant** : Fumarole backend is fed by multiple RPC Nodes and data is stored across multiple servers\nallowing redundancy and better read/write scalability.\n\n**gRPC**: Fumarole subscribe stream outs _Dragonsmouth_ `SubscribeUpdate` object so the learning curve\nfor fumarole stays low and can integrate easily into your code without too much changes.\n\n**Note**: You don't have to do anything to benefits from redundancy and persistence. It is done\nin the backend for you.\n\n### Filtering compatibility\n\nFumarole supports the exact same accounts and transactions filter as _Dragonsmouth_.\n\nHere's a reminder of _Dragonsmouth_ gRPC `SubscribeRequest`:\n\n```proto\nmessage SubscribeRequest {\n  map\u003cstring, SubscribeRequestFilterAccounts\u003e accounts = 1;\n  map\u003cstring, SubscribeRequestFilterSlots\u003e slots = 2;\n  map\u003cstring, SubscribeRequestFilterTransactions\u003e transactions = 3;\n  map\u003cstring, SubscribeRequestFilterTransactions\u003e transactions_status = 10;\n  map\u003cstring, SubscribeRequestFilterBlocks\u003e blocks = 4;\n  map\u003cstring, SubscribeRequestFilterBlocksMeta\u003e blocks_meta = 5;\n  map\u003cstring, SubscribeRequestFilterEntry\u003e entry = 8;\n  optional CommitmentLevel commitment = 6;\n  repeated SubscribeRequestAccountsDataSlice accounts_data_slice = 7;\n  optional SubscribeRequestPing ping = 9;\n  optional uint64 from_slot = 11;\n}\n```\n\nHere's Fumarole `SubscribeRequest`\n\n```proto\nmessage SubscribeRequest {\n  string consumer_group_label = 1; // name of the consumer group\n  optional uint32 consumer_id = 2; //  #num consumer group member, 0 by default\n  map\u003cstring, geyser.SubscribeRequestFilterAccounts\u003e accounts = 3;  // Same as Dragonsmouth\n  map\u003cstring, geyser.SubscribeRequestFilterTransactions\u003e transactions = 4; // Same as Dragonsmouth\n}\n```\n\n### Coding examples\n\nTo see the difference between Dragonsmouth and fumarole compare two files [dragonsmouth.rs](examples/rust/src/bin/dragonsmouth.rs) and\n[client.rs](examples/rust/src/bin/client.rs). \n\nMore precisely the only difference between the two is how you subscribe.\n\n\nHere is Dragonsmouth:\n\n```rust\nlet endpoint = config.endpoint.clone();\n\nlet mut geyser = GeyserGrpcBuilder::from_shared(endpoint)\n    .expect(\"Failed to parse endpoint\")\n    .x_token(config.x_token)\n    .expect(\"x_token\")\n    .tls_config(ClientTlsConfig::new().with_native_roots())\n    .expect(\"tls_config\")\n    .connect()\n    .await\n    .expect(\"Failed to connect to geyser\");\n\n// This request listen for all account updates and transaction updates\nlet request = SubscribeRequest {\n    accounts: HashMap::from(\n        [(\"f1\".to_owned(), SubscribeRequestFilterAccounts::default())]\n    ),\n    transactions: HashMap::from(\n        [(\"f1\".to_owned(), SubscribeRequestFilterTransactions::default())]\n    ),\n    ..Default::default()\n};\nlet (_sink, mut rx) = geyser.subscribe_with_request(Some(request)).await.expect(\"Failed to subscribe\");\n```\n\n\nAnd here's the more concise Fumarole version:\n\n\n```rust\nlet requests = yellowstone_fumarole_client::SubscribeRequestBuilder::default()\n    .build(args.cg_name);\n\nlet fumarole = FumaroleClientBuilder::default().connect(config);\nlet rx = fumarole\n    .subscribe_with_request(request)\n    .await\n    .expect(\"Failed to subscribe to Fumarole service\");\n```\n\n\nIf you want better control of your `SubscribeRequest` build process, you can fallback to the its _de-sugar_ form, without the builder pattern:\n\n```rust\nlet request = yellowstone_fumarole_client::proto::SubscribeRequest {\n    consumer_group_label: \"my_group\".to_string(),\n    accounts: HashMap::from(\n        [(\"f1\".to_owned(), SubscribeRequestFilterAccounts::default())]\n    ),\n    transactions: HashMap::from(\n        [(\"f1\".to_owned(), SubscribeRequestFilterTransactions::default())]\n    ),\n}\n\nlet fumarole = FumaroleClientBuilder::default().connect(config);\nlet rx = fumarole\n    .subscribe_with_request(request)\n    .await\n    .expect(\"Failed to subscribe to Fumarole service\");\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frpcpool%2Fyellowstone-fumarole","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frpcpool%2Fyellowstone-fumarole","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frpcpool%2Fyellowstone-fumarole/lists"}