{"id":28766280,"url":"https://github.com/validatorsdao/solana-stream","last_synced_at":"2026-04-08T19:38:33.034Z","repository":{"id":296206448,"uuid":"992530323","full_name":"ValidatorsDAO/solana-stream","owner":"ValidatorsDAO","description":"Solana Stream SDK","archived":false,"fork":false,"pushed_at":"2025-06-05T15:13:16.000Z","size":402,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-05T16:26:32.920Z","etag":null,"topics":["blockchain","geyser","geyser-plugin","grpc","http2","quic","rpc","rust","shreds","shredstream","solana","typescript","web3"],"latest_commit_sha":null,"homepage":"https://www.npmjs.com/package/@validators-dao/solana-stream-sdk","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ValidatorsDAO.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-29T09:56:37.000Z","updated_at":"2025-06-05T15:13:17.000Z","dependencies_parsed_at":"2025-05-29T14:18:46.469Z","dependency_job_id":null,"html_url":"https://github.com/ValidatorsDAO/solana-stream","commit_stats":null,"previous_names":["validatorsdao/solana-stream"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ValidatorsDAO/solana-stream","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ValidatorsDAO%2Fsolana-stream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ValidatorsDAO%2Fsolana-stream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ValidatorsDAO%2Fsolana-stream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ValidatorsDAO%2Fsolana-stream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ValidatorsDAO","download_url":"https://codeload.github.com/ValidatorsDAO/solana-stream/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ValidatorsDAO%2Fsolana-stream/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260355369,"owners_count":22996463,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blockchain","geyser","geyser-plugin","grpc","http2","quic","rpc","rust","shreds","shredstream","solana","typescript","web3"],"created_at":"2025-06-17T12:00:41.446Z","updated_at":"2026-04-08T19:38:33.026Z","avatar_url":"https://github.com/ValidatorsDAO.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://slv.dev/\" target=\"_blank\"\u003e\n    \u003cimg src=\"https://storage.validators.solutions/SolanaStreamSDK.jpg\" alt=\"SolanaStreamSDK\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://twitter.com/intent/follow?screen_name=ValidatorsDAO\" target=\"_blank\"\u003e\n    \u003cimg src=\"https://img.shields.io/twitter/follow/ValidatorsDAO.svg?label=Follow%20@ValidatorsDAO\" alt=\"Follow @ValidatorsDAO\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://crates.io/crates/solana-stream-sdk\"\u003e\n    \u003cimg alt=\"Crate\" src=\"https://img.shields.io/crates/v/solana-stream-sdk?label=solana-stream-sdk\u0026color=fc8d62\u0026logo=rust\"\u003e\n    \u003c/a\u003e\n  \u003ca href=\"https://www.npmjs.com/package/@validators-dao/solana-stream-sdk\"\u003e\n    \u003cimg alt=\"NPM Version\" src=\"https://img.shields.io/npm/v/@validators-dao/solana-stream-sdk?color=268bd2\u0026label=version\u0026logo=npm\"\u003e\n  \u003c/a\u003e\n  \u003ca aria-label=\"License\" href=\"https://github.com/ValidatorsDAO/solana-stream/blob/main/LICENSE.txt\"\u003e\n    \u003cimg alt=\"\" src=\"https://badgen.net/badge/license/Apache/blue\"\u003e\n  \u003c/a\u003e\n  \u003ca aria-label=\"Code of Conduct\" href=\"https://github.com/ValidatorsDAO/solana-stream/blob/main/CODE_OF_CONDUCT.md\"\u003e\n    \u003cimg alt=\"\" src=\"https://img.shields.io/badge/Contributor%20Covenant-2.1-4baaaa.svg\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n# Solana Stream SDK\n\nA collection of Rust and TypeScript packages for Solana stream data, operated by ValidatorsDAO. This repository is published as open-source software (OSS) and is freely available for anyone to use.\n\n\u003ca href=\"https://solana.com/\"\u003e\n  \u003cimg src=\"https://storage.slv.dev/PoweredBySolana.svg\" alt=\"Powered By Solana\" width=\"200px\" height=\"95px\"\u003e\n\u003c/a\u003e\n\n## Overview\n\nThis project provides libraries and tools for streaming real-time data from the Solana blockchain. It supports both Geyser and Shreds approaches, making it easier for developers to access Solana data streams.\n\n## Choose Your Path\n\n- Geyser gRPC (TypeScript/Rust): production-ready streaming with resilient reconnects\n- Shreds gRPC (TypeScript/Rust): raw shreds over gRPC for high-throughput ingestion\n- UDP Shreds (Rust): lowest-latency signal for detection and trading workflows\n\n## What's New (TypeScript v1.1.0)\n\n- Refreshed starter layout and docs to highlight trading hooks\n- Yellowstone Geyser gRPC connection upgraded to an NAPI-RS-powered client for better backpressure\n- NAPI-powered Shreds client/decoder so TypeScript can tap Rust-grade throughput\n- Improved backpressure handling and up to 4x streaming efficiency (400% improvement)\n- Faster real-time Geyser streams for TypeScript clients with lower overhead\n\n## Production-Ready Geyser Clients (TypeScript + Rust)\n\n- Ping/Pong handling to keep Yellowstone gRPC streams alive\n- Exponential reconnect backoff plus gap recovery (`fromSlot` / `from_slot`)\n- Bounded queues/channels with drop logging for backpressure safety\n- Code-based subscription filters in TypeScript\n- Optional runtime metrics logging (TypeScript)\n- Default filters drop vote/failed transactions to reduce traffic\n- Rust Geyser client ships the same safeguards and powers UDP Shreds for fastest signal\n\nTip: start with slots, then add filters as needed. When resuming from `fromSlot`,\nduplicates are expected.\n\n## Rust Highlight: UDP Shreds (Fastest Signal)\n\nIf you have **ERPC Dedicated Shreds**, you can forward raw Shreds over UDP to your own listener.\nThis is Solana’s fastest observation layer—before Geyser gRPC and far ahead of RPC/WebSocket.\nThe SDK includes a simple Rust sample; pump.fun is used only because it’s the most common\nquestion we get.\n\n### Why this is the fastest path\n\n- Shreds arrive first: validator-to-validator Shreds land before Geyser gRPC or RPC/WebSocket,\n  so latency-critical flows see events earliest.\n- UDP keeps overhead tiny: no connection setup, retransmit, or ordering; matches the on-wire\n  format between validators.\n- Optional latency monitoring uses a DashMap-backed slot tracker to reduce lock contention.\n- Trade-off: pre-finalization data can be missing/out-of-order/failed—handle that as part of the\n  speed bargain.\n\nNote: the shared Shreds gRPC endpoint runs over TCP, so it’s slower than UDP Shreds.\n\n### Try it with Solana Stream SDK\n\n- Sample code (`shreds-udp-rs`, Rust): pump.fun is just a common example—swap in your own target.  \n  https://github.com/ValidatorsDAO/solana-stream/tree/main/temp-release/shreds-udp-rs\n- Quick start requires `settings.jsonc` plus env (e.g., `SOLANA_RPC_ENDPOINT`); see the sample README.\n- Dedicated Shreds users: point your Shreds sender to the sample’s `ip:port` to see detections.\n- Not on UDP yet? Run it locally or on your own server to explore logs and customize hooks.\n\n### Pump.fun example log\n\n![pump.fun hits over UDP Shreds](https://storage.validators.solutions/SolanaStreamSDKUDPClientExample.jpg)\n\nThis example comes from the SDK sample; clone and run it to see hits, or swap in your own target.\n\n### Resources\n\n- All code and README docs are in the Solana Stream SDK repo:  \n  https://github.com/ValidatorsDAO/solana-stream\n\n## Package Structure\n\n### Rust Clients\n\n- **client/geyser-rs/**: Rust client using Geyser plugin (gRPC)\n- **client/shreds-rs/**: Rust client for Shredstream over gRPC\n- **client/shreds-udp-rs/**: Minimal UDP shred listener; includes pump.fun token-mint detection example\n\n### TypeScript Clients\n\n- **client/geyser-ts/**: TypeScript client using Geyser plugin (gRPC)\n- **client/shreds-ts/**: TypeScript client for Shredstream over gRPC\n\n### SDK Packages\n\n- **crate/solana-stream-sdk/**: Rust SDK for Solana stream functionality\n- **package/solana-stream-sdk/**: TypeScript SDK for Solana stream functionality\n\n## Getting Started\n\n### Prerequisites\n\n- Node.js 22 or 24 (LTS) for TypeScript packages\n- Rust (for Rust packages)\n- pnpm (for package management)\n\n### Installation\n\nFor the entire workspace:\n\n```bash\ngit clone https://github.com/ValidatorsDAO/solana-stream.git\ncd solana-stream\npnpm install\n```\n\n### Geyser Client Example – TypeScript\n\nCreate a `.env` file at `client/geyser-ts/.env` with your environment variables:\n\n```env\n# Optional: required only if your endpoint enforces auth\nX_TOKEN=YOUR_X_TOKEN\nGEYSER_ENDPOINT=https://grpc-ams.erpc.global\nSOLANA_RPC_ENDPOINT=\"https://edge.erpc.global?api-key=YOUR_API_KEY\"\n```\n\n⚠️ **Please note:** This endpoint is a sample and cannot be used as is. Please obtain and configure the appropriate endpoint for your environment.\n\nNext, build and run the client:\n\n```bash\npnpm -F @validators-dao/solana-stream-sdk build\npnpm -F geyser-ts dev\n```\n\n- A 1-day free trial for the Geyser gRPC endpoints is available by joining the Validators DAO Discord community. Please try it out:\n\n[https://discord.gg/C7ZQSrCkYR](https://discord.gg/C7ZQSrCkYR)\n\n### Quick Start Guide for Sample Shreds Client - Rust\n\n**Create a `.env` file** (placed in the project root)\n\n```env\nSHREDS_ENDPOINT=https://shreds-ams.erpc.global\nSOLANA_RPC_ENDPOINT=\"https://edge.erpc.global?api-key=YOUR_API_KEY\"\n```\n\n⚠️ **Please note:** This endpoint is a sample and cannot be used as is. Please obtain and configure the appropriate endpoint for your environment.\n\n**Run the sample client**\n\n```bash\ncargo run -p shreds-rs\n```\n\nThe sample code can be found at:\n\n[https://github.com/ValidatorsDAO/solana-stream/blob/main/client/shreds-rs/src/main.rs](https://github.com/ValidatorsDAO/solana-stream/blob/main/client/shreds-rs/src/main.rs)\n\nA 1-day free trial for the Shreds endpoints is available by joining the Validators DAO Discord community. Please try it out: [https://discord.gg/C7ZQSrCkYR](https://discord.gg/C7ZQSrCkYR)\n\n#### Usage with solana-stream-sdk\n\nYou can also use the published crate in your own projects:\n\n```toml\n[dependencies]\nsolana-stream-sdk = \"1.2.0\"\ntokio = { version = \"1\", features = [\"rt-multi-thread\", \"macros\"] }\ndotenvy = \"0.15\"\nsolana-entry = \"3.0.12\"\nbincode = \"1.3.3\"\n```\n\n```rust\nuse solana_stream_sdk::{CommitmentLevel, ShredstreamClient};\nuse std::env;\n\n#[tokio::main]\nasync fn main() -\u003e Result\u003c(), Box\u003cdyn std::error::Error\u003e\u003e {\n    // Load environment variables\n    dotenvy::dotenv().ok();\n\n    // Connect to shreds endpoint\n    let endpoint = env::var(\"SHREDS_ENDPOINT\")\n        .unwrap_or_else(|_| \"https://shreds-ams.erpc.global\".to_string());\n    let mut client = ShredstreamClient::connect(\u0026endpoint).await?;\n\n    // Create subscription for specific account\n    let request = ShredstreamClient::create_entries_request_for_account(\n        \"6EF8rrecthR5Dkzon8Nwu78hRvfCKubJ14M5uBEwF6P\",\n        Some(CommitmentLevel::Processed),\n    );\n\n    // Subscribe to entries stream\n    let mut stream = client.subscribe_entries(request).await?;\n\n    // Process incoming entries\n    while let Some(entry) = stream.message().await? {\n        let entries = bincode::deserialize::\u003cVec\u003csolana_entry::entry::Entry\u003e\u003e(\u0026entry.entries)?;\n        println!(\"Slot: {}, Entries: {}\", entry.slot, entries.len());\n\n        for entry in entries {\n            println!(\"  Entry has {} transactions\", entry.transactions.len());\n        }\n    }\n\n    Ok(())\n}\n```\n\nFor specific packages, navigate to the package directory and install dependencies.\n\n## Shreds UDP Pump.fun Watcher (Rust)\n\n`client/shreds-udp-rs` listens for Shredstream over **UDP** and highlights watched programs (defaults to pump.fun). Settings live in `client/shreds-udp-rs/settings.jsonc` and are embedded at build time; secrets like RPC can be overridden via environment variables.\n\nQuick start:\n\n```bash\nexport SOLANA_RPC_ENDPOINT=https://api.mainnet-beta.solana.com   # pass secrets via env only\ncargo run -p shreds-udp-rs                                       # settings already in settings.jsonc\n```\n\nLog legend:\n\n- Prefix: `🎯` program hit, `🐣` authority hit (`🎯🐣` means both)\n- Action: `🐣` create, `🟢` buy, `🔻` sell, `🪙` other, `❓` unknown/missing amounts\n- Votes skipped by default (`skip_vote_txs=true`)\n- `pump_min_lamports` can suppress small pump.fun buy/sell logs\n- Pump.fun SOL values are instruction limits (max for buy/create, min for sell); actual fills require event/meta data (e.g., Geyser/RPC).\n- UDP shreds are processed directly; not dependent on RPC commitment. Failed transactions may still appear; missing fields show as `❓`.\n\nComponents from `crate/solana-stream-sdk` (5 layers):\n\n- Config loader (`ShredsUdpConfig`): reads JSONC/env and builds `ProgramWatchConfig` (pump.fun defaults; composite mint finder = pump.fun accounts + SPL Token MintTo/Initialize). Use `watch_config_no_defaults()` to opt out of pump.fun fallbacks.\n- Receiver (`UdpShredReceiver`): minimal UDP socket reader with timestamps.\n- Pipeline (5 layers): ① receive/prefilter (`decode_udp_datagram`) → ② FEC buffer (`insert_shred` + `ShredsUdpState`) → ③ deshred (`deshred_shreds_to_entries`) → ④ watcher/detail (`collect_watch_events` + detailers) → ⑤ sink (logs/custom hooks).\n- One-call convenience: `handle_pumpfun_watcher` wraps the same 5 layers (pump.fun defaults).\n- Customize sink/detailer: via `ProgramWatchConfig::with_detailers(...)` or replace the sink with your own hook.\n- Vote filtering: by default `skip_vote_txs=true`, so vote-only shreds/txs are dropped early.\n- Samples: `cargo run -p shreds-udp-rs` (pump.fun defaults, one-call wrapper) or `cargo run -p shreds-udp-rs --bin generic_logger` (pump.fun-free logger; set `GENERIC_WATCH_PROGRAM_IDS` / `GENERIC_WATCH_AUTHORITIES` to watch your own programs).\n\nDesign notes\n\n- Layered pipeline (5 layers): ① UDP receive → ② FEC buffer/pre-deshred → ③ deshred → ④ watcher (mint extraction) → ⑤ detailer/sink (labeling + log output). Each stage can be swapped or reused.\n- Pure UDP/FEC path: single-purpose deshredder tuned for Agave merkle sizing; leaves ledger/rpc out of the hot path.\n- Config is JSONC/env: secrets (RPC) in env, behavior (watch ids, logging) in JSONC; defaults prefill pump.fun watch ids.\n- Pump filters: optional `pump_min_lamports` to log only pump.fun buy/sell with SOL limit above a threshold; logs show `sol:` when the limit is parsed.\n- Composable stages: receiver → deshred → watcher → detailer → sink; each stage can be swapped or reused.\n- Signal-first logging: emoji at a glance, vote-filtered by default, and mint-level detail with adapters (pump.fun).\n- Small, dependency-light SDK crate backing a CLI client; intended to embed into larger consumers as well.\n\nQuick choices:\n\n- Want a one-call, pump.fun-ready loop? Use `handle_pumpfun_watcher` in your own binary and set watch IDs/env as needed. This matches the out-of-the-box behavior shown in the screenshots.\n- Need to act on detections (e.g., push to a queue, custom filtering, alternate watchers/detailers)? Use the modular pipeline (`decode_udp_datagram` → `insert_shred` → `deshred_shreds_to_entries` → `collect_watch_events`) and hook your own sink right after detection (see `client/shreds-udp-rs` custom hook example).\n\nMinimal usage example (Rust):\n\n```rust\nuse solana_stream_sdk::shreds_udp::{ShredsUdpConfig, ShredsUdpState, DeshredPolicy, handle_pumpfun_watcher};\nuse solana_stream_sdk::UdpShredReceiver;\nuse std::sync::Arc;\n\n#[tokio::main]\nasync fn main() -\u003e Result\u003c(), Box\u003cdyn std::error::Error + Send + Sync\u003e\u003e {\n    let cfg = ShredsUdpConfig::from_env(); // reads SHREDS_UDP_CONFIG jsonc too\n    let mut receiver = UdpShredReceiver::bind(\u0026cfg.bind_addr, None).await?;\n    let policy = DeshredPolicy { require_code_match: cfg.require_code_match };\n    let watch_cfg = Arc::new(cfg.watch_config());\n    let state = ShredsUdpState::new(\u0026cfg);\n    loop {\n        handle_pumpfun_watcher(\u0026mut receiver, \u0026state, \u0026cfg, policy, watch_cfg.clone()).await?;\n    }\n}\n```\n\nModular pipeline example (pump.fun opt-out):\n\n```rust\nuse solana_stream_sdk::shreds_udp::{\n    collect_watch_events, decode_udp_datagram, deshred_shreds_to_entries, insert_shred,\n    DeshredPolicy, ShredInsertOutcome, ShredSource, ShredsUdpConfig, ShredsUdpState,\n};\nuse solana_stream_sdk::txn::{ProgramWatchConfig, SplTokenMintFinder};\nuse solana_stream_sdk::UdpShredReceiver;\nuse std::sync::Arc;\n\n#[tokio::main]\nasync fn main() -\u003e Result\u003c(), Box\u003cdyn std::error::Error + Send + Sync\u003e\u003e {\n    let cfg = ShredsUdpConfig::from_env();\n    let mut receiver = UdpShredReceiver::bind(\u0026cfg.bind_addr, None).await?;\n    let policy = DeshredPolicy { require_code_match: cfg.require_code_match };\n    let state = ShredsUdpState::new(\u0026cfg);\n    let watch_cfg = Arc::new(\n        ProgramWatchConfig::new(vec![], vec![]) // opt-out of pump.fun defaults\n            .with_mint_finder(Arc::new(SplTokenMintFinder))\n            .with_detailers(Vec::new()),\n    );\n\n    loop {\n        let datagram = receiver.recv_raw().await?;\n        if let Some(decoded) = decode_udp_datagram(\u0026datagram, \u0026state, \u0026cfg).await {\n            if let ShredInsertOutcome::Ready(ready) =\n                insert_shred(decoded, \u0026datagram, \u0026state, \u0026cfg, \u0026policy).await\n            {\n                let entries = deshred_shreds_to_entries(\u0026ready.shreds)?;\n                let txs: Vec\u003c_\u003e = entries.iter().flat_map(|e| e.transactions.iter()).collect();\n                let _hits = collect_watch_events(ready.key.slot, \u0026txs, watch_cfg.as_ref(), 0);\n                state.remove_batch(\u0026ready.key).await;\n                if matches!(ready.source, ShredSource::Data) {\n                    state.mark_completed(ready.key).await;\n                }\n            }\n        }\n    }\n}\n```\n\n## ⚠️ Experimental Filtering Feature Notice\n\nFiltering remains experimental on the **Shreds gRPC** path (`shreds-rs`, `shreds-ts`): requests should send empty filter maps because shreds-side filters are not usable yet. Geyser gRPC filters are fine. For workloads that need filtering, prefer the high-speed, customizable UDP shreds pipeline described above. Occasionally, data may not be fully available, and filters may not be applied correctly on the shreds gRPC path.\n\nIf you encounter such cases, please report them by opening an issue at: https://github.com/ValidatorsDAO/solana-stream/issues\n\nYour feedback greatly assists our debugging efforts and overall improvement of this feature.\n\nOther reports and suggestions are also highly appreciated.\n\nYou can also join discussions or share feedback on Validators DAO's Discord community:\nhttps://discord.gg/C7ZQSrCkYR\n\n## Development\n\nThis project uses a monorepo structure with both Rust and TypeScript components:\n\n- **Rust packages**: Managed with Cargo\n- **TypeScript packages**: Managed with pnpm workspaces\n- **Unified configuration**: Shared TypeScript and Prettier configurations\n\n### Building\n\n```bash\n# Build all TypeScript packages\npnpm build\n\n# Build Rust packages\ncargo build\n```\n\n## Usage\n\nEach package contains its own documentation and usage examples. Please refer to the individual package READMEs for specific implementation details.\n\n## Contributing\n\nWe welcome contributions from the community! This project is continuously updated and improved.\n\n1. Fork the repository\n2. Create a feature branch\n3. Make your changes\n4. Submit a pull request\n\n## About ValidatorsDAO\n\nThis project is operated and maintained by ValidatorsDAO, focused on providing robust tools and infrastructure for the Solana ecosystem.\n\nhttps://discord.gg/pw7kuJNDKq\n\n## Updates\n\nThis repository is actively maintained and will receive continuous updates to improve functionality and add new features.\n\n## License\n\nThe package is available as open source under the terms of the\n[Apache-2.0 License](https://www.apache.org/licenses/LICENSE-2.0).\n\n## Code of Conduct\n\nEveryone interacting in the Validators DAO project’s codebases, issue trackers, chat rooms\nand mailing lists is expected to follow the\n[code of conduct](https://github.com/ValidatorsDAO/solana-stream/blob/main/CODE_OF_CONDUCT.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvalidatorsdao%2Fsolana-stream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvalidatorsdao%2Fsolana-stream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvalidatorsdao%2Fsolana-stream/lists"}