{"id":26153966,"url":"https://github.com/openSVM/zindexer","last_synced_at":"2025-03-11T08:03:02.954Z","repository":{"id":281586906,"uuid":"923301374","full_name":"openSVM/zindexer","owner":"openSVM","description":"solana indexer for clickhouse in zig","archived":false,"fork":false,"pushed_at":"2025-03-10T03:40:02.000Z","size":46,"stargazers_count":8,"open_issues_count":3,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-10T04:32:23.737Z","etag":null,"topics":["zig","zig-package","ziglana","ziglang"],"latest_commit_sha":null,"homepage":"","language":"Zig","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openSVM.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-28T01:25:04.000Z","updated_at":"2025-03-02T17:16:14.000Z","dependencies_parsed_at":"2025-03-10T04:32:27.531Z","dependency_job_id":"f661b368-64b2-4494-97ea-78a8557eb338","html_url":"https://github.com/openSVM/zindexer","commit_stats":null,"previous_names":["opensvm/zindexer"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openSVM%2Fzindexer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openSVM%2Fzindexer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openSVM%2Fzindexer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openSVM%2Fzindexer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openSVM","download_url":"https://codeload.github.com/openSVM/zindexer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242889327,"owners_count":20201947,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["zig","zig-package","ziglana","ziglang"],"created_at":"2025-03-11T08:02:10.931Z","updated_at":"2025-03-11T08:03:02.939Z","avatar_url":"https://github.com/openSVM.png","language":"Zig","funding_links":[],"categories":["Solana Data and Indexing"],"sub_categories":["Notes"],"readme":"# OpenSVM: High-Performance Solana Validator Monitor\n\n![License](https://img.shields.io/badge/license-MIT-blue.svg)\n![Build](https://img.shields.io/badge/build-passing-brightgreen.svg)\n![Language](https://img.shields.io/badge/language-Zig-orange.svg)\n\nA blazingly fast, memory-efficient Solana validator monitoring system written in Zig. Designed for high-throughput, low-latency block ingestion and real-time analytics.\n\n## 🚀 Performance Benchmarks\n\n| Metric | Performance |\n|--------|------------|\n| Block Ingestion Rate | 25,000+ blocks/sec |\n| Memory Usage | \u003c 50MB base, ~2MB/million slots |\n| CPU Usage | \u003c 2% on single core |\n| Latency | \u003c 0.5ms p99 |\n| WebSocket Reconnection | \u003c 10ms |\n| Concurrent Connections | 10,000+ |\n\n## ✨ Features\n\n- 🔥 Zero-copy block parsing\n- 🔄 Automatic WebSocket reconnection with exponential backoff\n- 🔒 TLS/WSS support with certificate verification\n- 📊 Real-time slot monitoring\n- 💾 ClickHouse integration for analytics\n- 🎯 Zero-allocation hot path\n- 🛡️ Memory-safe implementation\n- 📈 Built-in performance metrics\n\n## 🛠 Installation\n\n```bash\ngit clone https://github.com/yourusername/opensvm.git\ncd opensvm\nzig build -Drelease-fast\n```\n\n## 🚦 Quick Start\n\n```bash\n# Set environment variables\nexport SOLANA_RPC_WS=\"wss://your-rpc-endpoint\"\nexport CLICKHOUSE_URL=\"http://localhost:8123\"\n\n# Run the indexer\n./zig-out/bin/solana-indexer\n```\n\n## 🔧 Configuration\n\n| Environment Variable | Description | Default |\n|---------------------|-------------|---------|\n| SOLANA_RPC_WS | WebSocket RPC endpoint | - |\n| CLICKHOUSE_URL | ClickHouse connection URL | http://localhost:8123 |\n| CLICKHOUSE_USER | ClickHouse username | default |\n| CLICKHOUSE_PASS | ClickHouse password | - |\n| CLICKHOUSE_DB | ClickHouse database | default |\n\n## 📊 Architecture\n\n```\n┌──────────────┐     ┌──────────────┐     ┌──────────────┐\n│   WebSocket  │ --\u003e │    Parser    │ --\u003e │  ClickHouse  │\n│   Client     │     │    Engine    │     │   Storage    │\n└──────────────┘     └──────────────┘     └──────────────┘\n       ↑                    ↑                    ↑\n       │                    │                    │\n       └── Zero Copy ───────┴── Zero Alloc ─────┘\n```\n\n## 💫 Advanced Features\n\n### Zero-Copy Block Processing\nThe system uses Zig's comptime features to generate zero-copy parsers, allowing direct processing of network buffers without intermediate allocations.\n\n### Memory Management\n- Preallocated buffer pools\n- Arena allocators for batch processing\n- Comptime memory optimization\n- Zero heap allocations in hot paths\n\n### Error Handling\n- Automatic reconnection with exponential backoff\n- Comprehensive error reporting\n- Graceful degradation under load\n- Self-healing connection management\n\n## 🔍 Monitoring\n\nBuilt-in metrics available at `/metrics`:\n- Block processing latency\n- Memory usage\n- WebSocket connection status\n- Parse errors\n- System health\n\n## 🎯 Use Cases\n\n1. **Validator Monitoring**\n   - Real-time slot tracking\n   - Fork detection\n   - Performance analysis\n\n2. **Network Analysis**\n   - Block propagation metrics\n   - Network health monitoring\n   - Consensus participation tracking\n\n3. **Performance Testing**\n   - TPS benchmarking\n   - Latency analysis\n   - Resource utilization tracking\n\n## 🤝 Contributing\n\nContributions are welcome! Please read our [Contributing Guide](CONTRIBUTING.md) for details on our code of conduct and the process for submitting pull requests.\n\n## 📝 License\n\nThis project is licensed under strict territorial restrictions - see the [LICENSE](LICENSE) file for details.\n\n## 🌟 Acknowledgments\n\n- Solana Labs for the amazing blockchain platform\n- ClickHouse team for the high-performance analytics database\n- Zig language team for the powerful systems programming language","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FopenSVM%2Fzindexer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FopenSVM%2Fzindexer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FopenSVM%2Fzindexer/lists"}