https://github.com/opensvm/zindexer
solana indexer for clickhouse in zig
https://github.com/opensvm/zindexer
zig zig-package ziglana ziglang
Last synced: about 1 year ago
JSON representation
solana indexer for clickhouse in zig
- Host: GitHub
- URL: https://github.com/opensvm/zindexer
- Owner: openSVM
- License: other
- Created: 2025-01-28T01:25:04.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-03-10T03:40:02.000Z (over 1 year ago)
- Last Synced: 2025-03-10T04:32:23.737Z (over 1 year ago)
- Topics: zig, zig-package, ziglana, ziglang
- Language: Zig
- Homepage:
- Size: 44.9 KB
- Stars: 8
- Watchers: 1
- Forks: 0
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
# ZIndexer - Multi-Network SVM Indexer
ZIndexer is a high-performance indexer for Solana Virtual Machine (SVM) networks, capable of subscribing to and indexing data from multiple networks simultaneously.
## Features
- **Multi-Network Support**: Index multiple SVM networks (mainnet, devnet, testnet, localnet) simultaneously
- **Comprehensive Indexing**:
- Automated Market Makers (AMMs)
- Metaplex NFTs and marketplaces
- Token transfers and balances
- Account balance changes
- Transactions and instructions
- Blocks and slots
- Account activity
- **Real-time and Historical Modes**: Choose between real-time indexing or historical backfilling
- **Database Integration**: High-performance storage and querying of indexed data with support for ClickHouse and QuestDB
- **Interactive TUI**: Monitor indexing progress across all networks in real-time
## Requirements
- Zig 0.14.0 or later
- ClickHouse server or QuestDB server
- Internet connection to access SVM networks
## Building
```bash
zig build
```
## Configuration
The indexer uses two configuration files:
- `src/rpc_nodes.json`: HTTP RPC endpoints for each network
- `src/ws_nodes.json`: WebSocket endpoints for each network
You can customize these files to add or remove networks, or to use different RPC providers.
## Usage
```bash
# Run in real-time mode (default)
./zig-out/bin/zindexer
# Run in historical mode
./zig-out/bin/zindexer --mode historical
# Customize ClickHouse connection
./zig-out/bin/zindexer --database-type clickhouse --database-url localhost:9000 --database-user default --database-password "" --database-name solana
# Use QuestDB instead
./zig-out/bin/zindexer --database-type questdb --database-url localhost:9000 --database-user admin --database-password "quest" --database-name solana
# Show help
./zig-out/bin/zindexer --help
```
### Command Line Options
- `-m, --mode `: Indexer mode (historical or realtime)
- `-r, --rpc-nodes `: RPC nodes configuration file
- `-w, --ws-nodes `: WebSocket nodes configuration file
- `-t, --database-type `: Database type (clickhouse or questdb)
- `-c, --database-url `: Database server URL
- `-u, --database-user `: Database username
- `-p, --database-password `: Database password
- `-d, --database-name `: Database name
- `-b, --batch-size `: Batch size for historical indexing
- `--max-retries `: Maximum retry attempts
- `--retry-delay `: Delay between retries in milliseconds
#### Legacy ClickHouse Options (for backward compatibility)
- `--clickhouse-url `: ClickHouse server URL
- `--clickhouse-user `: ClickHouse username
- `--clickhouse-password `: ClickHouse password
- `--clickhouse-database `: ClickHouse database name
#### QuestDB Options
- `--questdb-url `: QuestDB server URL
- `--questdb-user `: QuestDB username
- `--questdb-password `: QuestDB password
- `--questdb-database `: QuestDB database name
- `-h, --help`: Show help message
## Schema Setup
The schema can be automatically applied to your database instance using the included script:
### For ClickHouse
```bash
DB_TYPE=clickhouse CLICKHOUSE_URL="http://localhost:8123" ./scripts/apply_schema.sh
```
### For QuestDB
```bash
DB_TYPE=questdb QUESTDB_URL="http://localhost:9000" ./scripts/apply_schema.sh
```
## Architecture
ZIndexer is built with a modular architecture:
- **Core Indexer**: Manages connections to multiple networks and coordinates indexing
- **RPC Client**: Handles communication with SVM networks via HTTP and WebSocket
- **Database Abstraction Layer**: Provides a common interface for different database backends
- **ClickHouse Client**: Manages data storage and retrieval in ClickHouse
- **QuestDB Client**: Manages data storage and retrieval in QuestDB
- **Indexing Modules**:
- Transaction Indexer: Processes transaction data
- Instruction Indexer: Processes instruction data
- Account Indexer: Tracks account changes
- Token Indexer: Tracks token transfers and balances
- DeFi Indexer: Tracks AMM and DeFi protocol activity
- NFT Indexer: Tracks NFT mints, sales, and marketplace activity
- Security Indexer: Monitors for suspicious activity
## Database Schema
ZIndexer creates several tables in ClickHouse:
- `transactions`: Basic transaction data
- `instructions`: Instruction data with program IDs
- `accounts`: Account state changes
- `account_activity`: Account usage statistics
- `token_transfers`: Token transfer events
- `token_accounts`: Token account balances
- `token_holders`: Token holder information
- `nft_mints`: NFT mint events
- `nft_sales`: NFT sale events
- `pool_swaps`: AMM swap events
- `liquidity_pools`: AMM pool information
## Continuous Integration & Deployment
ZIndexer uses GitHub Actions for CI/CD:
- **CI Workflow**: Automatically builds and tests the code on Ubuntu and macOS
- **Lint Workflow**: Checks code formatting using `zig fmt`
- **Release Workflow**: Creates binary releases when a new tag is pushed
Status badges:
## License
[MIT License](LICENSE)