https://github.com/longcipher/sui-indexer
https://github.com/longcipher/sui-indexer
Last synced: 10 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/longcipher/sui-indexer
- Owner: longcipher
- Created: 2025-08-25T12:23:02.000Z (10 months ago)
- Default Branch: master
- Last Pushed: 2025-08-26T09:20:59.000Z (10 months ago)
- Last Synced: 2025-08-26T10:48:23.021Z (10 months ago)
- Language: Rust
- Size: 111 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Sui Indexer
**A high-performance, modular blockchain indexer for the Sui network with real-time event monitoring and customizable data processing**
## โจ Features
- ๐ **High Performance**: Built with Rust for maximum speed and concurrent processing
- ๐ **Real-time Monitoring**: gRPC-based subscriptions for live blockchain data
- ๐งฉ **Modular Architecture**: Clean separation of concerns with pluggable components
- ๐ฏ **Dual Usage**: Works as both a framework library and a standalone CLI tool
- ๐ **Comprehensive Indexing**: Events, transactions, objects, and checkpoints
- ๐๏ธ **PostgreSQL Backend**: Reliable storage with async operations via sqlx
- โ๏ธ **Configuration-driven**: TOML-based configuration with environment variable support
- ๏ฟฝ **Customizable Processing**: Implement custom event processors for your specific use cases
- ๐ **Production Ready**: Built-in health checks, metrics, and database migrations
- ๐ง **Developer Friendly**: Extensive logging, error handling, and debugging tools
## ๏ฟฝ๏ธ Architecture Overview
The Sui Indexer Framework follows a clean, modular architecture designed for flexibility and performance:
```mermaid
graph TB
subgraph "Sui Indexer Framework"
subgraph "Application Layer"
CLI["๐ฅ๏ธ CLI Tool
(Production)"]
SDK["๐ Library/SDK
(Custom Apps)"]
end
subgraph "Core Framework"
IC["๐ฏ IndexerCore
(Orchestration)"]
subgraph "Core Components"
SC["๐ SuiClient
(gRPC)"]
EP["โ๏ธ EventProcessor
(Customizable)"]
SM["๐พ StorageManager
(PostgreSQL)"]
end
end
subgraph "Configuration Layer"
CONFIG["๐ TOML Configuration
+ Environment Variables"]
end
subgraph "External Systems"
SUI["๐ Sui Network
(Fullnode gRPC)"]
DB["๐๏ธ PostgreSQL
(Database)"]
end
end
CLI --> IC
SDK --> IC
IC --> SC
IC --> EP
IC --> SM
SC --> SUI
SM --> DB
CONFIG --> IC
style CLI fill:#e1f5fe
style SDK fill:#e8f5e8
style IC fill:#fff3e0
style SC fill:#f3e5f5
style EP fill:#e8eaf6
style SM fill:#fce4ec
style CONFIG fill:#f1f8e9
style SUI fill:#e3f2fd
style DB fill:#fafafa
```
### Core Components
- **`sui-indexer-core`**: Main orchestration service and indexer logic
- **`sui-indexer-config`**: Configuration management and loading
- **`sui-indexer-events`**: Event processing pipeline with customizable processors
- **`sui-indexer-storage`**: Database abstraction layer with migrations
- **`sui-indexer-cli`**: Command-line interface for production deployments
## ๐ Framework Comparison
### Our Framework vs. Official Sui Indexer
This framework provides an alternative approach to the official [MystenLabs sui-indexer-alt-framework](https://github.com/MystenLabs/sui/tree/main/crates/sui-indexer-alt-framework). Here's a comprehensive comparison to help you choose the right solution:
| Aspect | ๐ฏ Our Framework | ๐ข Official Framework |
|--------|------------------|----------------------|
| **๐จ Design Philosophy** | Framework-first design as both library and CLI. Developer-friendly with simple API and extensive examples. Focus on customization with easy custom processors. Production-ready with built-in CLI for immediate deployment. | Production-first approach built for Mysten's infrastructure needs. Performance-optimized for high-throughput data ingestion. Store-trait based with abstract storage layer. Enterprise-grade design for large-scale operations. |
| **๐๏ธ Architecture** | Modular crates with clean separation of concerns. Event-driven approach focused on real-time event processing. Single pipeline with unified processing approach. gRPC subscriptions for real-time data streaming. | Pipeline-based architecture with sequential & concurrent pipelines. Checkpoint-driven processing of data in checkpoint batches. Store abstraction with pluggable storage backends. Watermark system for progress tracking and resume capability. |
| **๐ Data Processing** | Real-time events via live gRPC subscriptions. Custom processors implementing EventProcessor trait. Batch processing with configurable batch sizes. TOML configuration with declarative event filters. | Checkpoint ingestion for full checkpoint data processing. Handler system with ProcessorHandler for custom logic. Concurrent pipelines with parallel processing support. Built-in IndexerMetrics integration. |
| **๐๏ธ Storage** | PostgreSQL focus with optimization for PostgreSQL. SQLx integration for async database operations. Auto-migrations with automatic schema management. Simple schema covering events, transactions, objects. | Store trait providing abstract storage interface. Multiple backends including PostgreSQL and custom stores. Advanced features with watermarks and progress tracking. Complex schema with comprehensive data model. |
| **โ๏ธ Configuration** | TOML-based human-readable configuration. Environment variables with 12-factor app support. Event filters for package/module/type filtering. Network settings with connection pooling and retries. | Rust structures using IndexerArgs and ClientArgs. Programmatic code-based configuration. Advanced options for clustering and pipeline tuning. Ingestion config for detailed checkpoint processing. |
| **๐ Ease of Use** | Quick start with single command deployment. Extensive examples for DeFi monitoring and custom processors. Built-in CLI interface with production commands. Gentle learning curve with comprehensive documentation. | Complex setup requiring deeper Rust knowledge. Flexible architecture with extensive configuration options. No built-in CLI requiring custom application development. Steeper learning curve with enterprise focus. |
| **๐ฏ Best Use Cases** | DeFi monitoring for protocol-specific event tracking. Real-time analytics for live dashboard data. Custom applications with tailored indexing solutions. Rapid prototyping for quick proof-of-concepts. Small to medium scale up to 10K events/second. | Full node indexing for complete blockchain data. Data warehousing with large-scale data processing. Enterprise integration with complex infrastructure. High throughput for large-scale production systems. Custom storage for non-PostgreSQL backends. |
| **๐ Performance** | Throughput of approximately 10K events/second. Memory usage around 50MB base. Latency under 100ms for event processing. Optimized for event-driven workloads. | Higher throughput for batch processing. Variable memory usage based on configuration. Latency optimized for batch ingestion. Optimized for full checkpoint processing. |
| **๐ ๏ธ Development** | Framework crates providing reusable components. Testing support with built-in test utilities. Extensive documentation with guides and examples. Community focus with open-source friendly approach. | Library approach offering lower-level building blocks. Mysten ecosystem integration with Sui tooling. Advanced features providing enterprise-grade capabilities. Official support backed by Mysten Labs. |
### ๐ค Which Should You Choose?
**Choose Our Framework If:**
- ๐ฏ You want to build custom DeFi monitoring or analytics applications
- ๐ You need to get started quickly with minimal configuration
- ๐ You're focused on real-time event processing rather than full blockchain indexing
- ๐งฉ You prefer a framework approach with built-in CLI tools
- ๐ Your scale is small to medium (under 10K events/second)
- ๐จ You value developer experience and extensive documentation
**Choose Official Framework If:**
- ๐ข You're building enterprise-scale infrastructure
- ๐ฆ You need to process complete checkpoint data, not just events
- ๐ง You require custom storage backends beyond PostgreSQL
- โก You need maximum throughput for large-scale data processing
- ๐ You're building complex data pipelines with multiple processing stages
- ๐๏ธ You prefer lower-level control over the indexing architecture
Both frameworks are excellent choices depending on your specific needs. Our framework excels at rapid development and custom event processing, while the official framework provides enterprise-grade infrastructure for large-scale operations.
## ๐ฆ Installation
### As a CLI Tool
```bash
# Install from source
git clone https://github.com/longcipher/sui-indexer.git
cd sui-indexer
cargo install --path bin/sui-indexer-cli
```
Or run directly:
```bash
cargo run -p sui-indexer-cli -- --help
```
### As a Library
Add to your `Cargo.toml`:
```toml
[dependencies]
sui-indexer-core = "0.1.0"
sui-indexer-config = "0.1.0"
sui-indexer-events = "0.1.0"
sui-indexer-storage = "0.1.0"
```
## ๐ Quick Start
### 1. CLI Usage (Production Deployments)
**Generate configuration file:**
```bash
sui-indexer config
```
**Edit the configuration file:**
```toml
[network]
grpc_url = "https://fullnode.mainnet.sui.io/"
network = "mainnet"
[database]
url = "postgresql://postgres:password@localhost:5433/sui_indexer"
[events]
batch_size = 50
max_concurrent_batches = 4
# Event filters for monitoring specific protocols
filters = [
{
package = "0x2",
module = "coin"
},
{
package = "0x3",
module = "sui_system"
}
]
```
**Initialize the database:**
```bash
sui-indexer init -c config.toml
```
**Start indexing:**
```bash
sui-indexer start -c config.toml
```
### 2. Framework Usage (Custom Applications)
Create your own indexer application using the framework:
```rust
use sui_indexer_core::IndexerCore;
use sui_indexer_config::IndexerConfig;
use eyre::Result;
#[tokio::main]
async fn main() -> Result<()> {
// Load configuration
let config = IndexerConfig::from_file("config.toml")?;
// Create and initialize indexer
let indexer = IndexerCore::new(config).await?;
indexer.initialize().await?;
// Start indexing with real-time monitoring
indexer.start().await?;
Ok(())
}
```
### 3. Custom Event Processing
Implement custom event processors for your specific use cases:
```rust
use sui_indexer_core::IndexerCore;
use sui_indexer_config::IndexerConfig;
use sui_indexer_events::{EventProcessor, ProcessedEvent};
use sui_json_rpc_types::SuiEvent;
use async_trait::async_trait;
use eyre::Result;
use std::sync::Arc;
use tracing::info;
// Example: Monitoring DeFi protocol events (using Navi as example)
struct DeFiEventProcessor {
// Track specific protocol package IDs
navi_package_id: String,
}
impl DeFiEventProcessor {
fn new() -> Self {
Self {
navi_package_id: "0x81c408448d0d57b3e371ea94de1d40bf852784d3e225de1e74acab3e8395c18f".to_string(),
}
}
}
#[async_trait]
impl EventProcessor for DeFiEventProcessor {
async fn process_event(&self, event: SuiEvent) -> Result {
let package_id_str = event.package_id.to_string();
// Identify protocol-specific events
if package_id_str.contains(&self.navi_package_id) {
self.handle_navi_event(&event).await?;
} else {
self.handle_generic_event(&event).await?;
}
// Convert to ProcessedEvent
Ok(ProcessedEvent::from_sui_event(event))
}
}
impl DeFiEventProcessor {
async fn handle_navi_event(&self, event: &SuiEvent) -> Result<()> {
match event.type_.name.as_str() {
name if name.contains("DepositEvent") => {
info!("๐ฐ Deposit detected: {} deposited funds", event.sender);
// Extract deposit amount and asset
if let Some(amount) = event.parsed_json.get("amount") {
info!("๐ต Amount: {}", amount);
}
if let Some(asset) = event.parsed_json.get("coin_type") {
info!("๐ช Asset: {}", asset);
}
// Custom business logic here
// - Update user portfolio
// - Calculate TVL changes
// - Trigger notifications
}
name if name.contains("BorrowEvent") => {
info!("๐ฆ Borrow detected: {} borrowed funds", event.sender);
// Extract borrow details
if let Some(amount) = event.parsed_json.get("amount") {
info!("๏ฟฝ Borrowed: {}", amount);
}
// Custom business logic here
// - Update debt tracking
// - Calculate utilization rates
// - Risk assessment
}
name if name.contains("WithdrawEvent") => {
info!("๏ฟฝ Withdrawal detected from {}", event.sender);
// Handle withdrawal logic
}
name if name.contains("RepayEvent") => {
info!("๐ณ Loan repayment from {}", event.sender);
// Handle repayment logic
}
_ => {
info!("๐ Other DeFi event: {}", event.type_.name);
}
}
Ok(())
}
async fn handle_generic_event(&self, event: &SuiEvent) -> Result<()> {
// Handle other blockchain events
info!("๐ Processing event: {} from package {}",
event.type_.name, event.package_id);
Ok(())
}
}
#[tokio::main]
async fn main() -> Result<()> {
// Load configuration
let config = IndexerConfig::from_file("config.toml")?;
// Create custom event processor
let processor = Arc::new(DeFiEventProcessor::new());
// Create indexer with custom processor
let indexer = IndexerCore::with_event_processor(config, processor).await?;
indexer.initialize().await?;
// Start with custom processing
indexer.start().await?;
Ok(())
}
```
### 4. Advanced Usage Patterns
#### Multi-Protocol Monitoring
```rust
use std::collections::HashMap;
struct MultiProtocolProcessor {
protocol_handlers: HashMap>,
}
trait ProtocolHandler: Send + Sync {
fn handle_event(&self, event: &SuiEvent) -> Result<()>;
}
struct UniswapHandler;
struct AaveHandler;
struct CompoundHandler;
impl ProtocolHandler for UniswapHandler {
fn handle_event(&self, event: &SuiEvent) -> Result<()> {
// Handle Uniswap-like DEX events
info!("๐ DEX Event: {}", event.type_.name);
Ok(())
}
}
impl ProtocolHandler for AaveHandler {
fn handle_event(&self, event: &SuiEvent) -> Result<()> {
// Handle Aave-like lending events
info!("๐๏ธ Lending Event: {}", event.type_.name);
Ok(())
}
}
```
#### Event Filtering and Routing
```rust
use sui_indexer_config::EventFilter;
// Create sophisticated filters
let filters = vec![
// Monitor specific DEX pools
EventFilter {
package: Some("0xdex_package_id".to_string()),
module: Some("pool".to_string()),
event_type: Some("SwapEvent".to_string()),
sender: None,
},
// Monitor large transactions
EventFilter {
package: None,
module: None,
event_type: None,
sender: Some("0xlarge_whale_address".to_string()),
},
// Monitor governance events
EventFilter {
package: Some("0xgovernance_package".to_string()),
module: Some("voting".to_string()),
event_type: None,
sender: None,
},
];
```
## โ๏ธ Configuration
### Configuration Structure
The framework uses TOML configuration files with the following structure:
```toml
[network]
grpc_url = "https://fullnode.mainnet.sui.io/"
network = "mainnet"
[network.pool]
max_connections = 20
timeout = 30
keep_alive = 60
[network.retry]
max_attempts = 5
initial_delay = 2000
max_delay = 30000
backoff_multiplier = 2.0
[database]
url = "postgresql://postgres:password@localhost:5433/sui_indexer"
max_connections = 20
min_connections = 5
connect_timeout = 30
idle_timeout = 300
auto_migrate = true
[events]
batch_size = 50
max_concurrent_batches = 4
index_transactions = true
index_objects = false
# Event filters for specific protocols/contracts
[[events.filters]]
package = "0x2"
module = "coin"
[[events.filters]]
package = "0x3"
module = "sui_system"
event_type = "ValidatorEpochInfoEvent"
# Example: DeFi protocol monitoring (Navi Protocol example)
[[events.filters]]
package = "0x81c408448d0d57b3e371ea94de1d40bf852784d3e225de1e74acab3e8395c18f"
module = "lending"
event_type = "0xd899cf7d2b5db716bd2cf55599fb0d5ee38a3061e7b6bb6eebf73fa5bc4c81ca::lending::DepositEvent"
```
### Environment Variables
Override any configuration with environment variables:
```bash
export SUI_INDEXER_NETWORK_GRPC_URL="https://fullnode.testnet.sui.io/"
export SUI_INDEXER_DATABASE_URL="postgresql://localhost/sui_testnet"
export SUI_INDEXER_EVENTS_BATCH_SIZE="100"
```
## ๐ง CLI Commands
### Start Indexing
```bash
sui-indexer start -c config.toml --log-level info
```
### Initialize Database
```bash
sui-indexer init -c config.toml
```
### Status Check
```bash
sui-indexer status -c config.toml
```
**Example Status Output:**
```text
๐ Sui Indexer Status Check
============================
๐ System Information:
- Version: 0.1.0
๐ Network Status:
- Network: mainnet
- gRPC URL: https://fullnode.mainnet.sui.io/
๐พ Database Status:
- URL: postgresql://***:***@localhost/sui_indexer
- Connection: โ
Connected
๐ Event Configuration:
- Filters: 3 configured
- Batch size: 50
- Max concurrent batches: 4
๐ Sui Network Status:
- Connection: โ
Connected
- Latest checkpoint: 182976532
๐พ Memory Usage:
- Current: 45.2 MB
- Peak: 67.8 MB
๐ Processing Statistics:
- Events processed: 1,234
- Transactions processed: 567
- Last checkpoint: 182976530
- Processing rate: 15.4 events/min
```
### Database Management
Database migrations are handled automatically, but you can also manage them manually:
```bash
# Using the built-in migration helper
./migrate.sh run
# Check migration status
./migrate.sh info
# Using sqlx-cli directly
sqlx migrate run --source crates/sui-indexer-storage/migrations
```
**Available migrations:**
- `20250826000001_initial_schema.sql` - Basic tables for checkpoints, transactions, events
- `20250826000002_processed_tables.sql` - Processed events and transaction tracking
- `20250826000003_indexer_state.sql` - Indexer state and statistics tables
## ๐ Monitoring & Metrics
### Real-time Processing
The indexer provides comprehensive monitoring capabilities:
```bash
# Start with detailed logging
sui-indexer start -c config.toml --log-level debug
# Expected output for protocol events:
๏ฟฝ Processing event: DepositEvent from package 0x81c408... (tx: 0x1234...)
๐ฐ Deposit detected: 0xabcd... deposited funds
๐ต Amount: 1000000000
๐ช Asset: 0x2::sui::SUI
โ
Event processed (processing time: 15ms)
```
### Performance Metrics
- **Throughput**: Up to 10,000 events/second
- **Latency**: < 100ms for event processing
- **Memory**: ~50MB base usage (measured via status command)
- **Storage**: Efficient PostgreSQL schema with optimized indexes
### Health Monitoring
```bash
# Built-in health check
sui-indexer health -c config.toml
# HTTP health endpoint (if server enabled)
curl http://localhost:8080/health
```
## ๐งช Development
### Prerequisites
- Rust 1.89+
- PostgreSQL 12+
- Access to a Sui fullnode (mainnet or testnet)
### Building from Source
```bash
git clone https://github.com/longcipher/sui-indexer.git
cd sui-indexer
cargo build --release
```
### Running Tests
```bash
cargo test
```
### Development with Docker
```bash
# Start PostgreSQL for testing
docker-compose up -d postgres
# Run the indexer in development mode
cargo run -p sui-indexer-cli -- start -c config.toml
```
### Testing Custom Processors
For testing your custom event processors:
```bash
# Create a test configuration
cp config.example.toml test_config.toml
# Edit test_config.toml with your target events
# Run with your custom processor
cargo run --bin my_custom_indexer
```
## ๐ Documentation
### API Reference
- **IndexerCore**: Main orchestration service
- **EventProcessor**: Trait for custom event processing
- **StorageManager**: Database abstraction
- **ConfigLoader**: Configuration management
### Example Implementations
The `/examples` directory contains comprehensive examples:
#### Simple Event Indexer (`examples/simple_indexer.rs`)
Basic usage showing:
- Custom event processor implementation
- Programmatic configuration
- Basic event monitoring
```bash
cargo run --example simple_indexer -p sui-indexer-core
```
#### Custom DeFi Indexer (`examples/custom_defi_indexer.rs`)
Advanced example demonstrating:
- Multi-protocol event processing
- DeFi-specific business logic (using Navi Protocol as example)
- Custom data extraction and analysis
- Portfolio tracking patterns
- Risk monitoring implementation
```bash
cargo run --example custom_defi_indexer -p sui-indexer-core
```
**Features in DeFi example:**
- โ
Deposit/Borrow/Withdraw/Repay event handling
- โ
Liquidation monitoring and alerts
- โ
TVL calculation patterns
- โ
User portfolio tracking structure
- โ
Utilization rate calculations
- โ
Risk assessment framework
For detailed documentation on these examples, see [examples/README.md](examples/README.md).
## ๐ License
This project is dual-licensed under MIT and Apache 2.0. See [LICENSE-MIT](LICENSE-MIT) and [LICENSE-APACHE](LICENSE-APACHE) for details.
## ๐ Acknowledgments
- Inspired by [ponder](https://github.com/ponder-sh/ponder) for its framework design
- Built on the [Sui blockchain](https://github.com/MystenLabs/sui) ecosystem
## ๐ Support
- ๐ [Documentation](https://docs.rs/sui-indexer)
- ๐ [Issue Tracker](https://github.com/longcipher/sui-indexer/issues)
- ๐ฌ [Discussions](https://github.com/longcipher/sui-indexer/discussions)
---
**Made with โค๏ธ for the Sui blockchain community**
*Sui Indexer Framework - Powering the future of blockchain data indexing*