{"id":31788589,"url":"https://github.com/0xintuition/research-surreal","last_synced_at":"2025-10-10T14:24:03.847Z","repository":{"id":312286834,"uuid":"1046949003","full_name":"0xIntuition/research-surreal","owner":"0xIntuition","description":"Experimental Intuition backend powered by rindexer, redis streams and surrealdb","archived":false,"fork":false,"pushed_at":"2025-08-29T15:17:19.000Z","size":68,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-29T17:27:32.158Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/0xIntuition.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-29T13:43:46.000Z","updated_at":"2025-08-29T15:17:22.000Z","dependencies_parsed_at":"2025-08-29T17:31:18.888Z","dependency_job_id":"ad43733d-4732-4c57-a9ff-05943c463514","html_url":"https://github.com/0xIntuition/research-surreal","commit_stats":null,"previous_names":["0xintuition/research-surreal"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/0xIntuition/research-surreal","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0xIntuition%2Fresearch-surreal","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0xIntuition%2Fresearch-surreal/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0xIntuition%2Fresearch-surreal/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0xIntuition%2Fresearch-surreal/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/0xIntuition","download_url":"https://codeload.github.com/0xIntuition/research-surreal/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0xIntuition%2Fresearch-surreal/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279004170,"owners_count":26083688,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-10T02:00:06.843Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-10T14:24:01.831Z","updated_at":"2025-10-10T14:24:03.841Z","avatar_url":"https://github.com/0xIntuition.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Research Surreal\n\nA high-performance blockchain event indexing and data pipeline system for the Intuition protocol, featuring real-time synchronization between Redis streams and SurrealDB with comprehensive monitoring.\n\n## Overview\n\nThis project provides a complete data pipeline solution that:\n\n1. **Indexes blockchain events** from the Intuition testnet MultiVault contract using Rindexer\n2. **Streams events** to Redis for high-throughput processing\n3. **Synchronizes data** to SurrealDB for persistent storage and querying\n4. **Monitors pipeline** health with Prometheus and Grafana\n5. **Visualizes data** through a Next.js dashboard (Jupiter)\n\n## Architecture\n\n```\n┌───────────┐    ┌──────────┐    ┌──────────────┐    ┌───────────────────┐    ┌───────────┐\n│ Blockchain│───▶│ Rindexer │───▶│Redis Streams │───▶│Redis-SurrealDB    │───▶│ SurrealDB │\n└───────────┘    └──────────┘    └──────────────┘    │      Sync         │    └─────┬─────┘\n                                                     └──────────┬────────┘          │\n                                                                │                   │\n                                                                ▼                   ▼\n                                                       ┌─────────────┐    ┌──────────────────┐\n                                                       │ Prometheus  │    │Jupiter Dashboard │\n                                                       │  (Metrics)  │    └──────────────────┘\n                                                       └─────┬───────┘\n                                                             ▼\n                                                       ┌──────────┐\n                                                       │ Grafana  │\n                                                       └──────────┘\n```\n\n## Components\n\n### 1. Rindexer\n- **Purpose**: Blockchain event indexer for Intuition testnet\n- **Contract**: MultiVault (0x5a0A023F08dF301DCCE96166F4185Ec77DF6a87a)\n- **Events**: AtomCreated, TripleCreated, Deposited, Redeemed\n- **Output**: Redis streams for real-time processing\n\n### 2. Redis-SurrealDB Sync (Rust)\n- **Purpose**: High-performance event processor and data synchronizer\n- **Features**:\n  - Multi-stream Redis consumer with batching\n  - Circuit breaker for reliability\n  - Prometheus metrics integration\n  - Health check endpoints\n  - Graceful shutdown handling\n- **Performance**: Configurable batch processing with 4 worker threads\n\n### 3. SurrealDB\n- **Purpose**: Modern multi-model database for data persistence\n- **Features**:\n  - RocksDB backend for performance\n  - WebSocket and HTTP API support\n  - Optimized configuration for production workloads\n  - CORS and WebSocket middleware via Traefik\n\n### 4. Jupiter Dashboard (Next.js)\n- **Purpose**: Real-time data visualization dashboard\n- **Features**:\n  - Live atom count display\n  - SurrealDB connection status\n  - Responsive Tailwind CSS design\n  - TypeScript implementation\n\n### 5. Monitoring Stack\n- **Prometheus**: Metrics collection and storage\n- **Grafana**: Dashboard visualization with pre-configured panels\n- **Alertmanager**: Alert management (configured but not detailed)\n\n## Quick Start\n\n### Prerequisites\n- Docker and Docker Compose\n- Environment variables (see Configuration section)\n\n### Running the Stack\n\n1. Clone the repository:\n```bash\ngit clone https://github.com/0xIntuition/research-surreal.git \ncd research-surreal\n```\n\n2. Start all services:\n```bash\ndocker compose up -d\n```\n\n4. Access the services:\n- Jupiter Dashboard: `http://localhost:3033/`\n- Grafana: `http://localhost:3009/`\n- Surrealist: `http://localhost:3006/`\n\n\n\n## Monitoring\n\n### Metrics Available\n- Event processing rates and latencies\n- Redis connection status\n- SurrealDB query performance\n- System resource utilization\n\n### Grafana Dashboards\n- Pre-configured Redis-SurrealDB sync dashboard\n- Automatic Prometheus data source provisioning\n- Custom panels for pipeline monitoring\n\n### Alerts\n- Service health monitoring\n- Processing lag detection\n- Error rate thresholds\n\n## Data Flow\n\n1. **Blockchain Events**: MultiVault contract emits events on Intuition testnet\n2. **Event Indexing**: Rindexer captures and processes events from block 2009988\n3. **Stream Processing**: Events are pushed to Redis streams by category\n4. **Batch Processing**: Redis-SurrealDB sync consumes and batches events\n5. **Data Storage**: Processed events are stored in SurrealDB tables\n6. **Visualization**: Jupiter dashboard queries SurrealDB for real-time metrics\n\n\n## License\n\nMIT License - see [LICENSE](LICENSE) file for details.\n\n## Contributing\n\nThis is a research project for the Intuition protocol. Contributions should focus on performance optimization, monitoring improvements, and data pipeline reliability.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F0xintuition%2Fresearch-surreal","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F0xintuition%2Fresearch-surreal","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F0xintuition%2Fresearch-surreal/lists"}