{"id":26125844,"url":"https://github.com/opensvm/zindexer","last_synced_at":"2025-04-13T16:04:13.928Z","repository":{"id":281586906,"uuid":"923301374","full_name":"openSVM/zindexer","owner":"openSVM","description":"solana indexer for clickhouse in zig","archived":false,"fork":false,"pushed_at":"2025-03-10T03:40:02.000Z","size":46,"stargazers_count":8,"open_issues_count":3,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-10T04:32:23.737Z","etag":null,"topics":["zig","zig-package","ziglana","ziglang"],"latest_commit_sha":null,"homepage":"","language":"Zig","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openSVM.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-28T01:25:04.000Z","updated_at":"2025-03-02T17:16:14.000Z","dependencies_parsed_at":"2025-03-10T04:32:27.531Z","dependency_job_id":"f661b368-64b2-4494-97ea-78a8557eb338","html_url":"https://github.com/openSVM/zindexer","commit_stats":null,"previous_names":["opensvm/zindexer"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openSVM%2Fzindexer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openSVM%2Fzindexer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openSVM%2Fzindexer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openSVM%2Fzindexer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openSVM","download_url":"https://codeload.github.com/openSVM/zindexer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242889327,"owners_count":20201947,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["zig","zig-package","ziglana","ziglang"],"created_at":"2025-03-10T17:12:37.546Z","updated_at":"2025-04-13T16:04:13.897Z","avatar_url":"https://github.com/openSVM.png","language":"Zig","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ZIndexer - Multi-Network SVM Indexer\n\nZIndexer is a high-performance indexer for Solana Virtual Machine (SVM) networks, capable of subscribing to and indexing data from multiple networks simultaneously.\n\n## Features\n\n- **Multi-Network Support**: Index multiple SVM networks (mainnet, devnet, testnet, localnet) simultaneously\n- **Comprehensive Indexing**:\n  - Automated Market Makers (AMMs)\n  - Metaplex NFTs and marketplaces\n  - Token transfers and balances\n  - Account balance changes\n  - Transactions and instructions\n  - Blocks and slots\n  - Account activity\n- **Real-time and Historical Modes**: Choose between real-time indexing or historical backfilling\n- **Database Integration**: High-performance storage and querying of indexed data with support for ClickHouse and QuestDB\n- **Interactive TUI**: Monitor indexing progress across all networks in real-time\n\n## Requirements\n\n- Zig 0.14.0 or later\n- ClickHouse server or QuestDB server\n- Internet connection to access SVM networks\n\n## Building\n\n```bash\nzig build\n```\n\n## Configuration\n\nThe indexer uses two configuration files:\n\n- `src/rpc_nodes.json`: HTTP RPC endpoints for each network\n- `src/ws_nodes.json`: WebSocket endpoints for each network\n\nYou can customize these files to add or remove networks, or to use different RPC providers.\n\n## Usage\n\n```bash\n# Run in real-time mode (default)\n./zig-out/bin/zindexer\n\n# Run in historical mode\n./zig-out/bin/zindexer --mode historical\n\n# Customize ClickHouse connection\n./zig-out/bin/zindexer --database-type clickhouse --database-url localhost:9000 --database-user default --database-password \"\" --database-name solana\n\n# Use QuestDB instead\n./zig-out/bin/zindexer --database-type questdb --database-url localhost:9000 --database-user admin --database-password \"quest\" --database-name solana\n\n# Show help\n./zig-out/bin/zindexer --help\n```\n\n### Command Line Options\n\n- `-m, --mode \u003cmode\u003e`: Indexer mode (historical or realtime)\n- `-r, --rpc-nodes \u003cfile\u003e`: RPC nodes configuration file\n- `-w, --ws-nodes \u003cfile\u003e`: WebSocket nodes configuration file\n- `-t, --database-type \u003ctype\u003e`: Database type (clickhouse or questdb)\n- `-c, --database-url \u003curl\u003e`: Database server URL\n- `-u, --database-user \u003cuser\u003e`: Database username\n- `-p, --database-password \u003cpass\u003e`: Database password\n- `-d, --database-name \u003cdb\u003e`: Database name\n- `-b, --batch-size \u003csize\u003e`: Batch size for historical indexing\n- `--max-retries \u003ccount\u003e`: Maximum retry attempts\n- `--retry-delay \u003cms\u003e`: Delay between retries in milliseconds\n\n#### Legacy ClickHouse Options (for backward compatibility)\n\n- `--clickhouse-url \u003curl\u003e`: ClickHouse server URL\n- `--clickhouse-user \u003cuser\u003e`: ClickHouse username\n- `--clickhouse-password \u003cpass\u003e`: ClickHouse password\n- `--clickhouse-database \u003cdb\u003e`: ClickHouse database name\n\n#### QuestDB Options\n\n- `--questdb-url \u003curl\u003e`: QuestDB server URL\n- `--questdb-user \u003cuser\u003e`: QuestDB username\n- `--questdb-password \u003cpass\u003e`: QuestDB password\n- `--questdb-database \u003cdb\u003e`: QuestDB database name\n- `-h, --help`: Show help message\n\n## Schema Setup\n\nThe schema can be automatically applied to your database instance using the included script:\n\n### For ClickHouse\n\n```bash\nDB_TYPE=clickhouse CLICKHOUSE_URL=\"http://localhost:8123\" ./scripts/apply_schema.sh\n```\n\n### For QuestDB\n\n```bash\nDB_TYPE=questdb QUESTDB_URL=\"http://localhost:9000\" ./scripts/apply_schema.sh\n```\n\n## Architecture\n\nZIndexer is built with a modular architecture:\n\n- **Core Indexer**: Manages connections to multiple networks and coordinates indexing\n- **RPC Client**: Handles communication with SVM networks via HTTP and WebSocket\n- **Database Abstraction Layer**: Provides a common interface for different database backends\n  - **ClickHouse Client**: Manages data storage and retrieval in ClickHouse\n  - **QuestDB Client**: Manages data storage and retrieval in QuestDB\n- **Indexing Modules**:\n  - Transaction Indexer: Processes transaction data\n  - Instruction Indexer: Processes instruction data\n  - Account Indexer: Tracks account changes\n  - Token Indexer: Tracks token transfers and balances\n  - DeFi Indexer: Tracks AMM and DeFi protocol activity\n  - NFT Indexer: Tracks NFT mints, sales, and marketplace activity\n  - Security Indexer: Monitors for suspicious activity\n\n## Database Schema\n\nZIndexer creates several tables in ClickHouse:\n\n- `transactions`: Basic transaction data\n- `instructions`: Instruction data with program IDs\n- `accounts`: Account state changes\n- `account_activity`: Account usage statistics\n- `token_transfers`: Token transfer events\n- `token_accounts`: Token account balances\n- `token_holders`: Token holder information\n- `nft_mints`: NFT mint events\n- `nft_sales`: NFT sale events\n- `pool_swaps`: AMM swap events\n- `liquidity_pools`: AMM pool information\n\n## Continuous Integration \u0026 Deployment\n\nZIndexer uses GitHub Actions for CI/CD:\n\n- **CI Workflow**: Automatically builds and tests the code on Ubuntu and macOS\n- **Lint Workflow**: Checks code formatting using `zig fmt`\n- **Release Workflow**: Creates binary releases when a new tag is pushed\n\nStatus badges:\n\n![Build Status](https://github.com/openSVM/zindexer/workflows/ZIndexer%20CI/badge.svg)\n![Lint Status](https://github.com/openSVM/zindexer/workflows/ZIndexer%20Lint/badge.svg)\n![Release Status](https://github.com/openSVM/zindexer/workflows/ZIndexer%20Release/badge.svg)\n\n## License\n\n[MIT License](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopensvm%2Fzindexer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopensvm%2Fzindexer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopensvm%2Fzindexer/lists"}