{"id":30715914,"url":"https://github.com/bryanlabs/snapshots","last_synced_at":"2025-09-03T07:09:55.878Z","repository":{"id":304254963,"uuid":"1008669061","full_name":"bryanlabs/snapshots","owner":"bryanlabs","description":null,"archived":false,"fork":false,"pushed_at":"2025-08-10T18:50:22.000Z","size":1449,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-10T20:38:10.064Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bryanlabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"docs/SECURITY-HEADERS.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-25T23:04:25.000Z","updated_at":"2025-08-10T18:50:29.000Z","dependencies_parsed_at":"2025-07-12T00:27:53.307Z","dependency_job_id":"65ebdc34-4ff8-417f-bbfe-e7c2e2c9a211","html_url":"https://github.com/bryanlabs/snapshots","commit_stats":null,"previous_names":["bryanlabs/snapshots"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/bryanlabs/snapshots","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanlabs%2Fsnapshots","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanlabs%2Fsnapshots/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanlabs%2Fsnapshots/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanlabs%2Fsnapshots/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bryanlabs","download_url":"https://codeload.github.com/bryanlabs/snapshots/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanlabs%2Fsnapshots/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273404434,"owners_count":25099405,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-03T02:00:09.631Z","response_time":76,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-03T07:09:54.499Z","updated_at":"2025-09-03T07:09:55.867Z","avatar_url":"https://github.com/bryanlabs.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Blockchain Snapshots Service\n\nA production-grade blockchain snapshot hosting service providing reliable, bandwidth-managed access to blockchain snapshots with tiered user system. Built with Next.js 15, nginx storage backend, and deployed on Kubernetes.\n\n## 🚀 Overview\n\nThe Blockchain Snapshots Service provides high-speed access to blockchain node snapshots for the Cosmos ecosystem. It features:\n\n- **Tiered Access**: Free tier (50 Mbps shared) and Premium tier (250 Mbps shared)\n- **Multiple Authentication**: Email/password and Cosmos wallet (Keplr) authentication\n- **Compression Support**: Both ZST and LZ4 compressed snapshots\n- **Resume Support**: Interrupted downloads can be resumed\n- **Real-time Monitoring**: Prometheus metrics and Grafana dashboards\n- **High Availability**: Redundant deployments with automatic failover\n- **Security**: NextAuth.js authentication, secure download links, and IP restrictions\n\n## 📋 Table of Contents\n\n- [Key Features](#-key-features)\n- [Tech Stack](#-tech-stack)\n- [Architecture](#-architecture)\n- [Getting Started](#-getting-started)\n- [Development](#-development)\n- [API Reference](#-api-reference)\n- [Testing](#-testing)\n- [Deployment](#-deployment)\n- [Integration with Snapshot Processor](#-integration-with-snapshot-processor)\n- [Monitoring](#-monitoring)\n- [Contributing](#-contributing)\n- [License](#-license)\n\n## ✨ Key Features\n\n### Core Functionality\n- **Multiple Chain Support**: Host snapshots for 30+ Cosmos chains\n- **Dual Compression**: Support for both ZST and LZ4 compressed snapshots\n- **Bandwidth Management**: Dynamic per-connection bandwidth allocation\n- **Download Resume**: Support for interrupted download resumption\n- **Real-time Updates**: Automated snapshot processing via snapshot-processor\n- **User Management**: Full account system with profile, billing, and download history\n\n### User Experience\n- **Instant Access**: No registration required for free tier\n- **Premium Tier**: 5x faster downloads for authenticated users\n- **Multiple Auth Methods**: Email/password or Cosmos wallet authentication\n- **Search \u0026 Filter**: Find snapshots by chain name, type, or compression\n- **Download Progress**: Real-time download statistics\n- **Mobile Responsive**: Optimized for all device sizes\n\n### Technical Features\n- **Secure Downloads**: Time-limited download URLs with nginx secure_link module\n- **Rate Limiting**: Prevent abuse with configurable limits\n- **Health Checks**: Automated monitoring and alerting\n- **Metrics Export**: Prometheus-compatible metrics\n- **GitOps Ready**: Kubernetes manifests managed in bare-metal repository\n\n## 🛠️ Tech Stack\n\n### Frontend\n- **Next.js 15**: React framework with App Router\n- **TypeScript 5**: Type-safe development\n- **Tailwind CSS 4**: Utility-first styling\n- **React 19**: Latest React features\n- **NextAuth.js v5**: Authentication system\n\n### Backend\n- **Next.js API Routes**: Full-stack capabilities\n- **Nginx**: Static file storage with secure_link module\n- **Prisma ORM**: Database management\n- **SQLite**: User and session storage\n- **Redis**: Session caching and rate limiting\n- **JWT**: API authentication\n\n### Infrastructure\n- **Kubernetes**: Container orchestration\n- **TopoLVM**: Dynamic volume provisioning\n- **Snapshot Processor**: Go-based automated snapshot processing\n- **Prometheus**: Metrics collection\n- **Grafana**: Metrics visualization\n- **GitHub Actions**: CI/CD pipeline\n\n## 🏗️ Architecture\n\n### High-Level Overview\n```\n┌─────────────┐     ┌─────────────┐     ┌─────────────┐     ┌─────────────┐\n│   Browser   │────▶│  Next.js    │────▶│   Nginx     │◀────│  Snapshot   │\n└─────────────┘     │  Web App    │     │  Storage    │     │  Processor  │\n                    └─────────────┘     └─────────────┘     └─────────────┘\n                            │                    │                    │\n                            ▼                    ▼                    ▼\n                    ┌─────────────┐     ┌─────────────┐     ┌─────────────┐\n                    │   SQLite    │     │  TopoLVM    │     │ Kubernetes  │\n                    │  Database   │     │   Storage   │     │    Jobs     │\n                    └─────────────┘     └─────────────┘     └─────────────┘\n```\n\n### Component Interaction\n1. **User browses** available snapshots via Next.js frontend\n2. **Authentication** checked via NextAuth.js for tier determination\n3. **Snapshot data** fetched from nginx autoindex API\n4. **Download URLs** generated with nginx secure_link module\n5. **Direct download** from nginx with bandwidth management\n6. **Metrics** collected for monitoring and analytics\n7. **Snapshot creation** handled by separate snapshot-processor service\n\n### Integration with Snapshot Processor\nThe web app works in conjunction with the [snapshot-processor](https://github.com/bryanlabs/snapshot-processor):\n- **Processor creates** snapshots on schedule or request\n- **Compresses** with ZST or LZ4 based on configuration\n- **Uploads** to nginx storage at `/snapshots/[chain-id]/`\n- **Web app displays** available snapshots from nginx\n- **Users download** directly from nginx storage\n\n## 🚀 Getting Started\n\n### Prerequisites\n- Node.js 20.x or higher\n- npm or yarn\n- Docker (for development database)\n- Kubernetes cluster (for production)\n\n### Quick Start (Development)\n\n1. **Clone the repository**\n   ```bash\n   git clone https://github.com/bryanlabs/snapshots.git\n   cd snapshots\n   ```\n\n2. **Install dependencies**\n   ```bash\n   npm install\n   ```\n\n3. **Initialize database**\n   ```bash\n   ./scripts/init-db-proper.sh\n   ```\n\n4. **Set up environment variables**\n   ```bash\n   cp .env.example .env.local\n   # Edit .env.local with your configuration\n   ```\n\n5. **Run development server**\n   ```bash\n   npm run dev\n   ```\n\n6. **Open browser**\n   Navigate to [http://localhost:3000](http://localhost:3000)\n\n### Test Accounts\n- **Email**: test@example.com\n- **Password**: snapshot123\n\n## 💻 Development\n\n### Project Structure\n```\nsnapshots/\n├── app/                    # Next.js app directory\n│   ├── api/               # API routes\n│   │   ├── account/       # Account management\n│   │   ├── admin/         # Admin endpoints\n│   │   ├── auth/          # NextAuth endpoints\n│   │   ├── v1/            # Public API v1\n│   │   └── health         # Health checks\n│   ├── (auth)/            # Auth pages layout\n│   ├── (public)/          # Public pages layout\n│   ├── account/           # User account pages\n│   └── page.tsx           # Homepage\n├── components/            # React components\n├── lib/                   # Utilities and helpers\n│   ├── auth/             # Authentication logic\n│   ├── nginx/            # Nginx storage client\n│   └── bandwidth/        # Bandwidth management\n├── prisma/               # Database schema\n├── __tests__/            # Test files\n├── docs/                 # Documentation\n└── public/              # Static assets\n```\n\n### Environment Variables\n```bash\n# Nginx Storage Configuration\nNGINX_ENDPOINT=nginx\nNGINX_PORT=32708\nNGINX_USE_SSL=false\nNGINX_EXTERNAL_URL=https://snapshots.bryanlabs.net\nSECURE_LINK_SECRET=your-secure-link-secret\n\n# Authentication\nNEXTAUTH_SECRET=your-nextauth-secret\nNEXTAUTH_URL=https://snapshots.bryanlabs.net\nDATABASE_URL=file:/app/prisma/dev.db\n\n# Legacy Auth (for API compatibility)\nPREMIUM_USERNAME=premium_user\nPREMIUM_PASSWORD_HASH=$2a$10$...\n\n# Bandwidth Limits (Mbps)\nBANDWIDTH_FREE_TOTAL=50\nBANDWIDTH_PREMIUM_TOTAL=250\n\n# Redis Configuration\nREDIS_HOST=redis\nREDIS_PORT=6379\n\n# Download Limits\nDAILY_DOWNLOAD_LIMIT=10\n\n# API Configuration\nNEXT_PUBLIC_API_URL=https://snapshots.bryanlabs.net\n```\n\n### Development Commands\n```bash\n# Start development server\nnpm run dev\n\n# Run tests\nnpm test\n\n# Run tests in watch mode\nnpm run test:watch\n\n# Run e2e tests\nnpm run test:e2e\n\n# Build for production\nnpm run build\n\n# Start production server\nnpm start\n\n# Run linting\nnpm run lint\n\n# Database commands\nnpx prisma migrate dev    # Run migrations\nnpx prisma studio         # Open database GUI\nnpx prisma generate       # Generate Prisma client\n```\n\n## 📚 API Reference\n\nSee [API Routes Documentation](./API_ROUTES.md) for detailed endpoint information.\n\n### Public API (v1)\n- `GET /api/v1/chains` - List all chains with snapshots\n- `GET /api/v1/chains/[chainId]` - Get chain details\n- `GET /api/v1/chains/[chainId]/snapshots` - List chain snapshots\n- `GET /api/v1/chains/[chainId]/snapshots/latest` - Get latest snapshot\n- `POST /api/v1/chains/[chainId]/download` - Generate download URL\n- `POST /api/v1/auth/login` - Legacy JWT authentication\n- `POST /api/v1/auth/wallet` - Wallet authentication\n\n### NextAuth API\n- `POST /api/auth/signin` - Sign in with credentials or wallet\n- `GET /api/auth/signout` - Sign out\n- `GET /api/auth/session` - Get current session\n- `POST /api/auth/register` - Register new account\n\n### Account API\n- `GET /api/account/avatar` - Get user avatar\n- `POST /api/account/link-email` - Link email to wallet account\n\n### Admin API\n- `GET /api/admin/stats` - System statistics\n- `GET /api/admin/downloads` - Download analytics\n\n## 🧪 Testing\n\n### Test Structure\n```\n__tests__/\n├── api/              # API route tests\n├── components/       # Component tests\n├── integration/      # Integration tests\n└── lib/             # Library tests\n```\n\n### Running Tests\n```bash\n# Unit tests\nnpm test\n\n# Integration tests\nnpm run test:integration\n\n# E2E tests (requires running app)\nnpm run test:e2e\n\n# Test coverage\nnpm run test:coverage\n\n# Run specific test file\nnpm test -- auth.test.ts\n```\n\n## 🚢 Deployment\n\n### Kubernetes Deployment\n\nThe application is deployed as part of the BryanLabs bare-metal infrastructure:\n\n1. **Repository Structure**\n   ```\n   bare-metal/\n   └── cluster/\n       └── chains/\n           └── cosmos/\n               └── fullnode/\n                   └── snapshot-service/\n                       └── webapp/\n                           ├── deployment.yaml\n                           ├── configmap.yaml\n                           ├── secrets.yaml\n                           ├── pvc.yaml\n                           └── kustomization.yaml\n   ```\n\n2. **Deploy with Kustomize**\n   ```bash\n   cd /path/to/bare-metal\n   kubectl apply -k cluster\n   ```\n\n3. **Verify deployment**\n   ```bash\n   kubectl get pods -n fullnodes -l app=webapp\n   kubectl get svc -n fullnodes webapp\n   ```\n\n### Docker Build\n\n```bash\n# Build for production (AMD64)\ndocker buildx build --builder cloud-bryanlabs-builder \\\n  --platform linux/amd64 \\\n  -t ghcr.io/bryanlabs/snapshots:v1.5.0 \\\n  --push .\n\n# Build for local testing\ndocker build -t snapshots:local .\n```\n\n### CI/CD Pipeline\nThe project uses GitHub Actions for automated deployment:\n- Tests run on every push\n- Docker images built and pushed to GitHub Container Registry\n- Kubernetes manifests in bare-metal repo updated\n- Automatic rollback on failure\n\n## 📊 Monitoring\n\n### Health Checks\n- `/api/health` - Application health status\n- Kubernetes liveness/readiness probes configured\n- Database connection monitoring\n- Nginx storage availability checks\n\n### Metrics Collection\nThe service exports Prometheus metrics at `/api/metrics`:\n- Request counts and latencies\n- Download statistics by tier and chain\n- Bandwidth usage metrics\n- Authentication success/failure rates\n- Database query performance\n\n### Grafana Dashboards\nPre-built dashboards available:\n- Service Overview\n- User Analytics\n- Download Statistics\n- Error Tracking\n- Performance Metrics\n\n## 🔗 Integration with Snapshot Processor\n\nThe web app displays snapshots created by the [snapshot-processor](https://github.com/bryanlabs/snapshot-processor):\n\n### How it Works\n1. **Processor Configuration** defines snapshot schedules per chain\n2. **Processor creates** VolumeSnapshots on schedule\n3. **Processor compresses** snapshots (ZST or LZ4)\n4. **Processor uploads** to nginx storage at `/snapshots/[chain-id]/`\n5. **Web app reads** nginx autoindex to list available snapshots\n6. **Web app generates** secure download URLs for users\n\n### File Naming Convention\n- Scheduled: `[chain-id]-[YYYYMMDD]-[HHMMSS].tar.[compression]`\n- On-demand: `[chain-id]-[block-height].tar.[compression]`\n- Examples: \n  - `noble-1-20250722-174634.tar.zst`\n  - `osmosis-1-12345678.tar.lz4`\n\n## 🤝 Contributing\n\nWe welcome contributions! Please see our [Contributing Guide](./CONTRIBUTING.md) for details.\n\n### Development Process\n1. Fork the repository\n2. Create a feature branch\n3. Make your changes\n4. Add tests\n5. Submit a pull request\n\n### Code Style\n- Follow TypeScript best practices\n- Use ESLint and Prettier\n- Write meaningful commit messages\n- Add JSDoc comments for public APIs\n\n## 📄 License\n\nThis project is licensed under the MIT License - see the [LICENSE](./LICENSE) file for details.\n\n## 🙏 Acknowledgments\n\n- BryanLabs team for infrastructure support\n- Cosmos ecosystem for blockchain technology\n- Open source contributors\n\n## 📞 Support\n\n- **Documentation**: [docs/](./docs/)\n- **Issues**: [GitHub Issues](https://github.com/bryanlabs/snapshots/issues)\n- **Discord**: [BryanLabs Discord](https://discord.gg/bryanlabs)\n- **Email**: support@bryanlabs.net","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbryanlabs%2Fsnapshots","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbryanlabs%2Fsnapshots","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbryanlabs%2Fsnapshots/lists"}