{"id":28710636,"url":"https://github.com/marvinified/sharded","last_synced_at":"2026-03-11T02:01:41.388Z","repository":{"id":295534126,"uuid":"990285891","full_name":"Marvinified/sharded","owner":"Marvinified","description":"Sharded - Make any database fast","archived":false,"fork":false,"pushed_at":"2025-11-28T22:08:12.000Z","size":205,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-01T00:52:22.466Z","etag":null,"topics":["database","performance","prisma","sqlite"],"latest_commit_sha":null,"homepage":"https://www.npmjs.com/package/sharded","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Marvinified.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-05-25T21:46:02.000Z","updated_at":"2025-11-28T22:08:15.000Z","dependencies_parsed_at":"2025-09-21T02:32:27.928Z","dependency_job_id":"12c46dd3-8b95-4372-9d57-d690aa6ac3ee","html_url":"https://github.com/Marvinified/sharded","commit_stats":null,"previous_names":["marvinified/sharded"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Marvinified/sharded","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Marvinified%2Fsharded","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Marvinified%2Fsharded/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Marvinified%2Fsharded/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Marvinified%2Fsharded/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Marvinified","download_url":"https://codeload.github.com/Marvinified/sharded/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Marvinified%2Fsharded/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30367799,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-10T21:41:54.280Z","status":"online","status_checked_at":"2026-03-11T02:00:07.027Z","response_time":84,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","performance","prisma","sqlite"],"created_at":"2025-06-14T21:07:20.729Z","updated_at":"2026-03-11T02:01:41.376Z","avatar_url":"https://github.com/Marvinified.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Sharded - Make any database fast\n\n[![npm version](https://badge.fury.io/js/sharded.svg)](https://badge.fury.io/js/sharded)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n**Sharded** is a SQLite-based write buffer and caching system for Prisma that provides high-performance data operations with automatic synchronization to your main database.\n\n## 🚀 Performance Benefits\n\n- **Dramatic Speed Improvement**: Reduce write \u0026 read times from \u003e100ms to \u003c10ms\n- **10x faster reads** from SQLite cache vs. network database calls\n- **Reduced database load** through write buffering\n- **Improved user experience** with instant data access\n- **Scalable architecture** supporting multiple worker nodes\n\n## 🚀 Features\n\n- **Write Buffering**: Buffer write operations in fast SQLite databases before syncing to your main database\n- **Intelligent Caching**: Cache frequently accessed data for lightning-fast reads\n- **Automatic Sync**: Background synchronization using Redis queues (BullMQ)\n- **Prisma Integration**: Seamless integration with existing Prisma workflows\n- **Multi-Node Support**: Master/worker architecture for distributed systems\n- **Schema Generation**: CLI tools to generate optimized schemas for your blocks\n- **WAL Mode**: Automatic WAL optimizations prevent \"phantom data\" issues ([details](./WAL-OPTIMIZATIONS.md))\n\n## 🎯 When to Use Sharded\n\n### ✅ Perfect Use Cases\n\n- **Real-time applications** where sub-10ms response times are critical\n- **High-frequency read/write operations** on specific data subsets\n  - **Chat/messaging systems** with frequent message operations\n  - **Gaming applications** requiring ultra-fast state updates\n  - **Live collaboration tools** with real-time document editing\n  - **Analytics dashboards** with frequently accessed metric\n\n### ❌ When NOT to Use Sharded\n\n- **Infrequent database operations** (\u003c 10 operations per minute)\n- **Simple CRUD applications** without performance bottlenecks\n- **Applications with simple, linear data access patterns**\n- **One-time data processing** or batch operations\n- **Systems where network latency isn't a concern**\n- **Applications with mostly write-once, read-rarely data**\n- **Your database is fast enough** for your use case\n\n## 📦 Installation\n\n```bash\nyarn add sharded\n# or\nnpm install sharded\n```\n\n### Prerequisites\n\n- Node.js 16+\n- Prisma 6.7.0+\n- Redis (for queue management)\n- SQLite support\n\n### Docker Deployment\n\nWhen deploying with Docker, persist your blocks across redeployments by mounting the blocks data directory:\n\n```dockerfile\n# Dockerfile\nFROM node:18-alpine\n\nWORKDIR /app\nCOPY package*.json ./\nRUN yarn install\n\nCOPY . .\nRUN yarn build\n\n# Create blocks directory\nRUN mkdir -p /app/prisma/blocks/data\n\nEXPOSE 3000\nCMD [\"yarn\", \"start\"]\n```\n\n```yaml\n# docker-compose.yml\nversion: '3.8'\nservices:\n  app:\n    build: .\n    ports:\n      - \"3000:3000\"\n    volumes:\n      # Persist blocks across container restarts/redeployments\n      - blocks_data:/app/prisma/blocks/data\n    environment:\n      - DATABASE_URL=postgresql://user:pass@db:5432/mydb\n      - REDIS_URL=redis://redis:6379\n    depends_on:\n      - db\n      - redis\n\n  db:\n    image: postgres:15\n    environment:\n      POSTGRES_DB: mydb\n      POSTGRES_USER: user\n      POSTGRES_PASSWORD: pass\n    volumes:\n      - postgres_data:/var/lib/postgresql/data\n\n  redis:\n    image: redis:7-alpine\n    volumes:\n      - redis_data:/data\n\nvolumes:\n  blocks_data:    # Persists your Sharded blocks\n  postgres_data:  # Persists your main database\n  redis_data:     # Persists Redis queue data\n```\n\n**Important**: Without persisting `/app/prisma/blocks/data`, your blocks will be recreated on every deployment, losing cached data and requiring full reloads.\n\n## 🛠️ Quick Start\n\n### 1. Generate Block Schema\n\nFirst, generate a subset schema for the models you want to cache/buffer:\n\n```bash\n# Generate schema for specific models\nnpx sharded generate --schema=./prisma/schema.prisma --models=User,Order\n\n# Or generate for all models\nnpx sharded generate --schema=./prisma/schema.prisma --all-models\n```\n\nThis creates:\n\n- `prisma/blocks/block.prisma` - Optimized schema for SQLite\n- `prisma/blocks/template.sqlite` - Template database\n- `prisma/blocks/generated/` - Generated Prisma client\n\n### 2. Create a Block\n\n```typescript\nimport { Block } from \"sharded\";\nimport { PrismaClient } from \"@prisma/client\";\n\nconst mainClient = new PrismaClient();\n\n// Define how to load initial data into the block\nconst loader = async (blockClient: PrismaClient, mainClient: PrismaClient) =\u003e {\n  // Load users from main database\n  const users = await mainClient.user.findMany();\n  for (const user of users) {\n    await blockClient.user.create({ data: user });\n  }\n\n  // Load orders from main database\n  const orders = await mainClient.order.findMany();\n  for (const order of orders) {\n    await blockClient.order.create({ data: order });\n  }\n};\n\n// Create block client\nconst blockClient = await Block.create({\n  blockId: \"user-orders-cache\",\n  client: mainClient,\n  loader,\n  ttl: 3600, // Cache TTL in seconds (optional)\n  connection: {\n    host: \"localhost\",\n    port: 6379,\n    // password: 'your-redis-password'\n  },\n});\n```\n\n### 3. Use the Block Client\n\nThe block client works exactly like a regular Prisma client:\n\n```typescript\n// Create operations are buffered and synced asynchronously\nconst user = await blockClient.user.create({\n  data: {\n    email: \"user@example.com\",\n    name: \"John Doe\",\n  },\n});\n\n// Read operations use cached data when available\nconst users = await blockClient.user.findMany({\n  include: {\n    orders: true,\n  },\n});\n\n// Updates are buffered and synced\nawait blockClient.user.update({\n  where: { id: user.id },\n  data: { name: \"Jane Doe\" },\n});\n\n// Deletes are buffered and synced\nawait blockClient.user.delete({\n  where: { id: user.id },\n});\n```\n\n## 🏗️ Architecture\n\nBlock clients queue operations locally and to Redis. The `Block.watch()` method handles all background synchronization by automatically creating sync workers for blocks with pending operations.\n\n```typescript\n// Every block client works the same way\nconst blockClient = await Block.create({\n  blockId: \"my-cache\",\n  client: mainClient,\n  loader,\n  ttl: 3600, // Optional: 1 hour cache\n});\n\n// Background sync and cleanup handled by Block.watch()\nawait Block.watch({\n  ttl: 3600,\n  intervals: {\n    invalidation: 10000,   // Check TTL every 10 seconds\n    syncCheck: 2000,       // Check sync workers every 2 seconds\n    cleanup: 3600000,      // Clean Redis every hour (prevents performance degradation)\n  },\n  mainClient: prismaClient, // Used for sync workers\n  connection: {\n    host: \"localhost\",\n    port: 6379,\n  },\n});\n```\n\n### Data Flow\n\n1. **Writes**: Buffered in SQLite → Queued in Redis → Synced to main DB\n2. **Reads**: Check SQLite cache → Fallback to main DB → Cache result\n\n## 💡 Use Cases\n\n- **High-traffic applications** requiring fast read access\n- **Microservices** needing local data caching\n- **Real-time applications** with frequent database operations\n- **Analytics workloads** requiring fast aggregations\n- **Multi-tenant applications** with isolated data blocks\n\n## 🎯 Block Scoping Strategy\n\n**Important**: Sharded is designed for **subsections** of your application that need fast read/writes, not entire databases. Loading your entire database into a block would be inefficient and defeat the purpose.\n\n### ✅ Good Block Scoping Examples\n\n#### 1. **Per-User Blocks** (User Dashboard)\n\n```typescript\n// Block for a specific user's data\nconst userBlock = await Block.create({\n  blockId: `user-${userId}`,\n  client: mainClient,\n  loader: async (blockClient, mainClient) =\u003e {\n    // Load only this user's data\n    const user = await mainClient.user.findUnique({\n      where: { id: userId },\n      include: {\n        profile: true,\n        settings: true,\n        recentActivity: { take: 50 },\n      },\n    });\n\n    if (user) {\n      await blockClient.user.create({ data: user });\n    }\n  },\n});\n```\n\n#### 2. **Per-Chat Blocks** (Messaging App)\n\n```typescript\n// Block for a specific chat room\nconst chatBlock = await Block.create({\n  blockId: `chat-${chatId}`,\n  client: mainClient,\n  loader: async (blockClient, mainClient) =\u003e {\n    // Load chat and recent messages\n    const chat = await mainClient.chat.findUnique({\n      where: { id: chatId },\n      include: {\n        messages: {\n          take: 100, // Last 100 messages\n          orderBy: { createdAt: \"desc\" },\n        },\n        participants: true,\n      },\n    });\n\n    if (chat) {\n      await blockClient.chat.create({ data: chat });\n    }\n  },\n});\n\n// Fast message operations\nawait chatBlock.message.create({\n  data: {\n    content: \"Hello!\",\n    chatId: chatId,\n    userId: senderId,\n  },\n});\n```\n\n#### 3. **Per-Session Blocks** (E-commerce Cart)\n\n```typescript\n// Block for user's shopping session\nconst sessionBlock = await Block.create({\n  blockId: `session-${sessionId}`,\n  client: mainClient,\n  loader: async (blockClient, mainClient) =\u003e {\n    // Load cart, wishlist, and recently viewed\n    const session = await mainClient.session.findUnique({\n      where: { id: sessionId },\n      include: {\n        cart: { include: { items: true } },\n        wishlist: { include: { items: true } },\n        recentlyViewed: { take: 20 },\n      },\n    });\n\n    if (session) {\n      await blockClient.session.create({ data: session });\n    }\n  },\n});\n```\n\n#### 4. **Per-Game Blocks** (Gaming Application)\n\n```typescript\n// Block for active game state\nconst gameBlock = await Block.create({\n  blockId: `game-${gameId}`,\n  client: mainClient,\n  loader: async (blockClient, mainClient) =\u003e {\n    // Load game state and player data\n    const game = await mainClient.game.findUnique({\n      where: { id: gameId },\n      include: {\n        players: true,\n        gameState: true,\n        moves: { take: 50 }, // Recent moves\n      },\n    });\n\n    if (game) {\n      await blockClient.game.create({ data: game });\n    }\n  },\n});\n\n// Ultra-fast game moves\nawait gameBlock.move.create({\n  data: {\n    gameId,\n    playerId,\n    action: \"attack\",\n    coordinates: { x: 10, y: 15 },\n  },\n});\n```\n\n#### 5. **Per-Workspace Blocks** (Collaboration Tools)\n\n```typescript\n// Block for team workspace\nconst workspaceBlock = await Block.create({\n  blockId: `workspace-${workspaceId}`,\n  client: mainClient,\n  loader: async (blockClient, mainClient) =\u003e {\n    // Load workspace with active documents and team members\n    const workspace = await mainClient.workspace.findUnique({\n      where: { id: workspaceId },\n      include: {\n        documents: {\n          where: { status: \"active\" },\n          take: 50,\n        },\n        members: true,\n        recentActivity: { take: 100 },\n      },\n    });\n\n    if (workspace) {\n      await blockClient.workspace.create({ data: workspace });\n    }\n  },\n});\n```\n\n### ❌ Avoid These Patterns\n\n```typescript\n// ❌ DON'T: Load entire database\nconst badBlock = await Block.create({\n  blockId: \"entire-app\",\n  loader: async (blockClient, mainClient) =\u003e {\n    // This will be slow and memory-intensive\n    const allUsers = await mainClient.user.findMany(); // Could be millions\n    const allOrders = await mainClient.order.findMany(); // Could be millions\n    // ... loading everything\n  },\n});\n\n// ❌ DON'T: Overly broad scoping\nconst tooBroadBlock = await Block.create({\n  blockId: \"all-users-data\",\n  loader: async (blockClient, mainClient) =\u003e {\n    // Loading all users when you only need one\n    const users = await mainClient.user.findMany({\n      include: { orders: true, profile: true },\n    });\n  },\n});\n```\n\n### 🎯 Scoping Best Practices\n\n1. **Scope by User/Session**: Create blocks per user session or user context\n2. **Scope by Feature**: Create blocks for specific features (chat, cart, game)\n3. **Limit Data Size**: Only load what you need (recent messages, active items)\n4. **Use TTL Wisely**: Set appropriate cache expiration based on data freshness needs\n5. **Monitor Block Size**: Keep blocks under 100MB for optimal performance\n\n### 🔄 Block Lifecycle Management\n\n```typescript\n// Create block when user starts session\nconst userBlock = await Block.create({\n  blockId: `user-${userId}-${sessionId}`,\n  // ... config\n});\n\n// Use throughout session for fast operations\nawait userBlock.user.update({ ... });\nawait userBlock.activity.create({ ... });\n\n// Clean up when session ends\nawait Block.delete_block(`user-${userId}-${sessionId}`);\n```\n\n## 📚 API Reference\n\n### Block.create(config)\n\nCreates a new block instance.\n\n```typescript\ninterface BlockConfig\u003cT\u003e {\n  blockId: string;                    // Unique identifier for the block\n  client: T;                         // Main Prisma client\n  loader: (blockClient: T, mainClient: T) =\u003e Promise\u003cvoid\u003e; // Data loader function\n  debug?: boolean;                   // Enable debug logging\n  prismaOptions?: Prisma.PrismaClientOptions; // Additional Prisma options\n  connection?: {                     // Redis connection options\n    host: string;\n    port: number;\n    password?: string;\n  };\n  ttl?: number;                      // Cache TTL in seconds (used by watch() for invalidation)\n};\n```\n\n### Block.invalidate(blockId)\n\nManually invalidate a block cache:\n\n```typescript\nawait Block.invalidate(\"user-orders-cache\");\n```\n\n### Block.delete_block(blockId)\n\nDelete a block and its associated files:\n\n```typescript\nawait Block.delete_block(\"user-orders-cache\");\n```\n\n### Block.watch(options)\n\nStart watching for cache invalidation, sync worker management, and automatic Redis cleanup.\n\n**Interface:**\n```typescript\ninterface WatchOptions\u003cT\u003e {\n  ttl?: number;                  // Default TTL in seconds for all blocks\n  intervals?: {\n    invalidation?: number;       // Check TTL invalidation (ms, default: 10000)\n    syncCheck?: number;          // Check sync workers (ms, default: 2000)\n    cleanup?: number;            // Redis cleanup (ms, default: 3600000, set 0 to disable)\n  };\n  mainClient: T;                 // Required: Main Prisma client for sync worker creation\n  connection?: {                 // Redis connection options\n    host: string;\n    port: number;\n    password?: string;\n  };\n}\n```\n\n**Example:**\n```typescript\nawait Block.watch({\n  ttl: 3600,                    // Cache TTL in seconds\n  intervals: {\n    invalidation: 10000,        // Check TTL invalidation every 10 seconds\n    syncCheck: 2000,            // Check sync workers every 2 seconds\n    cleanup: 3600000,           // Clean Redis every 1 hour (default, optional)\n  },\n  mainClient: prismaClient,     // Required for sync worker creation\n  connection: {\n    host: \"localhost\",\n    port: 6379,\n  },\n});\n```\n\n**Automatic Redis Cleanup**: `Block.watch()` includes automatic cleanup to prevent performance degradation from accumulated stale data (failed jobs, orphaned keys, etc.). It runs every hour by default and cleans up:\n- Old failed jobs (older than 1 hour)\n- Stale operation keys for deleted blocks\n- Orphaned Redis metadata (`last_seen`, `block_ttl`)\n- Dead letter queues for non-existent blocks\n\n**Customize cleanup interval** based on your workload:\n```typescript\n// High-throughput apps: every 30 minutes\nintervals: { cleanup: 1800000 }\n\n// Normal workload: every 1 hour (default)\nintervals: { cleanup: 3600000 }\n\n// Disable automatic cleanup\nintervals: { cleanup: 0 }\n```\n\n**Important**: The `mainClient` parameter is crucial for sync worker creation. When `Block.watch()` detects blocks with pending operations, it uses this client to create sync workers that process queued operations and sync them to the main database.\n\n### Block.cleanup()\n\nManually trigger Redis cleanup (also runs automatically via `Block.watch()`):\n\n```typescript\n// Run immediate cleanup\nconst result = await Block.cleanup();\nconsole.log('Cleaned:', result);\n// { staleOperationKeys: 5, oldFailedJobs: 23, orphanedKeys: 8 }\n```\n\nUseful for:\n- Serverless environments (scheduled via cron)\n- Immediate cleanup when Redis is slow\n- Custom cleanup schedules outside of `Block.watch()`\n\n**📖 For detailed Redis maintenance guide**, see [REDIS-MAINTENANCE.md](./REDIS-MAINTENANCE.md)\n\n## ⚠️ Known Limitations\n\n- **Cache Consistency**: If records are modified directly in the main database, the block cache won't be aware until invalidation\n\n\n## 📋 TODO\n\n- **Multi-Machine Sync**: Currently, blocks with the same ID across different machines don't sync with each other. Multi-process on the same machine works fine as they share the same SQLite file location, but cross-machine block synchronization needs to be implemented.\n\n## 🤝 Contributing\n\n1. Fork the repository\n2. Create a feature branch: `git checkout -b feature/amazing-feature`\n3. Commit your changes: `git commit -m 'Add amazing feature'`\n4. Push to the branch: `git push origin feature/amazing-feature`\n5. Open a Pull Request\n\n\n## 🧪 Testing\n\n```bash\n# Run tests\nyarn test\n\n# Run in development mode\nyarn dev\n```\n\n## 📁 Project Structure\n\n```\nsharded/\n├── cli/                 # Command-line interface\n│   ├── generate.ts     # Schema generation\n│   └── index.ts        # CLI entry point\n├── runtime/            # Core runtime\n│   └── Block.ts        # Main Block class\n├── tests/              # Test files\n├── prisma/             # Example schema\n└── dist/               # Compiled output\n```\n\n\n## 📄 License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## 🔗 Links\n\n- [npm package](https://www.npmjs.com/package/sharded)\n- [GitHub repository](https://github.com/marvinified/sharded)\n- [Issues](https://github.com/marvinified/sharded/issues)\n\n---\n\nMade with ❤️ by the Sharded team\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarvinified%2Fsharded","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarvinified%2Fsharded","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarvinified%2Fsharded/lists"}