{"id":29100494,"url":"https://github.com/thirdweb-dev/engine-core","last_synced_at":"2026-04-21T13:01:13.909Z","repository":{"id":301636467,"uuid":"981427942","full_name":"thirdweb-dev/engine-core","owner":"thirdweb-dev","description":"Production-grade blockchain transaction infrastructure","archived":false,"fork":false,"pushed_at":"2026-04-20T14:50:01.000Z","size":1039,"stargazers_count":22,"open_issues_count":0,"forks_count":12,"subscribers_count":4,"default_branch":"main","last_synced_at":"2026-04-20T16:42:06.971Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://thirdweb.com","language":"Rust","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thirdweb-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-05-11T04:41:22.000Z","updated_at":"2026-04-20T14:47:05.000Z","dependencies_parsed_at":"2025-06-27T23:19:59.385Z","dependency_job_id":"aa93a7bd-cff4-4ffa-9da1-f3e89a81b084","html_url":"https://github.com/thirdweb-dev/engine-core","commit_stats":null,"previous_names":["thirdweb-dev/engine-core"],"tags_count":32,"template":false,"template_full_name":null,"purl":"pkg:github/thirdweb-dev/engine-core","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thirdweb-dev%2Fengine-core","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thirdweb-dev%2Fengine-core/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thirdweb-dev%2Fengine-core/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thirdweb-dev%2Fengine-core/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thirdweb-dev","download_url":"https://codeload.github.com/thirdweb-dev/engine-core/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thirdweb-dev%2Fengine-core/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32093156,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-21T11:25:29.218Z","status":"ssl_error","status_checked_at":"2026-04-21T11:25:28.499Z","response_time":128,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-28T18:14:15.195Z","updated_at":"2026-04-21T13:01:13.846Z","avatar_url":"https://github.com/thirdweb-dev.png","language":"Rust","readme":"# Thirdweb Engine Core\n\nProduction-grade blockchain transaction infrastructure built in Rust. Engine Core is a high-performance, horizontally-scalable system designed for developers building serious Web3 applications that require reliable smart contract interactions, Account Abstraction support, and enterprise-level transaction processing.\n\n## Why Engine Core?\n\n### Performance \u0026 Scalability\nBuilt with Rust's zero-cost abstractions and memory safety guarantees. The architecture supports horizontal scaling through Redis-backed job queues with configurable worker pools and lease-based concurrency control.\n\n### Production-Ready Infrastructure\n- Redis-backed message queues with atomic operations and retry logic\n- Graceful shutdown handling with job completion guarantees  \n- Comprehensive error handling and transaction rollback mechanisms\n- Built-in monitoring and observability through structured logging\n\n### Developer Experience\n- Complete OpenAPI specification with interactive documentation\n- Type-safe configuration system with environment variable overrides\n- Modular architecture allowing selective component deployment\n- Extensive test coverage including integration tests with Redis\n\n## 🏗️ Architecture\n\nEngine Core implements a microservices-like architecture within a single binary, using Rust's workspace system for clean module separation:\n\n### Core Infrastructure (`core/`)\n**Purpose**: Fundamental blockchain operations and abstractions\n- **Chain Management** (`chain.rs`): Multi-chain RPC client management with automatic failover\n- **Transaction Primitives** (`transaction.rs`): Raw transaction building, signing, and broadcasting\n- **UserOperation Support** (`userop.rs`): Complete ERC-4337 implementation with v0.6/v0.7 compatibility\n- **RPC Clients** (`rpc_clients/`): Specialized clients for bundlers, paymasters, and JSON-RPC endpoints\n- **Error Handling** (`error.rs`): Comprehensive error types with context preservation\n\n### Account Abstraction Engine (`aa-core/`)\n**Purpose**: Complete ERC-4337 Account Abstraction implementation\n- **Smart Account Management** (`smart_account/`): Account factory integrations and deployment\n- **UserOperation Builder** (`userop/`): Gas estimation, signature aggregation, and bundler submission\n- **Account Factory Support** (`account_factory/`): Pluggable factory implementations (default, chained)\n- **Signature Handling** (`signer.rs`): Multi-signature support with Vault integration\n\n### HTTP API Server (`server/`)\n**Purpose**: REST API layer with comprehensive endpoint coverage\n- **Contract Operations**: Read, write, and encode smart contract functions\n- **Transaction Management**: Raw transaction sending with AA support\n- **Message Signing**: EIP-712 typed data and personal message signing\n- **Dynamic ABI**: Runtime contract introspection and interaction\n- **OpenAPI Documentation**: Auto-generated specs with Scalar UI\n\n### Background Job System (`executors/` + `twmq/`)\n**Purpose**: Reliable asynchronous processing with Redis persistence\n\n#### TWMQ (Thirdweb Message Queue)\nAdvanced Redis-backed job queue with enterprise features:\n- **Lease-Based Concurrency**: Prevents job duplication across worker instances\n- **Atomic Operations**: All queue operations use Lua scripts for consistency\n- **Retry Logic**: Configurable backoff strategies with failure categorization\n- **Job Lifecycle Management**: Pending → Active → Success/Failed with full audit trail\n- **Delayed Jobs**: Schedule jobs for future execution with precise timing\n- **Cancellation Support**: Cancel jobs in any state with immediate or pending cancellation\n\n#### Executor Types\n- **Webhook Delivery**: Reliable HTTP webhook notifications with configurable retries\n- **Transaction Confirmation**: Block confirmation tracking with reorganization handling  \n- **External Bundler Integration**: UserOperation submission and status monitoring\n- **EOA Transaction Processing**: Production-grade EOA (Externally Owned Account) transaction management with advanced nonce handling, crash recovery, and optimal throughput\n\n##### EOA Executor Deep Dive\n\nThe EOA executor implements a sophisticated single-worker-per-EOA architecture that ensures transaction consistency while maximizing throughput:\n\n**Key Features:**\n- **Crash-Resilient Recovery**: Borrowed transaction pattern prevents loss during worker restarts\n- **Intelligent Nonce Management**: Optimistic nonce allocation with recycling for failed transactions\n- **Three-Phase Processing**: Recovery → Confirmation → Send phases ensure complete transaction lifecycle management\n- **Adaptive Capacity Control**: Dynamic in-flight transaction limits based on network conditions\n- **Health Monitoring**: Automatic EOA balance checking with funding state awareness\n\n**Transaction Flow:**\n1. **Recovery Phase**: Rebroadcasts any prepared transactions from crashes\n2. **Confirmation Phase**: Efficiently tracks transaction confirmations using nonce progression\n3. **Send Phase**: Processes new transactions with recycled nonce prioritization and capacity management\n\n**Error Classification:**\n- **Deterministic Failures**: Immediate requeue with nonce recycling (invalid signatures, malformed transactions)\n- **Success Cases**: Transaction tracking for known/duplicate transactions\n- **Indeterminate Cases**: Optimistic handling for network timeouts and unknown errors\n\nThis architecture provides strong consistency guarantees while handling high-volume transaction processing with graceful degradation under network stress.\nFor more details, see [README_EOA.md](README_EOA.md).\n\n### Thirdweb Service Integration (`thirdweb-core/`)\n**Purpose**: First-party service integrations\n- **Vault SDK**: Hardware-backed private key management\n- **IAW (In-App Wallets)**: Embedded wallet creation and management\n- **ABI Service**: Dynamic contract ABI resolution and caching\n\n## 🚀 Getting Started\n\n### System Requirements\n\n- **Rust 1.70+** (2021 edition with async support)\n- **Redis 6.0+** (required for job queue persistence and atomic operations)\n- **Thirdweb API Credentials** (secret key and client ID from dashboard)\n\n### Quick Setup\n\n```bash\n# Clone and build\ngit clone \u003crepo-url\u003e \u0026\u0026 cd engine-core\ncargo build --release\n\n# Start Redis (Docker recommended for development)\ndocker run -d --name redis -p 6379:6379 redis:7-alpine\n\n# Configure credentials\nexport APP__THIRDWEB__SECRET=\"your_secret_key\"\nexport APP__THIRDWEB__CLIENT_ID=\"your_client_id\"\n\n# Launch engine\nRUST_LOG=info ./target/release/thirdweb-engine\n```\n\n### Configuration System\n\nEngine Core uses a hierarchical configuration system: YAML files + environment variables with full type safety and validation.\n\n#### Configuration Layers\n1. **Base Configuration** (`server_base.yaml`) - Default values\n2. **Environment-Specific** (`server_development.yaml`, `server_production.yaml`)\n3. **Environment Variables** - Highest priority, prefix with `APP__`\n\n#### Essential Configuration\n\n```yaml\n# server/configuration/server_local.yaml\nserver:\n  host: \"0.0.0.0\"\n  port: 3069\n\nthirdweb:\n  secret: \"your_thirdweb_secret_key\"\n  client_id: \"your_thirdweb_client_id\"\n  urls:\n    vault: \"https://vault.thirdweb.com\"\n    bundler: \"bundler.thirdweb.com\" \n    paymaster: \"bundler.thirdweb.com\"\n\nredis:\n  url: \"redis://localhost:6379\"\n\nqueue:\n  webhook_workers: 50\n  external_bundler_send_workers: 20\n  userop_confirm_workers: 10\n  local_concurrency: 100\n  polling_interval_ms: 100\n  lease_duration_seconds: 600\n```\n\n#### Environment Variable Override Examples\n\n```bash\n# Scale worker pools for high throughput\nexport APP__QUEUE__WEBHOOK_WORKERS=200\nexport APP__QUEUE__LOCAL_CONCURRENCY=500\n\n# Custom Redis configuration\nexport APP__REDIS__URL=\"redis://redis-cluster:6379\"\n\n# Debug logging for development\nexport RUST_LOG=\"thirdweb_engine=debug,twmq=debug\"\n```\n\n### Docker Deployment\n\n```dockerfile\nFROM rust:1.70 as builder\nWORKDIR /app\nCOPY . .\nRUN cargo build --release\n\nFROM debian:bookworm-slim\nRUN apt-get update \u0026\u0026 apt-get install -y ca-certificates\nCOPY --from=builder /app/target/release/thirdweb-engine /usr/local/bin/\nEXPOSE 3069\nCMD [\"thirdweb-engine\"]\n```\n\n### Health Checks \u0026 Monitoring\n\n```bash\n# API health check\ncurl http://localhost:3069/v1/api.json\n\n# Queue statistics (if monitoring endpoint enabled)\ncurl http://localhost:3069/v1/queue/stats\n\n# Structured logging output\nRUST_LOG=\"thirdweb_engine=info,twmq=warn\" ./thirdweb-engine\n```\n\n## 🌩️ Thirdweb Engine Cloud\n\n**Want Engine without the ops overhead?** [**Thirdweb Engine Cloud**](https://thirdweb.com/engine) is our fully-managed, production-ready service built on Engine Core with enterprise enhancements:\n\n### ⚡ Enhanced Features Beyond Core\n- **Auto Execution Resolution**: Smart execution strategy selection with cached Account Abstraction details\n- **Streamlined Wallet Management**: Convenient wallet creation and management through dashboard\n- **Smart Account Cache**: Pre-resolved AA configurations (signer addresses, factory details, gas policies)\n- **Global Edge Network**: Optimized RPC routing and intelligent caching for sub-100ms response times\n- **Advanced Analytics**: Real-time transaction monitoring, gas usage insights, and performance metrics\n- **Zero-Config Account Abstraction**: Automatic paymaster selection and gas sponsorship\n\n### 🛡️ Production-Ready Operations\n- **High Availability** with automated failover and disaster recovery\n- **Horizontal Auto-Scaling** based on transaction volume and queue depth\n- **Enterprise Security**: Encryption at rest/transit, comprehensive audit logging\n- **Expert Support** with dedicated technical assistance\n- **Custom Rate Limits** and priority processing for high-volume applications\n\n### 🚀 Get Started Instantly\n```bash\n# No infrastructure setup required\ncurl -X POST \"https://api.engine.thirdweb.com/contract/write\" \\\n  -H \"Authorization: Bearer YOUR_ACCESS_TOKEN\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"executionOptions\": {\n      \"chainId\": 137,\n      \"type\": \"auto\",\n      \"from\": \"0x...\"\n    },\n    \"params\": [{\n      \"contractAddress\": \"0x...\",\n      \"functionName\": \"mint\",\n      \"args\": [\"0x...\", \"1\"]\n    }]\n  }'\n```\n\n**[Start Building →](https://thirdweb.com/engine)** | **[View Cloud API Reference →](https://engine.thirdweb.com/reference)**\n\n---\n\n## 📚 Self-Hosted API Documentation\n\nFor self-hosted Engine Core instances:\n- **Interactive Documentation**: `http://localhost:3069/reference`\n- **OpenAPI Specification**: `http://localhost:3069/api.json`\n\n## 🔧 Development\n\n### Running Tests\n\n```bash\n# Run all tests\ncargo test\n\n# Run tests for a specific component\ncargo test -p twmq\ncargo test -p engine-core\n\n# Run with Redis integration tests\ncargo nextest run -p twmq --profile ci\n```\n\n### Code Structure\n\n```\n├── server/           # Main HTTP server\n│   ├── src/\n│   │   ├── http/     # REST API routes and handlers\n│   │   ├── queue/    # Queue management\n│   │   └── config.rs # Configuration management\n│   └── configuration/ # YAML config files\n├── core/             # Core blockchain functionality\n│   └── src/\n│       ├── userop.rs # Account Abstraction support\n│       ├── chain.rs  # Chain management\n│       └── error.rs  # Error types\n├── twmq/             # Message queue system\n│   └── src/\n│       ├── queue.rs  # Queue implementation\n│       └── job.rs    # Job processing\n├── executors/        # Background job handlers\n│   └── src/\n│       ├── webhook/  # Webhook delivery\n│       └── external_bundler/ # AA bundler integration\n│       └── eoa/      # EOA transaction processing\n│           ├── worker/       # Main worker logic\n│           ├── store/        # Redis-backed state management\n│           ├── events.rs     # Transaction lifecycle events\n│           └── error_classifier.rs # Error categorization\n└── thirdweb-core/    # Thirdweb integrations\n```\n\n### Adding New Features\n\n1. **New API Endpoints**: Add routes in `server/src/http/routes/`\n2. **Background Jobs**: Implement `DurableExecution` trait in `executors/`\n3. **Chain Support**: Extend `ThirdwebChainService` in `server/src/chains.rs`\n\n## 🔍 Monitoring \u0026 Operations\n\n### Queue Statistics\n\nThe system provides comprehensive queue monitoring for webhook delivery, transaction sending, and confirmation processing:\n\n```rust\n// Queue statistics are available through the QueueManager\nlet stats = queue_manager.get_stats().await?;\nprintln!(\"Pending jobs: {}\", stats.webhook.pending);\n```\n\n### Logging\n\nConfigure logging levels using the `RUST_LOG` environment variable:\n\n```bash\n# Detailed debugging\nRUST_LOG=\"thirdweb_engine=debug,twmq=debug,engine_executors=debug\"\n\n# Production logging\nRUST_LOG=\"thirdweb_engine=info,twmq=warn\"\n```\n\n### Health Checks\n\nThe server provides graceful shutdown handling and can be monitored for:\n\n- HTTP server health\n- Redis connectivity\n- Background worker status\n\n## 🔒 Security Considerations\n\n- **Vault Integration**: All private keys are managed through Thirdweb Vault\n- **API Authentication**: Requests are authenticated using Thirdweb secret keys\n- **Network Security**: Configure appropriate CORS policies for production\n- **Redis Security**: Secure your Redis instance with authentication and network restrictions\n\n## 🌐 Production Deployment\n\n### Environment Configuration\n\nSet `APP_ENVIRONMENT=production` and create `server_production.yaml`:\n\n```yaml\nserver:\n  host: \"0.0.0.0\"\n  port: 3069\n\nredis:\n  url: \"redis://your-redis-cluster\"\n\nqueue:\n  webhook_workers: 50\n  external_bundler_send_workers: 20\n  userop_confirm_workers: 10\n```\n\n### Resource Requirements\n\n- **CPU**: 2+ cores recommended\n- **Memory**: 1GB+ RAM (more for high-throughput scenarios)\n- **Storage**: Minimal (logs and temporary data only)\n- **Network**: Stable internet connection for blockchain RPC calls\n\n### Scaling Considerations\n\n- **Horizontal Scaling**: Multiple instances can share the same Redis backend\n- **Queue Workers**: Adjust worker counts based on throughput requirements\n- **Redis**: Consider Redis clustering for high availability\n\n## 📦 Dependencies\n\n### Key External Dependencies\n\n- **Alloy** - Ethereum library for Rust\n- **Axum** - Web framework\n- **Redis** - Message queue backend\n- **Tokio** - Async runtime\n- **Thirdweb Vault SDK** - Secure wallet management\n\n## 🤝 Contributing\n\n1. Fork the repository\n2. Create a feature branch\n3. Make your changes\n4. Add tests where appropriate\n5. Run the test suite: `cargo test`\n6. Submit a pull request\n\n## 📄 License\n\nMIT\n\n## 🆘 Support\n\nFor issues and questions:\n\n1. Check the API documentation at `/reference`\n2. Review server logs for error details\n3. Ensure Redis is running and accessible\n4. Verify Thirdweb credentials are valid\n\n---\n\n**Built with ❤️ by the Thirdweb team**\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthirdweb-dev%2Fengine-core","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthirdweb-dev%2Fengine-core","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthirdweb-dev%2Fengine-core/lists"}