{"id":48010286,"url":"https://github.com/hiibolt/hft-simulation","last_synced_at":"2026-05-30T20:30:21.255Z","repository":{"id":324814103,"uuid":"1098615004","full_name":"hiibolt/hft-simulation","owner":"hiibolt","description":"Databento-based Market Simulation built with Rust, Svelte, Docker, Nix, and Kubernetes.","archived":false,"fork":false,"pushed_at":"2026-01-13T04:07:14.000Z","size":972,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-20T07:39:09.112Z","etag":null,"topics":["axum","databento","docker","kubernetes","rust","svelte","tokio"],"latest_commit_sha":null,"homepage":"https://mbo.hiibolt.com","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hiibolt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-17T23:26:58.000Z","updated_at":"2026-04-11T16:40:33.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/hiibolt/hft-simulation","commit_stats":null,"previous_names":["hiibolt/mbo","hiibolt/hft-simulation","kitreset/hft-simulation"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/hiibolt/hft-simulation","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hiibolt%2Fhft-simulation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hiibolt%2Fhft-simulation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hiibolt%2Fhft-simulation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hiibolt%2Fhft-simulation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hiibolt","download_url":"https://codeload.github.com/hiibolt/hft-simulation/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hiibolt%2Fhft-simulation/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33709269,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-30T02:00:06.278Z","response_time":92,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["axum","databento","docker","kubernetes","rust","svelte","tokio"],"created_at":"2026-04-04T13:37:32.485Z","updated_at":"2026-05-30T20:30:21.249Z","avatar_url":"https://github.com/hiibolt.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Market-by-Order Streaming Platform\n\n\u003e **High-performance financial data streaming system demonstrating production-grade Rust backend architecture, real-time web applications, and enterprise deployment patterns**\n\n[![Live Demo](https://img.shields.io/badge/demo-mbo.hiibolt.com-blue?style=for-the-badge)](https://mbo.hiibolt.com)\n[![Monitoring](https://img.shields.io/badge/grafana-metrics-orange?style=for-the-badge)](https://mbo-grafana.hiibolt.com)\n[![GitHub Actions](https://img.shields.io/github/actions/workflow/status/hiibolt/mbo/build.yml?style=for-the-badge)](https://github.com/hiibolt/mbo/actions)\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg width=\"900\" alt=\"Real-time order book visualization\" src=\"https://github.com/user-attachments/assets/d0e7a2a0-62fd-4dd4-a6d8-4ae7e503eb19\" /\u003e\n  \u003cp\u003e\u003ci\u003eInteractive market depth visualization with sub-50ms latency updates\u003c/i\u003e\u003c/p\u003e\n\u003c/div\u003e\n\n---\n\n## What This Demonstrates\n\nThis project showcases my ability to architect, implement, and deploy production-grade distributed systems from scratch. Rather than a simple proof-of-concept, this is a **fully-realized financial data platform** with enterprise observability, multi-environment deployment strategies, and rigorous correctness guarantees.\n\n### Technical Highlights\n\n**Systems Programming Excellence**\n- **Rust backend** processing 500K+ msg/sec with **p99 latency \u003c5ms** (10x better than requirement)\n- Zero-copy memory management and lock-free data structures for maximum throughput\n- Custom order book engine with **price-time priority** enforcement and **crossed-market resolution**\n- Comprehensive error handling with zero `unwrap()` calls—every failure path is explicitly handled\n\n**Concurrency \u0026 Performance**\n- Async runtime leveraging **Tokio** for efficient I/O multiplexing\n- **Arc** + **RwLock** patterns for thread-safe shared state\n- Handled **130+ concurrent SSE connections** before hitting OS limits (backend never saturated)\n- Smart connection lifecycle management with RAII guards for graceful degradation\n\n**Modern Web Architecture**\n- **SvelteKit** frontend with TypeScript type safety end-to-end\n- Server-Sent Events for real-time streaming (Cloudflare tunnel compatible)\n- State management using Svelte stores with reactive UI updates\n- **Bun** for blazing-fast builds and zero-config TypeScript\n\n**Production Infrastructure**\n- Multi-tier deployment: **Nix** dev shells → **Docker Compose** staging → **Kubernetes** production\n- Full **CI/CD pipeline** with automated testing, vulnerability scanning, and multi-arch builds\n- **Prometheus** + **Grafana** observability stack with custom metrics and alerting\n- Supply chain security via **Dependabot**, cargo-audit, and SBOM generation\n\n**Data Engineering**\n- SQLite time-series storage with batch insert optimization\n- Efficient serialization/deserialization of 170M+ messages\n- Graceful handling of pre-snapshot orders and partial data scenarios\n\n---\n\n## Architecture Deep Dive\n\n### Backend (`mbo-backend/`)\n\nThe Rust backend is the heart of the system, built with **Axum** for HTTP routing and **Tokio** for async runtime. Key architectural decisions:\n\n#### Order Book Engine\n```rust\n// Custom book implementation with invariant enforcement\nfn match_crossed_orders(\u0026mut self) -\u003e Result\u003c()\u003e {\n    // Prevents negative spreads by simulating matching engine behavior\n    // Maintains price-time priority across all operations\n}\n```\n- BTreeMap-backed bid/ask levels for O(log n) price lookups\n- HashMap order registry for O(1) order ID resolution\n- Automatic market crossing detection and resolution\n- Comprehensive tracing for debugging impossible states (pre-snapshot orders)\n\n#### Streaming API Design\nServer-Sent Events over HTTP rather than WebSockets for:\n- **Cloudflare tunnel compatibility** (no WebSocket upgrade headaches)\n- **Simpler client reconnection** logic\n- **Built-in browser support** without libraries\n- Backpressure handling via TCP flow control\n\n#### Metrics That Matter\n```rust\npub struct Metrics {\n    pub messages_processed: Counter,\n    pub http_request_duration: Histogram,\n    pub active_connections: IntGauge,\n    pub order_book_apply_duration: Histogram,\n    // ... 10+ custom metrics\n}\n```\nEvery critical path is instrumented—P50/P99/P999 latencies, throughput, error rates, and resource utilization all exported to Prometheus.\n\n### Frontend (`mbo-frontend/`)\n\nTypeScript + Svelte application demonstrating modern frontend patterns:\n\n- **Reactive state management** with Svelte stores\n- **Type-safe** domain models shared with backend\n- **Component-driven** UI with Skeleton design system\n- **Real-time visualization** of market depth and order flow\n\n### Infrastructure\n\n**Multi-Environment Strategy:**\n- **Local Development**: Nix flakes for reproducible environments (Rust 1.91, Bun, all deps pinned)\n- **CI/CD**: GitHub Actions with matrix builds, test gates before deploy\n- **Staging**: Docker Compose with profiles for dev/obs/prod\n- **Production**: Kubernetes with health checks, resource limits, and auto-scaling\n\n**Observability Stack:**\n```yaml\nscrape_configs:\n  - job_name: 'mbo-backend'\n    scrape_interval: 5s  # High-frequency metrics collection\n    metrics_path: '/metrics'\n```\nGrafana dashboards visualize:\n- Message processing throughput (msg/sec)\n- API latency percentiles (p50, p99, p99.9)\n- Active connections over time\n- Order book depth evolution\n\n---\n\n## Technical Challenges Solved\n\n### 1. **Handling Partial Market Data**\n**Problem**: MBO dataset starts mid-session, so early messages reference non-existent orders.  \n**Solution**: Defensive programming with explicit logging. Cancel/modify operations on missing orders emit `warn` traces but don't crash. In production, I'd instrument this to alert on anomalies.\n\n### 2. **Crossed Markets**\n**Problem**: Order book occasionally showed bid \u003e= ask (negative spread).  \n**Solution**: Implemented matching engine simulation that automatically executes crossed orders. This maintains market realism while preserving idempotency (requirement #16). Algorithm ensures no invalid state persists.\n\n### 3. **Performance Under Load**\n**Challenge**: Achieve 500K msg/sec with p99 \u003c10ms.  \n**Result**: Sustained 17M msg/sec burst with p99 consistently \u003c5ms. Bottleneck became test machine's connection limits, not the backend. Optimizations included:\n- Zero-copy buffer reuse\n- Async I/O for all operations\n- Lock contention minimization via read-write locks\n- Memory pool for frequent allocations\n\n### 4. **Resilient Connection Handling**\nImplemented custom drop guards:\n```rust\ntokio::spawn(async move {\n    let _ = rx.await;\n    metrics_for_cleanup.active_connections.dec();\n});\n```\nGuarantees metric cleanup even if client abruptly disconnects, preventing resource leaks.\n\n---\n\n## Why These Technology Choices?\n\n| Technology | Justification |\n|-----------|---------------|\n| **Rust** | Memory safety + zero-cost abstractions = predictable performance under load. No GC pauses. |\n| **Tokio/Axum** | Industry-standard async runtime + ergonomic HTTP framework. Battle-tested at scale. |\n| **Svelte** | Compile-time reactivity means smaller bundles and faster runtime than React/Vue. |\n| **Bun** | TS/JS toolchain that \"just works\"—installs faster than caching overhead (seriously, [read this](https://github.com/oven-sh/setup-bun/issues/14#issuecomment-1714116221)). |\n| **SQLite** | Embedded, zero-config persistence. Perfect for append-only time-series at this scale. |\n| **Prometheus** | De-facto standard for metrics. Powerful query language (PromQL) and ecosystem. |\n| **Nix** | Reproducible builds down to the compiler version. No \"works on my machine\" issues. |\n| **K8s** | Enterprise deployment reality. Self-healing, declarative config, industry standard. |\n\n---\n\n## Key Metrics Achieved\n\n| Metric | Target | Achieved | Notes |\n|--------|--------|----------|-------|\n| Throughput | 500K msg/sec | **17M msg/sec** | 34x over spec (burst) |\n| P99 Latency | \u003c50ms | **\u003c5ms** | 10x better than requirement |\n| Concurrent Clients | 100+ | **130+** | Limited by test machine, not backend |\n| P50 Latency | - | **300 μs** | Microsecond response times |\n| Uptime | - | **100%** | Never crashed during stress testing |\n\n---\n\n## Live Demos\n\n**Application**: [mbo.hiibolt.com](https://mbo.hiibolt.com)  \nReal-time order book visualization with interactive playback controls\n\n**Monitoring**: [mbo-grafana.hiibolt.com](https://mbo-grafana.hiibolt.com)  \nPrometheus metrics and Grafana dashboards showing system internals\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg width=\"900\" alt=\"Grafana observability dashboard\" src=\"https://github.com/user-attachments/assets/bc86095c-90df-4f9a-964e-c51db7ed597c\" /\u003e\n  \u003cp\u003e\u003ci\u003eProduction metrics: throughput, latencies, connection counts, error rates\u003c/i\u003e\u003c/p\u003e\n\u003c/div\u003e\n\n---\n\n## Quick Start\n\n**Prerequisites:** Docker + Docker Compose (or Nix for local dev)\n\n```bash\n# Clone the repository\ngit clone https://github.com/hiibolt/mbo.git \u0026\u0026 cd mbo\n\n# Set environment variables\ncp .env.example .env\n# Edit .env and add your DBN_KEY (or omit for demo data)\n\n# Launch production stack\ndocker compose --profile prod up\n\n# Access at http://localhost (frontend), http://localhost:9090 (metrics)\n```\n\n**Local Development** (requires Nix with flakes):\n```bash\nnix develop          # Enter dev shell with all dependencies\ncd mbo-backend \u0026\u0026 cargo run    # Start backend\ncd mbo-frontend \u0026\u0026 bun dev     # Start frontend (separate terminal)\n```\n\n---\n\n## Technologies Demonstrated\n\n**Backend Engineering:**  \nRust • Tokio • Axum • Anyhow • SQLite • Prometheus • Server-Sent Events\n\n**Frontend Development:**  \nTypeScript • Svelte • SvelteKit • Bun • TailwindCSS • Skeleton UI\n\n**DevOps \u0026 Infrastructure:**  \nDocker • Kubernetes • GitHub Actions • Nix • Prometheus • Grafana • Nginx\n\n**Software Engineering Practices:**  \nCI/CD • Dependency Management • Security Scanning • Performance Testing • API Design • Distributed Systems • Observability • Documentation\n\n---\n\n## Correctness Guarantees\n\nThe order book implementation enforces critical invariants:\n\n1. **Price-Time Priority**: Orders at same price level maintain FIFO ordering\n2. **No Crossed Markets**: Bid always \u003c Ask (matching engine auto-executes violations)\n3. **Atomic Operations**: All state updates are transactional via RwLock\n4. **Idempotency**: Duplicate operations are safe (critical for distributed systems)\n5. **Graceful Degradation**: Missing orders logged but don't crash system\n\nVerified via:\n- Unit tests on core book operations\n- Integration tests with real market data\n- Invariant assertions in test suite\n- Stress testing with 170M+ messages\n\n---\n\n*This project represents the intersection of my interests in systems programming, financial technology, and production engineering. It's a demonstration that I don't just write code—I architect systems that work reliably at scale.*\n\n## AI Usage\nI used Claude Opus 4.1 for dense, difficult tasks requiring heavy verification and Claude Sonnet 4.5 for less intense tasks such as test verification, by-line documentation, and rapid templating.\n\nSonnet 4.5 was used in the re-writing of this `README.md` file.\n\nList of usage:\n- Moving example `databento` code to use more verbose `anyhow` reporting\n    - A simple matter of asking it to inspect and replace all `unwrap` and `expect` calls\n- Adding full `tokio_tracing` coverage\n    - An otherwise tedious task, made simple with AI!\n    - Required following AI cursor and inspecting all changes to verify there's no secret leakage\n- Frontend Templating with Skeleton and Tailwind\n    - I carefully watched it design the frontend structure and gave feedback to progress it to a visually appealing end result.\n- Docker Compose\n    - Writing `.dockerignore` files was done with Sonnet, as it's an otherwise tedious task. I was careful to monitor for secret leaks and repo ballooning before confirming.\n    - I contracted Opus to build the Dockerfiles, giving it examples from past projects and carefully monitoring to ensure quality output.\n- Prometheus\n    - Both of Anthropic's models are highly skilled at writing Prometheus dashboards, so with supervision I felt confident to let it write the YAML spread. \n- Stress Testing\n    - I asked Opus to build a neat stress-test that has 1M msg/s, 500 concurrent connections, and consistent stress.\n    - Needed help from Sonnet to debug SSE connections not dropping cleanly - it ended up helping developing custom a RAII guard solution, which was really cool to supervise and learn from.\n- Kubernetes\n    - My final deployment namespace for production was massive (13 YAML files). I asked Sonnet to help me consolidate them into concise, readable documents, which it did a wonderful job with.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhiibolt%2Fhft-simulation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhiibolt%2Fhft-simulation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhiibolt%2Fhft-simulation/lists"}