{"id":50383565,"url":"https://github.com/zoraizmohammad/agentic-pollination-uav","last_synced_at":"2026-05-30T13:30:38.948Z","repository":{"id":348108695,"uuid":"1196471620","full_name":"zoraizmohammad/agentic-pollination-uav","owner":"zoraizmohammad","description":"Advanced UAV Engineering Drone System","archived":false,"fork":false,"pushed_at":"2026-04-14T20:53:34.000Z","size":20740,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-14T22:26:29.045Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://agentic-pollination-uav.vercel.app","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zoraizmohammad.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-30T18:24:23.000Z","updated_at":"2026-04-14T20:53:38.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/zoraizmohammad/agentic-pollination-uav","commit_stats":null,"previous_names":["zoraizmohammad/drone-simulation","zoraizmohammad/agentic-pollination-uav"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/zoraizmohammad/agentic-pollination-uav","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zoraizmohammad%2Fagentic-pollination-uav","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zoraizmohammad%2Fagentic-pollination-uav/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zoraizmohammad%2Fagentic-pollination-uav/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zoraizmohammad%2Fagentic-pollination-uav/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zoraizmohammad","download_url":"https://codeload.github.com/zoraizmohammad/agentic-pollination-uav/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zoraizmohammad%2Fagentic-pollination-uav/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33694714,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-30T02:00:06.278Z","response_time":92,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-30T13:30:38.212Z","updated_at":"2026-05-30T13:30:38.928Z","avatar_url":"https://github.com/zoraizmohammad.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Smart Pollinator Drone — Mission Simulation \u0026 Agentic CV System\n\nA full-stack autonomous pollinator drone platform built in two mirrored halves: a browser-based mission dashboard (Preact + TypeScript + Vite) that runs a real-time simulation of a drone pollinating flowers across a 20m × 20m garden, and a production-grade autonomous flight and computer vision system (Python FastAPI, Raspberry Pi 4, Pixhawk 2.4.8, Google Coral TPU) that can fly the same mission on real hardware.\n\nThe two halves share identical mission logic, sensor models, and TSP path-planning algorithms. An LLM agent (Claude Haiku via LangChain `ChatAnthropic`) sits alongside the inference server and provides real-time planning decisions, adaptive confidence thresholds, and streaming mission commentary. LangChain callbacks route every LLM thought and tool invocation to the live terminal panel. Completed missions are embedded into a persistent Chroma vector store so the agent learns from past flights.\n\n---\n\n## Table of Contents\n\n1. [Architecture Overview](#architecture-overview)\n2. [How the Two Modes Work](#how-the-two-modes-work)\n3. [Frontend — Source Layout](#frontend--source-layout)\n4. [Data Layer](#data-layer)\n5. [Simulation Engine](#simulation-engine)\n6. [Sensor Models](#sensor-models)\n7. [UI Panels](#ui-panels)\n8. [Backend — Inference Server](#backend--inference-server)\n9. [Backend — Agent Server](#backend--agent-server)\n10. [LangChain Integration](#langchain-integration)\n11. [RAG Mission Memory](#rag-mission-memory)\n12. [UCB1 Confidence Bandit](#ucb1-confidence-bandit)\n13. [End-to-End Data Flow](#end-to-end-data-flow)\n14. [Quick Start](#quick-start)\n15. [Configuration](#configuration)\n16. [Physical Drone Hardware Stack](#physical-drone-hardware-stack)\n17. [Firmware \u0026 MAVLink Integration](#firmware--mavlink-integration)\n18. [Computer Vision Pipeline (Real Drone)](#computer-vision-pipeline-real-drone)\n19. [ML Model — YOLOv8 Flower Detection](#ml-model--yolov8-flower-detection)\n20. [Pollination Manager](#pollination-manager)\n21. [Hardware Assembly \u0026 Setup](#hardware-assembly--setup)\n22. [Tech Stack](#tech-stack)\n\n---\n\n## Architecture Overview\n\n```\n┌─────────────────────────────────────────────────────────────────────────┐\n│                          Browser — Preact + TypeScript                  │\n│                                                                         │\n│  ┌──────────────────┐  ┌────────────────┐  ┌──────────────────────────┐ │\n│  │  TopDownView     │  │  SideView      │  │  TelemetryPanel          │ │\n│  │  SVG 20×20m      │  │  altitude      │  │  sensor HUD + AI section │ │\n│  │  garden, drone   │  │  cross-section │  │  EKF, OF, battery, CV    │ │\n│  │  trail, TSP+AI   │  │  sparkline     │  │  decisions, overrides    │ │\n│  │  route overlays  │  └────────────────┘  └──────────────────────────┘ │\n│  └──────────────────┘                                                   │\n│  ┌──────────────────┐  ┌──────────────────────────────────────────────┐ │\n│  │  CameraAnalysis  │  │  AgentCommentaryPanel                        │ │\n│  │  layered SVG     │  │  streaming commentary, decision badge,       │ │\n│  │  reticle, heatmap│  │  confidence gauge, decision history          │ │\n│  │  flow vectors    │  └──────────────────────────────────────────────┘ │\n│  └──────────────────┘                                                   │\n│  ┌─────────────────────────────────────────────────────────────────────┐│\n│  │  TerminalPanel   [SYS] [PHASE] [→WS] [←WS] [DET] [TSP] [NAV] [AI]   ││\n│  │  real-time color-coded flight log — AI tab shows LangChain events   ││\n│  └─────────────────────────────────────────────────────────────────────┘│\n│                                                                         │\n│  ┌──────────────────────────────────────────────────────────────────┐   │\n│  │  liveInferenceEngine  (requestAnimationFrame loop, 30 fps)       │   │\n│  │   ├─ AutonomousNavigator  ←── applyAgentDecision()               │   │\n│  │   │   13-phase FSM, lawnmower scan, TSP, proximity detection     │   │\n│  │   ├─ WsClient            WebSocket → :8765  (inference)          │   │\n│  │   └─ AgentClient         HTTP/SSE/WS → :8766  (agent)            │   │\n│  │       ├─ POST /decide    (debounced 200ms, every 30 frames)      │   │\n│  │       ├─ GET  /stream    (SSE commentary on phase transitions)   │   │\n│  │       ├─ WS  /terminal   (LangChain callback drain, 100ms tick)  │   │\n│  │       └─ POST /mission/save (fire-and-forget on nav.done)        │   │\n│  └──────────────────────────────────────────────────────────────────┘   │\n└──────────────┬──────────────────────────────┬───────────────────────────┘\n               │                              │\n     :8766 HTTP/SSE/WS                  :8765 WebSocket\n               │                              │\n┌──────────────▼────────────┐  ┌──────────────▼─────────────────────────┐\n│  Agent Server             │  │  Inference Server                      │\n│  agent_server.py          │  │  inference_server.py                   │\n│  FastAPI + LangChain      │  │  FastAPI WebSocket                     │\n│                           │  │                                        │\n│  POST /decide             │  │  WS /inference                         │\n│    ChatAnthropic          │  │    ① receive drone + flowers JSON      │\n│    .bind_tools()          │  │    ② scene_renderer.py → 640×640 frame │\n│    RAG context injected   │  │    ③ DetectionBridge.detect()          │\n│    max 3 tool rounds      │  │       Coral TPU → ONNX → mock           |\n│    callbacks attached     │  │    ④ _compute_tsp_suggestion()         │\n│                           │  │    ⑤ _phase_suggestion()               │\n│  LangChain Tools:         │  │    ⑥ return detections + TSP + phase   │\n│  - compute_tsp_route      │  └─────────────────────────────────────────┘\n│  - estimate_battery_range │\n│  - recommend_conf_thresh  │  ┌────────────────────────────────────────┐\n│  - plan_scan_pattern      │  │  Chroma RAG Store                      │\n│                           │  │  drone-cv-system/mission_history/      │\n│  WS /terminal             │  │                                        │\n│    DroneTerminalCallback  │  │  Embeddings: all-MiniLM-L6-v2          │\n│    drain queue 100ms      │  │  local CPU — no API key needed         │\n│    → frontend AI tab      │  │                                        │\n│                           │  │  Saved after each mission completes    │\n│  GET /stream  (SSE)       │  │  Retrieved before each /decide call    │\n│  POST /mission/save       │  │  Top-3 similar missions injected into  │\n│  POST /feedback (bandit)  │  │  Claude system prompt                  │\n│  GET /health /metrics     │  └────────────────────────────────────────┘\n│  UCB1 ConfidenceBandit    │\n└───────────────────────────┘\n```\n\n---\n\n## How the Two Modes Work\n\nThe app boots to a mode selector. There are two execution paths that share all the same UI panels and TypeScript types but drive them from completely different engines.\n\n### Replay Mode\n\nReplay mode runs a deterministic pre-generated mission. On first render, `getMissionFrames()` is called once and caches all **2700 frames** (30 fps × 90 seconds) in a `useRef`. A `requestAnimationFrame` loop in `useReplayEngine` accumulates wall-clock delta time, multiplies by the speed setting (1×, 2×, 4×), and advances a frame index. The loop reads `frames.current[frameIndex]` and sets it as the current `ReplayFrame` in React state. All panels receive this frame as a prop — they are pure rendering functions with no simulation logic of their own.\n\nThe frame index is the single source of truth. Position history is a sliding window of the last 90 drone XY coordinates read directly from the pre-generated frames. Altitude history samples every 5th frame. Events accumulated from `frame.events` arrays are kept as a rolling window of 100.\n\nSeeking works by jumping `frameIndex` directly and rebuilding the history windows by slicing `frames.current[0..targetIndex]`. This is O(n) but runs fast because it's just array iteration over pre-computed data.\n\n### Live Mode\n\nLive mode runs a real autonomous mission. `useLiveInferenceEngine` creates:\n\n- An `AutonomousNavigator` instance with a randomly seeded garden\n- A `WsClient` connected to the inference server at `ws://localhost:8765/inference`\n- An `AgentClient` connected to the agent server at `http://localhost:8766`\n\nThe same `requestAnimationFrame` loop ticks the navigator with `dt` (delta seconds, capped at 100ms to prevent spiral-of-death on tab focus). Each tick:\n\n1. `nav.tick(dt, latestInference)` — advance the state machine, detect flowers, move the drone, return a `LiveFrame`\n2. `ws.send(lf.drone, lf.flowers, lf.phase)` — push state to inference server\n3. Every 30 frames: `agent.requestDecision(lf)` — fire debounced POST to `/decide`\n4. On phase change: `agent.startCommentaryStream(lf)` — open SSE to `/stream`\n5. When `nav.done`: `agent.saveMission(lf)` — embed mission into Chroma\n\nThe `LiveFrame` is adapted to a `ReplayFrame` shape via `liveToReplay()` in `App.tsx` so all four existing panels render without modification. The agent state (`AgentState`) is threaded separately via additional props to panels that need it.\n\n---\n\n## Frontend — Source Layout\n\n```\n╔═══════════════════════════════════════════════════════════════════════════╗\n║                        BROWSER  (Web Simulator)                           ║\n║                                                                           ║\n║   MODE 1 — Replay                    MODE 2 — Live Inference              ║\n║   ─────────────────                  ─────────────────────────            ║\n║   missionGenerator.ts                randomMissionGenerator.ts            ║\n║   (2700 ReplayFrames,                (random garden, 6–10 flowers,        ║\n║    seeded PRNG, 30 fps)               lawnmower scan, seeded PRNG)        ║\n║         ↓                                      ↓                          ║\n║   useReplayEngine                    useLiveInferenceEngine               ║\n║   (RAF loop, seek,                   (RAF loop, AutonomousNavigator,      ║\n║    speed multiplier)                  WsClient, terminal buffer)          ║\n║         ↓                                      ↓                          ║\n║   ┌─────────────────── liveToReplay() adapter ─────────────────┐          ║\n║   │           4 SVG Panels (shared between both modes)          │         ║\n║   │   TopDownView · SideView · TelemetryPanel · CameraAnalysis  │         ║\n║   └──────────────────────────────────────────────────────────── ┘         ║\n║                               TerminalPanel (Mode 2 only)                 ║\n╚═══════════════════════════════════════════════════════════════════════════╝\n                      ↕  WebSocket  ws://localhost:8765/inference\n╔═══════════════════════════════════════════════════════════════════════════╗\n║               Python Inference Server  (localhost / same machine)         ║\n║                                                                           ║\n║   FastAPI + uvicorn WebSocket endpoint                                    ║\n║         ↓                                                                 ║\n║   scene_renderer.py  →  PIL 640×640 photorealistic frame                  ║\n║         ↓                                                                 ║\n║   DetectionBridge                                                         ║\n║     ├─ CoralBridge   →  Google Coral USB TPU (_edgetpu.tflite)            ║\n║     ├─ OnnxDetector  →  YOLOv8n ONNX CPU  →  bbox parse  →  geo-match     ║\n║     └─ MockDetector  →  physics confidence (altitude + distance)          ║\n║         ↓                                                                 ║\n║   Planning Agent (_compute_tsp_suggestion)  →  TSP route                  ║\n║         ↓                                                                 ║\n║   JSON response  {detections, phaseSuggestion, tspSuggestion, framePng}   ║\n╚═══════════════════════════════════════════════════════════════════════════╝\n\n╔═══════════════════════════════════════════════════════════════════════════╗\n║                 Real Drone CV System  (Raspberry Pi 4)                    ║\n║                                                                           ║\n║   RPi Camera v2 (CSI)                                                     ║\n║         ↓                                                                 ║\n║   FramePreprocessor  →  640×640 RGB normalized tensor                     ║\n║         ↓                                                                 ║\n║   FlowerDetector (YOLOv8n ONNX, ONNX Runtime)                             ║\n║         ↓                                                                 ║\n║   OpticalFlowTracker (Lucas-Kanade, EMA smoothing)                        ║\n║         ↓                                                                 ║\n║   DepthEstimator  (bbox size + altitude fusion)                           ║\n║         ↓                                                                 ║\n║   StateMachine (13 phases, 20 Hz tick loop)                               ║\n║         ↓                                                                 ║\n║   FlightController  ──→  MAVLinkInterface  ──→  Pixhawk 2.4.8             ║\n║         ↓                        ↑                                        ║\n║   PollinationManager          Telemetry stream (ATTITUDE, GPS,            ║\n║   (MAVLink AUX OUT 1          OPTICAL_FLOW_RAD, EKF_STATUS, …)            ║\n║    → pollen-dispenser servo)                                              ║\n╚═══════════════════════════════════════════════════════════════════════════╝\n```\n\n---\n\n## Data Layer\n\n### Replay Mission Generator (`missionGenerator.ts`)\n\nThe replay mission is entirely pre-computed. `getMissionFrames()` runs once (lazily, then cached) and returns an array of 2700 `ReplayFrame` objects covering 90 seconds at 30 fps.\n\n**Garden layout.** 10 flower clusters at fixed positions within a 20m × 20m space. Clusters are identified `f1`–`f10`. The drone home base is at `(2, 2)`. 8 of the 10 clusters are visited during the mission (f8 and f10 are unvisited bystanders, visible but never targeted).\n\n**Timeline segments.** The 90-second mission is divided into named segments:\n\n```\n0–2s      idle       — drone stationary, systems initializing\n2–4s      arming     — pre-flight sequence\n4–8s      takeoff    — climb from 0m to 8m patrol altitude\n8–14s     transit    — fly from home to first waypoint\n14–22s    scanning   — hover scan of cluster\n22–26s    candidate  — confidence building (0.2→0.6)\n26–30s    target_lock— high confidence achieved (0.6→0.92)\n30–34s    descent    — descend 8m → 1.5m\n34–37s    hover_align— XY precision alignment\n37–41s    pollinating— pollination mechanism active\n41–45s    ascent     — climb back to 8m\n45–90s    (repeat for remaining 7 clusters with adjusted timing)\n```\n\n**Per-frame computation.** For each frame index `i`:\n1. Determine which timeline segment it falls in\n2. Compute `t = (i - segmentStart) / segmentDuration` (normalized 0→1)\n3. Lerp drone X/Y between segment start and end waypoints\n4. Add a sinusoidal wobble (`sin(t * 7.3 * 2π) × 0.03m`) for realism\n5. Compute altitude as a piecewise function of phase\n6. Look up sensor state via `getSensorAtDistance(drone.z * 39.37)`\n7. Compute optical flow state via `computeOpticalFlowState(sample, i)`\n8. Compute detection confidence as a phase-specific ramp × sensor coupling\n9. Update flower states (unscanned → discovered → candidate → locked → pollinated)\n10. Emit events at phase transitions\n\n**Seeded RNG.** The `seededRandom(42)` PRNG (multiplicative linear congruential: `s = s × 16807 mod 2147483647`) ensures every run produces identical frames. This makes regression testing against visual output reliable.\n\n### Random Garden Generator (`randomMissionGenerator.ts`)\n\nUsed by live mode to generate a fresh garden for each session.\n\n**`generateRandomGarden(seed)`** places 6–10 flowers using a different LCG (`s = s × 1664525 + 1013904223`). Each flower is rejected if it falls within 3m of the home base at `(2,2)` or within 2.8m of another flower. Positions are rounded to 0.1m. Returns `LiveFlower[]` — all start as `'undiscovered'`.\n\n**`generateLawnmowerPath(spacing = 4.5)`** generates a boustrophedon (alternating S→N / N→S) set of waypoints spaced `spacing` meters apart from `x=3.0` to `x=18.0`. At default 4.5m spacing with a 4.5m proximity detection radius, every point in the flower zone `[2.5m, 17.5m]` is covered. The spacing parameter is overridden by agent decisions when the scan pattern tool recommends tighter or wider passes.\n\n**`computeTSPRoute(flowers, discoveredIds)`** runs a greedy nearest-neighbor heuristic starting from home `(2,2)`. At each step it picks the unvisited discovered flower closest to the current position. O(n²) but n ≤ 10 so this is negligible. Returns an ordered array of flower IDs.\n\n### Optical Flow Dataset (`opticalFlowDataset.ts`)\n\nReal sensor data from `raw_opticalflow_data.csv` is merged with synthetically generated midpoint rows to give 6-inch step resolution from 0 to 315 inches. The merge strategy:\n\n1. Load the 24 real rows (0–276 inches at 12-inch intervals)\n2. Generate synthetic midpoint rows between each consecutive real pair\n3. Generate 3 extrapolated rows from 276 to 315 inches (patrol altitude coverage)\n4. Merge: real rows always override synthetic rows at matching distances\n5. Sort the combined dataset by `distance_in`\n\nThe resulting dataset has ~50 rows and is cached in a module-level constant.\n\n---\n\n## Simulation Engine\n\n### Replay Engine (`replayEngine.ts`)\n\n`useReplayEngine()` is a Preact hook that wraps a `requestAnimationFrame` loop.\n\n**State.** All mutable loop state lives in refs (not React state) to avoid triggering re-renders on every frame tick:\n- `frameIndexRef` — current index into the frames array\n- `isPlayingRef` — controls whether the RAF loop continues\n- `speedRef` — 1, 2, or 4 (multiplied into accumulated delta time)\n- `accumulatedTimeRef` — fractional frame accumulator (handles non-integer fps)\n- `lastTimeRef` — previous `requestAnimationFrame` timestamp\n\nReact state is only set when visible output changes: `currentFrameData`, `positionHistory`, `altitudeHistory`, `accumulatedEvents`.\n\n**Tick loop.** Each RAF callback:\n1. Computes `delta = timestamp - lastTimestamp` in ms\n2. Adds `(delta / 1000) × speed` to `accumulatedTime`\n3. Advances `frameIndex` by `floor(accumulatedTime × 30)` frames\n4. Subtracts the integer portion back out of `accumulatedTime`\n5. Updates histories and emits events if the new frame has any\n\n**Seek.** `seekTo(time)` converts time to a frame index and sets it directly. Histories are rebuilt by iterating `frames[0..targetIndex]` and re-applying the same update logic.\n\n### Live Inference Engine (`liveInferenceEngine.ts`)\n\n`useLiveInferenceEngine()` owns all live simulation state. It creates and manages three worker objects (`AutonomousNavigator`, `WsClient`, `AgentClient`) and runs a RAF loop that drives them together.\n\n**Terminal buffer.** To avoid O(n) React re-renders on every terminal push, entries are accumulated in a `useRef` buffer (`termBufRef`) and synced to React state every 250ms via a `setInterval`. The terminal panel therefore has at most 4 state updates per second regardless of how many events fire.\n\n**Phase transition detection.** `lastPhaseRef` stores the previous frame's phase. When `lf.phase !== lastPhaseRef.current`, a new SSE commentary stream is started by calling `agent.startCommentaryStream(lf)`.\n\n**Agent decision loop.** `frameIdxRef` increments each RAF tick. When `frameIdxRef % 30 === 0` (once per second at 30fps), `agent.requestDecision(lf)` fires. The decision is debounced inside `AgentClient` to 200ms so rapid phase transitions don't flood the server.\n\n**Mission completion.** When `nav.done` is true, the RAF loop stops and `agent.saveMission(lf)` fires a background POST to embed the completed mission into Chroma.\n\n### Autonomous Navigator (`autonomousNavigator.ts`)\n\n`AutonomousNavigator` is a class (not a hook) that implements the 13-phase live mission as a state machine. It is instantiated once per live session.\n\n**State.** All drone physics state, flower states, route planning state, and history windows live inside the class instance. The `tick(dt, inference)` method is the single entry point called each RAF frame.\n\n**Phases and transitions:**\n\n```\nidle\n  └─ immediately → arming (armTimer starts)\n\narming (2s dwell)\n  └─ → takeoff\n\ntakeoff\n  └─ climb at 1.8 m/s until z ≥ 7.9m → scanning\n\nscanning\n  └─ moveToward(lawnmower[scanWpIdx], 2.5 m/s)\n     doProximityDetection() each frame\n     on waypoint reached: scanWpIdx++\n     when all lawnmower WPs done: scanComplete=true → planning\n\nplanning (2.5s dwell)\n  └─ computeTSPRoute(flowers, discoveredIds)\n     fallback: if zero CV detections, add all flowers\n     → approach (or mission_complete if route empty)\n\napproach\n  └─ moveToward(currentTarget, 2.0 m/s) at PATROL_ALT\n     doProximityDetection() continues during approach\n     on arrival (\u003c 0.4m): → descent\n\ndescent\n  └─ lower z at 1.8 m/s to 1.5m\n     slow XY drift toward target (0.5 m/s)\n     when z ≤ 1.65m: → hover_align\n\nhover_align\n  └─ moveToward(target, 0.3 m/s) at 1.5m\n     track XY error; when \u003c 0.18m for 0.5s: → pollinating\n\npollinating (3s dwell)\n  └─ pollinateTimer counts\n     on complete: push to pollinatedIds, flower.state = 'pollinated'\n     send feedback POST to agent → bandit update\n     → ascent\n\nascent\n  └─ climb at 1.8 m/s to 8m\n     when z ≥ 7.9m: → resume\n\nresume\n  └─ tspIdx++\n     if more targets remain: → approach\n     else: → mission_complete\n\nmission_complete\n  └─ fly back toward home (2, 2) at 2.0 m/s\n     on arrival: → landing\n\nlanding\n  └─ descend at 1.1 m/s\n     when z ≤ 0.01m: nav.done = true, RAF loop stops\n```\n\n**Proximity detection.** Each frame during `scanning` and `approach`, every undiscovered flower is checked: if `hypot(drone.x - flower.x, drone.y - flower.y) \u003c 4.5m`, the flower transitions to `'discovered'` and is added to `discoveredIds`. Confidence at detection = `0.9 - (dist / 4.5) × 0.6` (0.9 at zero offset, 0.3 at edge of radius). Whenever a new discovery is made, `computeTSPRoute` is re-run immediately.\n\n**Inference integration.** If the WebSocket server returns an `InferenceResult`, `processInference(inf)` runs: for each detection, the confidence is updated on the matching flower, flower state is promoted (`scanned → candidate → locked`) based on `currentConfidenceThreshold` (dynamically set by the agent), and any server-suggested TSP IDs not yet in `discoveredIds` are added.\n\n**Agent integration.** `applyAgentDecision(decision)`:\n- If `decision.priorityOverride` is non-empty and `!planningComplete`, rebuilds `tspRoute` with the agent-suggested order (filtering to only valid undiscovered/unpollinated IDs)\n- If `decision.altitudeOverride` is set, logs it to the terminal\n- `currentConfidenceThreshold` is updated from `decision.confidenceThreshold` on every decision\n- `scanSpacing` is updated from `decision.scanSpacing` (triggers lawnmower regeneration on next scan pass)\n\n### Agent Client (`agentClient.ts`)\n\n`AgentClient` manages all communication with the agent server. All methods fail silently so the simulation continues normally when the server is offline.\n\n**Health polling.** On `connect()`, a `setInterval` checks `GET /health` every 3 seconds. Status transitions: `disconnected → connecting → connected` (on 200 OK) or `error` (on non-200). When status becomes `'connected'`, `openTerminalWs()` is called automatically if `connectTerminalStream()` has already been called.\n\n**Decision channel.** `requestDecision(frame)` sets a 200ms `setTimeout`. If a new call arrives before it fires, the timer is reset (debounce). On fire, a `POST /decide` is sent with the drone state, flowers, phase, sensor, pollinated/discovered IDs, and battery. Response is an `AgentDecision` JSON.\n\n**Commentary channel.** `startCommentaryStream(frame)` aborts any existing SSE stream and opens a new `GET /stream` with query params extracted from the current frame. The response body is read as a `ReadableStream`. Each `data: {...}` SSE line is parsed. Text delta chunks are accumulated and the `onCommentary` callback fires on each chunk (with `streaming: true`) and on completion (with `streaming: false`). Incomplete lines are buffered across chunks.\n\n**Terminal WebSocket channel.** `connectTerminalStream(onEvent)` stores the callback and calls `openTerminalWs()`. The WS connects to `ws://localhost:8766/terminal`. Each incoming `{events: [{type, text}]}` message routes each event to `onTerminalEvent(type, text)`, which calls `pushTerminal()` in `liveInferenceEngine`. This is how LangChain callback events appear as emerald **AI** entries in the terminal. If the WS closes, it reconnects after 3 seconds while agent status is `'connected'`.\n\n**Mission save channel.** `saveMission(frame)` POSTs `{events, telemetry}` to `/mission/save`. Fire-and-forget with 5s timeout.\n\n**Feedback channel.** `sendFeedback(success, state)` POSTs to `/feedback`. Called by `AutonomousNavigator` after each successful pollination to update the UCB1 bandit.\n\n---\n\n## Sensor Models\n\n### Sensor Interpolation (`sensorInterpolation.ts`)\n\n`getSensorAtDistance(distanceInches)` looks up the optical flow sensor state at the current drone altitude. The altitude in meters is converted to inches (`z × 39.37`) before calling this function.\n\n**Algorithm:**\n1. Clamp input to `[0, 315]` inches (no extrapolation)\n2. Binary search the dataset to find the bracketing pair `(lower, upper)` such that `lower.distance_in ≤ input ≤ upper.distance_in`\n3. Compute normalized parameter `t = (input - lower.distance_in) / (upper.distance_in - lower.distance_in)`\n4. Apply smooth-step easing: `st = t² × (3 − 2t)` — this prevents sharp transitions at dataset boundaries\n5. Lerp all scalar fields: `sensor_distance`, `strength`, `precision`, `flow_vel_x`, `flow_vel_y`, `flow_quality`\n6. Return an `OpticalFlowSample` with the interpolated values\n\nThe smooth-step easing means sensor readings don't jump abruptly between dataset rows — they ease in and out of transitions, mimicking the continuous nature of physical sensor response.\n\n### Optical Flow Physics Model (`opticalFlowModel.ts`)\n\n`computeOpticalFlowState(sample, frameIndex)` takes an interpolated sensor sample and applies physics-based processing to produce the full `OpticalFlowState`.\n\n**Base physics:**\n```\nvx = sample.flow_vel_x × (distance_in / 1000)\nvy = sample.flow_vel_y × (distance_in / 1000)\n```\nOptical flow apparent motion scales linearly with altitude — a flower appears to move faster in the image at higher altitude for the same ground speed. The `/1000` factor normalizes the sensor's raw velocity values.\n\n**Stability and quality derivation:**\n```\nstability         = min(1, flow_quality / 150)        // 150 is peak quality from real data at ~3m\nnoise             = (1 − stability) × 0.15\nnormalizedStrength = strength / 255\nprecisionWeight   = 1 / max(1, precision)              // lower precision number = better\neffectiveQuality  = flow_quality × normalizedStrength × precisionWeight\n```\n\n**Degradation above 5m (197 inches):**\n```\nexcess            = (distance_in − 197) / 118         // 0→1 over the 197–315in range\ndegradedStability = stability × (1 − excess × 0.60)\ndegradedQuality   = effectiveQuality × (1 − excess × 0.70)\n```\nAt patrol altitude (8m ≈ 315 inches), the sensor is at 40% of nominal stability and 30% of nominal quality, modeling real sensor degradation from range.\n\n**Low-strength noise amplification:**  \nWhen `strength \u003c 60`, noise increases: `finalNoise += (1 − strength/60) × 0.25`. Weak return signal produces noisy measurements.\n\n**Deterministic drift (quality \u003c 50):**  \nWhen the flow quality falls below 50, the sensor is unreliable. Drift is injected using a deterministic pseudo-random function seeded by `frameIndex`:\n```\ndriftX = (pseudoRand(frameIndex × 0.03)       − 0.5) × 0.4\ndriftY = (pseudoRand(frameIndex × 0.03 + 100) − 0.5) × 0.4\n```\n`pseudoRand(seed) = frac(sin(seed × 127.1 + 311.7) × 43758.5453)` — a hash-like function that produces stable values without a PRNG object. The drift is reproducible per frame index, so seeking in replay mode gives identical results.\n\n**Hover instability (altitude \u003c 3m / 118 inches):**  \n```\nhoverX = sin(t × 4.3) × 0.05    where t = frameIndex / 30\nhoverY = cos(t × 3.7) × 0.05\n```\nLow-altitude hover introduces sinusoidal ground-effect oscillation in the velocity readings. The 4.3 and 3.7 frequencies create a slightly irregular wobble rather than a perfect circle.\n\n**CV coupling.** Detection confidence is modulated each frame by optical flow quality:\n```\nstabilityFactor = 0.6 + 0.4 × stability          // never below 60%\nstrengthFactor  = 0.6 + 0.4 × (strength / 255)   // never below 60%\nconfidence     *= stabilityFactor × strengthFactor\n```\nIf `|vx| \u003e 1.5 or |vy| \u003e 1.5 m/s`: blur penalty reduces confidence. If `flow_quality \u003c 50`: heavy reduction `× 0.6`. If `stability \u003e 0.7 and altitude \u003c 3m` (stable hover): 15% boost.\n\n---\n\n## UI Panels\n\n### Top-Down Garden View\n\nRenders the full 20m × 20m garden as a scaled SVG. The coordinate transform maps meters → pixels using a fixed scale factor. Features:\n\n- **Flower clusters** — each cluster renders 4–7 individual flower SVGs using `FlowerClusterRenderer`. Petal count, angle, and position are seeded from the cluster ID so they look different but are always identical across renders.\n- **State coloring** — `unscanned` (dark, low opacity), `discovered` (amber ring), `candidate` (orange ring + glow), `locked` (cyan ring + bright), `pollinated` (green ring + full opacity)\n- **Drone body** — a hexagonal body with 4 arm stubs. Each arm tip has a small circle representing a rotor with a CSS `spin` animation. The body rotates to match `drone.yaw`.\n- **Motion trail** — a polyline through the last 90 position history points. Opacity fades toward the oldest point.\n- **TSP route** — dashed lines connecting the current planned visit order in the drone's current accent color.\n- **AI route overlay** — when `agent.lastDecision.priorityOverride` is non-empty, a second dashed route is drawn in purple `#a78bfa` with numbered stop labels.\n- **Home base** — small marker at `(2, 2)` with \"HOME\" label.\n\n### Side View (Altitude Profile)\n\nSVG cross-section showing altitude over time. The X axis spans the altitude history (up to 150 samples, sampled every 5 frames ≈ every 0.83 seconds). The Y axis spans 0–10m. A polyline connects the altitude history points. A small drone silhouette marker sits at the rightmost (current) position. Phase-colored horizontal bands show the patrol altitude (8m) and hover altitude (1.5m).\n\n### Telemetry Panel\n\nA grid of labeled metric rows organized into sections. All values read directly from the current `ReplayFrame.sensor` and `ReplayFrame.drone` objects — no computation happens here.\n\nSections: Navigation (x, y, z, speed, yaw, yawRate), Optical Flow (all extended fields from `ofStrength`, `ofPrecision`, `ofStability`, `ofNoise`, `ofEffectiveQuality`, `distanceInches`, `sensorDistanceMm`), Rangefinder (rangefinderDistance, sonarEstimate), EKF Confidence bar, Battery bar, Signal Strength bar, CV Detection (confidence bar, current target ID, flowers in view, target locked status), Mission State (phase chip, pollinated/total, elapsed).\n\nIn live mode, an additional AI AGENT section shows: agent connection dot (color-coded), total decisions made, number of route overrides applied, current dynamic confidence threshold with a small bar, and the last decision action + truncated reasoning text.\n\n### Camera / Flower Analysis Panel\n\nThe most visually complex panel. `CameraAnalysisPanel` first runs `computeAnalysisFrame(frame)` to transform the `ReplayFrame` into an `AnalysisFrame` — this is the only panel with a data transformation step.\n\n**`computeAnalysisFrame` pipeline:**\n1. Maps each `FlowerCluster` to a `FlowerRenderState` with fixed camera-space positions from the `CAMERA_POSITIONS` lookup (800×500 viewbox coordinate system)\n2. Determines if the current target flower should zoom to scene center `(400, 230)` at scale 2.0 during `target_lock`, `descent`, `hover_align`, `pollinating`, `ascent`\n3. Computes `FrustumState.tightness` from the phase tightness map (0 = wide, 1.0 = maximally tight, reached during `pollinating`)\n4. Passes the resulting `AnalysisFrame` to `CameraAnalysisScene`\n\n**Scene layers (bottom to top):**\n1. Dark background with radial vignette\n2. `FlowerClusterRenderer` — one group per visible flower, with jitter applied (`jx/jy` random offsets) when `of_quality \u003c 50`, and SVG `feGaussianBlur` motion blur when `|velocity| \u003e 1.0 m/s`\n3. `DetectionHeatmap` — per-flower radial gradient blobs, opacity driven by `confidence × qualityIntensity`\n4. `PollinationEffect` — only active during `pollinating`: 8 orbiting sparkle particles + 3 concentric pulse rings\n5. `FlowVectorOverlay` — a 5×4 grid of small flow lines plus one large velocity arrow at scene center. Color and animation class depend on stability tier.\n6. `DetectionReticle` — corner brackets and crosshair, tightness controlled by phase\n7. `MissionPhaseOverlay` — phase-specific banner text\n8. `AnalysisHud` — bottom strip with phase chip, confidence sparkline, lock indicator\n9. `OpticalFlowHud` — top-right mini panel with sensor readings\n\nIf a `livePng` base64 JPEG is provided (from the inference server), it is rendered as an `\u003cimage\u003e` tag under all SVG layers, showing the actual synthetic camera frame that was fed to the ML detector.\n\n### Terminal Panel\n\nA fixed-bottom overlay showing the rolling flight computer log. Entries are color-coded:\n\n| Type | Color | Content |\n|---|---|---|\n| `sys` | Slate | Session/connection events |\n| `phase` | Purple | State machine transitions |\n| `ws-out` | Blue | WebSocket frames sent |\n| `ws-in` | Cyan | WebSocket frames received |\n| `detect` | Green | CV flower detections |\n| `tsp` | Amber | Route planning updates |\n| `nav` | Gray | Proximity detection, navigation |\n| `error` | Red | Connection/inference errors |\n| `agent` | Emerald | LangChain LLM thoughts, tool calls, RAG hits |\n\nThe AI filter tab isolates the emerald `agent` entries, showing only what the LLM and its tools are doing. Each entry shows a timestamp (seconds since session start), type label, and truncated text.\n\n---\n\n## Backend — Inference Server\n\n### `inference_server.py` (port 8765)\n\nFastAPI app with a single WebSocket endpoint `/inference`. The server is started automatically by the Vite dev server via `/api/start-inference-server` when live mode is selected.\n\n**WebSocket protocol:**\n\nClient → Server (every ~100ms, sent from `WsClient.send()`):\n```json\n{\n  \"drone\":   { \"x\": 12.3, \"y\": 8.4, \"z\": 8.0, \"yaw\": 45.2 },\n  \"flowers\": [{ \"id\": \"r1\", \"x\": 5.5, \"y\": 9.2, \"radius\": 0.8, ... }],\n  \"phase\":   \"scanning\"\n}\n```\n\nServer → Client:\n```json\n{\n  \"detections\":      [{ \"id\": \"r1\", \"confidence\": 0.73, \"cls\": \"flower_open\", \"bbox\": [210,190,430,410] }],\n  \"phaseSuggestion\": \"approach\",\n  \"targetId\":        \"r1\",\n  \"inferenceMs\":     12.4,\n  \"inferenceMode\":   \"mock\",\n  \"framePng\":        \"\u003cbase64 JPEG\u003e\",\n  \"tspSuggestion\":   [\"r1\", \"r3\", \"r2\"]\n}\n```\n\n**Processing pipeline per message:**\n1. Parse JSON, extract `drone`, `flowers`, `phase`\n2. `render_frame(drone, flowers)` — run in a thread executor to avoid blocking the event loop\n3. `bridge.detect(frame_arr, drone, flowers)` — run in executor, returns `(detections, mode, elapsed_ms)`\n4. Optionally encode frame as base64 JPEG via `frame_to_base64()`\n5. `_phase_suggestion(detections, phase)` — simple confidence threshold logic\n6. `_compute_tsp_suggestion(detections, flowers, drone_x, drone_y)` — greedy NN TSP\n7. Assemble response dict and `send_json()`\n\n### Scene Renderer (`scene_renderer.py`)\n\nGenerates a 640×640 top-down synthetic camera frame using PIL (Pillow). Each flower is projected from garden coordinates into pixel space using the pinhole camera model:\n\n```python\nrel_x = flower.x - drone.x\nrel_y = flower.y - drone.y\n\n# Rotate by drone yaw\ncam_x = rel_x * cos(yaw_rad) + rel_y * sin(yaw_rad)\ncam_y = -rel_x * sin(yaw_rad) + rel_y * cos(yaw_rad)\n\n# Project (90° FOV → focal length = IMG_SIZE / 2 = 320)\nu = 320 * cam_x / altitude + 320\nv = 320 * cam_y / altitude + 320\nradius_px = max(3, int(flower.radius / altitude * 320))\n```\n\nFlowers outside the image bounds are skipped. For each visible flower, `_draw_flower()` renders:\n- A green ellipse shadow offset by 3px\n- 6 petals drawn as ellipses rotated around the center, angles seeded by `hashlib.md5(f'{seed}-{n}')` for deterministic per-flower variation\n- A darkened center circle (the nectary)\n- A curved stem and two leaf ellipses below the bloom\n- A Gaussian blur pass via PIL for anti-aliasing\n\nThe resulting NumPy float32 `[640, 640, 3]` array is returned for inference and optionally JPEG-encoded to base64.\n\n### Detection Bridge (`detection_bridge.py`)\n\nThree-tier detection hierarchy:\n\n**Tier 1: `CoralBridge` (Google Coral USB TPU)**  \nWraps `cv/coral_detector.py` → `pycoral.utils.edgetpu.make_interpreter()`. Loads an EdgeTPU-compiled `_edgetpu.tflite` INT8 model. The input frame (float32 [0,1]) is converted to uint8 for the TPU. After inference, detected bounding boxes are scaled from the model's input resolution back to 640×640 and matched to projected garden flowers using nearest-centroid matching (box center within `radius × 2.5` pixels of projected flower center). Target latency: ~5ms.\n\n**Tier 2: `OnnxDetector` (ONNX Runtime CPU)**  \nLoads `flower_detector.onnx` via `onnxruntime.InferenceSession` with 2 threads. The YOLOv8n output format is `[1, 84, 8400]`: first 4 values are `cx, cy, w, h` in pixels, next 80 are per-class scores. The model was trained with 3 classes: `flower_open`, `flower_closed`, `flower_cluster`. After confidence filtering (`\u003e 0.20`) and NMS, boxes are matched to projected garden flowers the same way as Coral. Target latency: ~30ms.\n\n**Tier 3: `MockDetector` (physics-based)**  \nNo ML required. For each flower, projects its position into camera space, computes horizontal distance from drone, and scores confidence as:\n```python\nbase = max(0, 1.0 - hdist / (alt * 1.8))\nconf = base * (0.6 + 0.4 * stability) * (0.6 + 0.4 * strength / 255)\n```\nFlowers with confidence \u003c 0.12 are excluded. Results sorted by confidence descending. Always available.\n\n**Fallback behavior.** If Coral inference fails (hardware disconnected, timeout \u003e 2s), the bridge nulls out the Coral instance and falls to ONNX. If ONNX fails, it nulls out ONNX and falls to mock. These transitions are permanent for the session — no retry.\n\n---\n\n## Backend — Agent Server\n\n### `agent_server.py` (port 8766)\n\nFastAPI app with five endpoint types serving the LangChain planning agent.\n\n### `POST /decide` — LangChain Planning Decision\n\nThe main planning endpoint. `_llm_decide(state)` runs:\n\n**Step 1 — RAG retrieval:**\nA semantic query string is assembled from the current state:\n```\n\"phase=scanning battery=87 stability=0.72 discovered=3 pollinated=1\"\n```\n`MissionStore.retrieve_context(query, k=3)` embeds this query and returns the 3 most similar past missions as formatted text. If the store is empty or unavailable, this returns an empty string and the step is skipped.\n\n**Step 2 — System prompt construction:**\n```\n\"You are an autonomous drone mission planner AI for a pollinator drone.\nYour goal is to maximise flower pollination efficiency while managing\nbattery life and sensor conditions. You have access to planning tools.\nBe concise and decisive.\n\n[RAG context appended here if available]\"\n```\n\n**Step 3 — User message construction:**\nThe current mission state is serialized into a concise natural-language message describing phase, position, altitude, battery, optical flow stability, discovered/pollinated counts, and remaining targets.\n\n**Step 4 — LangChain `ChatAnthropic.invoke()` with callbacks:**\n```python\nlc_model = ChatAnthropic(model=\"claude-haiku-4-5-20251001\", api_key=...).bind_tools(lc_tools)\nresponse = lc_model.invoke(messages, config={\"callbacks\": [_terminal_callback]})\n```\n`bind_tools()` converts the LangChain `StructuredTool` list into the Anthropic tools format and attaches them to every call. The callback fires `on_chat_model_start` as the call begins and `on_llm_end` when it returns.\n\n**Step 5 — Tool dispatch loop (max 3 rounds):**\n```python\nwhile tool_calls exist and round \u003c 3:\n    for each tool_call in response.tool_calls:\n        result = dispatch(tool_call.name, tool_call.args)\n        # Extract planning data from results\n        if name == \"compute_tsp_route\": priority_override = result[\"route\"]\n        if name == \"recommend_confidence_threshold\": conf_threshold = result[\"threshold\"]\n        if name == \"plan_scan_pattern\": scan_spacing = result[\"spacing\"]\n    append AIMessage + ToolMessages to conversation\n    invoke again\n```\nEach tool dispatch fires `on_tool_start` and `on_tool_end` callbacks.\n\n**Step 6 — Decision assembly:**\nThe final `AIMessage` content string is the reasoning. Action type is inferred from which tools were called and from keywords in the reasoning text. Returns:\n```json\n{\n  \"action\":              \"replan\",\n  \"reasoning\":           \"Battery at 72%, 4 flowers remain. TSP route optimized...\",\n  \"priorityOverride\":    [\"r2\", \"r4\", \"r1\", \"r3\"],\n  \"altitudeOverride\":    null,\n  \"confidenceThreshold\": 0.60,\n  \"scanSpacing\":         null,\n  \"decisionMs\":          340.2,\n  \"modelUsed\":           \"langchain/claude-haiku-4-5-20251001\"\n}\n```\n\n**Tool implementations** (called locally in Python, never sent to the LLM for execution):\n\n| Tool | Logic |\n|---|---|\n| `compute_tsp_route` | Greedy nearest-neighbor from drone position. If `prioritize_confidence=True`, scores each flower by `confidence / distance` and picks the best combined score at each step. Caps route length by estimated battery range. |\n| `estimate_battery_range` | `reachable = (battery − 20) / (8 + avg_dist × 0.5)`. The 20% floor is the minimum safe return battery. 8 = approx cost per flower visit in % units. |\n| `recommend_confidence_threshold` | Delegates to `ConfidenceBandit.select_threshold()` if available, else applies a heuristic: base 0.40 (scanning), 0.75 (approach), adjusted ±0.05 by stability, ±0.03 by battery, ±0.05 by remaining count. |\n| `plan_scan_pattern` | Spacing 3.5m if \u003e3 flowers already found, 5.5m if 0 found, 4.5m otherwise. Computes `passes = garden_size / spacing`. |\n\n**Graceful degradation.** If `ANTHROPIC_API_KEY` is unset or LangChain is not installed, `_mock_decision()` returns immediately with `action='continue'`, empty override, and fixed 0.75 threshold.\n\n### `WS /terminal` — LangChain Callback Stream\n\nDrains `_terminal_callback.drain()` (the global `DroneTerminalCallbackHandler` instance) every 100ms and sends batches:\n```json\n{ \"events\": [\n  { \"type\": \"agent\", \"text\": \"THINK  [claude-haiku-4-5-20251001]  Current mission state (T+34.2s)…\" },\n  { \"type\": \"tsp\",   \"text\": \"TOOL:compute_tsp_route  drone_x=12.3 drone_y=8.4 battery=72…\" },\n  { \"type\": \"ws-in\", \"text\": \"TOOL-RESULT  {\\\"route\\\": [\\\"r2\\\", \\\"r4\\\", \\\"r1\\\"], \\\"estimated_…\" },\n  { \"type\": \"agent\", \"text\": \"RESULT  With battery at 72%, prioritizing the two closest…\" }\n]}\n```\nMultiple frontend clients can connect simultaneously. The drain is non-blocking and returns an empty list if no events are queued.\n\n### `GET /stream` — SSE Commentary\n\nAn SSE endpoint that streams a 1–2 sentence mission narration for the current phase. Uses the raw `anthropic` SDK's `client.messages.stream()` context manager (not LangChain) since streaming and callbacks together require the raw SDK. Each text chunk is yielded as:\n```\ndata: {\"text\": \"The drone is descending\", \"done\": false}\ndata: {\"text\": \" to 1.5m hover altitude,\", \"done\": false}\ndata: {\"text\": \"\", \"done\": true}\n```\nThe system prompt instructs Claude to be \"technical but accessible\" and to focus on the most interesting aspect of the current moment. The user message includes a rich natural-language description of the phase, battery level interpretation, optical flow status, and mission progress.\n\n### `POST /mission/save` — RAG Store Write\n\nReceives `{events, telemetry}` and calls `MissionStore.save_mission()`. The telemetry includes `pollinatedIds`, `discoveredIds`, `battery_pct`, and `time`. Returns `{ok, total_missions}`.\n\n### `POST /feedback` — Bandit Reward Signal\n\nReceives `{phase, of_stability, battery_pct, success}` from the frontend (sent after each pollination). Calls `bandit.update_reward()` to adjust the UCB1 arm weights for the matching context bucket.\n\n---\n\n## LangChain Integration\n\n### `DroneTerminalCallbackHandler` (`drone_callback.py`)\n\nSubclasses `langchain_core.callbacks.BaseCallbackHandler`. A thread-safe `collections.deque(maxlen=300)` accumulates events. A `threading.Lock` protects both `_push()` and `drain()` since LangChain callbacks can fire from executor threads.\n\n**Hooks implemented:**\n\n```python\non_chat_model_start(serialized, messages, ...)\n  # Fires as the LLM call starts — extracts model name and last message preview\n  # → type=\"agent\", text=\"THINK  [claude-haiku-4-5-20251001]  {first 70 chars}…\"\n\non_llm_end(response, ...)\n  # Fires when LLM returns — extracts first generation text\n  # → type=\"agent\", text=\"RESULT  {first 150 chars}\"\n\non_tool_start(serialized, input_str, ...)\n  # Fires before each tool execution — maps tool name to terminal type\n  # compute_tsp_route/estimate_battery_range → type=\"tsp\"\n  # recommend_confidence_threshold/plan_scan_pattern → type=\"detect\"\n  # → text=\"TOOL:{name}  {first 80 chars of input}\"\n\non_tool_end(output, ...)\n  # Fires after tool returns — shows result\n  # → type=\"ws-in\", text=\"TOOL-RESULT  {first 100 chars}\"\n\non_agent_action(action, ...)     → type=\"tsp\",   text=\"AGENT-ACT  tool={name}  input={input}\"\non_agent_finish(finish, ...)     → type=\"agent\", text=\"AGENT-FIN  {output}\"\non_llm_error(error, ...)         → type=\"error\", text=\"LLM-ERROR  {message}\"\non_tool_error(error, ...)        → type=\"error\", text=\"TOOL-ERROR  {message}\"\n```\n\n**Fallback.** If `langchain_core` is not installed, `drone_callback.py` defines no-op stub classes for `BaseCallbackHandler` and `LLMResult`. The rest of the server imports `DroneTerminalCallbackHandler` and it works — it just does nothing. The `CALLBACK_AVAILABLE` flag in `agent_server.py` controls whether it's attached to model calls.\n\n---\n\n## RAG Mission Memory\n\n### `MissionStore` (`mission_store.py`)\n\n**Embeddings.** `HuggingFaceEmbeddings(model_name=\"sentence-transformers/all-MiniLM-L6-v2\")` from `langchain-community`. This model produces 384-dimensional vectors, runs entirely on CPU (no GPU required), and downloads ~90MB on first run. `normalize_embeddings=True` ensures cosine similarity is equivalent to dot-product similarity, which Chroma uses internally.\n\n**Vector store.** `Chroma(collection_name=\"drone_missions\", persist_directory=\".../mission_history/\")` from `langchain-community`. Chroma persists to SQLite under the hood. The collection accumulates mission documents across server restarts — the drone's knowledge of past flights grows over time.\n\n**Document format.** Each saved mission becomes one Chroma document. The page content is a prose summary built from the event log and final telemetry:\n```\n\"Successfully pollinated 6/8 discovered flowers (75% success rate).\nMission duration 124s. moderate battery use (final 61%).\nPhase sequence: Arming sequence → Taking off to 8m → Scanning — lawnmower pass\n  → Computing optimal route → Approaching target → ...\nKey events: Flower detected — r1 (3.2m lateral); Flower detected — r3 (1.8m lateral);\n  POLLINATION COMPLETE — r1; POLLINATION COMPLETE — r3; ...\"\n```\nMetadata stored: `pollinated` (int), `discovered` (int), `battery_final` (float), `duration_s` (float). These appear alongside retrieved documents for quick interpretation.\n\n**Retrieval.** `similarity_search(query, k=min(3, collection_count))` returns the top-k documents by cosine similarity. The query is a compact state descriptor: `\"phase=approach battery=72 stability=0.71 discovered=5 pollinated=2\"`. The returned documents are formatted as:\n```\nRelevant past mission experiences:\n  [1] Successfully pollinated 7/8 ... (pollinated=7, battery_end=58%, duration=118s)\n  [2] Successfully pollinated 5/7 ... (pollinated=5, battery_end=42%, duration=145s)\n  [3] Successfully pollinated 4/8 ... (pollinated=4, battery_end=39%, duration=162s)\n```\nThis text block is appended to the Claude system prompt before each `/decide` call.\n\n**Graceful degradation.** If `langchain-community`, `chromadb`, or `sentence-transformers` are not installed, `MissionStore.available = False` and all methods are silent no-ops. The `RAG_AVAILABLE` flag in `agent_server.py` skips retrieval entirely when false.\n\n---\n\n## UCB1 Confidence Bandit\n\n### `ConfidenceBandit` (`confidence_bandit.py`)\n\nA contextual multi-armed bandit that learns optimal detection confidence thresholds from mission experience.\n\n**Context bucketing.** The state space is factored into three independent dimensions:\n- **Phase tier**: `scanning` (scanning/planning), `approach` (descent/approach/target_lock/candidate_detected), `hover` (hover_align/pollinating)\n- **Quality tier**: `high` (of_stability \u003e 0.7), `med` (0.4–0.7), `low` (\u003c 0.4)\n- **Battery tier**: `high` (≥ 50%), `low` (\u003c 50%)\n\nThis gives 3 × 3 × 2 = 18 possible contexts. Each is identified by a string key like `\"approach_med_high\"`.\n\n**Arms.** Three threshold options per context: `[0.40, 0.60, 0.75]`. A lower threshold is more permissive (transitions happen earlier but may be premature). A higher threshold is more conservative (may be too slow to commit).\n\n**UCB1 score.** Each arm tracks `[pulls, reward_sum]`. The UCB1 selection formula:\n```\nucb_score(arm) = reward_sum / pulls + sqrt(2 × log(total_pulls) / pulls)\n```\nThe exploration bonus `sqrt(2 × log(total) / pulls)` is large for rarely-tried arms, encouraging exploration. As more data is gathered, the exploitation term (`reward_sum / pulls`) dominates and the bandit converges to the best arm.\n\n**Initialization.** New contexts start with `[[1, 0.5], [1, 0.5], [1, 0.5]]` — one phantom pull with neutral reward 0.5. This prevents division by zero and ensures all arms are initially explored roughly equally.\n\n**Reward signal.** `update_reward(phase, of_stability, battery_pct, success=True/False)` converts `success` to `+1.0` / `-1.0` and adds it to the currently-best arm for that context. In practice, `success=True` is sent after each pollination, `success=False` could be sent on timeout (not currently wired but the endpoint exists via `/feedback`).\n\nOver many missions, the bandit learns: for example, in `approach_low_low` (degraded sensor, low battery), a 0.40 threshold might get more reward than 0.75 because committing quickly before battery runs out is better than waiting for high confidence that may never come.\n\n---\n\n## End-to-End Data Flow\n\nThis is the complete flow for a single live mode frame tick:\n\n```\nrequestAnimationFrame(ts)\n│\n├── dt = (ts - lastTs) / 1000   // capped at 100ms\n│\n├── nav.tick(dt, latestInference)\n│   ├── time += dt, frameIdx++\n│   ├── if inference: processInference(inf)\n│   │   ├── update flower confidences from detections\n│   │   ├── promote flower states (scanned → candidate → locked)\n│   │   └── add server-discovered flowers to discoveredIds\n│   ├── stepPhase(dt)   // advance FSM based on current phase\n│   │   └── doScanning → doProximityDetection\n│   │       ├── check each undiscovered flower \u003c 4.5m\n│   │       ├── mark discovered, update confidence\n│   │       └── rerun computeTSPRoute if new discovery\n│   ├── updateHistory()  // append to posHistory, altHistory\n│   └── buildFrame()\n│       ├── getSensorAtDistance(z * 39.37)\n│       ├── computeOpticalFlowState(sample, frameIdx)\n│       └── return LiveFrame { drone, sensor, flowers, phase, ... }\n│\n├── ws.send(lf.drone, lf.flowers, lf.phase)   // → :8765\n│   └── inference_server receives, renders frame, detects, returns\n│       └── latestInference updated on next RAF tick\n│\n├── if frameIdx % 30 == 0:\n│   agent.requestDecision(lf)   // debounced 200ms → POST :8766/decide\n│       └── agent returns AgentDecision\n│           ├── nav.applyAgentDecision(decision)\n│           │   ├── tspRoute = decision.priorityOverride (if valid)\n│           │   └── currentConfidenceThreshold = decision.confidenceThreshold\n│           └── agentState updated\n│\n├── if lf.phase != lastPhase:\n│   agent.startCommentaryStream(lf)   // SSE :8766/stream\n│       └── streams text chunks → onCommentary → AgentCommentaryPanel\n│\n├── liveToReplay(lf) → ReplayFrame   // adapter for existing panels\n│\n└── setFrame(lfWithAgent)   // React state update → re-render all panels\n\nEvery 100ms (in agent server):\n  _terminal_callback.drain()\n  → send to all /terminal WebSocket clients\n  → liveInferenceEngine.pushTerminal(type, text)\n  → terminal buffer → TerminalPanel (synced every 250ms)\n\nOn nav.done (mission complete):\n  agent.saveMission(lf)   // POST :8766/mission/save\n  → MissionStore.save_mission(events, telemetry)\n  → build prose document from event log\n  → embed with all-MiniLM-L6-v2\n  → add to Chroma collection\n  → available for retrieval in next session\n```\n\n---\n\n## Quick Start\n\n### Prerequisites\n\n- Node.js 18+\n- Python 3.11+\n- (Optional) `ANTHROPIC_API_KEY` for LLM features\n\n### 1. Install frontend dependencies\n```bash\nnpm install\n```\n\n### 2. Install Python backend dependencies\n```bash\n# Inference server (minimal)\npip install -r drone-cv-system/server/requirements_server.txt\n\n# Agent server (LangChain + Chroma + sentence-transformers)\npip install -r drone-cv-system/server/requirements_agent.txt\n```\n\n\u003e **Note:** `sentence-transformers` downloads `all-MiniLM-L6-v2` (~90MB) on first use. This is a one-time download cached in `~/.cache/huggingface/`.\n\n### 3. Set Anthropic API key (optional — falls back to mock)\n```bash\nexport ANTHROPIC_API_KEY=sk-ant-...\n```\n\nWithout an API key, the agent server runs in mock mode: decisions return `action='continue'` with fixed 0.75 threshold, SSE commentary returns placeholder text, and all LangChain imports are skipped. The simulation is fully functional either way.\n\n### 4. Start both backend servers\n```bash\n# Option A — single script\nbash drone-cv-system/server/start_servers.sh\n\n# Option B — separate terminals\npython3 drone-cv-system/server/inference_server.py   # :8765\npython3 drone-cv-system/server/agent_server.py       # :8766\n```\n\n### 5. Start the frontend\n```bash\nnpm run dev\n```\n\nOpen http://localhost:5173. Select **REPLAY MODE** for the pre-generated deterministic simulation or **LIVE MODE** for the real-time autonomous mission with WebSocket inference and LLM agent.\n\nIn live mode, click the **TERMINAL** button in the header to open the terminal panel. Use the **AI** filter tab to see only LangChain callback events — LLM thoughts, tool calls, and tool results appear in real time as the agent reasons about the mission.\n\n---\n\n## Configuration\n\n| Variable | Default | Description |\n|---|---|---|\n| `ANTHROPIC_API_KEY` | — | Claude API key. Unset = mock mode |\n| Inference server port | `8765` | WebSocket endpoint |\n| Agent server port | `8766` | HTTP/SSE/WS agent endpoints |\n| Vite dev server port | `5173` | Frontend |\n| RAG persist dir | `drone-cv-system/mission_history/` | Chroma SQLite files |\n| Lawnmower default spacing | `4.5m` | Overridable by agent `plan_scan_pattern` tool |\n| Agent decision interval | 30 frames (~1s) | `AGENT_DECISION_EVERY_N_FRAMES` in `liveInferenceEngine.ts` |\n| Terminal drain interval | 100ms | Callback queue poll in `/terminal` WS handler |\n| Terminal state sync | 250ms | `setInterval` in `liveInferenceEngine` |\n| Decision debounce | 200ms | In `AgentClient.requestDecision()` |\n| Proximity detect radius | 4.5m | `PROXIMITY_DETECT_RADIUS` in `autonomousNavigator.ts` |\n| Patrol altitude | 8.0m | `PATROL_ALT` |\n| Hover altitude | 1.5m | `HOVER_ALT` |\n| Pollination dwell | 3.0s | `POLLINATE_DWELL_S` |\n| Max LLM tool rounds | 3 | In `_llm_decide()` |\n| RAG retrieval count | 3 | `k=3` in `retrieve_context()` |\n\n---\n\n## Physical Drone Hardware Stack\n\n### Airframe \u0026 Propulsion\n\n| Component | Spec |\n|---|---|\n| Frame | F450 glass-fiber quad, 450 mm wheelbase, plastic landing skid |\n| Motors | 4× brushless — 2212 920 KV or 2213 935 KV |\n| ESCs | 4× 20 A ESC |\n| Propellers | 4× 9450 self-tightening (2× CW, 2× CCW) |\n| Battery | 11.1 V 3S LiPo, 4200 mAh |\n| Power module | 5 V 2 A BEC — powers Pixhawk; **separate** 5 V BEC for RPi and servo |\n\n**Motor layout (ArduCopter X-quad):**\n\n```\n       FRONT\n  1 (CCW) · 3 (CW)\n     ·         ·\n  2 (CCW) · 4 (CW)\n       REAR\n```\n\n### RC Control (Manual Override Only)\n\nThe **FS-i6X transmitter + FS-iA6B receiver** connects to Pixhawk RC IN in PPM mode and is used exclusively for manual override. The entire autonomous mission runs without any RC input. The RC link provides:\n\n- **CH5** → Flight mode switching (GUIDED ↔ STABILIZE/LOITER)\n- **CH7** → RTL/failsafe\n- **RC failsafe:** If signal is lost, Pixhawk executes RTL automatically\n- `is_rc_override_active()` in `FlightController` detects when the pilot has switched out of GUIDED and pauses autonomous position commands\n\n### Flight Controller — Pixhawk 2.4.8 (ArduCopter)\n\n- **IMU fusion:** 3-axis accelerometer + gyroscope, fused by EKF2 extended Kalman filter\n- **Barometer:** MS5611 for coarse altitude hold\n- **GPS:** M8N u-blox module with integrated compass on foldable mast\n- **MAVLink output:** TELEM2 port (SERIAL1 in ArduCopter) at 921 600 baud\n- **Motor mixing:** PWM outputs MAIN OUT 1–4 drive the 4× 20 A ESCs. All stabilization loops (rate PID, attitude PID, altitude hold) run on Pixhawk — the companion computer only sends high-level `SET_POSITION_TARGET_LOCAL_NED` setpoints\n- **AUX OUT 1:** Drives the pollen-dispenser servo via MAVLink `DO_SET_SERVO` (`SERVO9_FUNCTION = 0`, passthrough)\n- **GUIDED mode:** Required for autonomous mission; RC override to any other mode pauses Python position commands immediately\n\n### Companion Computer — Raspberry Pi 4\n\n- Runs full Python CV + mission stack at ~20 Hz mission tick\n- Communicates with Pixhawk over hardware UART (`/dev/ttyAMA0`, 921 600 baud) via pymavlink — Bluetooth must be disabled to free the hardware UART: `dtoverlay=disable-bt` in `/boot/config.txt`\n- Sends position setpoints + arming commands to Pixhawk via MAVLink\n- Triggers pollination servo via `FlightController.trigger_aux_servo()` → MAVLink `DO_SET_SERVO` on AUX OUT 1\n\n### Google Coral USB Accelerator\n\nThe Coral USB Edge TPU is the primary inference accelerator:\n\n| Property | Value |\n|---|---|\n| Interface | USB 3.0 (USB-C to USB-A on RPi) |\n| Performance | ~4 TOPS (INT8), ~15–30 FPS on 320×320 MobileNet-SSD |\n| Model format | EdgeTPU-compiled `.tflite` (INT8 quantized, uint8 NHWC input) |\n| Runtime | `libedgetpu1-std` (apt) + `pycoral` (pip) |\n| Compile | `edgetpu_compiler flower_detector.tflite → flower_detector_edgetpu.tflite` |\n\n**Important tensor format difference:** Coral requires `[1, 320, 320, 3]` **uint8 NHWC** input — completely different from ONNX which requires `[1, 3, 640, 640]` float32 NCHW. The `CoralDetector` and `FramePreprocessor.read_frame_for_coral()` handle this conversion.\n\n### Camera\n\nDownward-facing camera (CSI or USB) capturing the garden below the drone:\n\n- Center-cropped and resized to model input size (640×640 for ONNX, 320×320 for Coral)\n- BGR → RGB conversion before inference\n- Field of view at 1.5 m hover altitude: ~1.7 m × 1.7 m — sufficient to frame one flower cluster\n\n### Pollination Mechanism\n\nA **micro servo** arm actuates a lightweight pollen-dispenser assembly (pollen reservoir + dispensing gate + mounting bracket) mounted below the drone frame. The servo arm physically positions the dispenser over the flower and a gravity/air-puff mechanism releases pollen during the 2.5-second dwell.\n\n**Actuation sequence (via `FlightController.trigger_aux_servo()`):**\n\n1. Pixhawk receives `DO_SET_SERVO` → AUX OUT 1 → 1700 µs PWM → servo arm deploys\n2. Hold 2.5 s (pollen transfer dwell)\n3. Pixhawk receives `DO_SET_SERVO` → AUX OUT 1 → 1000 µs PWM → servo arm retracts\n\nHardware PWM from Pixhawk AUX ensures timing accuracy and the command is logged in DataFlash. A GPIO fallback (RPi PWM on pin 18) is available for bench testing without a Pixhawk.\n\n### Wiring Diagram\n\n```\nRaspberry Pi UART                       Pixhawk 2.4.8 TELEM2\n─────────────────                       ────────────────────\nGPIO 14 (TX, /dev/ttyAMA0) ──────────→ TELEM2 RX\nGPIO 15 (RX, /dev/ttyAMA0) ←────────── TELEM2 TX\nGND                        ─────────── GND\n(921600 baud, disable-bt overlay required)\n\nPixhawk 2.4.8 AUX OUT                  Pollen-Dispenser Servo\n──────────────────────                  ──────────────────────\nAUX OUT 1 (signal) ──────────────────→ Servo signal wire (orange)\nDedicated 5V BEC (+) ────────────────→ Servo power (red)   ← NOT RPi 5V pin\nCommon GND ──────────────────────────→ Servo ground (black)\n(SERVO9_FUNCTION = 0 in ArduCopter params)\n\nFS-iA6B Receiver (PPM)                 Pixhawk 2.4.8 RC IN\n──────────────────────                  ───────────────────\nPPM output ──────────────────────────→ RC IN (CH5 = flight mode, CH7 = RTL)\n\nCoral USB Edge TPU                      Raspberry Pi\n──────────────────                      ────────────\nUSB-C connector ─────────────────────→ USB 3.0 port\n\nDownward camera (CSI) ───────────────→ RPi CSI connector (or USB)\n```\n\n---\n\n## Firmware \u0026 MAVLink Integration\n\n### MAVLink Interface (`mavlink_interface.py`)\n\nThe lowest layer of the drone-cv-system. Wraps pymavlink into a clean Python API with a **background reader thread** that continuously pulls MAVLink packets and writes them into a thread-safe `TelemetrySnapshot` object. The main mission thread reads the snapshot without blocking.\n\n**Connection:**\n\n```python\nconnection = mavutil.mavlink_connection(\n    '/dev/ttyAMA0',   # RPi UART\n    baud=921600,\n    source_system=1\n)\n```\n\n**`TelemetrySnapshot` fields written by background thread:**\n\n```python\nroll, pitch, yaw          # radians (ATTITUDE message)\nrollspeed, pitchspeed, yawspeed\nlat, lon, alt_msl         # GLOBAL_POSITION_INT\nrelative_alt              # altitude above home, mm → m\nvx, vy, vz                # cm/s → m/s\nheading                   # centidegrees → degrees\nflow_x, flow_y            # integrated optical flow (OPTICAL_FLOW_RAD)\nflow_quality              # 0-255\nground_distance           # meters (from Pixhawk-fused rangefinder)\nrangefinder_distance      # raw DISTANCE_SENSOR distance, meters\nekf_flags                 # EKF_STATUS_REPORT bitmask\nvelocity_variance         # EKF velocity variance\npos_horiz_variance        # EKF horizontal position variance\nbattery_voltage           # millivolts → volts\nbattery_remaining         # 0-100 %\narmed                     # bool\n```\n\n**Heartbeat monitoring:** A watchdog timer detects if no heartbeat arrives within 3 seconds and raises a `ConnectionLostError`.\n\n### Flight Controller Wrapper (`flight_controller.py`)\n\nBuilds on `MAVLinkInterface` to provide mission-level commands:\n\n| Method | MAVLink command | Notes |\n|---|---|---|\n| `pre_flight_checks()` | Reads telemetry snapshot | Verifies EKF healthy, GPS lock, battery \u003e20%, not armed |\n| `arm()` | `MAV_CMD_COMPONENT_ARM_DISARM (1)` | Requires GUIDED mode active first |\n| `disarm()` | `MAV_CMD_COMPONENT_ARM_DISARM (0)` | Only if landed; force flag for emergency |\n| `takeoff(alt_m)` | `MAV_CMD_NAV_TAKEOFF` | Blocks until altitude reached ±0.5 m |\n| `goto_ned(n,e,d)` | `SET_POSITION_TARGET_LOCAL_NED` | NED frame, velocity feed-forward |\n| `trigger_aux_servo(ch, pwm)` | `MAV_CMD_DO_SET_SERVO` | Drives AUX OUT 1 → pollen dispenser |\n| `is_rc_override_active()` | Reads `telem.mode` | True if pilot switched out of GUIDED |\n| `precision_hover(x, y, alt)` | `SET_POSITION_TARGET_LOCAL_NED` | Tighter tolerance (±0.1m XY, ±0.05m Z) for hover_align |\n| `land()` | `MAV_CMD_NAV_LAND` | Blocks until landed + disarmed |\n| `set_mode(mode)` | `SET_MODE` | Switches GUIDED / STABILIZE / LOITER |\n\n### Telemetry Messages Consumed\n\n| MAVLink Message | Rate | Fields Used |\n|---|---|---|\n| `ATTITUDE` (30) | 50 Hz | roll, pitch, yaw, yawspeed |\n| `GLOBAL_POSITION_INT` (33) | 10 Hz | lat, lon, relative_alt, vx, vy, vz, hdg |\n| `OPTICAL_FLOW_RAD` (106) | 20 Hz | integrated_x/y, quality, time_delta_distance_us, ground_distance |\n| `DISTANCE_SENSOR` (132) | 20 Hz | current_distance, min/max_distance, type |\n| `EKF_STATUS_REPORT` (193) | 2 Hz | flags, velocity_variance, pos_horiz_variance, terrain_alt_variance |\n| `SYS_STATUS` (1) | 1 Hz | voltage_battery, current_battery, battery_remaining |\n| `HEARTBEAT` (0) | 1 Hz | armed state, flight mode |\n\n---\n\n## Computer Vision Pipeline (Real Drone)\n\n### Frame Preprocessor (`frame_preprocessor.py`)\n\nHandles camera input regardless of source (USB UVC, CSI ribbon, file). Steps applied to every frame:\n\n1. Capture raw frame (BGR from OpenCV)\n2. Center-crop to square aspect ratio\n3. Resize to 640×640 using `INTER_LINEAR`\n4. Convert BGR → RGB\n5. Normalize pixel values to `[0, 1]` float32\n6. Expand dims to `[1, 3, 640, 640]` NCHW tensor layout for ONNX, or `[1, 320, 320, 3]` uint8 NHWC for Coral\n\n### YOLOv8 Flower Detector (`flower_detector.py`)\n\nWraps the ONNX Runtime session and post-processes raw YOLOv8 output into clean Python detection objects.\n\n**Input:** `[1, 3, 640, 640]` float32 tensor  \n**Raw output:** `[1, 84, 8400]` — 8400 anchor proposals, each with 4 bbox coords + 80 class scores  \n**Classes used:** `flower_open` (class 0), `flower_closed` (class 1), `flower_cluster` (class 2)\n\n**Post-processing steps:**\n\n1. Transpose output to `[8400, 84]`\n2. Extract `cx, cy, w, h` from columns 0–3\n3. Extract class scores from columns 4–83; take `argmax` and `max` as class ID + confidence\n4. Filter: confidence \u003e 0.25 and class in {0, 1, 2}\n5. Convert `cx, cy, w, h` → `x1, y1, x2, y2` pixel coordinates\n6. Apply NMS (IoU threshold 0.45) to remove overlapping boxes\n7. Build `Detection` objects:\n\n```python\n@dataclass\nclass Detection:\n    x1, y1, x2, y2: float       # pixel bbox\n    confidence: float            # 0–1\n    class_id: int                # 0/1/2\n    class_name: str              # flower_open / flower_closed / flower_cluster\n    center_x, center_y: float   # bbox center\n    width, height: float\n    area: float\n    bearing: Optional[np.ndarray]       # unit vector camera→flower\n    distance_estimate: Optional[float]  # meters (from DepthEstimator)\n```\n\n### Optical Flow Tracker (`optical_flow_tracker.py`)\n\nProvides temporal continuity for detections across frames using **Lucas-Kanade sparse optical flow** (OpenCV `calcOpticalFlowPyrLK`).\n\n**Algorithm:**\n\n1. Maintain a list of active `Track` objects (one per flower being tracked)\n2. Each `Track` stores: bbox, confidence, age (frames active), last seen frame\n3. On each new frame:\n   - Extract bounding box corners as keypoints\n   - Run LK optical flow to predict new keypoint positions\n   - Recompute bboxes from moved keypoints\n   - Match YOLOv8 detections to predicted tracks using IoU\n   - **EMA smoothing** on matched tracks: `bbox = 0.6 × new_detection + 0.4 × predicted`\n   - Increment age on matched tracks, mark unmatched tracks as stale after 10 frames\n4. Return merged list of smoothed active tracks\n\nThis prevents the jitter from per-frame YOLO detections and maintains stable IDs across frames.\n\n### Depth Estimator (`depth_estimator.py`)\n\nEstimates physical distance to detected flowers (meters) from:\n\n1. **Apparent bbox size:** The physical width of a flower cluster (~0.3m) divided by the bounding box width in pixels, scaled by focal length\n2. **Altitude fusion:** When rangefinder altitude is available, the depth estimate is blended with a geometry-based estimate using the drone's current tilt angle\n3. **Output:** Absolute distance in meters, written into each `Detection.distance_estimate`\n\n---\n\n## ML Model — YOLOv8 Flower Detection\n\n### Model Architecture\n\n**YOLOv8n (nano)** — chosen for inference speed on the Raspberry Pi 4:\n\n| Property | Value |\n|---|---|\n| Parameters | 3.2 M |\n| Model size | ~6.3 MB (ONNX) |\n| Input | 640×640×3 |\n| Backbone | CSPDarknet (5 stages, C2f bottleneck blocks) |\n| Neck | PAN-FPN (3 detection scales: 80×80, 40×40, 20×20) |\n| Head | Decoupled detection head (separate cls/reg branches) |\n| RPi inference time | ~180 ms (ONNX Runtime, CPU) |\n| Laptop inference time | ~12 ms (GPU) |\n\n**Classes:**\n\n| ID | Name | Description |\n|---|---|---|\n| 0 | `flower_open` | Fully open flower, visible pistil/stamens |\n| 1 | `flower_closed` | Budded or partially open flower |\n| 2 | `flower_cluster` | Dense group of flowers treated as a single target |\n\n### Training Pipeline\n\n```\n1. Dataset collection\n   Sources: COCO (flowers subset), iNaturalist flower images, Roboflow community datasets\n   Annotation: LabelImg, YOLO format (.txt per image, normalized bbox + class)\n   Split: 80% train / 10% val / 10% test\n\n2. Augmentation (Ultralytics built-in)\n   Mosaic (4-image), random flip, HSV jitter (hue ±0.015, sat ±0.7, val ±0.4)\n   Scale (±50%), translate (±10%), cutmix (close-range images)\n\n3. Fine-tuning command\n   python train.py --model yolov8n.pt --data flowers.yaml \\\n     --epochs 100 --imgsz 640 --batch 16 --lr0 0.01 \\\n     --patience 20 --device 0\n\n4. Evaluation targets\n   mAP@50:    \u003e0.72\n   mAP@50-95: \u003e0.45\n   Precision: \u003e0.70\n   Recall:    \u003e0.65\n\n5. Export to ONNX\n   from ultralytics import YOLO\n   model = YOLO('best.pt')\n   model.export(format='onnx', imgsz=640, simplify=True, opset=12)\n   # Output: best.onnx  (~6.3 MB)\n\n6. Export to EdgeTPU TFLite (for Coral)\n   model.export(format='tflite', int8=True, imgsz=320)\n   edgetpu_compiler flower_detector.tflite\n   # Output: flower_detector_edgetpu.tflite\n```\n\nONNX Runtime on the Raspberry Pi 4 runs the model in **CPU execution provider** mode with `opset=12` for maximum compatibility. The session is loaded once at startup and reused across all frames (first load ~800ms).\n\n### Detection Dataflow\n\n```\nCamera frame (640×640 RGB)\n         ↓\nFramePreprocessor  →  [1, 3, 640, 640] float32 tensor\n         ↓\nONNX Runtime sess.run()  →  [1, 84, 8400] raw output\n         ↓\nTranspose + confidence filter (\u003e0.25) + NMS (IoU 0.45)\n         ↓\nList[Detection] with pixel bboxes + class + confidence\n         ↓\nOpticalFlowTracker  →  temporally smoothed + stable IDs\n         ↓\nDepthEstimator     →  each detection gets distance_estimate\n         ↓\nStateMachine.process_detections()\n```\n\n---\n\n## Pollination Manager\n\n`drone-cv-system/mission/pollination_manager.py` tracks flower target state and controls the physical pollination hardware.\n\n**Garden state tracking:**\n\n```python\n@dataclass\nclass FlowerTarget:\n    id: str\n    x: float          # meters in garden space\n    y: float\n    visited: bool = False\n    pollinated: bool = False\n    detection_count: int = 0\n    last_confidence: float = 0.0\n```\n\n**Pollination sequence** (triggered by `StateMachine` on entering `pollinating` phase):\n\n```python\ndef trigger_pollination(self, flower_id: str):\n    # 1. Extend servo arm to contact position via MAVLink AUX channel\n    flight_controller.trigger_aux_servo(channel=9, pwm=1700)  # deploy\n\n    # 2. Dwell 2.5 seconds for pollen transfer\n    time.sleep(2.5)\n\n    # 3. Retract servo\n    flight_controller.trigger_aux_servo(channel=9, pwm=1000)  # stow\n\n    # 4. Mark flower pollinated\n    self.targets[flower_id].pollinated = True\n```\n\nGPIO fallback for bench testing (no Pixhawk required):\n\n```python\nGPIO.setup(SERVO_PIN, GPIO.OUT)\npwm = GPIO.PWM(SERVO_PIN, 50)   # 50 Hz\npwm.start(7.5)                  # neutral\npwm.ChangeDutyCycle(10.0)       # deploy (~1700µs @ 50Hz)\ntime.sleep(2.5)\npwm.ChangeDutyCycle(2.5)        # stow (~1000µs @ 50Hz)\n```\n\n---\n\n## Hardware Assembly \u0026 Setup\n\n### Assembly Checklist\n\n1. **Frame:** F450 glass-fiber quad, 450 mm wheelbase, landing skid attached\n2. **Flight controller:** Pixhawk 2.4.8 on vibration damping plate (top deck)\n3. **Power:** 11.1 V 3S 4200 mAh LiPo in middle deck; power module to Pixhawk POWER port\n4. **ESCs:** 4× 20 A ESC connected to MAIN OUT 1–4 and to LiPo distribution\n5. **Motors:** 4× 2212 920 KV brushless; 9450 self-tightening props (CCW on 1/2, CW on 3/4)\n6. **GPS + Compass:** M8N GPS on foldable mast, connected to Pixhawk GPS port\n7. **RC receiver:** FS-iA6B in PPM mode → Pixhawk RC IN (CH5 flight mode, CH7 RTL)\n8. **Companion computer:** Raspberry Pi mounted on bottom deck; **dedicated 5 V BEC** for power (not from Pixhawk BEC)\n9. **Coral USB TPU:** Google Coral USB Accelerator in RPi USB 3.0 port\n10. **Camera:** Downward-facing camera (CSI or USB) on RPi, pointed directly down\n11. **UART bridge:** RPi GPIO 14/15 → Pixhawk TELEM2 (921 600 baud)\n12. **Pollination payload:** Micro servo on bracket; servo signal → Pixhawk AUX OUT 1; servo power → dedicated 5 V BEC (shared ground)\n\n### Software Setup on Raspberry Pi\n\n```bash\n# Flash Raspberry Pi OS Lite (64-bit) to SD card\n\n# 1. Enable UART, disable Bluetooth (to free /dev/ttyAMA0 for Pixhawk)\necho \"enable_uart=1\" \u003e\u003e /boot/config.txt\necho \"dtoverlay=disable-bt\" \u003e\u003e /boot/config.txt\nsudo reboot\n\n# 2. Install system dependencies\nsudo apt update \u0026\u0026 sudo apt install -y python3-pip python3-opencv libopencv-dev\n\n# 3. Install Google Coral USB runtime\necho \"deb https://packages.cloud.google.com/apt coral-edgetpu-stable main\" | \\\n    sudo tee /etc/apt/sources.list.d/coral-edgetpu.list\ncurl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -\nsudo apt update \u0026\u0026 sudo apt install libedgetpu1-std\n\n# 4. Install Python packages\npip3 install pymavlink onnxruntime pillow loguru RPi.GPIO pyyaml\n# Coral packages (from Coral index):\npip3 install pycoral tflite-runtime \\\n    --extra-index-url https://google-coral.github.io/py-packages/\n\n# 5. Upload models\nscp flower_detector.onnx pi@drone-pi.local:~/drone-cv-system/models/\nscp flower_detector_edgetpu.tflite pi@drone-pi.local:~/drone-cv-system/models/\n\n# 6. Start mission\npython3 drone-cv-system/main.py\n```\n\n### ArduCopter Parameters Required\n\n| Parameter | Value | Reason |\n|---|---|---|\n| `SERIAL1_BAUD` | 921 | TELEM2 baud rate (921600) |\n| `SERIAL1_PROTOCOL` | 2 | MAVLink 2.0 on TELEM2 |\n| `SERVO9_FUNCTION` | 0 | AUX OUT 1 passthrough for pollen servo |\n| `WPNAV_RADIUS` | 100 | 1 m waypoint acceptance radius |\n| `GUIDED_OPTIONS` | 0 | Accept velocity setpoints from companion |\n| `FS_THR_ENABLE` | 1 | RC throttle failsafe → RTL on signal loss |\n| `FLTMODE_CH` | 5 | CH5 for flight mode switching |\n\n---\n\n## Tech Stack\n\n| Layer | Technology | Notes |\n|---|---|---|\n| Frontend framework | Preact 10 + TypeScript 5 | React-compatible, smaller bundle |\n| Build tool | Vite 5 + `@preact/preset-vite` | HMR, proxy middleware |\n| Styling | Tailwind CSS v4 + `@tailwindcss/vite` | Utility classes + custom animations |\n| Rendering | Pure SVG (no canvas/WebGL) | All panels are SVG with CSS animations |\n| State management | Preact hooks only | No Redux/Zustand/MobX |\n| Backend framework | FastAPI + uvicorn | Async WebSocket + HTTP + SSE |\n| LLM integration | `langchain-anthropic` + `langchain-core` | `ChatAnthropic`, `BaseCallbackHandler`, `StructuredTool` |\n| LLM model | Claude claude-haiku-4-5-20251001 | Fast + cheap for real-time decisions |\n| RAG embeddings | `sentence-transformers/all-MiniLM-L6-v2` | 384-dim, local CPU, ~90MB |\n| RAG vector store | Chroma (`langchain-community`) | SQLite persistence, cosine similarity |\n| Computer vision | ONNX Runtime + YOLOv8n | 3.2M params, 6.3MB, COCO mAP50 37.3 |\n| TPU inference | Google Coral pycoral + EdgeTPU | INT8 TFLite, ~5ms/frame |\n| Bandit | UCB1 (custom Python) | No ML framework dependency |\n| Streaming | SSE + WebSocket | Commentary (SSE), terminal + inference (WS) |\n| Hardware target | Raspberry Pi 4 + Pixhawk 6 + Coral USB | Production autonomous flight |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzoraizmohammad%2Fagentic-pollination-uav","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzoraizmohammad%2Fagentic-pollination-uav","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzoraizmohammad%2Fagentic-pollination-uav/lists"}