{"id":51526592,"url":"https://github.com/oaslananka/sky-track-vision-dev","last_synced_at":"2026-07-08T22:00:59.638Z","repository":{"id":367116569,"uuid":"1279321226","full_name":"oaslananka/sky-track-vision-dev","owner":"oaslananka","description":"LLM-powered autonomous drone perception \u0026 mission orchestration on AirSim — YOLOv8 detection, Kalman tracking, cascade-PID visual servoing, and an LLM mission pilot with deterministic safety gates.","archived":false,"fork":false,"pushed_at":"2026-06-24T17:09:48.000Z","size":157,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-24T17:15:16.795Z","etag":null,"topics":["airsim","autonomous-drones","computer-vision","drone","kalman-filter","llm","llm-agents","mission-planning","object-tracking","openai","python","robotics","unreal-engine","visual-servoing","yolov8"],"latest_commit_sha":null,"homepage":"https://deepwiki.com/oaslananka/sky-track-vision-dev","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oaslananka.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":["oaslananka"],"buy_me_a_coffee":"oaslananka"}},"created_at":"2026-06-24T15:20:50.000Z","updated_at":"2026-06-24T17:12:25.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/oaslananka/sky-track-vision-dev","commit_stats":null,"previous_names":["oaslananka/sky-track-vision-dev"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/oaslananka/sky-track-vision-dev","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oaslananka%2Fsky-track-vision-dev","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oaslananka%2Fsky-track-vision-dev/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oaslananka%2Fsky-track-vision-dev/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oaslananka%2Fsky-track-vision-dev/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oaslananka","download_url":"https://codeload.github.com/oaslananka/sky-track-vision-dev/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oaslananka%2Fsky-track-vision-dev/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35279442,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-08T02:00:06.796Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airsim","autonomous-drones","computer-vision","drone","kalman-filter","llm","llm-agents","mission-planning","object-tracking","openai","python","robotics","unreal-engine","visual-servoing","yolov8"],"created_at":"2026-07-08T22:00:58.697Z","updated_at":"2026-07-08T22:00:59.618Z","avatar_url":"https://github.com/oaslananka.png","language":"Python","funding_links":["https://github.com/sponsors/oaslananka","https://buymeacoffee.com/oaslananka","https://www.buymeacoffee.com/oaslananka","https://img.buymeacoffee.com/button-api/?text=Buy%20me%20a%20coffee\u0026emoji=%E2%98%95\u0026slug=oaslananka\u0026button_colour=FFDD00\u0026font_colour=000000\u0026font_family=Arial\u0026outline_colour=000000\u0026coffee_colour=ffffff"],"categories":[],"sub_categories":[],"readme":"# SkyTrackVision\n\n\u003cdiv align=\"center\"\u003e\n\n**LLM-Powered Autonomous Drone Perception \u0026 Mission Orchestration System**\n\n[![Python 3.11–3.13](https://img.shields.io/badge/python-3.11%E2%80%933.13-blue.svg)](https://www.python.org/downloads/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)\n[![Tests](https://github.com/oaslananka/sky-track-vision-dev/actions/workflows/ci.yml/badge.svg)](https://github.com/oaslananka/sky-track-vision-dev/actions)\n[![Code Style: Ruff](https://img.shields.io/badge/code%20style-ruff-000000.svg)](https://github.com/astral-sh/ruff)\n[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/oaslananka/sky-track-vision-dev)\n\n\u003cbr/\u003e\n\n_An open-source framework for building autonomous drone intelligence using YOLOv8 perception, Kalman-filtered visual tracking, cascade PID visual servoing, and LLM-based mission orchestration — all running on top of [AirSim](https://github.com/microsoft/AirSim) simulation._\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://www.buymeacoffee.com/oaslananka\"\u003e\n    \u003cimg src=\"https://img.buymeacoffee.com/button-api/?text=Buy%20me%20a%20coffee\u0026emoji=%E2%98%95\u0026slug=oaslananka\u0026button_colour=FFDD00\u0026font_colour=000000\u0026font_family=Arial\u0026outline_colour=000000\u0026coffee_colour=ffffff\" alt=\"Buy me a coffee\" /\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\u003c/div\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/demo.gif\" alt=\"SkyTrackVision Demo\" width=\"720\"/\u003e\n\u003c/p\u003e\n\n---\n\n## 🎯 Purpose\n\nSkyTrackVision is designed as a **learning and research platform** for developers who want to:\n\n- **Develop agent management skills** — orchestrate multi-layered autonomous systems where an LLM pilot issues high-level commands and deterministic controllers handle real-time execution.\n- **Explore AI engineering patterns** — study how perception, control, safety, and planning layers compose in a real robotics pipeline.\n- **Contribute to open-source autonomy** — extend detection classes, add new mission modes, improve the IBVS controller, or build entirely new LLM tool integrations.\n\n\u003e This project is intentionally structured as a **demo-ready, extensible framework** rather than a production system, making it ideal for educational exploration and community-driven development.\n\n## 🏗️ Architecture\n\nThe system is organized into **three frequency bands**:\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│                     SkyPilot (LLM Layer)                       │\n│  Natural language task → FSM transitions → Mission orchestration│\n│  Frequency: ~1 Hz (LLM inference cadence)                      │\n├─────────────────────────────────────────────────────────────────┤\n│                  Perception \u0026 Control Layer                     │\n│  YOLOv8 + BoT-SORT → Kalman Tracker → Cascade PID (IBVS)      │\n│  Frequency: ~15-30 Hz (camera frame rate)                      │\n├─────────────────────────────────────────────────────────────────┤\n│                     AirSim I/O Layer                            │\n│  Camera frames, LiDAR, proximity sensors, velocity commands     │\n│  Frequency: ~30-60 Hz (simulation tick rate)                   │\n└─────────────────────────────────────────────────────────────────┘\n```\n\n### Package Map\n\n| Package           | Responsibility                                               | AirSim Dependency |\n| ----------------- | ------------------------------------------------------------ | ----------------- |\n| `autonomy/`       | Contracts, IBVS controller, FSM, safety evaluator, reporting | ❌ None           |\n| `vision/`         | YOLOv8 detector, Kalman tracker, frame annotator             | ❌ None           |\n| `agents/`         | Thin facade agents (scene, control, mission, safety)         | ❌ None           |\n| `airsim_control/` | Camera, sensors, movement, client management                 | ✅ Required       |\n| `skypilot/`       | LLM pilot loop, tool dispatcher, AirSim bridge, HUD          | ✅ Required       |\n| `config/`         | Typed dataclass configs, YAML merge, structured logging      | ❌ None           |\n| `demo/`           | Synthetic frames \u0026 sensor snapshots for offline demos        | ❌ None           |\n| `ui/`             | Overlay renderer for classic runtime                         | ❌ None           |\n\n\u003e **Key design principle:** The `autonomy/` package is **completely independent** of AirSim. All mission logic, safety gates, and IBVS control can be unit-tested with pure Python — no simulator needed.\n\n## ⚙️ How It Works\n\n### Classic Runtime (`main.py`)\n\nReal-time loop that captures AirSim frames, runs YOLO detection + Kalman tracking, computes IBVS velocity commands, and applies safety-gated movement. Supports keyboard-driven manual control and a demo mode that runs without AirSim.\n\n### SkyPilot Runtime (`python -m skypilot`)\n\nLLM-powered mission executor. You give it a natural language task, and the GPT pilot:\n\n1. **Plans** — converts the task into a sequence of tool calls\n2. **Executes** — manages FSM transitions (SCAN → TRACK → MONITOR → REPORT)\n3. **Controls** — delegates to the IBVS controller for real-time tracking\n4. **Guards** — every movement passes through the `SafetyEvaluator` before reaching AirSim, while a mission-level `MissionWatchdog` enforces the flight envelope (timeout, geofence, altitude, battery) and can force a universal `EMERGENCY` safe-abort with no human in the loop\n5. **Verifies** — the task is parsed into measurable acceptance criteria, and the mission is only scored a success if those objectives are met (not merely if the closing protocol ran) — see [Semantic Completion Verification](docs/architecture.md#semantic-completion-verification)\n\n```\n\"Take off, scan for trucks, follow the nearest one for 30 seconds, then land.\"\n    ↓\nrequest_takeoff → request_move_to_altitude(4m) → request_scan → [target locked]\n    → request_follow → wait_seconds(30) → set_mission_state(REPORT) → request_land\n```\n\n## 🚀 Quick Start\n\n### Prerequisites\n\n- **Python 3.11–3.13** for the core library, tests, and demo mode.\n  - ⚠️ **Live AirSim requires Python 3.11 specifically.** The `airsim` package's RPC stack relies on the stdlib `asyncore` module, which was removed in Python 3.12 ([PEP 594](https://peps.python.org/pep-0594/)). Every AirSim import in the codebase is guarded, so the library still runs on 3.12/3.13 — only the live simulator bridge is unavailable there.\n- **Unreal Engine 5.x + AirSim plugin** (for live simulation) — [AirSim Setup Guide](https://microsoft.github.io/AirSim/)\n- **OpenAI API key** (for SkyPilot LLM runtime)\n\n### Installation\n\n```bash\n# Clone the repository\ngit clone https://github.com/oaslananka/sky-track-vision-dev.git\ncd sky-track-vision-dev\n\n# Create virtual environment\npython -m venv .venv\nsource .venv/bin/activate  # Linux/macOS\n# .venv\\Scripts\\Activate.ps1  # Windows PowerShell\n\n# Install core dependencies (AirSim-free; works on Python 3.11–3.13)\npip install -r requirements.txt\n\n# For development (lint, test, type-check)\npip install -r requirements-dev.txt\n\n# For live AirSim simulation only (Python 3.11) — see file header for build notes\npip install -r requirements-sim.txt\n```\n\n### Configuration\n\n```bash\n# Copy environment template and add your API key\ncp .env.example .env\n# Edit .env and set your OPENAI_API_KEY\n\n# Sync AirSim settings (copies settings.json to ~/Documents/AirSim/)\n./sync_settings.sh      # Linux/macOS\n./sync_settings.ps1     # Windows\n```\n\nAll runtime parameters are in [`pilot.yaml`](pilot.yaml). Defaults are defined as typed dataclasses in [`config/settings.py`](config/settings.py).\n\n### Running\n\n```bash\n# Classic runtime (requires AirSim)\npython main.py\n\n# Demo mode (no AirSim needed — synthetic frames)\npython main.py --demo\n\n# SkyPilot mission (requires AirSim + OpenAI API key)\npython -m skypilot \"Scan for pedestrians and report\"\npython -m skypilot \"Take off, find a truck, follow it for 30 seconds, then land\"\n\n# Quick smoke test\npython smoke_test.py\n\n# Record a demo video (rendered HUD + overlays)\npython main.py --demo --record outputs/demo.mp4 --record-fps 30\n\n# Record a SkyPilot mission HUD video (requires AirSim)\npython -m skypilot \"Scan for pedestrians and report\" --record outputs/mission.mp4 --record-fps 30\n\nThe recorder writes the final rendered OpenCV frame, including HUD and overlays,\nusing a wall-clock based fixed-FPS writer so demo playback stays close to real\ntime even if processing jitter occurs. Use --no-hud to run SkyPilot without the\ndisplay; --record requires the HUD window and will error if combined with --no-hud.\n```\n\n### Keyboard Controls (Classic Runtime)\n\n| Key   | Action               | Key     | Action              |\n| ----- | -------------------- | ------- | ------------------- |\n| `W/S` | Forward / Backward   | `R/F`   | Ascend / Descend    |\n| `A/D` | Strafe Left / Right  | `J/L`   | Yaw Left / Right    |\n| `X`   | Toggle Auto-Follow   | `H`     | Emergency Hover     |\n| `G`   | Land                 | `U`     | Toggle Overlay Mode |\n| `O`   | Toggle Overlay       | `P`     | Screenshot          |\n| `N/B` | Next/Prev Demo Stage | `Q/Esc` | Quit                |\n\n## 🧪 Testing\n\n```bash\n# Run all tests\npython -m pytest tests/ -v\n\n# Run with coverage\npython -m pytest tests/ --cov=. --cov-report=term-missing\n\n# Lint \u0026 format\nruff check .\nruff format .\n\n# Type checking\nmypy .\n```\n\n### Mission Benchmark\n\nScore missions on success rate, **zero-intervention rate**, and safety-violation\nrate — the metrics that matter for unattended autonomy:\n\n```bash\npython scripts/benchmark.py --demo   # offline, deterministic scorecard\n```\n\nCollect [`MissionTrial`](autonomy/benchmark.py) records from live runs (one per\nseeded scenario) and pass them to `score_trials` to benchmark the real pilot.\n\n## 🤝 Contributing\n\nWe welcome contributions! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.\n\n**Areas where contributions are especially welcome:**\n\n- 🎯 **New detection classes** — extend YOLO target list and add mission modes\n- 🧠 **LLM tool integrations** — add new tools to the `ToolDispatcher`\n- 🎮 **Control algorithms** — improve or replace the IBVS controller\n- 🔐 **Safety features** — geofence, altitude ceiling, and a pluggable battery source now ship in `MissionWatchdog`; add wind estimation, or a Re-ID-based target re-acquisition channel\n- 📊 **Visualization** — dashboards, 3D trajectory plots, mission replay\n- 📖 **Documentation** — tutorials, architecture deep-dives, video demos\n\n## 📁 Project Structure\n\n```\nsky-track-vision-dev/\n├── main.py                  # Classic runtime entry point\n├── skypilot.py              # SkyPilot convenience launcher\n├── pilot.yaml               # Runtime configuration overrides\n├── settings.json            # AirSim vehicle/sensor configuration\n├── autonomy/                # AirSim-free core logic\n│   ├── contracts.py         # Typed dataclass contracts\n│   ├── ibvs.py              # Cascade PID visual servoing\n│   ├── follow_controller.py # Motion primitive resolver\n│   ├── mission.py           # Mission FSM with state graph (+ EMERGENCY safe-abort)\n│   ├── mission_spec.py      # NL→objective parser + semantic completion verifier\n│   ├── watchdog.py          # Mission-envelope watchdog (timeout/geofence/battery)\n│   ├── safety.py            # Deterministic per-frame safety evaluator\n│   ├── reporting.py         # Mission telemetry collector\n│   └── scene_reasoner.py    # Detection-to-insight summarizer\n├── vision/                  # Computer vision pipeline\n│   ├── detector.py          # YOLOv8 + BoT-SORT facade\n│   ├── tracker.py           # Kalman-filtered target tracker\n│   ├── annotator.py         # Frame overlay renderer\n│   └── utils.py             # Geometry \u0026 FPS utilities\n├── agents/                  # Facade agent wrappers\n├── airsim_control/          # AirSim client, camera, sensors, movement\n├── skypilot/                # LLM mission orchestration\n│   ├── __main__.py          # SkyPilot entry point\n│   ├── pilot.py             # LLM conversation loop\n│   ├── tools.py             # Tool dispatcher (14 tools)\n│   ├── llm_client.py        # OpenAI adapter with retry logic\n│   ├── airsim_bridge.py     # Safety-gated movement bridge\n│   ├── pilot_display.py     # Real-time pilot HUD\n│   └── hybrid.py            # Inter-tick deterministic controller\n├── config/                  # Settings \u0026 logging\n├── demo/                    # AirSim-free demo director\n├── ui/                      # Classic runtime overlay\n├── tests/                   # Pytest suite (172 tests)\n├── .github/                 # CI/CD \u0026 issue templates\n└── docs/                    # Extended documentation\n```\n\n## 🔧 Tech Stack\n\n| Component        | Technology                                                                                                                       |\n| ---------------- | -------------------------------------------------------------------------------------------------------------------------------- |\n| Simulation       | [Unreal Engine](https://www.unrealengine.com/) + [AirSim](https://github.com/microsoft/AirSim)                                   |\n| Object Detection | [Ultralytics YOLOv8](https://github.com/ultralytics/ultralytics) with [BoT-SORT](https://github.com/NirAharon/BoT-SORT) tracking |\n| Visual Servoing  | Cascade PID — Image-Based Visual Servoing (IBVS)                                                                                 |\n| Target Tracking  | Constant-velocity Kalman filter with EMA smoothing                                                                               |\n| Mission Planning | LLM function-calling via [OpenAI API](https://platform.openai.com/)                                                              |\n| Safety           | Per-frame deterministic safety evaluator (dynamic stopping distance) + mission-envelope watchdog with universal EMERGENCY abort     |\n| Verification     | Deterministic NL→objective parser with semantic completion gate                                                                  |\n| Visualization    | [OpenCV](https://opencv.org/) real-time HUD                                                                                      |\n| Language         | Python 3.11–3.13 with full type annotations (live AirSim: 3.11)                                                                  |\n\n## 📜 License\n\nThis project is licensed under the MIT License — see [LICENSE](LICENSE) for details.\n\n## 🙏 Acknowledgments\n\n- **[Microsoft AirSim](https://github.com/microsoft/AirSim)** — open-source drone/car simulation platform built on Unreal Engine\n- **[Ultralytics](https://github.com/ultralytics/ultralytics)** — YOLOv8 real-time object detection framework\n- **[BoT-SORT](https://github.com/NirAharon/BoT-SORT)** — state-of-the-art multi-object tracking algorithm\n- **[OpenAI](https://openai.com/)** — GPT models powering the SkyPilot mission orchestration\n- **[OpenCV](https://opencv.org/)** — computer vision and image processing library\n\n## 📧 Contact\n\n**Maintainer:** [@oaslananka](https://github.com/oaslananka)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foaslananka%2Fsky-track-vision-dev","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foaslananka%2Fsky-track-vision-dev","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foaslananka%2Fsky-track-vision-dev/lists"}