{"id":50781031,"url":"https://github.com/kylefoxaustin/keyhole","last_synced_at":"2026-06-12T03:03:39.753Z","repository":{"id":350524835,"uuid":"1206971533","full_name":"kylefoxaustin/keyhole","owner":"kylefoxaustin","description":null,"archived":false,"fork":false,"pushed_at":"2026-06-11T02:58:47.000Z","size":952,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-11T04:21:23.931Z","etag":null,"topics":["computer-vision","edge-ai","npu","object-detection","python","video-analytics"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kylefoxaustin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-10T12:48:25.000Z","updated_at":"2026-06-11T02:58:51.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/kylefoxaustin/keyhole","commit_stats":null,"previous_names":["kylefoxaustin/keyhole"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/kylefoxaustin/keyhole","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kylefoxaustin%2Fkeyhole","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kylefoxaustin%2Fkeyhole/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kylefoxaustin%2Fkeyhole/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kylefoxaustin%2Fkeyhole/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kylefoxaustin","download_url":"https://codeload.github.com/kylefoxaustin/keyhole/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kylefoxaustin%2Fkeyhole/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34226634,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-12T02:00:06.859Z","response_time":109,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","edge-ai","npu","object-detection","python","video-analytics"],"created_at":"2026-06-12T03:03:37.691Z","updated_at":"2026-06-12T03:03:39.747Z","avatar_url":"https://github.com/kylefoxaustin.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Keyhole\n\n**v1.0.0** · Open-source edge AI video analytics platform. Cameras detect; a\nshared NPU enriches detections with vision + open-vocabulary labeling;\noperators query events in natural language.\n\nInspired by the Ubiquiti UniFi AI Key architecture: lightweight detection on\nthe camera, deeper enrichment on a compute node, semantic + literal search\nover the resulting event store.\n\n---\n\n## Headline result\n\n**SAM 3 → Hybrid V2 architectural pivot delivers 515× lower DRAM bandwidth\nper shipping pipeline frame**, measured by Nsight Compute (not projected).\nOn NPU Mid stock LPDDR5X, the recommended pipeline (TRT FP8 YOLO-seg + TRT\nFP8 CLIP @ 1 Hz keyframe debounce) reaches **36 FPS at 720p** — first\nreal-time edge result in the campaign.\n\nThe SAM 3 baseline this replaces was bandwidth-bound everywhere: 119 GB\nDRAM per forward, 0.4 FPS edge, no escape via quantization, prompt\nreduction, or resolution cut. The lever turned out to be architectural\nreplacement, not optimization.\n\n| Stage | What it does | RTX 5090 ms | NPU Mid edge ms | NPU Mid FPS |\n|---|---|---|---|---|\n| SAM 3 baseline (BF16) | open-vocab segmentation, monolithic | ~120 | ~1700 | **0.4** |\n| Hybrid V2 BF16 (YOLO-seg + CLIP) | two-stage, FP-baseline | ~62 | ~890 | ~17 |\n| **Hybrid V2 TRT FP8** + 1 Hz CLIP | shipping recipe | ~28 | ~28 | **36** |\n\n`±15%` sensitivity band on the 36 FPS number reflects the `0.70` bandwidth\nefficiency assumption used to scale from 5090 reference to NPU Mid.\n\n## Three operational modes\n\nThe deployment surface acknowledges three modes (the `keyhole-sizer`\ncompanion app surfaces them as a UI toggle):\n\n1. **Vision-only** — vision pipeline runs; LLM off. Default video-analytics\n   deployment.\n2. **Vision + LLM** — both running on a shared NPU. The LLM acts on vision\n   data (NLQ over the event store, agentic scene queries). The engineering\n   question is NPU coexistence — duty cycle, bandwidth share, multi-stream.\n3. **LLM-only** — vision pipeline off; LLM standalone. Equivalent to the\n   Skippy product running inside `keyhole-sizer`. Perf matches the Skippy\n   deck exactly.\n\n## LLM identity\n\nThe LLM in Keyhole is the **Skippy product artifact, unmodified** —\nQwen3-30B-A3B base, Q4_K_M quantization, identical shipping recipe. Keyhole\n*uses* the artifact; the training story (recipe taxonomy, fine-tuning\ncampaign coverage, headline-erosion methodology arc, cross-family\nbase-selection) is documented in the [Skippy / personal-AI-framework\ndeck](https://github.com/kylefoxaustin/personal-ai-framework) and is not\nreproduced here.\n\n## NPU tier model (canonical)\n\nPAI deck slide 11 is the golden source for NPU tier specs; both\n`keyhole-sizer` and the Keyhole deck pull from the same model.\n\n| Tier | Memory bus | BW theo | BW eff (70%) | TOPS | Note |\n|---|---|---|---|---|---|\n| NPU Low-LP5-32bit / i.MX 95 | 32-bit LPDDR5 @ 6.4 GT/s | 25.6 | 17.92 | 2 INT8 | Neutron-class |\n| NPU Low-LP5-64bit | 64-bit LPDDR5 @ 6.4 GT/s | 51.2 | 35.84 | 2 INT8 | INT8-only |\n| NPU Low-LP5X | 64-bit LPDDR5X @ 8.4 GT/s | 67.2 | 47.04 | 50 / 100 / 100 BF/INT8/FP8 | FP-capable lite |\n| **NPU Mid** | 128-bit LPDDR5X @ 8.4 GT/s | 134.4 | 94.08 | **200 INT8 (no FP)** | Recommended target — INT8-only |\n| **NPU High** | **same bus as Mid** | 134.4 | 94.08 | 200 / 400 / 400 BF/INT8/FP8 | FP-capable — full Hybrid V2 |\n| RTX 5090 (reference) | 512-bit GDDR7 @ 28 GT/s | 1792 | 1523.2 (85%) | ~105 / ~210 / DP4A | Measurement anchor |\n\nMid is INT8-only; High differentiates on compute + capacity + TDP, **not\nbandwidth** (shared 8.4 GT/s class). FP recipes (CLIP, ViT alternatives,\nEfficientSAM3 variants) pin to NPU High. Memory upgrades (LPDDR5T-11.2,\nLPDDR6) lift decode on both tiers in lockstep.\n\n## What's in this release\n\n### Bake-off catalog\n\n- **TensorRT YOLO-seg** at INT8 + FP8 on Blackwell — full-model Conv quant\n  via TRT 10.16 (torchao's 1×128 block-size constraint had blocked this).\n  Recall = 1.000, matched IoU = 0.998 vs FP16 engine.\n- **TensorRT CLIP visual tower** at FP16 + FP8 — 3× speedup, top-1\n  concept-tag agreement 0.964 vs BF16.\n- **Mask-model bake-off** — MobileSAM, EfficientSAM-tiny/small, YOLO-seg\n  scored against SAM 3 reference; YOLO-seg wins on combined detection +\n  segmentation footprint.\n- **EfficientSAM3 community variants** (Apr 2026) — bench against the\n  shipping pipeline; community SAM 3 Lite is real but ~13× slower than the\n  recommended Hybrid V2 stack at edge.\n- **YOLOE-26 one-model open-vocab** — alternative to Hybrid V2; TRT FP8\n  gives ~17% speedup vs 3× on YOLO-seg (kernel-launch-bound at small\n  parameter count). Real but doesn't displace the two-stage pipeline.\n- **ViT alternatives** (RT-DETR-L, DETR-ResNet50, OWLv2, Grounding DINO) —\n  what-if analysis; camera-side ViTs don't fit NPU Mid stock memory.\n  OWLv2 fits a 1 Hz agentic-query budget.\n\n### Measurement validation\n\n- **Nsight Compute pipeline** — measured DRAM per forward for 23 workloads;\n  log-scale chart + full table in the deck. The 515× SAM 3 → Hybrid V2 win\n  is measured (118,975 MB / 231 MB), not projected.\n- **End-to-end pipeline latency budget** — every stage (FFmpeg decode,\n  preprocess, YOLO, CLIP, SQLite INSERT) on 5090 reference + projected to\n  NPU Mid. On pure-NPU edge boards (no ISP / 2D-GPU offload), CPU stages\n  crowd out the 36 FPS budget; production SoCs with offloads sustain it.\n- **i.MX 95 anchor** — NXP eIQ Neutron NPU measurement on yolov8n-seg INT8\n  @ 1080p (29.2 FPS measured vs 18.3 FPS BW-projected — 1.6× delta\n  documents pure-BW projection limits on weak silicon).\n\n### Companion artifacts\n\n- **Plain deck** (63 slides, dark-bg) and **NXP-branded deck** (63 slides,\n  corporate template via `pptx_template_converter` theme swap).\n- **`PRESENTER_SCRIPT.md`** — speaker text for 45–60 min technical-\n  management walkthrough.\n- **`docs/CHANGES_2026-05-17.md`** — reviewer-orientation doc.\n- **`docs/ALIGNMENT_PLAN.md`** — conceptual-frame alignment history;\n  ALL PHASES DONE.\n- **`docs/PRIVATE_DECK.md`** — operator-discipline reference for the\n  `--include-private` build that surfaces measured silicon anchor values\n  from a gitignored `.streamlit/secrets.toml` file.\n\n### Companion repo\n\n- [**`keyhole-sizer`**](https://github.com/kylefoxaustin/keyhole-sizer) —\n  Streamlit app for what-if NPU sizing across all five tiers + the three\n  operational modes. Self-contained; pulls from the same NPU model the\n  deck does.\n\n---\n\n## Quick Start\n\n### Prerequisites\n\n- NVIDIA GPU with ≥16 GB VRAM (RTX 5090 reference)\n- Python 3.10+ (3.12+ recommended for SAM 3 / TRT 10.16)\n- PyTorch 2.7+ with CUDA support\n- FFmpeg\n- HuggingFace account with SAM 3 access (only for SAM 3 baseline pipelines)\n- TensorRT 10.16 (Blackwell) for FP8 + INT8 compile path\n\n### Installation\n\n```bash\ngit clone https://github.com/kylefoxaustin/keyhole.git\ncd keyhole\n\npython -m venv .venv \u0026\u0026 source .venv/bin/activate\npip install -r requirements.txt\n\n# SAM 3 (only needed for the baseline)\nmkdir -p third_party \u0026\u0026 cd third_party\ngit clone https://github.com/facebookresearch/sam3.git\ncd sam3 \u0026\u0026 pip install -e \".[notebooks]\"\npip install einops ninja\npip install flash-attn-3 --no-deps --index-url https://download.pytorch.org/whl/cu128\ncd ../..\n\n# Lightweight SAM + CLIP\npip install git+https://github.com/ChaoningZhang/MobileSAM.git\npip install open-clip-torch\nmkdir -p weights\nwget -O weights/mobile_sam.pt \\\n  https://github.com/ChaoningZhang/MobileSAM/raw/master/weights/mobile_sam.pt\n\ncp .env.example .env\nhf auth login\n```\n\n### Running\n\n```bash\n# SAM 3 baseline (the slow one — for reference / re-measurement)\npython -m src.main process --video data/videos/clip.mp4 --single-pass --render --profile\n\n# Hybrid V2 BF16 (the architectural pivot, pre-TRT)\npython -m src.main process --video data/videos/clip.mp4 \\\n  --hybrid-v2 yolo11n-seg.pt --render --profile\n\n# Bake-offs\npython scripts/bakeoff_trt_yolo.py        # TRT YOLO INT8 + FP8 full-model\npython scripts/bakeoff_trt_clip.py        # TRT CLIP visual tower FP16 + FP8\npython scripts/bakeoff_llm_anchors.py     # 5090 LLM anchor catalog\npython scripts/profile_ncu.py             # Nsight Compute DRAM measurements\n\n# Edge projection + comparison\npython -m src.main emulate --compare-all\npython scripts/compare_models.py --video data/videos/clip.mp4 --max-frames 10\n\n# Decks\npython scripts/build_deck.py                       # plain (dark-bg)\nKEYHOLE_DECK_MERGE_TARGET=1 python scripts/build_deck.py  # merge-ready (for branded conversion)\n\n# HTTP API server (for keyhole-UI or any HTTP client)\npython -m src.main serve\n```\n\n### Branded deck rebuild (Phase E workflow)\n\n```bash\n# 1. Build merge-ready plain deck\nKEYHOLE_DECK_MERGE_TARGET=1 python scripts/build_deck.py\ncp data/output/keyhole_results.pptx \\\n  ~/Documents/GitHub/pptx_template_converter/input/keyhole_merge_ready.pptx\n\n# 2. Apply NXP corporate template + color map\ncd ~/Documents/GitHub/pptx_template_converter\npython convert.py \\\n  --input    input/keyhole_merge_ready.pptx \\\n  --template template/corporate_template.pptx \\\n  --output   output/keyhole_deck_branded.pptx \\\n  --color-map mappings/keyhole_to_corporate.json\n\n# 3. Copy back + restore plain (dark) variant\ncp output/keyhole_deck_branded.pptx \\\n  ~/Documents/GitHub/keyhole/data/output/keyhole_deck_branded.pptx\ncd ~/Documents/GitHub/keyhole\npython scripts/build_deck.py\n```\n\n`scripts/update_branded_deck.py` is obsolete — was an in-place patcher for\na static 58-slide variant; superseded by the full-rebuild flow above.\n\n---\n\n## HTTP API\n\nKeyhole exposes a FastAPI server (default `http://localhost:8777`) consumed\nby the [keyhole-UI](https://github.com/kylefoxaustin/keyhole-UI) Next.js\nfrontend but usable by any HTTP client. Start it with `python -m src.main\nserve`.\n\nKey endpoints (full contract in [`API.md`](./API.md)):\n\n| Endpoint | Purpose |\n|---|---|\n| `GET /api/health` | Server + GPU status |\n| `GET /api/videos` | List videos with per-item status |\n| `POST /api/videos` | Upload a video |\n| `GET /api/videos/{id}/thumbnail` | First-frame JPEG |\n| `GET /api/videos/{id}/annotated` | Annotated MP4 stream |\n| `GET /api/videos/{id}/density?buckets=N` | Timeline heatmap |\n| `DELETE /api/videos/{id}` | Remove a video and its events |\n| `GET /api/events?q=...` | Hybrid semantic + literal search |\n| `GET /api/events/{id}/frame` | Annotated frame JPEG |\n| `GET /api/events/{id}/clip?before=5\u0026after=5\u0026format=mp4\\|gif` | Event clip |\n| `GET /api/concepts`, `GET /api/classes` | Autocomplete vocabularies |\n| `WS /api/ws/processing` | Live processing-worker events |\n\nOpenAPI schema at `/api/openapi.json`. Uploads are processed by a single\nasync worker; the WS emits `queue_position` for client UIs.\n\n---\n\n## Project Structure\n\n```\nkeyhole/\n├── src/\n│   ├── main.py              # CLI entry point\n│   ├── ingest/video.py      # FFmpeg frame extraction\n│   ├── detect/\n│   │   ├── yolo.py          # YOLO 11 detection\n│   │   ├── sam3_detect.py   # SAM 3 single-pass (baseline)\n│   │   └── hybrid_v2.py     # YOLO-seg + CLIP (recommended)\n│   ├── enrich/sam3.py       # SAM 3 concept enrichment (sequential reference)\n│   ├── render/video.py      # Annotated video + GIF output\n│   ├── emulate/\n│   │   ├── npu_emulator.py  # Bandwidth-aware edge projection\n│   │   ├── sam3_reference.py\n│   │   └── layer_profiler.py\n│   ├── anchors/             # Private NPU + CNN measured-silicon anchor loaders\n│   ├── store/               # SQLite + SQLAlchemy metadata store\n│   ├── query/nlq.py         # LLM-backed natural language query\n│   └── api/server.py        # FastAPI + web UI\n├── scripts/\n│   ├── build_deck.py        # 63-slide deck generator (KEYHOLE_DECK_MERGE_TARGET=1 for branded)\n│   ├── bakeoff_*.py         # Bake-off harnesses (TRT YOLO, TRT CLIP, LLM anchors, ncu, ...)\n│   ├── profile_ncu.py       # Nsight Compute pipeline\n│   └── compare_models.py    # Multi-model benchmark\n├── docs/\n│   ├── PRESENTER_SCRIPT.md  # 45–60 min walkthrough script\n│   ├── CHANGES_2026-05-17.md\n│   ├── ALIGNMENT_PLAN.md    # Conceptual-frame alignment history\n│   └── PRIVATE_DECK.md      # --include-private operator discipline\n├── data/output/             # Decks + bake-off artifacts (gitignored)\n└── third_party/             # SAM 3 source install (gitignored)\n```\n\n---\n\n## Anchor secrets discipline\n\nMeasured silicon anchor values (NPU Mid/High decode rates, CNN ms/forward)\nlive in a gitignored `.streamlit/secrets.toml` file Kyle populates locally.\nThe `--include-private` build flag surfaces them on an extra deck slide at\nruntime; values are **never** committed to source, logged, surfaced in\nexception messages, or quoted on the cross-session bus. Refer by KEY, not\nVALUE. See [`docs/PRIVATE_DECK.md`](./docs/PRIVATE_DECK.md) for the full\noperator-discipline reference.\n\nThe `keyhole_results_PRIVATE.pptx` output of `--include-private` is\ngitignored and must not be pushed to public surfaces (my-stuff,\ngdrive:skippy_files). NXP-internal-only destinations.\n\n---\n\n## Version history\n\n| Version | Date | Notes |\n|---|---|---|\n| **v1.0.0** | 2026-05-18 | First tagged release. Recovery point ahead of cross-repo engine-extraction work. 63-slide deck (plain + NXP-branded). PAI golden NPU tier framing held. Skippy training content removed (cross-referenced in deck slide 48). Three-operational-modes framing landed (deck slide 6). Phase E branded rebuild via `pptx_template_converter`. Reciprocal cross-reference live with personal-ai-framework deck slide 18. |\n\n---\n\n## Companion / cross-repo links\n\n- [**`personal-ai-framework`**](https://github.com/kylefoxaustin/personal-ai-framework) —\n  Skippy product (LLM artifact + training story). The Keyhole LLM layer is\n  this project's `v4` Qwen3-30B-A3B Q4_K_M unmodified. Cross-reference slide:\n  PAI deck slide 18 ↔ Keyhole deck slide 48.\n- [**`keyhole-sizer`**](https://github.com/kylefoxaustin/keyhole-sizer) —\n  Streamlit what-if sizer for the three operational modes across all five\n  NPU tiers.\n- [**`personal-ai-assistant-sizer`**](https://github.com/kylefoxaustin/personal-ai-assistant-sizer) —\n  Sister sizer focused on the LLM-only mode (Skippy product deployments).\n- [**`pptx_template_converter`**](https://github.com/kylefoxaustin/pptx_template_converter) —\n  Local theme-swap tool used for the NXP-branded deck rebuild.\n\n---\n\n## Maintainer\n\n**Kyle Fox** ([@kylefoxaustin](https://github.com/kylefoxaustin))\n\n## License\n\nMIT License — see LICENSE file.\n\n## TTA — Trust the Awesomeness\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkylefoxaustin%2Fkeyhole","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkylefoxaustin%2Fkeyhole","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkylefoxaustin%2Fkeyhole/lists"}