{"id":51472624,"url":"https://github.com/machinefi/trio-retina","last_synced_at":"2026-07-06T19:30:32.091Z","repository":{"id":365490496,"uuid":"1270538745","full_name":"machinefi/trio-retina","owner":"machinefi","description":"Model-agnostic state layer for world models — turn any detector (YOLO · VLM · DINO) into one standard, queryable stream of events + latent state. numpy-only, runs on CPU at the edge.","archived":false,"fork":false,"pushed_at":"2026-06-17T16:44:56.000Z","size":17855,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-17T16:49:19.788Z","etag":null,"topics":["computer-vision","edge-ai","event-stream","object-detection","object-tracking","perception","python","rtsp","video-analytics","vlm","world-models","yolo"],"latest_commit_sha":null,"homepage":"https://machinefi.github.io/trio-retina/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/machinefi.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-15T20:13:33.000Z","updated_at":"2026-06-17T16:45:57.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/machinefi/trio-retina","commit_stats":null,"previous_names":["machinefi/trio-retina"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/machinefi/trio-retina","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/machinefi%2Ftrio-retina","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/machinefi%2Ftrio-retina/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/machinefi%2Ftrio-retina/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/machinefi%2Ftrio-retina/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/machinefi","download_url":"https://codeload.github.com/machinefi/trio-retina/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/machinefi%2Ftrio-retina/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35204409,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-06T02:00:07.184Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","edge-ai","event-stream","object-detection","object-tracking","perception","python","rtsp","video-analytics","vlm","world-models","yolo"],"created_at":"2026-07-06T19:30:27.064Z","updated_at":"2026-07-06T19:30:32.054Z","avatar_url":"https://github.com/machinefi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# Trio Retina\n\n\u003cimg src=\"https://raw.githubusercontent.com/machinefi/trio-retina/main/media/stack.png\" width=\"840\" alt=\"The world-model stack: perception backbones (YOLO, DINOv2, V-JEPA 2, SAM, VLMs) feed Trio Retina — the encoder and standardized WorldState — which world models for dynamics and control build on top of\"\u003e\n\n**The state layer of the world-model stack** — bring any perception model on top, get one standard, model-agnostic `WorldState`, build any dynamics underneath. Swap the model or the dynamics; **Retina is the constant in the middle.**\n\n[![CI](https://github.com/machinefi/trio-retina/actions/workflows/ci.yml/badge.svg)](https://github.com/machinefi/trio-retina/actions/workflows/ci.yml) [![PyPI](https://img.shields.io/pypi/v/trio-retina.svg)](https://pypi.org/project/trio-retina/) [![Docs](https://img.shields.io/badge/docs-live-brightgreen.svg)](https://machinefi.github.io/trio-retina/) [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE) [![Python](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/)\n\n**[docs](https://machinefi.github.io/trio-retina/)** · **[quickstart](#-quickstart)** · **[the world-model stack](#-the-world-model-stack)** · **[benchmark](BENCHMARK.md)** · **[examples](examples/)** · **[notebooks](notebooks/)**\n\n\u003c/div\u003e\n\n*A lightweight, model-agnostic **computer-vision pipeline** for **object detection \u0026 tracking** that emits structured **events** — zone intrusion, line-crossing, dwell, people-counting — from **YOLO**, **VLM**, or **Grounding DINO** detectors over video, files, or **RTSP**. Runs on CPU at the **edge**; feeds **digital twins**, dynamics models, and LLMs.*\n\n\u003e Just want camera events (zone intrusion, line-crossing) pushed to a webhook? → jump to the [5-line quickstart](#-quickstart), or copy [`examples/rtsp_to_webhook.py`](examples/rtsp_to_webhook.py).\n\n## 👋 hello\n\n**Trio Retina** (Retina for short) turns raw signals — video, sensor — into a **queryable world-state**: readable **events** (`zone.enter`, `dwell`, `line.cross`) *plus* a standardized **latent** `vec` channel on the same records, on one small model-agnostic standard. The latent channel is a real, serializable interface (attach your own embedding — see [`examples/latent_vec.py`](examples/latent_vec.py)), and the automatic *producers* now ship: `DinoV2Embedder` fills per-object `entity.vec` and `VJepa2Embedder` fills the scene latent `ws.scene`. Bring any model (YOLO, V-JEPA, DINO, a VLM, or none); Retina assembles its output into state a dynamics model, rule engine, or LLM can consume — and a small example dynamics model [imagines the future off that state](#-the-world-model-stack).\n\nThink **OpenTelemetry for perception** — it doesn't build the sensors, it normalizes any of them into one state. In world-model terms it's the **encoder** (`s = Enc(x)`), and *only* the encoder; dynamics and policy build on top. Retina isn't trying to win a vertical — driving, games, and robotics each already have their own stack — it's the **neutral state standard those structured, multi-sensor world models can share**. → see [`DESIGN.md`](DESIGN.md).\n\n## 💻 install\n\n```bash\npip install trio-retina            # core: numpy only\npip install 'trio-retina[yolo]'    # + Ultralytics YOLO adapter\npip install 'trio-retina[video]'   # + OpenCV frame source (files / RTSP / webcam)\npip install 'trio-retina[all]'     # everything\n```\n\nThen try the CLI — `retina demo` runs a synthetic dock scene and prints the `retina.event` stream (numpy only, no model/GPU/video):\n\n```bash\npip install trio-retina \u0026\u0026 retina demo        # instant event stream, no model\nretina validate events.jsonl                  # check a JSONL stream against the spec\nretina --version                              # also: retina run / retina bench\n```\n\n## 🔥 quickstart\n\nRuns on a bare `pip install trio-retina` (numpy only) — no model, no GPU, no video file. A stand-in detector walks one \"person\" across a dock zone; Retina emits the real `retina.event` stream:\n\n```python\nimport numpy as np\n\nfrom retina import CountRule, IoUTracker, Retina, Zone, ZoneRule\nfrom retina.detect import Detection\n\n\nclass ScriptedDetector:\n    \"\"\"A stand-in model: one 'person' walking across a dock zone.\"\"\"\n\n    def __init__(self):\n        self._xs = list(range(0, 102, 6))\n\n    def __call__(self, frame):\n        x = self._xs.pop(0) if self._xs else 100\n        return [Detection(label=\"person\", bbox=(x - 10, 40, x + 10, 60), confidence=0.9)]\n\n\ndock = Zone(\"dock\", [(40, 0), (60, 0), (60, 100), (40, 100)])\n\ncam = Retina(\n    source_id=\"cam_01\",\n    detector=ScriptedDetector(),\n    tracker=IoUTracker(min_hits=2),\n    rules=[\n        ZoneRule(dock, classes={\"person\"}, dwell_s=2.0),\n        CountRule(1, classes={\"person\"}),\n    ],\n)\n\nframes = [(np.zeros((100, 100, 3), dtype=np.uint8), float(i)) for i in range(18)]\nfor event in cam.run(frames):\n    print(event.to_json())\n    # {\"type\":\"count.threshold\",\"t\":1.0,\"src\":\"cam_01\",\"n\":1,\"frame\":1,...}\n    # {\"type\":\"zone.enter\",\"t\":7.0,\"src\":\"cam_01\",\"id\":1,\"label\":\"person\",...}\n    # {\"type\":\"zone.dwell\",\"t\":7.0,...,\"zone\":\"dock\",\"dur\":2.0,...}\n    # {\"type\":\"zone.exit\",\"t\":7.0,...,\"zone\":\"dock\",\"dur\":3.0,...}\n```\n\n**▶ with a real model + video** — `pip install 'trio-retina[yolo]'` (add `[video]` for the frame source), then point it at your clip:\n\n```python\nfrom retina import Retina, Zone, ZoneRule, YoloDetector\nfrom retina.sources import video_frames\n\ndock = Zone(\"dock\", [(0.3, 0.2), (0.7, 0.2), (0.7, 0.9), (0.3, 0.9)], normalized=True)\n\ncam = Retina(\n    source_id=\"cam_01\",\n    detector=YoloDetector(\"yolo11n.pt\", classes={\"person\"}),\n    rules=[ZoneRule(dock, classes={\"person\"}, dwell_s=30)],\n)\nfor event in cam.run(video_frames(\"your.mp4\")):\n    print(event.to_json())\n    # {\"type\":\"zone.dwell\",\"t\":1718254799.8,\"src\":\"cam_01\",\"id\":42,\n    #  \"label\":\"person\",\"zone\":\"dock\",\"dur\":31.0,\"conf\":0.91}\n```\n\nMore no-model examples ship with the source (not the wheel) — `git clone` the repo and run `python examples/quickstart.py` (the forecast / video demos need `[video]` + a clip).\n\n**▶️ Or run it in your browser — no install:**\n\n| notebook | what it shows |\n|---|---|\n| [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/machinefi/trio-retina/blob/main/notebooks/01_quickstart_events.ipynb) | **quickstart** — detector → `zone` / `line` / `count` / `dwell` events + `validate()` |\n| [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/machinefi/trio-retina/blob/main/notebooks/02_camera_to_webhook.ipynb) | **camera → webhook** — a restricted-zone alert pushed to your endpoint |\n| [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/machinefi/trio-retina/blob/main/notebooks/03_from_supervision.ipynb) | **from Supervision** — pipe your existing `sv.Detections` straight in |\n\n### compose models with `|`\n\nWire models like n8n / LangChain, no GUI. Add a cheap gate and a VLM enricher anywhere in the chain:\n\n```python\nfrom retina import MotionGate, GateNode, YoloDetector, IoUTracker, EnricherNode, ZoneRule, JsonlSink\n\npipe = (\n    GateNode(MotionGate())                 # skip static frames (cut model calls)\n    | YoloDetector(\"yolo11n.pt\", classes={\"person\", \"forklift\"})\n    | IoUTracker()\n    | EnricherNode(my_vlm_describe)        # attach a VLM read to frame.user\n    | ZoneRule(dock, dwell_s=30)\n    | JsonlSink(\"events.jsonl\")\n)\n```\n\n\u003cdetails\u003e\n\u003csummary\u003eTwo more ways to wire it (explicit list · declarative JSON) + the node catalog\u003c/summary\u003e\n\n```python\n# explicit node list\nfrom retina import Pipeline, DetectorNode, TrackerNode, RuleNode\npipe = Pipeline([DetectorNode(yolo), TrackerNode(), RuleNode(ZoneRule(dock))])\n\n# declarative workflow file (shareable, no code)\npipe = Pipeline.from_json(\"workflow.json\")   # see examples/workflow.json\n```\n\n| node | what it does | wraps |\n|---|---|---|\n| `DetectorNode` | image → detections | any `callable(image)-\u003e[Detection]` |\n| `TrackerNode` | detections → tracks | `IoUTracker` / `NorfairTracker` |\n| `RuleNode` | tracks → events | `ZoneRule` / `LineRule` / `CountRule` |\n| `GateNode` | drop uninteresting frames | any `callable(image,t)-\u003ebool` (e.g. `MotionGate`) |\n| `EnricherNode` | attach context to `frame.user` | any `callable(frame)-\u003edict` (VLM / V-JEPA) |\n| `SinkNode` | emit events | `JsonlSink` / `WebhookSink` |\n\nRegister your own for `from_json` with `register_node(\"my_type\", builder)`.\n\u003c/details\u003e\n\n## 🎛️ supported models\n\nRetina imports no model — **any** detector plugs in, and out comes one standard event stream. That seam *is* the point:\n\n| plug in any detector… | → | …out comes one `retina.event` stream |\n|---|:---:|---|\n| **YOLO** (Ultralytics: v5–v12, RT-DETR) | → | `{\"type\":\"zone.enter\", \"id\":42, \"label\":\"person\", …}` |\n| **any VLM** (GPT-4o · Qwen-VL · Gemini · Claude) | → | `{\"type\":\"line.cross\", \"dir\":\"a_to_b\", …}` |\n| **Grounding DINO** (open-vocab, no training) | → | `{\"type\":\"zone.dwell\", \"dur\":31.0, …}` |\n| your existing **`sv.Detections`** (Supervision) | → | `{\"type\":\"count.threshold\", \"n\":12, …}` |\n| any **`callable(image) -\u003e [Detection]`** | → | …+ an optional latent `vec` on the same record |\n\nSupervision gives you boxes on a screen; Retina turns *any* of those into a serializable state + event stream the next layer (dynamics, twin, agent) can consume. Batteries-included adapters:\n\n- **YOLO family** — `YoloDetector(\"\u003cweights\u003e.pt\")` (Ultralytics): YOLOv5/8/9/10/11/12, RT-DETR. Open-vocab via YOLO-World.\n- **Open-vocab from text** — `GroundingDinoDetector([\"forklift\", \"hard hat\"])`, no training.\n- **Any VLM** — `VlmDetector(client, prompt)` (Qwen-VL / Gemini / GPT-4o / Claude / local), as a detector or an event-source enricher.\n- **Supervision interop** — `Detection.from_supervision(sv_detections)` ingests a Roboflow `sv.Detections`, so anything that already converts to Supervision pipes straight into Retina's event layer.\n- **Latent producers (shipped)** — `DinoV2Embedder()` fills per-object `entity.vec` (frozen DINOv2, `pip install 'trio-retina[dino]'`); `VJepa2Embedder()` fills the scene latent `ws.scene` from a rolling clip (frozen V-JEPA 2 video encoder, `pip install 'trio-retina[vjepa]'`). Swap either underneath the same fixed state schema — see [`examples/world_model/multi_encoder.py`](examples/world_model/multi_encoder.py).\n\nTrackers are pluggable too: `IoUTracker` (pure-Python default) or `NorfairTracker`.\n\n## 📦 the event format\n\nThe `retina.event` standard is tiny, like a JWT — three required fields, everything else optional and omitted when absent. Full spec in [`SPEC.md`](SPEC.md).\n\n```json\n{\"type\":\"zone.dwell\",\"t\":1718254799.8,\"src\":\"cam_01\",\"id\":42,\"label\":\"person\",\"zone\":\"dock\",\"dur\":31.0}\n```\n\n```python\nfrom retina import validate\nvalidate(event)   # -\u003e [] if valid, else a list of problems  (pure-Python, ships a JSON Schema)\n```\n\n## 🌍 the world-model stack\n\nRetina is the **encoder** (`s = Enc(x)`) in a world model. It doesn't try to win\nany one vertical — driving, games, and robotics each already have their own\nstack; it's the **neutral state layer those structured, multi-sensor world models\nplug into**, in four ways:\n\n- **One contract for many sensors.** A camera embedding, a radar return, an IMU stream, a WiFi CSI latent — heterogeneous encoders all land in the *same* `WorldState`, fused on a model-tagged `vec` and a typed `locus` / `scene`. Retina doesn't build the sensors; it normalizes any of them into one state.\n- **Swap the front or the back, never both.** Encoder and dynamics meet on a frozen state contract — change the perception model without retraining the dynamics, or change the dynamics without touching perception. The seam is the product.\n- **A state you can read, log, and verify.** Small, serializable, half-symbolic — the state doubles as the world model's observability layer: eval it against ground truth, stream it to a digital twin, alert on a `retina.event`, even when the dynamics itself is a black box.\n- **Cheap at the edge.** The encode-to-state step runs on CPU; the heavy dynamics lives wherever you like. Retina is the lightweight front door that turns raw signals into state before the expensive layer sees them.\n\nThe scope is deliberate — Retina is for world models that **reason over\nstructured, multi-sensor state**, not monolithic pixel-to-pixel video generators.\nWith the latent producers shipped, that seam is now demonstrable end to end — on a\nsynthetic scene, as a small but honest proof of concept ([`examples/world_model/`](examples/world_model/)):\n\n**1 · swap the encoder, the state is constant.** The same pipeline, run three\nways — symbolic-only, `+ DinoV2Embedder` (per-object `entity.vec`), and\n`+ VJepa2Embedder` (scene-level `ws.scene`) — yields the *identical* WorldState\nschema; only which model filled the latent changes. → [`multi_encoder.py`](examples/world_model/multi_encoder.py)\n\n**2 · a dynamics model imagines the future off that state.** A small transformer\ntrained offline on recorded `WorldState` sequences predicts where each entity is\nheaded, and rolls out *imagination* trajectories inside the learned model. The\nhonest ablation — does Retina's appearance latent actually help? — on **held-out**\ndata with **real DINOv2** vecs (Mac Studio, MPS), mean held-out 7-step position\nerror (px, lower is better):\n\n| dynamics input | 7-step error |\n|---|---|\n| constant-velocity baseline | 7.68 px |\n| learned, pos-only | 1.45 px |\n| **learned, pos + appearance latent** | **1.33 px** |\n\n**The latent channel measurably improves prediction: +83% over constant-velocity,\n+8% over pos-only at horizon 7** — and the edge *widens with the horizon*, because\nthat's where local velocity runs out and object *type* (legible only from\nappearance) decides the future. → [`dynamics.py`](examples/world_model/dynamics.py), full grid in [`BENCHMARK.md`](BENCHMARK.md)\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/machinefi/trio-retina/main/media/world_model_soccer.gif\" width=\"840\" alt=\"Left: raw broadcast soccer clip. Middle: a WorldState arrow. Right: a top-down tactical radar where each player is a team-coloured dot with a brand-indigo predicted next run and a faint gray past trail.\"\u003e\u003c/p\u003e\n\n\u003e **Raw video → one standardized Retina `WorldState` → predicted player runs.** Left is a real broadcast clip (Roboflow's MIT-licensed [`sports`](https://github.com/roboflow/sports) sample, originally DFL Bundesliga). It goes through a real YOLO detector + tracker and a frozen DINOv2-small appearance encoder, and comes out as one model-agnostic `WorldState`; the right panel renders that state as a **top-down tactical radar**, and the small dynamics transformer — trained offline on those sequences — draws each player's **predicted next run** ahead in brand indigo (faint gray = where they came from). Teams are coloured by clustering the players' DINOv2 appearance vectors into two groups — the latent knows who's who. The radar is a stylized perspective top-down (no Roboflow pitch-keypoint weights on this host, so a fixed homography from the clip's pitch landmarks, not per-frame). Honest by design: player motion is stochastic, so at this short horizon the learned model roughly *ties* a constant-velocity baseline on held-out error — the appearance latent's *measurable* win lives in the cleaner synthetic ablation above, not on free-running humans. Real pipeline, end to end — [`examples/world_model/soccer/`](examples/world_model/soccer/). The synthetic car rollout (held-out, where the latent earns its keep) lives in [`make_demo_gif.py`](examples/world_model/make_demo_gif.py) · [`media/rollout.png`](examples/world_model/media/rollout.png).\n\n**3 · the state layer is signal-agnostic — the same schema carries a WiFi CSI world model.** The `WorldState`/`Vec` that carried DINOv2 *appearance* above also carries a **WiFi CSI channel latent**, with **zero core schema change** to express an RF world model: the global channel latent drops into `ws.scene`, and the subject's metric room position rides the typed `Entity.locus` (metres, distinct from the pixel `bbox`). On **synthetic CSI** (a documented forward model, fully offline) we reproduce the *recipe* of two CSI world-model papers — a JEPA that predicts the next latent ([arXiv:2409.10045](https://arxiv.org/abs/2409.10045)) under an action-conditioned homomorphic transition ([arXiv:2603.20048](https://arxiv.org/abs/2603.20048)) — and route it through Retina's state layer. Real numbers from one run (seed 0, CPU): action-conditioning improves next-latent prediction **+72.9%** over an action-blind ablation; the latent self-organizes into a **metric room map (~0.33 m** probe error in a 6×5 m room); and the imagination rollout **beats a constant-velocity baseline by ~22%** over a 14-step horizon, with the edge *growing* past ~7 steps. Honest scope: those papers are channel/comms-side world models trained on **real** CSI; **our** contribution is showing the *recipe runs through a signal-agnostic state layer* — this is a synthetic-CSI proof of concept, **not** a real-WiFi result. → [`examples/world_model/csi/`](examples/world_model/csi/)\n\n**4 · front + back compose through one standard.** Any encoder in front, any\ndynamics behind, meeting on one serializable state — a pip-installable world-model\nseam you can run in one script. → [`end_to_end.py`](examples/world_model/end_to_end.py)\n\n```bash\npip install 'trio-retina[dynamics,dino]'\npython examples/world_model/dataset.py --n 12 --len 24 --out examples/world_model/data/sequences.json\npython examples/world_model/end_to_end.py   # encoder → WorldState → dynamics → imagined rollout\n```\n\n\u003e Honest scope: a synthetic scene, a tiny model, reproducible on MPS with\n\u003e run-to-run variance. The *producers* ship; the *trained dynamics* is a small\n\u003e example, not a product. The point is the seam — that front and back compose\n\u003e through one standardized state.\n\n## 🎬 demos\n\nTwo more ways Retina's state feeds the next layer — same standard, different consumers:\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003ctd width=\"50%\" valign=\"top\"\u003e\n\n**Forecast — the dynamics layer on Retina**\n\n\u003cimg src=\"https://raw.githubusercontent.com/machinefi/trio-retina/main/media/retina_demo.gif\" width=\"100%\" alt=\"Trio Retina: YOLO object tracking with two dynamics models forecasting entity trajectories from one world-state — gray constant-velocity baseline vs magenta learned model\"\u003e\n\nOne world-state → **two dynamics models forecast** where each entity is headed off the *same* state (gray = constant-velocity, magenta = learned). A dynamics model eats structured **state**, not pixels. → [`examples/forecast/`](examples/forecast/)\n\n\u003c/td\u003e\n\u003ctd width=\"50%\" valign=\"top\"\u003e\n\n**iTwin.js — a live, predictive digital twin**\n\n\u003cimg src=\"https://raw.githubusercontent.com/machinefi/trio-retina/main/examples/itwin/media/retina_itwin_demo.gif\" width=\"100%\" alt=\"Trio Retina perception events and forecast arrows rendered live on a Bentley iTwin.js digital twin (Baytown plant)\"\u003e\n\nRetina's entities, forecast arrows, and `retina.event` alerts on a real Bentley **iTwin.js** iModel (Baytown), one neutral JSON contract, fully headless — it gives the twin *live eyes*. → [`examples/itwin/`](examples/itwin/)\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eAll examples\u003c/summary\u003e\n\nThe examples live in this repo (not in the installed wheel) — `git clone` to run them. The top-level quickstarts run with **no model and no GPU** (synthetic detections):\n\n```bash\npython examples/quickstart.py          # zone / line / count / dwell events\npython examples/three_apps.py          # one stream -\u003e security, retail, safety\npython examples/any_model.py           # swap the detector, rest unchanged\npython examples/gate_savings.py        # a cheap gate cuts detector calls 100 -\u003e 23\npython examples/pipeline_compose.py    # compose with | (n8n without a GUI)\npython examples/rtsp_to_webhook.py     # camera -\u003e restricted-zone alert -\u003e webhook\npython examples/from_supervision.py    # ingest a Roboflow sv.Detections pipeline\npython examples/latent_vec.py          # populate the latent vec channel by hand\npython examples/dino_embeddings.py     # REAL DINOv2 per-object vecs (needs [dino])\n```\n\nReal-footage / dynamics demos need a clip and the extras — `pip install 'trio-retina[all]'`:\n\n```bash\npython examples/yolo_video.py v.mp4    # YOLO on a video file\nexamples/forecast/                     # dynamics layer on the WorldState stream (needs [video] + a clip)\nexamples/itwin/                        # events + forecasts on a Bentley iTwin.js iModel\n```\n\nThe **world-model stack** lives in [`examples/world_model/`](examples/world_model/) (needs `[dynamics]`, plus `[dino]`/`[vjepa]` for real encoders):\n\n```bash\npython examples/world_model/multi_encoder.py   # swap encoder, state schema stays constant\npython examples/world_model/dynamics.py        # train + the honest appearance ablation\npython examples/world_model/benchmark.py       # the front/back-end benchmark grid → BENCHMARK.md\npython examples/world_model/end_to_end.py      # encoder → WorldState → dynamics → imagined rollout\n```\n\u003c/details\u003e\n\n**Send events anywhere.** `WebhookSink(url)` POSTs each event as JSON (stdlib urllib, no `requests`); `JsonlSink(path)` streams to a file. For a live camera, `video_frames(src, live=True)` reads RTSP / HLS / webcam with wall-clock timestamps — see [`examples/rtsp_to_webhook.py`](examples/rtsp_to_webhook.py).\n\n## 🎯 use cases\n\nOne state layer, many domains — the *same* `retina.event` stream, read differently above:\n\n- **Security \u0026 intrusion detection** — `zone.enter` / `line.cross` on cameras and RTSP feeds.\n- **Retail analytics \u0026 people-counting** — footfall, queue dwell, zone occupancy from any detector.\n- **Workplace safety** — PPE, forklift, and restricted-zone alerts via open-vocab detectors.\n- **Smart city \u0026 traffic monitoring** — vehicle/pedestrian counting and crossings at the edge.\n- **Industrial digital twins** — feed live entities + forecasts into a twin ([iTwin.js demo](examples/itwin/)).\n\n## 🧠 how it works\n\nEverything flows through one append-only data unit, the **`Frame`**. Each stage *enriches* it and never overwrites upstream fields:\n\n```\n                      ┌──────────────── Frame (append-only) ───────────────┐\n frame ─► Detector ─► │ .detections ─► Tracker ─► .tracks ─► Rule ─► .events │ ─► Sink\n   ▲        ▲         │                  ▲                    ▲              │     ▲\n source   any model   │   Gate (skip?)   tracker     zone/line/count/dwell  │  jsonl/\n                      │   Enricher (VLM / V-JEPA → .user)                    │  webhook\n                      └─────────────────────────────────────────────────────┘\n```\n\n- The **detector** is the model-agnostic seam: any `callable(image) -\u003e [Detection]`.\n- The **tracker** gives objects identity over time; **rules** turn tracks into **events**; **enrichers** attach context; **gates** skip work; **sinks** push out.\n- Output is **dual**: a readable symbolic stream *and* an optional model-tagged latent channel — never collapsed.\n\n\u003cdetails\u003e\n\u003csummary\u003eWhy \"encoder\", the dual state, and how it compares to DeepStream / Supervision\u003c/summary\u003e\n\n**Two senses of \"encoder.\"** Foundation backbones (V-JEPA, DINO, SAM, YOLO) turn pixels into features — that race is theirs, and Retina rides it. Retina is the encoder *layer* on top: it **fuses** many models into one record, gives objects **persistent identity**, **structures** it into entities + relations + events, carries the **dual** symbolic + latent channels, as an **event-sourced stream** — one small, serializable, model-agnostic standard.\n\n**Dual state.** The same entities on two linked channels: *symbolic* (readable `events` / entity records, for rules / LLMs / dashboards) and *latent* (optional model-tagged embeddings, for a downstream dynamics model). Symbols you can read; vectors a model can predict on. The latent channel is a standardized, serializable interface — populate `entity.vec` with your own embedding ([`examples/latent_vec.py`](examples/latent_vec.py)), or let a built-in producer fill it: `DinoV2Embedder` (per-object) and `VJepa2Embedder` (scene-level) both ship today.\n\n**vs DeepStream / Holoscan** — same good ideas (event semantics, metadata model, composable graph), none of the weight:\n\n| | DeepStream / Holoscan | **Retina** |\n|---|---|---|\n| Install | CUDA + TensorRT + containers | `pip install trio-retina` |\n| Hardware | NVIDIA / Jetson locked | any machine — CPU is fine |\n| Model | tied to the NV stack | **bring any model** (or none) |\n| Shape | a platform you build *inside* | a library you `import` |\n| Core deps | a lot | **numpy** |\n\n**vs Supervision** — Supervision turns a model's output into detections + overlays (great toolbox, ends at the screen). Retina is a level up: it emits a serializable **state + event stream** that the *next* layer (dynamics, twin, agent) consumes. We compose Supervision / detectors, not compete with them.\n\nFull rationale, references, and the world-model stack: [`DESIGN.md`](DESIGN.md).\n\u003c/details\u003e\n\n## 🗺️ roadmap\n\nEarly but real (`v0.3.0`). Stable: the event layer + JSON Schema/validator, the composable pipeline (`|` / list / JSON), YOLO + open-vocab + VLM detectors (plus `from_supervision` interop), IoU + Norfair trackers, and jitter-robust rules (`exit_grace_s` · `anchor` · `min_frames`).\n\nNext: ByteTrack / OC-SORT · `proximity` / `anomaly` events · VLM-as-event-source · Kafka / MQTT sinks · **more encoders** behind the latent channel · **model-based RL / latent-rollout imagination** on the learned state · growing the [front/back-end benchmark](BENCHMARK.md). See [`CHANGELOG.md`](CHANGELOG.md).\n\nRetina is the open **perception encoder** extracted from [Trio](https://machinefi.com); the layers above (dynamics, policy / judgment) are Trio's commercial platform. Retina is, and stays, model-agnostic and free.\n\n## 🤝 contributing\n\nContributions that keep Retina small and beautiful are very welcome — see [`CONTRIBUTING.md`](CONTRIBUTING.md) for dev setup and how to add a detector / tracker / rule / sink. By participating you agree to the [Code of Conduct](CODE_OF_CONDUCT.md); to report a vulnerability see [`SECURITY.md`](SECURITY.md).\n\n## license\n\n[Apache-2.0](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmachinefi%2Ftrio-retina","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmachinefi%2Ftrio-retina","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmachinefi%2Ftrio-retina/lists"}