{"id":50679136,"url":"https://github.com/openvinotoolkit/physicalai","last_synced_at":"2026-06-08T17:30:42.802Z","repository":{"id":358536232,"uuid":"1159108540","full_name":"openvinotoolkit/physicalai","owner":"openvinotoolkit","description":null,"archived":false,"fork":false,"pushed_at":"2026-06-01T17:02:37.000Z","size":95043,"stargazers_count":12,"open_issues_count":8,"forks_count":5,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-01T17:17:39.425Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openvinotoolkit.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":"SUPPORT.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-16T10:24:57.000Z","updated_at":"2026-05-29T16:45:37.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/openvinotoolkit/physicalai","commit_stats":null,"previous_names":["openvinotoolkit/physicalai"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/openvinotoolkit/physicalai","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openvinotoolkit%2Fphysicalai","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openvinotoolkit%2Fphysicalai/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openvinotoolkit%2Fphysicalai/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openvinotoolkit%2Fphysicalai/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openvinotoolkit","download_url":"https://codeload.github.com/openvinotoolkit/physicalai/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openvinotoolkit%2Fphysicalai/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34073602,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-08T02:00:07.615Z","response_time":111,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-08T17:30:42.042Z","updated_at":"2026-06-08T17:30:42.781Z","avatar_url":"https://github.com/openvinotoolkit.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/openvinotoolkit/physicalai/main/docs/assets/physicalai.png\" alt=\"Physical AI\" width=\"100%\"\u003e\n\u003c/p\u003e\n\n\u003cdiv align=\"center\"\u003e\n\n**Runtime package for deploying robot policies trained with [Physical AI Studio](https://github.com/open-edge-platform/physical-ai-studio)**\n\n[Installation](#installation) •\n[Camera API](#camera-api) •\n[Robot API](#robot-api) •\n[Inference](#inference) •\n[Docs](#documentation)\n\n\u003c/div\u003e\n\n---\n\nPhysical AI Runtime provides the deployment-side components for running trained policies on real hardware. It handles camera capture, robot control, and policy inference with a unified API that works across different hardware vendors.\n\n**Key Features:**\n\n- **Unified Camera API** — Same interface for UVC, RealSense, Basler, and IP cameras\n- **Robot Protocol** — Structural typing for any robot; no inheritance required\n- **Inference Engine** — Load exported policies from Studio with auto-detected backends\n- **Policy Runtime** — Control loop with observation building and action dispatch\n\n---\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/openvinotoolkit/physicalai/main/docs/assets/inference_rerun.webp\" alt=\"Inference demo\" width=\"100%\"\u003e\n\u003c/p\u003e\n\n## Installation\n\n```bash\npip install physicalai\n```\n\nWith hardware-specific extras:\n\n```bash\npip install physicalai[realsense]   # Intel RealSense cameras\npip install physicalai[basler]      # Basler industrial cameras\npip install physicalai[so101]       # SO-101 robot arm\npip install physicalai[trossen]     # Trossen WidowX robots\n```\n\n---\n\n## Camera API\n\nAll cameras share a unified interface: `connect()`, `read()`, `read_latest()`, and context manager support. Switch hardware without changing application code.\n\n```python\nfrom physicalai.capture import UVCCamera\n\nwith UVCCamera(device=\"/dev/video0\", width=640, height=480, fps=30) as camera:\n    frame = camera.read_latest()\n    print(frame.data.shape)  # (480, 640, 3)\n    print(frame.timestamp)   # monotonic timestamp\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eIntel RealSense (RGB + Depth)\u003c/strong\u003e\u003c/summary\u003e\n\n```python\nfrom physicalai.capture import RealSenseCamera\n\nwith RealSenseCamera(serial_number=\"123456789\", width=640, height=480, fps=30) as camera:\n    rgb, depth = camera.read_rgbd()\n    print(rgb.data.shape)    # (480, 640, 3) RGB\n    print(depth.data.shape)  # (480, 640) depth in mm\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eBasler Industrial Camera\u003c/strong\u003e\u003c/summary\u003e\n\n```python\nfrom physicalai.capture import BaslerCamera\n\nwith BaslerCamera(serial_number=\"12345678\", width=1920, height=1080, fps=60) as camera:\n    frame = camera.read_latest()\n    print(frame.data.shape)  # (1080, 1920, 3)\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eMulti-Camera Sync\u003c/strong\u003e\u003c/summary\u003e\n\n```python\nfrom physicalai.capture import UVCCamera, RealSenseCamera, read_cameras\n\ncameras = {\n    \"wrist\": UVCCamera(device=\"/dev/video0\"),\n    \"overhead\": RealSenseCamera(serial_number=\"123456789\"),\n}\n\n# Connect all\nfor cam in cameras.values():\n    cam.connect()\n\n# Read from all cameras concurrently\nsynced = read_cameras(cameras)\nprint(synced.frames[\"wrist\"].data.shape)\nprint(synced.frames[\"overhead\"].data.shape)\n\n# Cleanup\nfor cam in cameras.values():\n    cam.disconnect()\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eCamera Discovery\u003c/strong\u003e\u003c/summary\u003e\n\n```python\nfrom physicalai.capture import discover_all, UVCCamera\n\n# Discover all connected cameras (returns dict of camera_type -\u003e list of devices)\nall_devices = discover_all()\nfor camera_type, devices in all_devices.items():\n    for dev in devices:\n        print(f\"{camera_type}: {dev.device_id} - {dev.name}\")\n\n# Discover specific type\nuvc_devices = UVCCamera.discover()\n```\n\n\u003c/details\u003e\n\n---\n\n## Robot API\n\nRobots implement a Protocol-based interface. Any class with `connect()`, `disconnect()`, `get_observation()`, `send_action()`, and `joint_names` works — no inheritance required.\n\n```python\nfrom physicalai.robot import SO101\n\nrobot = SO101(port=\"/dev/ttyUSB0\")\nrobot.connect()\n\nobs = robot.get_observation()\nprint(obs.joint_positions)  # [0.1, 0.2, 0.3, 0.4, 0.5, 0.6]\nprint(robot.joint_names)    # ['shoulder_pan', 'shoulder_lift', ...]\n\nrobot.send_action(target_positions, goal_time=0.1)\nrobot.disconnect()\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eTrossen WidowX-AI\u003c/strong\u003e\u003c/summary\u003e\n\n```python\nfrom physicalai.robot import WidowXAI\n\nrobot = WidowXAI()\nrobot.connect()\n\nobs = robot.get_observation()\nprint(obs.joint_positions)\n\nrobot.send_action(target_positions)\nrobot.disconnect()\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eBimanual WidowX-AI\u003c/strong\u003e\u003c/summary\u003e\n\n```python\nfrom physicalai.robot import BimanualWidowXAI\n\nrobot = BimanualWidowXAI()\nrobot.connect()\n\nobs = robot.get_observation()\n# Joint positions for both arms concatenated\nprint(obs.joint_positions.shape)\n\nrobot.send_action(bimanual_targets)\nrobot.disconnect()\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eRobot Verification\u003c/strong\u003e\u003c/summary\u003e\n\n```python\nfrom physicalai.robot import SO101, verify_robot\n\nrobot = SO101(port=\"/dev/ttyUSB0\")\nverify_robot(robot)  # Interactive joint-by-joint check\n```\n\n\u003c/details\u003e\n\n---\n\n## Inference\n\nLoad exported policies from [Physical AI Studio](https://github.com/open-edge-platform/physical-ai-studio). The `InferenceModel` class auto-detects the backend (OpenVINO or ONNX in this package; companion distributions may contribute additional adapters such as ExecuTorch) and handles action chunking automatically.\n\n```python\nfrom physicalai.inference import InferenceModel\n\n# Load exported policy\nmodel = InferenceModel.load(\"./exports/act_policy\")\n\n# Reset state for new episode\nmodel.reset()\n\n# Run inference\naction = model.select_action(observation)\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eWith Explicit Backend\u003c/strong\u003e\u003c/summary\u003e\n\n```python\nfrom physicalai.inference import InferenceModel\n\n# Force specific backend\nmodel = InferenceModel.load(\n    \"./exports/act_policy\",\n    backend=\"openvino\",\n    device=\"GPU\",\n)\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eLatency benchmarking\u003c/strong\u003e\u003c/summary\u003e\n\n```python\nimport json\n\nfrom physicalai.benchmark.performance import InferenceLatencyBenchmark\nfrom physicalai.inference import InferenceModel\n\nmodel = InferenceModel.load(\"./exports/act_policy\")\nmodel.reset()\nbenchmark = InferenceLatencyBenchmark(\n        max_iters=100,\n        warmup_iters=2,\n        max_duration=10000,\n    )\nmetrics = benchmark.run(model)\nprint(json.dumps(metrics, indent=2))\n\n```\n\n\u003c/details\u003e\n\n---\n\n## Policy Runtime\n\nThe `PolicyRuntime` orchestrates the full control loop: connecting hardware, reading cameras, building observations, running inference, and dispatching actions to the robot.\n\n```python\nfrom physicalai.runtime import PolicyRuntime, SyncExecution\nfrom physicalai.inference import InferenceModel\nfrom physicalai.capture import UVCCamera, RealSenseCamera\nfrom physicalai.robot import SO101\n\nruntime = PolicyRuntime(\n    fps=30,\n    robot=SO101(port=\"/dev/ttyACM0\"),\n    model=InferenceModel.load(\"./exports/act_policy\"),\n    cameras={\n        \"wrist\": UVCCamera(device=\"/dev/video0\", width=640, height=480),\n        \"overhead\": RealSenseCamera(serial_number=\"123456789\"),\n    },\n    execution=SyncExecution(),\n)\n\nwith runtime:\n    runtime.run(duration_s=60)\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eFrom YAML Config\u003c/strong\u003e\u003c/summary\u003e\n\n\u003e **Preview:** This API is not yet implemented.\n\n```python\nruntime = PolicyRuntime.from_config(\"runtime.yaml\")\nruntime.run(duration_s=60)\n```\n\n```yaml\n# runtime.yaml\nruntime:\n  class_path: physicalai.runtime.PolicyRuntime\n  init_args:\n    fps: 30\n    robot:\n      class_path: physicalai.robot.so101.SO101\n      init_args:\n        port: /dev/ttyACM0\n    model:\n      class_path: physicalai.inference.InferenceModel\n      init_args:\n        export_dir: ./exports/act_policy\n    cameras:\n      wrist:\n        class_path: physicalai.capture.UVCCamera\n        init_args:\n          device: /dev/video0\n          width: 640\n          height: 480\n    execution:\n      class_path: physicalai.runtime.SyncExecution\n      init_args:\n        mode: chunk\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eCLI\u003c/strong\u003e\u003c/summary\u003e\n\n```bash\nphysicalai run --config runtime.yaml --run.duration_s=60\n```\n\nThe runtime package owns the shared `physicalai` executable. Training packages\ncan add subcommands such as `fit` and `benchmark` through the\n`physicalai.cli.subcommands` entry-point group.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eAsync Execution\u003c/strong\u003e\u003c/summary\u003e\n\nAsync execution runs inference in a background thread while the main loop handles camera reads and robot commands at a fixed frequency. Useful when inference is slower than the control rate.\n\n```python\nfrom physicalai.runtime import PolicyRuntime, AsyncExecution\n\nruntime = PolicyRuntime(\n    fps=30,\n    robot=robot,\n    model=model,\n    cameras=cameras,\n    execution=AsyncExecution(fps=30),\n)\n\nwith runtime:\n    runtime.run(duration_s=60)\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eRemote Execution\u003c/strong\u003e\u003c/summary\u003e\n\nRemote execution sends observations to an inference server and receives actions over the network. Useful for running large models on a separate GPU machine.\n\n\u003e **Preview:** This API is not yet implemented.\n\n```python\nfrom physicalai.runtime import PolicyRuntime, RemoteExecution\n\nruntime = PolicyRuntime(\n    fps=30,\n    robot=robot,\n    cameras=cameras,\n    execution=RemoteExecution(endpoint=\"http://gpu-server:8080/infer\"),\n)\n\nruntime.run(duration_s=60)\n```\n\n\u003c/details\u003e\n\n---\n\n\u003e **Full walkthrough:** See [`examples/tutorials/collect_train_deploy.ipynb`](https://github.com/openvinotoolkit/physicalai/blob/main/examples/tutorials/collect_train_deploy.ipynb) for a complete collect -\u003e train -\u003e deploy guide.\n\n---\n\n## Documentation\n\n[Home](https://github.com/openvinotoolkit/physicalai/tree/main/docs) • [Getting Started](https://github.com/openvinotoolkit/physicalai/tree/main/docs/getting-started) • [How-To Guides](https://github.com/openvinotoolkit/physicalai/tree/main/docs/how-to) • [Concepts](https://github.com/openvinotoolkit/physicalai/tree/main/docs/explanation) • [API Reference](https://github.com/openvinotoolkit/physicalai/tree/main/docs/reference)\n\n## Contributing\n\nSee [CONTRIBUTING.md](https://github.com/openvinotoolkit/physicalai/blob/main/CONTRIBUTING.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenvinotoolkit%2Fphysicalai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenvinotoolkit%2Fphysicalai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenvinotoolkit%2Fphysicalai/lists"}