{"id":28753288,"url":"https://github.com/google-deepmind/physics-iq-benchmark","last_synced_at":"2025-06-17T00:39:27.988Z","repository":{"id":272936304,"uuid":"903246817","full_name":"google-deepmind/physics-IQ-benchmark","owner":"google-deepmind","description":"Benchmarking physical understanding in generative video models","archived":false,"fork":false,"pushed_at":"2025-05-20T13:54:42.000Z","size":63290,"stargazers_count":162,"open_issues_count":3,"forks_count":16,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-05-20T14:48:31.802Z","etag":null,"topics":["benchmark","generative-models","physical-understanding","video-generation"],"latest_commit_sha":null,"homepage":"https://physics-iq.github.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google-deepmind.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-12-14T05:07:19.000Z","updated_at":"2025-05-20T13:54:46.000Z","dependencies_parsed_at":"2025-05-20T14:50:54.569Z","dependency_job_id":null,"html_url":"https://github.com/google-deepmind/physics-IQ-benchmark","commit_stats":null,"previous_names":["google-deepmind/physics-iq-benchmark"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/google-deepmind/physics-IQ-benchmark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-deepmind%2Fphysics-IQ-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-deepmind%2Fphysics-IQ-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-deepmind%2Fphysics-IQ-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-deepmind%2Fphysics-IQ-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google-deepmind","download_url":"https://codeload.github.com/google-deepmind/physics-IQ-benchmark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-deepmind%2Fphysics-IQ-benchmark/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260268635,"owners_count":22983601,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","generative-models","physical-understanding","video-generation"],"created_at":"2025-06-17T00:39:27.047Z","updated_at":"2025-06-17T00:39:27.942Z","avatar_url":"https://github.com/google-deepmind.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/duckie.png\" width=\"20%\" alt=\"logo\"\u003e\n\u003c/p\u003e\n\n[Step A: Generating Videos](#step-a-generating-videos-for-physics-iq-test-cases-based-on-video-model) | [Step B: Evaluating Generated Videos](#step-b-evaluating-generated-videos-on-physics-iq-to-generate-benchmark-scores) | [Leaderboard](#leaderboard) | [Citation](#citation) | [License](#license-and-disclaimer)\n\n# Physics-IQ: Benchmarking physical understanding in generative video models\n\nPhysics-IQ is a high-quality, realistic, and comprehensive benchmark dataset for evaluating physical understanding in generative video models.\n\nProject website: [physics-iq.github.io](https://physics-iq.github.io/)\n\n### Key Features:\n- **Real-world videos**: All videos are captured with high-quality cameras, not rendered.\n- **Diverse scenarios**: Covers a wide range of physical phenomena, including collisions, fluid dynamics, gravity, material properties, light, shadows, magnetism, and more.\n- **Multiple perspectives**: Each scenario is filmed from 3 different angles.\n- **Variations**: Each scenario is recorded twice to capture natural physical variations.\n- **High resolution and frame rate**: Videos are recorded at 3840 × 2160 resolution and 30 frames per second.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/teaser1.gif\" width=\"23%\" alt=\"Teaser 1\"\u003e\n  \u003cimg src=\"assets/teaser2.gif\" width=\"23%\" alt=\"Teaser 2\"\u003e\n  \u003cimg src=\"assets/teaser3.gif\" width=\"23%\" alt=\"Teaser 3\"\u003e\n  \u003cimg src=\"assets/teaser4.gif\" width=\"23%\" alt=\"Teaser 4\"\u003e\n  \u003cimg src=\"assets/teaser5.gif\" width=\"23%\" alt=\"Teaser 5\"\u003e\n  \u003cimg src=\"assets/teaser6.gif\" width=\"23%\" alt=\"Teaser 6\"\u003e\n  \u003cimg src=\"assets/teaser7.gif\" width=\"23%\" alt=\"Teaser 7\"\u003e\n  \u003cimg src=\"assets/teaser8.gif\" width=\"23%\" alt=\"Teaser 8\"\u003e\n\u003c/p\u003e\n\n---\n\n## Leaderboard\nThe best possible score on Physics-IQ is 100.0%, this score would be achieved by physically realistic videos that differ only in physical randomness but adhere to all tested principles of physics.\n\nIf you test your model on Physics-IQ and would like your score/paper/model to be featured here in this table, feel free to open a pull request that adds a row to the table and we'll be happy to include it!\n\n| **#** | **Model** | **input type** | **Physics-IQ score** | **date added (YYYY-MM-DD)** |\n| -- | --- | --- | --- | --- |\n| 1 | [Magi-1](https://arxiv.org/abs/2505.13211)                             | multiframe (v2v) | **56.0 %** :1st_place_medal: | 2025-04-21 |\n| 2 | [Video-GPT](https://arxiv.org/abs/2505.12489)                          | multiframe (v2v) | 35.0 % :2nd_place_medal: | 2025-05-22 |\n| 3 | [Magi-1](https://arxiv.org/abs/2505.13211)                             | i2v              | 30.2 % :3rd_place_medal: | 2025-04-21 |\n| 4 | [VideoPoet](https://arxiv.org/abs/2312.14125)                          | multiframe (v2v) | 29.5 % | 2025-02-19 |\n| 5 | [Lumiere](https://arxiv.org/abs/2401.12945)                            | multiframe (v2v) | 23.0 % | 2025-02-19 |\n| 6 | [Runway Gen 3](https://runwayml.com/research/introducing-gen-3-alpha)  | i2v        | 22.8 %  | 2025-02-19 |\n| 7 | [VideoPoet](https://arxiv.org/abs/2312.14125)                          | i2v        | 20.3 % | 2025-02-19 |\n| 8 | [Lumiere](https://arxiv.org/abs/2401.12945)                            | i2v        | 19.0 % | 2025-02-19 |\n| 9 | [Stable Video Diffusion](https://arxiv.org/abs/2311.15127)             | i2v        | 14.8 % | 2025-02-19 |\n| 10 | [Pika](https://pika.art/)                                             | i2v        |  13.0 % | 2025-02-19 |\n| 11 | [Sora](https://openai.com/sora/)                                      | i2v        |  10.0 % | 2025-02-19 |\n\n*Note to early adopters of the benchmark: results from the paper were finalized on February 19, 2025; if you used the toolbox before please re-run since we changed and improved a few aspects. Likewise, if you downloaded the dataset before that date, it is recommended to re-download it, ensuring the ground truth video masks have a duration of five seconds.*\n\n---\n\n## Step A: Generating Videos for Physics-IQ Test Cases Based on Video Model\n\n### 1. Download Benchmark Dataset\n\nVisit the [Google Cloud Storage link](https://console.cloud.google.com/storage/browser/physics-iq-benchmark) to download the dataset, or run the following script:\n\n```bash\npip install gsutil\npython3 ./code/download_physics_iq_data.py\n```\n\n- If your desired FPS already exists in the dataset, it will be downloaded.\n- If it does not exist, the script will download 30 FPS files and generate your desired FPS videos by downsampling the 30 FPS version.\n\n---\n\n### 2. Running Video Model on Test Cases from Benchmark\n\nThis section explains how to generate videos using the provided benchmark and save them in the required format. Follow the instructions below based on your model type:\n\n#### 2.1 Image-to-Video (i2v) Models\n\n1. **Input Requirements**:\n   - **Initial Frame**: Use frames from `physics-iq-benchmark/switch-frames`.\n   - **Text Input (Optional)**: If required, use descriptions from `descriptions.csv`.\n\n2. **Steps to Run**:\n   - Generate videos using the initial frame (and text condition, if applicable).\n   - Save generated videos in the following structure, using any filename as long as the unique ID prefix from the test videos is kept (`0001_`, ..., `0198_`):\n     ```\n     .model_name/{ID}_{anything-you-like}.mp4\n     ```\n   - Refer to the `generated_video_name` column in `descriptions.csv` for file naming conventions.\n\n---\n\n#### 2.2 Multiframe-to-Video Models\n\n1. **Input Requirements**:\n   - **Conditioning Frames**:\n     - Available in `physics-iq-benchmark/split-videos/conditioning-videos`.\n     - Ensure the correct frame rate: `30FPS`, `24FPS`, `16FPS`, or `8FPS`.\n   - **Text Input (Optional)**: Use `descriptions.csv`.\n\n2. **Steps to Run**:\n   - Use conditioning frames to generate videos.\n   - Save generated videos in the structure:\n     ```\n     model_name/{ID}_{perspective}_{scenario_name}.mp4\n     example: model_name/{0001}_{perspective-left}_{trimmed-ball-and-block-fall}.mp4\n     ```\n   - Refer to the `generated_video_name` column in `descriptions.csv` for file naming conventions.\n\n---\n\n## Step B: Evaluating Generated Videos on Physics-IQ to Generate Benchmark Scores\n\n### 1. Installation\n\nEnsure you have Python 3 installed. Then, run the following command to install the necessary packages:\n\n```bash\npip install -r requirements.txt\n```\n\n### 2. Dataset Placement\n\n- Ensure you have downloaded and placed the `physics-iq-benchmark` dataset in your working directory. This dataset must include 30FPS videos and optionally your desired FPS. If your desired FPS does not exist in our dataset already, it will be automatically generated. You should have the following structure:\n\n```plaintext\nphysics-IQ-benchmark/\n├── full-videos/\n│   └── ...\n|\n├── split-videos/\n│   ├── conditioning-videos/\n│   │   └── 30FPS/\n│   │       ├── 0001_conditioning-videos_30FPS_perspective-left_take-1_trimmed-ball-and-block-fall.mp4\n│   │       ├── 0002_conditioning-videos_30FPS_perspective-center_take-1_trimmed-ball-and-block-fall.mp4\n│   │       └── ...\n│   └── testing-videos/\n│       └── 30FPS/\n│           ├── 0001_testing-videos_30FPS_perspective-left_take-1_trimmed-ball-and-block-fall.mp4\n│           ├── 0002_testing-videos_30FPS_perspective-center_take-1_trimmed-ball-and-block-fall.mp4\n│           └── ...\n├── switch-frames/\n│   ├── 0001_switch-frames_anyFPS_perspective-left_trimmed-ball-and-block-fall.jpg\n│   ├── 0002_switch-frames_anyFPS_perspective-center_trimmed-ball-and-block-fall.jpg\n│   └── ...\n└── video-masks/\n    └── real/\n        └── 30FPS/\n            ├── 0001_video-masks_30FPS_perspective-left_take-1_trimmed-ball-and-block-fall.mp4\n            ├── 0002_video-masks_30FPS_perspective-center_take-1_trimmed-ball-and-block-fall.mp4\n            └── ...\n```\n\n- the descriptions file which includes all file names and descriptions of the scenarios should be placed in your home directory as `descriptions.csv`.\n- Place your generated videos under `.model_name` directory.\n\n⚠️ **IMPORTANT:** Note that this script evaluates the **first 5 seconds** of your generated videos. Hence, make sure these are the 5 seconds generated right after the switch frame.\n\n### 3. Generate benchmark scores and plots\n\n```bash\npython3 code/run_physics_iq.py --input_folders \u003cgenerated_videos_dirs\u003e --output_folder \u003coutput_dir\u003e --descriptions_file \u003cdescriptions_file\u003e\n```\n**Parameters:**\n- `--input_folders`: The path to the directories containing input videos (in `.mp4` format), with one directory per model (`/model_name/video.mp4`).\n- `--output_folder`: The path to the directory where output CSV files will be saved.\n- `--descriptions_file`: The path to the `descriptions.csv` file.\n\n---\n\n\n## Citation\nIf you think this project is helpful, please feel free to leave a star ⭐️\n\n```latex\n@article{motamed2025physics,\n  title={Do generative video models understand physical principles?},\n  author={Saman Motamed and Laura Culp and Kevin Swersky and Priyank Jaini and Robert Geirhos},\n  journal={arXiv preprint arXiv:2501.09038},\n  year={2025}\n}\n```\n\n\n## License and disclaimer\n\nCopyright 2024 DeepMind Technologies Limited\n\nAll software is licensed under the Apache License, Version 2.0 (Apache 2.0);\nyou may not use this file except in compliance with the Apache 2.0 license.\nYou may obtain a copy of the Apache 2.0 license at:\nhttps://www.apache.org/licenses/LICENSE-2.0\n\nAll other materials are licensed under the Creative Commons Attribution 4.0\nInternational License (CC-BY). You may obtain a copy of the CC-BY license at:\nhttps://creativecommons.org/licenses/by/4.0/legalcode\n\nUnless required by applicable law or agreed to in writing, all software and\nmaterials distributed here under the Apache 2.0 or CC-BY licenses are\ndistributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,\neither express or implied. See the licenses for the specific language governing\npermissions and limitations under those licenses.\n\nThis is not an official Google product.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-deepmind%2Fphysics-iq-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle-deepmind%2Fphysics-iq-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-deepmind%2Fphysics-iq-benchmark/lists"}