{"id":21010510,"url":"https://github.com/hughplay/tvr","last_synced_at":"2025-10-12T12:33:20.031Z","repository":{"id":44388130,"uuid":"316128424","full_name":"hughplay/TVR","owner":"hughplay","description":":boom: Transformation Driven Visual Reasoning - CVPR 2021","archived":false,"fork":false,"pushed_at":"2023-05-27T13:40:42.000Z","size":5104,"stargazers_count":35,"open_issues_count":0,"forks_count":6,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-05-14T00:05:32.354Z","etag":null,"topics":["blender","clevr","cvpr2021","trance","tvr","visual-reasoning"],"latest_commit_sha":null,"homepage":"https://hongxin2019.github.io/TVR","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hughplay.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-11-26T05:06:49.000Z","updated_at":"2024-03-19T08:24:14.000Z","dependencies_parsed_at":"2023-02-14T04:46:12.233Z","dependency_job_id":null,"html_url":"https://github.com/hughplay/TVR","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hughplay%2FTVR","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hughplay%2FTVR/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hughplay%2FTVR/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hughplay%2FTVR/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hughplay","download_url":"https://codeload.github.com/hughplay/TVR/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225324946,"owners_count":17456631,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blender","clevr","cvpr2021","trance","tvr","visual-reasoning"],"created_at":"2024-11-19T09:21:03.427Z","updated_at":"2025-10-12T12:33:15.010Z","avatar_url":"https://github.com/hughplay.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TVR - Transformation driven Visual Reasoning\n\n\u003cbr\u003e\n\nOfficial repository for [\"Transformation driven Visual Reasoning\"](https://github.com/hughplay/TVR).\n\n\u003c!-- ![A fancy image here](docs/_static/imgs/logo.svg) --\u003e\n\u003cimg src=\"docs/_static/imgs/web.svg\" width=\"500\"\u003e\n\n**Figure:** *Given the initial state and the final state, the target is to infer the intermediate transformation.*\n\n\u003e **Transformation driven Visual Reasoning** \u003cbr\u003e\n\u003e Xin Hong, Yanyan Lan, Liang Pang, Jiafeng Guo, Xueqi Cheng \u003cbr\u003e\n\u003e *Published on 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*\n\n[![](imgs/project.svg)](https://hongxin2019.github.io/TVR/)\n[![](https://img.shields.io/badge/-code-green?style=flat-square\u0026logo=github\u0026labelColor=gray)](https://github.com/hughplay/TVR)\n[![](https://img.shields.io/badge/TRANCE-dataset-blue?style=flat-square\u0026labelColor=gray)](https://hongxin2019.github.io/TVR/dataset)\n[![](https://img.shields.io/badge/TRANCE-explore_dataset-blue?style=flat-square\u0026labelColor=gray)](https://hongxin2019.github.io/TVR/explore)\n[![](https://img.shields.io/badge/arXiv-2011.13160-b31b1b?style=flat-square)](https://arxiv.org/pdf/2011.13160.pdf)\n[![](https://img.shields.io/badge/PyTorch-ee4c2c?style=flat-square\u0026logo=pytorch\u0026logoColor=white)](https://pytorch.org/get-started/locally/)\n[![DeepCodebase](https://img.shields.io/badge/Deep-Codebase-2d50a5.svg?style=flat-square)](https://github.com/hughplay/DeepCodebase)\n\n\n\u003cbr\u003e\n\n## Description\n\n**Motivation:** Most existing visual reasoning tasks, such as CLEVR in VQA, are solely defined to test how well the machine understands the concepts and relations within static settings, like one image. We argue that this kind of **state driven visual reasoning** approach has limitations in reflecting whether the machine has the ability to infer the dynamics between different states, which has been shown as important as state-level reasoning for human cognition in Piaget’s theory.\n\n**Task:** To tackle aforementioned problem, we propose a novel **transformation driven visual reasoning** task. Given both the initial and final states, the target is to infer the corresponding single-step or multi-step transformation.\n\nIf you find this code useful, please consider to star this repo and cite us:\n\n```\n@inproceedings{hongTransformationDrivenVisual2021d,\n  title = {Transformation {{Driven Visual Reasoning}}},\n  booktitle = {2021 {{IEEE}}/{{CVF Conference}} on {{Computer Vision}} and {{Pattern Recognition}} ({{CVPR}})},\n  author = {Hong, Xin and Lan, Yanyan and Pang, Liang and Guo, Jiafeng and Cheng, Xueqi},\n  year = {2021},\n  pages = {6899--6908}\n}\n```\n\n\n\u003cbr\u003e\n\n## Environment Setup\n\nWe use docker to manage the environment. You need to build the docker image first and then enter the container to run the code.\n\n**0. Basic Setup**\n\nThe host machine should have installed following packages (need sudo previlege to install):\n- [Docker](https://docs.docker.com/engine/install)\n- [Nvidia-Docker](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker)\n\nAnd we also need to install [docker-compose](https://docs.docker.com/compose/install/) to manage the docker containers. Simply run the following command to install it:\n\n```bash\npip install docker-compose\n```\n\n**1. Start the docker container**\n\n```\npython docker.py startd\n```\n\nPlease follow the prompts to set the variables such as `PROJECT`, `DATA_ROOT`, `LOG_ROOT`. After that, a `.env` file will be generated in the root directory. You can also modify the variables in the `.env` file directly.\n\n`DATA_ROOT` is mapped to `/data` in the container. `LOG_ROOT` is mapped to `/log` in the container.\n\n**Tips:** when you first start the container, it will take some time to build the image. After that, it will be much faster. If you want to rebuild the image, you can run:\n\n```\npython docker.py startd --build\n```\n\n\n**2. Enter the container**\n\n``` sh\npython docker.py\n```\n\n\u003cbr\u003e\n\n## Data Preparation\n\n**1. Download TRANCE dataset**\n\nFollow the [steps in this page](https://hongxin2019.github.io/TVR/dataset) to download TRNACE from Kaggle. After that, decompress the package under `DATA_ROOT` (specified in `.env` file). The final dataset location should be `DATA_ROOT/trance` in the host machine and it will be mapped to `/data/trance` in the container.\n\n**2. Preprocess the data**\n\nPreprocess the data with the following command:\n\n```\npython scripts/preprocess.py /data/trance\n```\n\nThis will merge the raw images files and meta data into a single hdf5 file. After that, the directory should include the following files:\n\n\n```\ntrance\n├── data.h5\n├── properties.json\n└── values.json\n```\n\n\n\u003cbr\u003e\n\n## Training \u0026 testing\n\nAfter entering the container, you can run the following command to train a model:\n\n``` bash\npython train.py experiment=event_cnn_concat logging.wandb.tags=\"[event, base]\"\n```\n\nOr, you can training multiple models with available GPUs:\n\n``` bash\npython scripts/batch_train.py scripts/training/train_models.sh --gpus 0,1,2,3\n```\n\nPlease refer to the scripts under [`scripts/training`](scripts/training) for full training commands.\n\n**Notice:** We fixed a bug in TRANCE, therefore, the performance on Event and View is slightly higher (0.03~0.06 on Acc) than the results reported in our CVPR paper.\n\n\n\u003cbr\u003e\n\n\n## Demo\n\nWe provide a demo to explore the dataset and testing predictions of trained models.\n\n![](docs/_static/imgs/demo.png)\n\n**1. Launch the api server**\n\nEnter the project container and run the api server:\n\n``` bash\npython docker.py\nuvicorn src.demo.api_server.main:app --host 0.0.0.0 --port 8000 --reload\n```\n\n**Tips 1:** you can check the api docs by visiting `http://\u003chost_ip\u003e:8000/docs` in your browser. The `host_ip` is the ip address of the host machine.\n\n**Tips 2:** the default port is 8000. If you use another port, you need also to modify the port specified in [`src/demo/ui/src/js/api.js`](src/demo/ui/src/js/api.js).\n\n**2. Launch the web (UI) server**\n\nWe need another docker container to launch the ui. Run the command in another terminal window in the host machine (recommend tmux):\n\n``` sh\npython docker.py start --service demo\n```\n\nWhen you first start the container, besides the image building, it will also take some time to install the npm packages. After that, it will be much faster.\n\n\u003cbr\u003e\n\n## Data Generation\n\nWe provide the code to generate the dataset.\n\n**1. Build the docker image**\n\n```\npython docker.py prepare --service blender --build\n```\n\n**2. Enter the container**\n\n```\npython docker.py --service blender\n```\n\n**3. Generate the dataset**\n\n``` sh\n# with CPU\nblender --background --python render.py -- --config configs/standard.yaml --gpu false --render_tile_size 16\n\n# with GPU\nCUDA_VISIBLE_DEVICES=0 blender --background --python render.py -- --config configs/standard.yaml --gpu true --n_sample 1\n```\n\nThe speed of rendering can be affected by:\n- GPU or CPU. Gererally, GPU is more faster than CPU, unless your CPU has many cores.\n- `render_tile_size`. CPU prefers small tile size, while GPU prefers large tile size.\n- Balanced sampling. It has noting to do with blender rendering. However, sampling scene graph for rendering can also be time consuming.\n\n\u003cbr\u003e\n\n## LICENSE\n\nThe code is licensed under the [MIT license](./LICENSE) and the TRANCE dataset is licensed under the \u003ca rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc/4.0/\"\u003eCreative Commons Attribution-NonCommercial 4.0 International License\u003c/a\u003e.\n\nNotice: Some materials are directly inherited from [CLEVR](https://github.com/facebookresearch/clevr-dataset-gen) which are licensed under BSD License. More details can be found in [this document](trance/resource/README.md).\n\n\u003cbr\u003e\n\n*This is a project based on [DeepCodebase](https://github.com/hughplay/DeepCodebase) template.*\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhughplay%2Ftvr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhughplay%2Ftvr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhughplay%2Ftvr/lists"}