{"id":19199842,"url":"https://github.com/rose-stl-lab/dive","last_synced_at":"2025-05-09T01:28:58.540Z","repository":{"id":75657481,"uuid":"303441154","full_name":"Rose-STL-Lab/DIVE","owner":"Rose-STL-Lab","description":"Disentangled Imputed Video autoEncoder (DIVE) ","archived":false,"fork":false,"pushed_at":"2020-12-03T16:14:50.000Z","size":2090,"stargazers_count":4,"open_issues_count":1,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-31T20:36:11.151Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Rose-STL-Lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-10-12T15:54:39.000Z","updated_at":"2021-12-25T23:01:54.000Z","dependencies_parsed_at":"2023-02-27T22:15:58.649Z","dependency_job_id":null,"html_url":"https://github.com/Rose-STL-Lab/DIVE","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rose-STL-Lab%2FDIVE","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rose-STL-Lab%2FDIVE/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rose-STL-Lab%2FDIVE/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rose-STL-Lab%2FDIVE/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Rose-STL-Lab","download_url":"https://codeload.github.com/Rose-STL-Lab/DIVE/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253173695,"owners_count":21865740,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T12:29:10.297Z","updated_at":"2025-05-09T01:28:58.531Z","avatar_url":"https://github.com/Rose-STL-Lab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Disentangled Imputed Video autoEncoder (DIVE)\n\nCode for NeurIPS 2020 paper titled [Learning Disentangled Representations of Video with Missing Data](https://arxiv.org/abs/2006.13391), by Armand Comas, Chi Zhang, Zlatan Feric, Octavia Camps and Rose Yu.\n\nMissing data poses significant challenges while learning representations of video sequences.\nWe present DIVE, a deep generative model that imputes and predicts future video frames in the presence of missing data.\nSpecifically, DIVE contributes by:\n- Introducing a missingness latent variable,\n- Disentangling the hidden video representations into static and dynamic appearance, pose, and missingness factors for each object and\n- Imputing each object trajectory where data is missing.\nThis is done in an end-to-end training fashion and with using only self-supervision, by leveraging a VAE framework.\n\n## Poster, Slides and video\nThe poster is the file with name \"NeurIPS_DIVE_Poster_v3.pdf\". Slides and video will be available soon.\n\n## Demo\n\nFollowing we provide details to run both datasets presented in our paper. If you encounter a problem please feel free to contact the authors.\n### Deformed and Missing-Data Moving MNIST\n\nWe prepare the code to be trained and tested for Scenario 3 of the experiments for Moving MNIST which includes:\n- Out-of-scene (fully occluded) digits for 1 time-step\n- Varying appearance following an elastic transformation, which is reduced linearly in time from very severe (alpha = 100) to inexistent.\nAll arguments are set by default for this scenario, but they can be changed at the users convenience. To better understand them, use the help command \"-h\" or refer to the \"config.py\" file.\nFor this experiment, missing labels are set to be soft.\n\nA qualitative example of the expected results after 600 epochs (about 100k iterations) is:\n\u003cimg src=\"example_image_varying_MNIST.png\" width=\"800px\"/\u003e\n\nExpected quantitative results can be found in the paper. For Scenario 1, set the argument crop_size to [64, 32] and set the flag use_crop_size to True in the config.py file. For both Scenarios 1 and 2, fix the flag with_var_appearance to False and ini-et-alpha to 0.\n\nAfter setting up the environment, we can train and test the code with:\n```\ncd DIVE/\npython3 train.py\n```\nand\n```\npython3 eval.py\n```\nTo view the results (while training in this case), execute the following commands:\n```\ncd tensorboard/ckpt/moving_mnist/dive/\ntensorboard --logdir train_log --port 6006\n```\n\n## Installation\nThis codebase is trained and tested with Python 3.6+, Pytorch 1.2.0+ and Cuda 10.0+. We use tensorboardX 2.0. for visualization purposes.\nWe make use of [Pyro](http://pyro.ai/) 0.2 as our framework for probabilitstic programming. To better understand our model, we encourage the reader to browse their [Examples and Tutorials](https://pyro.ai/examples/).\n\n### Moving MNIST\nDownload MNIST dataset from by running:\n```\ncd DIVE\nmkdir moving_mnist\ncd moving_mnist\nwget http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz\n```\nFor this demo we don't provide our fixed test set. Instead we use an on-the-fly generated set for testing.\n\n### MOTSChallenge Pedestrian\nDownload the MOTSChallenge pre-processed dataset [here](https://drive.google.com/file/d/1t3Z4ebREwaSrE5WxziuKdvjrllM4p-NM/view?usp=sharing). Allocate it in the directory that you find convenient under a folder named after \u003cem\u003edset_name\u003c/em\u003e.\nSince the code is set for MovingMNIST experiments, some changes will have to be made to the config.py file.\nWe set image_size=[256, 256], crop_size=[256, 256], n_frames_output=5, dset_dir: \u003cem\u003edata directory\u003c/em\u003e, dset_name: \"pedestrian\", num_missing: 1, num_objects: 3, n_components: 3, hidden_size: 96, stn_scale_prior: 3.5, batch_size = 32 and gamma_switch_step = 5e3.\n\nWe use as the backbone of our implementation the available implementation for Decompositional Disentangled Predictive Auto-Encoder ([DDPAE](https://github.com/jthsieh/DDPAE-video-prediction)), as well as many of its functions, while it is built on Pyro.\n\n## Citation\n\nIf you find this repository useful in your research, please cite our paper:\n```\n@article{Comas2020Dive,\n  title={Learning Disentangled Representations of Video with Missing Data},\n  author={Armand Comas Massague and Chi Zhang and Zlatan Feric and O. Camps and R. Yu},\n  journal={Advances in neural information processing systems},\n  year={2020},\n}\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frose-stl-lab%2Fdive","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frose-stl-lab%2Fdive","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frose-stl-lab%2Fdive/lists"}