{"id":17259859,"url":"https://github.com/ztangent/multimodal-dmm","last_synced_at":"2025-04-14T06:30:42.470Z","repository":{"id":37605191,"uuid":"170830882","full_name":"ztangent/multimodal-dmm","owner":"ztangent","description":"A PyTorch implementation of Factorized Inference in Deep Markov Models for Incomplete Multimodal Time Series (https://arxiv.org/abs/1905.13570).","archived":false,"fork":false,"pushed_at":"2022-12-08T05:56:25.000Z","size":1243,"stargazers_count":22,"open_issues_count":8,"forks_count":9,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-27T20:12:14.415Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ztangent.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-02-15T08:40:56.000Z","updated_at":"2024-01-05T08:42:22.000Z","dependencies_parsed_at":"2023-01-24T14:45:51.022Z","dependency_job_id":null,"html_url":"https://github.com/ztangent/multimodal-dmm","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ztangent%2Fmultimodal-dmm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ztangent%2Fmultimodal-dmm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ztangent%2Fmultimodal-dmm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ztangent%2Fmultimodal-dmm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ztangent","download_url":"https://codeload.github.com/ztangent/multimodal-dmm/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248832916,"owners_count":21168790,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-15T07:46:23.286Z","updated_at":"2025-04-14T06:30:42.415Z","avatar_url":"https://github.com/ztangent.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Multimodal Deep Markov Models\n\nA PyTorch implementation of the Multimodal Deep Markov Model (MDMM) and associated inference methods described in [Factorized Inference in Deep Markov Models for Incomplete Multimodal Time Series](https://arxiv.org/abs/1905.13570). Please cite this paper if you use or modify any of this code.\n\nGeneralizes the Multimodal Variational Auto-Encoder (MVAE) by [Wu \u0026 Goodman](https://papers.nips.cc/paper/7801-multimodal-generative-models-for-scalable-weakly-supervised-learning) and the Deep Markov Model by [Krishnan et al](https://www.aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14215).\n\n## Setup\n\nAfter creating a virtual environment with `virtualenv` or `conda`, one can simply install the dependencies in `requirements.txt`. Compatible with both Python 2.7 and Python 3.\n\n```\nvirtualenv venv\nsource venv/bin/activate\npip install -r requirements.txt\n```\n\nAlternatively, one can install the following packages directly through `pip`:\n```\n# For basic functionality\npip install torch==1.1.0 pandas pyyaml matplotlib\n\n# To download and pre-process the Weizmann video dataset\nsudo apt-get install ffmpeg\npip install scipy scikit-video scikit-image requests tqdm opencv-python\n\n# To run the experiment scripts using Ray Tune\npip install ray psutil\n```\n\n## Datasets\n\nBefore training, the datasets need to be generated or downloaded.\n\nTo generate the Spirals dataset, make `datasets` the current directory, then run `python spirals.py`. For a list of options, run `python spirals.py -h`.  \n\nTo automatically download and preprocess the Weizmann video dataset of human actions, again make sure that `datasets` is the current directory, then run `python weizmann.py`.\n\nIf automated download fails, create a directory called `weizmann` in `datasets`, and download the zip files and segmentation masks from the [Weizmann dataset website](http://www.wisdom.weizmann.ac.il/~vision/SpaceTimeActions.html).\n\n## Models and Inference Methods\n\nThe `models` subdirectory contains three different inference methods that can be used with MDMM (or MDMM-like) architectures:\n\n- `dmm.py` implements the MDMM with Backward Forward Variational Inference (BFVI), as described in our paper. Refer to the included docstrings for a full list of options.\n\n- `dks.py` implements the MDMM with the RNN-based structured inference networks described by [Krishnan et al](https://www.aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14215). By providing different options to the constructor, one can use either forward or backward RNN networks, and toggle different methods for handling missing data. Refer to the docstrings for details.\n\n- `vrnn.py` implements a multimodal version of the Variational Recurrent Neural Network (VRNN) described by [Chung et al](https://papers.nips.cc/paper/5653-a-recurrent-latent-variable-model-for-sequential-data). This is similar to using `dks.py` with a forward RNN.\n\n## Training\n\nThe training code for the Spirals dataset can be run by calling:\n```python spirals.py```\nDefault hyper-parameters are used, run `python spirals.py -h` for a full list of options.\n\nThe training code for the Weizmann dataset can be run by calling:\n```python weizmann.py```\nAgain, default hyper-parameters are used, run `python weizmann.py -h` for a full list of options.\n\nTo specify which inference method to use, use the `--model` flag with either `dmm` or `dks`. To specify which modalities to load and train on, use the `--modalities` flag. To visualize predictions while training, add the `--visualize` flag. Pretrained models can be evaluated by adding `--load PATH/TO/MODEL`.\n\nAn abstract `Trainer` class can be found in `trainer.py`, allowing training code to easily written for other multimodal sequential datasets.\n\n## Experiments\n\nRay Tune can be used to easily run experiments across multiple sets of hyper-parameters over multiple trials. Make sure `ray` is installed for this to work. Install `tensorboard` and `tensorflow` as well if you would like to visualize the loss curves via Tensorboard.\n\n### Comparing different inference methods on a range of tasks\n\nFor the Spirals dataset:\n```python -m experiments.spirals_suite --trial_cpus N --trial_gpus N```\n\nFor the Weizmann dataset:\n```python -m experiments.weizmann_suite --trial_cpus N --trial_gpus N```\n\n### Learning with uniformly random missing data\n\nFor the Spirals dataset:\n```python -m experiments.spirals_partial --trial_cpus N --trial_gpus N```\n\nFor the Weizmann dataset:\n```python -m experiments.weizmann_partial --trial_cpus N --trial_gpus N```\n\n### Semi-supervised learning\n\nSemi-supervised learning refers to learning where some sequences have entire modalities removed.\n\nFor the Spirals dataset:\n```python -m experiments.spirals_semisup --trial_cpus N --trial_gpus N```\n\nFor the Weizmann dataset:\n```python -m experiments.weizmann_semisup --trial_cpus N --trial_gpus N```\n\n## Examples\n\nBelow are spiral reconstructions produced by the inference methods across different inference tasks. BFVI (our method) consistently produces good reconstructions across all inference tasks, unlike the RNN-based methods.\n\n![Comparison of spiral reconstructions](/images/spirals_comparison.png)\n\nVideo reconstructions from the Weizmann dataset are shown below, comparing BFVI to the next best method (B-Skip). Only video data is provided; the silhoutte masks and action labels have to be inferred. Again, it can be seen that BFVI produces better reconstructions, as well as better silhouette and action predictions.\n\n![Comparison of video reconstructions](/images/weizmann_comparison.png)\n\nRefer to the [paper](https://arxiv.org/abs/1905.13570) for more examples.\n\n## Bugs \u0026 Questions\n\nFeel free to raise issues, or email xuan [at] mit [dot] edu with questions.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fztangent%2Fmultimodal-dmm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fztangent%2Fmultimodal-dmm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fztangent%2Fmultimodal-dmm/lists"}