{"id":19462460,"url":"https://github.com/andrewowens/multisensory","last_synced_at":"2025-04-25T08:31:06.547Z","repository":{"id":149694292,"uuid":"129283202","full_name":"andrewowens/multisensory","owner":"andrewowens","description":"Code for the paper: Audio-Visual Scene Analysis with Self-Supervised Multisensory Features","archived":false,"fork":false,"pushed_at":"2019-07-17T06:57:17.000Z","size":1215,"stargazers_count":220,"open_issues_count":34,"forks_count":60,"subscribers_count":12,"default_branch":"master","last_synced_at":"2024-11-10T18:03:06.970Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://andrewowens.com/multisensory/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/andrewowens.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2018-04-12T16:47:42.000Z","updated_at":"2024-07-30T05:32:09.000Z","dependencies_parsed_at":null,"dependency_job_id":"5b0a75ba-0eb1-4684-9505-56197d7b76eb","html_url":"https://github.com/andrewowens/multisensory","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrewowens%2Fmultisensory","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrewowens%2Fmultisensory/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrewowens%2Fmultisensory/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrewowens%2Fmultisensory/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/andrewowens","download_url":"https://codeload.github.com/andrewowens/multisensory/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250782012,"owners_count":21486371,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T18:01:15.686Z","updated_at":"2025-04-25T08:31:06.247Z","avatar_url":"https://github.com/andrewowens.png","language":"Python","funding_links":[],"categories":["Citing","Computer Vision"],"sub_categories":["Contents","Audio"],"readme":"[[Paper]](https://arxiv.org/pdf/1804.03641.pdf)\n[[Project page]](http://andrewowens.com/multisensory)\n\nThis repository contains code for the [paper](https://arxiv.org/pdf/1804.03641.pdf):\n\nAndrew Owens, Alexei A. Efros. Audio-Visual Scene Analysis with Self-Supervised Multisensory Features. arXiv, 2018\n\n## Contents\nThis release includes code and models for:\n- **On/off-screen source separation**: separating the speech of an on-screen speaker from background sounds.\n- **Blind source separation**: audio-only source separation using [u-net](https://arxiv.org/pdf/1505.04597.pdf) and [PIT](https://arxiv.org/pdf/1607.00325).\n- **Sound source localization**: visualizing the parts of a video that correspond to sound-making actions.\n- **Self-supervised audio-visual features**: a pretrained 3D CNN that can be used for downstream tasks (e.g. action recognition, source separation).\n\n## Setup\n- Install [Python 2.7](https://www.python.org/download/releases/2.7)\n- Install [ffmpeg](https://www.ffmpeg.org/download.html)\n- Install [TensorFlow](https://www.tensorflow.org/), e.g. through pip:\n```bash\npip install tensorflow     # for CPU evaluation only\npip install tensorflow-gpu # for GPU support\n```\nWe used TensorFlow version 1.8, which can be installed with:\n```\npip install tensorflow-gpu==1.8\n```\n\n\n- Install other python dependencies\n```bash\npip install numpy matplotlib pillow scipy\n```\n- Download the pretrained models and sample data\n```bash\n./download_models.sh\n./download_sample_data.sh\n```\n\n## Pretrained audio-visual features\nWe have provided the features for our fused audio-visual network. These features were learned through self-supervised learning. Please see [shift_example.py](src/shift_example.py) for a simple example that uses these pretrained features.\n\n## Audio-visual source separation\nTo try the on/off-screen source separation model, run:\n```bash\npython sep_video.py ../data/translator.mp4 --model full --duration_mult 4 --out ../results/\n```\nThis will separate a speaker's voice from that of an off-screen speaker. It will write the separated video files to `../results/`, and will also display them in a local webpage, for easier viewing. This produces the following videos (click to watch):\n\n| Input | On-screen | Off-screen |\n| ----- | --------- | ---------- |\n| \u003ca href = \"https://youtu.be/4kVNzxFeboo\"\u003e\u003cimg src = \"doc/translator_input.jpg\" width = 200\u003e\u003c/a\u003e | \u003ca href = \"https://youtu.be/XvJVXsHyBKw\"\u003e\u003cimg src = \"doc/translator_input.jpg\" width = 200\u003e\u003c/a\u003e | \u003ca href = \"https://youtu.be/NFll7nfmwO8\"\u003e\u003cimg src = \"doc/translator_input.jpg\" width = 200\u003e\u003c/a\u003e |\n\nWe can visually mask out one of the two on-screen speakers, thereby removing their voice:\n```bash\npython sep_video.py ../data/crossfire.mp4 --model full --mask l --out ../results/\npython sep_video.py ../data/crossfire.mp4 --model full --mask r --out ../results/\n```\nThis produces the following videos (click to watch):\n\n| Source | Left | Right |\n| ------ | ---- | ----- |\n| \u003ca href = \"https://youtu.be/H9CgWJToF_s\"\u003e\u003cimg src=\"doc/crossfire_input.jpg\" width=\"200\"/\u003e\u003c/a\u003e | \u003ca href = \"https://youtu.be/9jPaA8ttI6A\"\u003e\u003cimg src=\"doc/crossfire_l.jpg\" width=\"200\"/\u003e\u003c/a\u003e | \u003ca href = \"https://youtu.be/M4ACgIWuiWM\"\u003e\u003cimg src=\"doc/crossfire_r.jpg\" width=\"200\"/\u003e\u003c/a\u003e |\n\n## Blind (audio-only) source separation\nThis baseline trains a [u-net](https://arxiv.org/pdf/1505.04597.pdf) model to minimize a [permutation invariant](https://arxiv.org/pdf/1607.00325) loss.\n```bash\npython sep_video.py ../data/translator.mp4 --model unet_pit --duration_mult 4 --out ../results/\n```\nThe model will write the two separated streams in an arbitrary order.\n\n## Visualizing the locations of sound sources\nTo view the self-supervised network's class activation map (CAM), use the `--cam` flag:\n```bash\npython sep_video.py ../data/translator.mp4 --model full --cam --out ../results/\n```\nThis produces a video in which the CAM is overlaid as a heat map:\n\n\u003ca href = \"https://youtu.be/u99MdLBDnJc\"\u003e\u003cimg src=\"doc/crossfire_cam.jpg\" width=\"300\"/\u003e\u003c/a\u003e\n\n## Action recognition and fine-tuning\nWe have provided example code for training an action recognition model (e.g. on the [UCF-101](http://crcv.ucf.edu/data/UCF101.php) dataset) in [videocls.py](src/videocls.py)). This involves fine-tuning our pretrained, audio-visual network. It is also possible to train this network with only visual data (no audio).\n\n## Citation\nIf you use this code in your research, please consider citing our paper:\n```\n@article{multisensory2018,\n  title={Audio-Visual Scene Analysis with Self-Supervised Multisensory Features},\n  author={Owens, Andrew and Efros, Alexei A},\n  journal={arXiv preprint arXiv:1804.03641},\n  year={2018}\n}\n```\n\n## Updates\n- 11/08/18: Fixed a bug in the class activation map example code. Added Tensorflow 1.9 compatibility.\n\n## Acknowledgements\nOur *u*-net code draws from [this implementation](https://github.com/affinelayer/pix2pix-tensorflow) of [pix2pix](https://arxiv.org/abs/1611.07004).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandrewowens%2Fmultisensory","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandrewowens%2Fmultisensory","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandrewowens%2Fmultisensory/lists"}