{"id":19385883,"url":"https://github.com/zhengpeng7/motion_magnification_learning-based","last_synced_at":"2025-04-23T22:32:12.834Z","repository":{"id":63942789,"uuid":"223970988","full_name":"ZhengPeng7/motion_magnification_learning-based","owner":"ZhengPeng7","description":"An unofficial implementation of \"Learning-based Video Motion Magnification\" in Pytorch.","archived":false,"fork":false,"pushed_at":"2025-03-18T14:16:32.000Z","size":26034,"stargazers_count":111,"open_issues_count":0,"forks_count":25,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-02T21:42:37.602Z","etag":null,"topics":["deep-learning","motion-magnification"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ZhengPeng7.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2019-11-25T14:42:10.000Z","updated_at":"2025-03-18T14:16:36.000Z","dependencies_parsed_at":"2022-11-29T11:00:14.522Z","dependency_job_id":"39e375bb-ba05-4ed9-a464-0b975ac651e8","html_url":"https://github.com/ZhengPeng7/motion_magnification_learning-based","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZhengPeng7%2Fmotion_magnification_learning-based","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZhengPeng7%2Fmotion_magnification_learning-based/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZhengPeng7%2Fmotion_magnification_learning-based/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZhengPeng7%2Fmotion_magnification_learning-based/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ZhengPeng7","download_url":"https://codeload.github.com/ZhengPeng7/motion_magnification_learning-based/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250527180,"owners_count":21445318,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","motion-magnification"],"created_at":"2024-11-10T10:03:31.717Z","updated_at":"2025-04-23T22:32:07.823Z","avatar_url":"https://github.com/ZhengPeng7.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Motion_magnification_learning-based\nThis is an unofficial implementation of \"[Learning-based Video Motion Magnification](https://arxiv.org/abs/1804.02684)\" in Pytorch (1.8.1~2.0).\n[Here is the official implementation in Tensorflow==1.8.0](https://github.com/12dmodel/deep_motion_mag).\n\n#### High recommendations on my friends' latest works, come and try them!\n+ \u003cins\u003eEvent-Based\u003c/ins\u003e Motion Magnification: [[paper](https://arxiv.org/pdf/2402.11957.pdf)] [[codes](https://github.com/OpenImagingLab/emm)] [[project](https://openimaginglab.github.io/emm/)]\n+ \u003cins\u003eFrequency Decoupling\u003c/ins\u003e for Motion Magnification via Multi-Level Isomorphic Architecture: [[paper](https://arxiv.org/pdf/2403.07347.pdf)] [[codes](https://github.com/Jiafei127/FD4MM)]\n\n# Update\n**(2023/11/05) Add notebook demo for offline inference. Feel free to email me or leave issues if you want any help I can do.**\n\n**(2023/04/07) I find there are still a few friends like you who have interests in this old repo, so I make a Colab demo for easy inference if you want. And I'm sorry for my stupid codes years ago, I felt painful when I used them for the Colab demo... And you know, some still exist 😂 But if you have any trouble with it, feel free to leave an issue or send an e-mail to me.**\n\nBesides, as tested, this repo can be run with **PyTorch 2.0**\n\n*Given the video, and amplify it with only one click for all steps:*\n\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1inOucehJXUAVBlRhZvo650SoOPLKQFNv#scrollTo=BjgKRohk7Q5M)\n\n\u003cp align=\"center\"\u003e\n\u003cimg width=\"768\" alt=\"VMM_colab_demo\" src=\"https://user-images.githubusercontent.com/25921713/230589573-68d65731-ff7a-4b34-9aa3-e29b4dc6db4b.png\"\u003e\n\u003c/p\u003e\n\n# Env\n`conda install pytorch==2.0.0 torchvision==0.15.1 pytorch-cuda=11.8 -c pytorch -c nvidia`\n\n`pip install -r requirements.txt`\n\n# Data preparation\n\n0. About the synthetic dataset for **training**, please refer to the official repository mentioned above or download [here](https://drive.google.com/drive/folders/19K09QLouiV5N84wZiTPUMdoH9-UYqZrX?usp=sharing).\n\n1. About the video datasets for **validation**, you can also download the preprocessed frames [here](https://drive.google.com/drive/folders/19K09QLouiV5N84wZiTPUMdoH9-UYqZrX?usp=sharing), which is named train_vid_frames.zip.\n\n2. Check the settings of val_dir in **config.py** and modify it if necessary.\n\n3. To convert the **validation** video into frames:\n\n    `mkdir VIDEO_NAME \u0026\u0026 ffmpeg -i VIDEO_NAME.mp4 -f image2 VIDEO_NAME/%06d.png`\n\n\u003e Tips: ffmpeg can also be installed by conda.\n\n4. Modify the frames into **frameA/frameB/frameC**:\n\n    `python make_frameACB.py `(remember adapt the 'if' at the beginning of the program to select videos.)\n\n# Little differences from the official codes\n\n1. **Poisson noise** is not used here because I was a bit confused about that in official code. Although I coded it in data.py, and it works exactly the same as the official codes as I checked by examples.\n2. About the **optimizer**, we kept it the same as that in the original paper -- Adam(lr=1e-4, betas=(0.9, 0.999)) with no weight decay, which is different from the official codes.\n3. About the \u003cimg src=\"https://latex.codecogs.com/svg.latex?\\lambda\" title=\"\\lambda\" /\u003e in loss, we also adhere to the original paper -- set to 0.1, which is different from the official codes.\n4. The **temporal filter** is currently a bit confusing for me, so I haven't made the part of testing with temporal filter, sorry for that:(...\n\n# One thing **important**\n\nIf you check the Fig.2-a in the original paper, you will find that the predicted magnified frame \u003cimg src=\"https://latex.codecogs.com/svg.latex?\\hat{Y}\" title=\"\\y_hat\" /\u003e is actually \u003cimg src=\"https://latex.codecogs.com/svg.latex?texture(X_b)+motion(X_a-\u003eX_b)*\\alpha\" title=\"texture(X_b)+motion(X_a-\u003eX_b)*\\alpha\" /\u003e, although the former one is theoretically same as \u003cimg src=\"https://latex.codecogs.com/svg.latex?texture(X_a)+motion(X_a-\u003eX_b)*(\\alpha+1)\" /\u003e   with the same  \u003cimg src=\"https://latex.codecogs.com/svg.latex?\\alpha\" title=\"\\alpha\" /\u003e .\n\n\u003cimg src=\"materials/Fig2-a.png\" alt=\"Fig2-a\" style=\"zoom:60%;\" div align=center /\u003e\n\nHowever, what makes it matter is that the authors used perturbation for regularization, and the images in the dataset given has 4 parts:\n\n1. frameA:  \u003cimg src=\"https://latex.codecogs.com/svg.latex?X_a\" /\u003e , unperturbed;\n2. frameB: perturbed frameC, is actually   \u003cimg src=\"https://latex.codecogs.com/svg.latex?X_{b}^{'}\" /\u003e  in the paper,\n3. frameC: the real   \u003cimg src=\"https://latex.codecogs.com/svg.latex?X_b\" /\u003e , unperturbed;\n4. **amplified**: represent both   \u003cimg src=\"https://latex.codecogs.com/svg.latex?Y\" /\u003e  and   \u003cimg src=\"https://latex.codecogs.com/svg.latex?Y^{'}\" /\u003e , perturbed.\n\nHere is the first training sample, where you can see clear that **no perturbation** between **A-C** nor between **B-amp**, and no motion between B-C:\n\n\u003cimg src=\"materials/dogs.png\" alt=\"dog\" style=\"zoom: 67%;\" div align=center /\u003e\n\nGiven that, we don't have the unperturbed amplified frame, so **we can only use the former formula**(with  \u003cimg src=\"https://latex.codecogs.com/svg.latex?texture(X_b)\" /\u003e ). Besides, if you check the **loss** in the original paper, you will find the   \u003cimg src=\"https://latex.codecogs.com/svg.latex?L_1(V_{b}^{'},V_{Y}^{'})\" /\u003e, where is the  \u003cimg src=\"https://latex.codecogs.com/svg.latex?V_{Y}^{'}\" /\u003e?... I also referred to some third-party reproductions on this problem which confused me a lot, but none of them solve it. And some just gave 0 to   \u003cimg src=\"https://latex.codecogs.com/svg.latex?L_1(V_{b}^{'},V_{Y}^{'})\" /\u003e  manually, so I think they noticed this problem too but didn't manage to understand it.\n\nHere are some links to the issues about this problem in the official repository, [issue-1](https://github.com/12dmodel/deep_motion_mag/issues/3), [issue-2](https://github.com/12dmodel/deep_motion_mag/issues/5), [issue-3](https://github.com/12dmodel/deep_motion_mag/issues/4), if you want to check them.\n\n# Run\n`bash run.sh` to train and test.\n\nIt took me around 20 hours to train for 12 epochs on a single TITAN-Xp.\n\nIf you don't want to use all the 100,000 groups to train, you can modify the `frames_train='coco100000'` in config.py to coco30000 or some other number.\n\nYou can **download the weights**-ep12 from [the release](https://github.com/ZhengPeng7/motion_magnification_learning-based/releases/tag/v1.0), and `python test_videos.py baby-guitar-yourself-...` to do the test.\n\n# Results\n\nHere are some results generated from the model trained on the whole synthetic dataset for **12** epochs. \n\nBaby, amplification factor = 50\n\n![baby](materials/baby_comp.gif)\n\nGuitar, amplification factor = 20\n\n![guitar](materials/guitar_comp.gif)\n\nAnd I also took a video on the face of myself with amplification factor 20, which showed a Chinese idiom called '夺眶而出'😂.\n\n![myself](materials/myself_comp.gif)\n\n\u003e Any question, all welcome:)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhengpeng7%2Fmotion_magnification_learning-based","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzhengpeng7%2Fmotion_magnification_learning-based","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhengpeng7%2Fmotion_magnification_learning-based/lists"}