{"id":15039325,"url":"https://github.com/hzwer/eccv2022-rife","last_synced_at":"2025-04-27T03:58:12.079Z","repository":{"id":37423395,"uuid":"312179734","full_name":"hzwer/ECCV2022-RIFE","owner":"hzwer","description":"ECCV2022 - Real-Time Intermediate Flow Estimation for Video Frame Interpolation","archived":false,"fork":false,"pushed_at":"2025-02-12T09:01:56.000Z","size":10990,"stargazers_count":4825,"open_issues_count":79,"forks_count":461,"subscribers_count":77,"default_branch":"main","last_synced_at":"2025-04-27T03:58:07.742Z","etag":null,"topics":["aigc","computer-vision","deep-learning","slomo-filter","video-interpolation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hzwer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-11-12T05:46:23.000Z","updated_at":"2025-04-27T03:09:42.000Z","dependencies_parsed_at":"2023-02-17T05:01:04.558Z","dependency_job_id":"685d4dd1-3827-48ad-896d-4653abff641d","html_url":"https://github.com/hzwer/ECCV2022-RIFE","commit_stats":null,"previous_names":["hzwer/eccv2022-rife","hzwer/arxiv2021-rife","hzwer/rife","megvii-research/eccv2022-rife"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hzwer%2FECCV2022-RIFE","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hzwer%2FECCV2022-RIFE/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hzwer%2FECCV2022-RIFE/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hzwer%2FECCV2022-RIFE/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hzwer","download_url":"https://codeload.github.com/hzwer/ECCV2022-RIFE/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251085194,"owners_count":21533841,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aigc","computer-vision","deep-learning","slomo-filter","video-interpolation"],"created_at":"2024-09-24T20:42:27.855Z","updated_at":"2025-04-27T03:58:12.061Z","avatar_url":"https://github.com/hzwer.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Real-Time Intermediate Flow Estimation for Video Frame Interpolation\n## Introduction\nThis project is the implement of [Real-Time Intermediate Flow Estimation for Video Frame Interpolation](https://arxiv.org/abs/2011.06294). Currently, our model can run 30+FPS for 2X 720p interpolation on a 2080Ti GPU. It supports arbitrary-timestep interpolation between a pair of images.\n\n**2024.08 - We find that [4.22.lite](https://github.com/hzwer/Practical-RIFE/tree/main?tab=readme-ov-file#model-list) is quite suitable for post-processing of [some diffusion model generated videos](https://drive.google.com/drive/folders/1hSzUn10Era3JCaVz0Z5Eg4wT9R6eJ9U9?usp=sharing).**\n\n2023.11 - We recently release new [v4.7-4.10](https://github.com/hzwer/Practical-RIFE/tree/main#model-list) optimized for anime scenes! We draw from [SAFA](https://github.com/megvii-research/WACV2024-SAFA/tree/main)’s research.\n\n2022.7.4 - Our paper is accepted by ECCV2022. Thanks to all relevant authors, contributors and users!\n\nFrom 2020 to 2022, we submitted RIFE for five submissions（rejected by CVPR21 ICCV21 AAAI22 CVPR22). Thanks to all anonymous reviewers, your suggestions have helped to significantly improve the paper!\n\n[ECCV Poster](https://drive.google.com/file/d/1xCXuLUCSwhN61kvIF8jxDvQiUGtLK0kN/view?usp=sharing) | [ECCV 5-min presentation](https://youtu.be/qdp-NYqWQpA) | [论文中文介绍](https://zhuanlan.zhihu.com/p/568553080) | [rebuttal (2WA1WR-\u003e3WA)](https://drive.google.com/file/d/16IVjwRpwbTuJbYyTn4PizKX8I257QxY-/view?usp=sharing)\n\n## [YouTube](https://www.youtube.com/results?search_query=rife+interpolation\u0026sp=CAM%253D) | [BiliBili](https://search.bilibili.com/all?keyword=SVFI\u0026order=stow\u0026duration=0\u0026tids_1=0) | [Colab](https://colab.research.google.com/github/hzwer/ECCV2022-RIFE/blob/main/Colab_demo.ipynb) | [Tutorial](https://www.youtube.com/watch?v=gf_on-dbwyU\u0026feature=emb_title) | [V2EX](https://www.v2ex.com/t/984548)\n\n**Pinned Software: [RIFE-App](https://grisk.itch.io/rife-app) | [FlowFrames](https://nmkd.itch.io/flowframes) | [SVFI (中文)](https://github.com/YiWeiHuang-stack/Squirrel-Video-Frame-Interpolation)**\n\n16X interpolation results from two input images: \n\n![Demo](./demo/I2_slomo_clipped.gif)\n![Demo](./demo/D2_slomo_clipped.gif)\n\n## Software\n[Flowframes](https://nmkd.itch.io/flowframes) | [SVFI(中文)](https://github.com/YiWeiHuang-stack/Squirrel-Video-Frame-Interpolation) | [Waifu2x-Extension-GUI](https://github.com/AaronFeng753/Waifu2x-Extension-GUI) | [Autodesk Flame](https://vimeo.com/505942142) | [SVP](https://www.svp-team.com/wiki/RIFE_AI_interpolation) | [MPV_lazy](https://github.com/hooke007/MPV_lazy) | [enhancr](https://github.com/mafiosnik777/enhancr)\n\n[RIFE-App(Paid)](https://grisk.itch.io/rife-app) | [Steam-VFI(Paid)](https://store.steampowered.com/app/1692080/SVFI/) \n\nWe are not responsible for and participating in the development of above software. According to the open source license, we respect the commercial behavior of other developers.\n\n[VapourSynth-RIFE](https://github.com/HolyWu/vs-rife) | [RIFE-ncnn-vulkan](https://github.com/nihui/rife-ncnn-vulkan) | [VapourSynth-RIFE-ncnn-Vulkan](https://github.com/styler00dollar/VapourSynth-RIFE-ncnn-Vulkan) | [vs-mlrt](https://github.com/AmusementClub/vs-mlrt)\n\n\u003cimg src=\"https://api.star-history.com/svg?repos=megvii-research/ECCV2022-RIFE,Justin62628/Squirrel-RIFE,n00mkrad/flowframes,nihui/rife-ncnn-vulkan,hzwer/Practical-RIFE\u0026type=Date\" height=\"320\" width=\"480\" /\u003e\n\nIf you are a developer, welcome to follow [Practical-RIFE](https://github.com/hzwer/Practical-RIFE), which aims to make RIFE more practical for users by adding various features and design new models with faster speed.\n\nYou may check [this pull request](https://github.com/megvii-research/ECCV2022-RIFE/pull/300) for supporting macOS.\n## CLI Usage\n\n### Installation\n\n```\ngit clone git@github.com:megvii-research/ECCV2022-RIFE.git\ncd ECCV2022-RIFE\npip3 install -r requirements.txt\n```\n\n* Download the pretrained **HD** models from [here](https://drive.google.com/file/d/1APIzVeI-4ZZCEuIRE1m6WYfSCaOsi_7_/view?usp=sharing). (百度网盘链接:https://pan.baidu.com/share/init?surl=u6Q7-i4Hu4Vx9_5BJibPPA 密码:hfk3，把压缩包解开后放在 train_log/\\*)\n\n* Unzip and move the pretrained parameters to train_log/\\*\n\n* This model is not reported by our paper, for our paper model please refer to [evaluation](https://github.com/hzwer/ECCV2022-RIFE#evaluation).\n\n### Run\n\n**Video Frame Interpolation**\n\nYou can use our [demo video](https://drive.google.com/file/d/1i3xlKb7ax7Y70khcTcuePi6E7crO_dFc/view?usp=sharing) or your own video. \n```\npython3 inference_video.py --exp=1 --video=video.mp4 \n```\n(generate video_2X_xxfps.mp4)\n```\npython3 inference_video.py --exp=2 --video=video.mp4\n```\n(for 4X interpolation)\n```\npython3 inference_video.py --exp=1 --video=video.mp4 --scale=0.5\n```\n(If your video has very high resolution such as 4K, we recommend set --scale=0.5 (default 1.0). If you generate disordered pattern on your videos, try set --scale=2.0. This parameter control the process resolution for optical flow model.)\n```\npython3 inference_video.py --exp=2 --img=input/\n```\n(to read video from pngs, like input/0.png ... input/612.png, ensure that the png names are numbers)\n```\npython3 inference_video.py --exp=2 --video=video.mp4 --fps=60\n```\n(add slomo effect, the audio will be removed)\n```\npython3 inference_video.py --video=video.mp4 --montage --png\n```\n(if you want to montage the origin video and save the png format output)\n\n**Extended Application**\n\nYou may refer to [#278](https://github.com/megvii-research/ECCV2022-RIFE/issues/278#event-7199085190) for **Optical Flow Estimation** and refer to [#291](https://github.com/hzwer/ECCV2022-RIFE/issues/291#issuecomment-1328685348) for **Video Stitching**.\n\n**Image Interpolation**\n\n```\npython3 inference_img.py --img img0.png img1.png --exp=4\n```\n(2^4=16X interpolation results)\nAfter that, you can use pngs to generate mp4:\n```\nffmpeg -r 10 -f image2 -i output/img%d.png -s 448x256 -c:v libx264 -pix_fmt yuv420p output/slomo.mp4 -q:v 0 -q:a 0\n```\nYou can also use pngs to generate gif:\n```\nffmpeg -r 10 -f image2 -i output/img%d.png -s 448x256 -vf \"split[s0][s1];[s0]palettegen=stats_mode=single[p];[s1][p]paletteuse=new=1\" output/slomo.gif\n```\n\n### Run in docker\nPlace the pre-trained models in `train_log/\\*.pkl` (as above)\n\nBuilding the container:\n```\ndocker build -t rife -f docker/Dockerfile .\n```\n\nRunning the container:\n```\ndocker run --rm -it -v $PWD:/host rife:latest inference_video --exp=1 --video=untitled.mp4 --output=untitled_rife.mp4\n```\n```\ndocker run --rm -it -v $PWD:/host rife:latest inference_img --img img0.png img1.png --exp=4\n```\n\nUsing gpu acceleration (requires proper gpu drivers for docker):\n```\ndocker run --rm -it --gpus all -v /dev/dri:/dev/dri -v $PWD:/host rife:latest inference_video --exp=1 --video=untitled.mp4 --output=untitled_rife.mp4\n```\n\n## Evaluation\nDownload [RIFE model](https://drive.google.com/file/d/1h42aGYPNJn2q8j_GVkS_yDu__G_UZ2GX/view?usp=sharing) or [RIFE_m model](https://drive.google.com/file/d/147XVsDXBfJPlyct2jfo9kpbL944mNeZr/view?usp=sharing) reported by our paper.\n\n**UCF101**: Download [UCF101 dataset](https://liuziwei7.github.io/projects/VoxelFlow) at ./UCF101/ucf101_interp_ours/\n\n**Vimeo90K**: Download [Vimeo90K dataset](http://toflow.csail.mit.edu/) at ./vimeo_interp_test\n\n**MiddleBury**: Download [MiddleBury OTHER dataset](https://vision.middlebury.edu/flow/data/) at ./other-data and ./other-gt-interp\n\n**HD**: Download [HD dataset](https://github.com/baowenbo/MEMC-Net) at ./HD_dataset. We also provide a [google drive download link](https://drive.google.com/file/d/1iHaLoR2g1-FLgr9MEv51NH_KQYMYz-FA/view?usp=sharing).\n```\n# RIFE\npython3 benchmark/UCF101.py\n# \"PSNR: 35.282 SSIM: 0.9688\"\npython3 benchmark/Vimeo90K.py\n# \"PSNR: 35.615 SSIM: 0.9779\"\npython3 benchmark/MiddleBury_Other.py\n# \"IE: 1.956\"\npython3 benchmark/HD.py\n# \"PSNR: 32.14\"\n\n# RIFE_m\npython3 benchmark/HD_multi_4X.py\n# \"PSNR: 22.96(544*1280), 31.87(720p), 34.25(1080p)\"\n```\n\n## Training and Reproduction\nDownload [Vimeo90K dataset](http://toflow.csail.mit.edu/).\n\nWe use 16 CPUs, 4 GPUs and 20G memory for training: \n```\npython3 -m torch.distributed.launch --nproc_per_node=4 train.py --world_size=4\n```\n\n## Revision History\n\n2021.3.18 [arXiv](https://arxiv.org/pdf/2011.06294v5.pdf): Modify the main experimental data, especially the runtime related issues.\n\n2021.8.12 [arXiv](https://arxiv.org/pdf/2011.06294v6.pdf): Remove pre-trained model dependency and propose privileged distillation scheme for frame interpolation. Remove [census loss](https://github.com/hzwer/arXiv2021-RIFE/blob/0e241367847a0895748e64c6e1604c94db54d395/model/loss.py#L20) supervision.\n\n2021.11.17 [arXiv](https://arxiv.org/pdf/2011.06294v11.pdf): Support arbitrary-time frame interpolation, aka RIFEm and add more experiments.\n\n## Recommend\nWe sincerely recommend some related papers:\n\nCVPR22 - [Optimizing Video Prediction via Video Frame Interpolation](https://openaccess.thecvf.com/content/CVPR2022/html/Wu_Optimizing_Video_Prediction_via_Video_Frame_Interpolation_CVPR_2022_paper.html)\n\nCVPR22 - [Video Frame Interpolation with Transformer](https://openaccess.thecvf.com/content/CVPR2022/html/Lu_Video_Frame_Interpolation_With_Transformer_CVPR_2022_paper.html)\n\nCVPR22 - [IFRNet: Intermediate Feature Refine Network for Efficient Frame Interpolation](https://openaccess.thecvf.com/content/CVPR2022/html/Kong_IFRNet_Intermediate_Feature_Refine_Network_for_Efficient_Frame_Interpolation_CVPR_2022_paper.html)\n\nCVPR23 - [A Dynamic Multi-Scale Voxel Flow Network for Video Prediction](https://huxiaotaostasy.github.io/DMVFN/)\n\nCVPR23 - [Extracting Motion and Appearance via Inter-Frame Attention for Efficient Video Frame Interpolation](https://arxiv.org/abs/2303.00440)\n\n## Citation\nIf you think this project is helpful, please feel free to leave a star or cite our paper:\n\n```\n@inproceedings{huang2022rife,\n  title={Real-Time Intermediate Flow Estimation for Video Frame Interpolation},\n  author={Huang, Zhewei and Zhang, Tianyuan and Heng, Wen and Shi, Boxin and Zhou, Shuchang},\n  booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},\n  year={2022}\n}\n```\n\n## Reference\n\nOptical Flow:\n[ARFlow](https://github.com/lliuz/ARFlow)  [pytorch-liteflownet](https://github.com/sniklaus/pytorch-liteflownet)  [RAFT](https://github.com/princeton-vl/RAFT)  [pytorch-PWCNet](https://github.com/sniklaus/pytorch-pwc)\n\nVideo Interpolation: \n[DVF](https://github.com/lxx1991/pytorch-voxel-flow)  [TOflow](https://github.com/Coldog2333/pytoflow)  [SepConv](https://github.com/sniklaus/sepconv-slomo)  [DAIN](https://github.com/baowenbo/DAIN)  [CAIN](https://github.com/myungsub/CAIN)  [MEMC-Net](https://github.com/baowenbo/MEMC-Net)   [SoftSplat](https://github.com/sniklaus/softmax-splatting)  [BMBC](https://github.com/JunHeum/BMBC)  [EDSC](https://github.com/Xianhang/EDSC-pytorch)  [EQVI](https://github.com/lyh-18/EQVI)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhzwer%2Feccv2022-rife","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhzwer%2Feccv2022-rife","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhzwer%2Feccv2022-rife/lists"}