{"id":13469229,"url":"https://github.com/google-research/frame-interpolation","last_synced_at":"2025-05-15T01:05:46.494Z","repository":{"id":37443204,"uuid":"447389581","full_name":"google-research/frame-interpolation","owner":"google-research","description":"FILM: Frame Interpolation for Large Motion, In ECCV 2022.","archived":false,"fork":false,"pushed_at":"2024-08-10T01:12:48.000Z","size":27595,"stargazers_count":2957,"open_issues_count":7,"forks_count":297,"subscribers_count":44,"default_branch":"main","last_synced_at":"2025-04-11T14:16:59.849Z","etag":null,"topics":["forward-warp","frame-interpolation","optical-flow","splatting","video"],"latest_commit_sha":null,"homepage":"https://film-net.github.io","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-12T22:29:45.000Z","updated_at":"2025-04-10T10:09:10.000Z","dependencies_parsed_at":"2024-07-31T15:10:35.415Z","dependency_job_id":"7b68e2ec-6fd9-4fd9-b779-5a10f3ccbaf1","html_url":"https://github.com/google-research/frame-interpolation","commit_stats":{"total_commits":49,"total_committers":6,"mean_commits":8.166666666666666,"dds":"0.22448979591836737","last_synced_commit":"69f8708f08e62c2edf46a27616a4bfcf083e2076"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fframe-interpolation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fframe-interpolation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fframe-interpolation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fframe-interpolation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google-research","download_url":"https://codeload.github.com/google-research/frame-interpolation/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254254039,"owners_count":22039792,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["forward-warp","frame-interpolation","optical-flow","splatting","video"],"created_at":"2024-07-31T15:01:29.786Z","updated_at":"2025-05-15T01:05:43.141Z","avatar_url":"https://github.com/google-research.png","language":"Python","funding_links":[],"categories":["Python","Uncategorized","视频生成_补帧_摘要"],"sub_categories":["Uncategorized","资源传输下载"],"readme":"# FILM: Frame Interpolation for Large Motion\n\n### [Website](https://film-net.github.io/) | [Paper](https://arxiv.org/pdf/2202.04901.pdf) | [Google AI Blog](https://ai.googleblog.com/2022/10/large-motion-frame-interpolation.html) | [Tensorflow Hub Colab](https://www.tensorflow.org/hub/tutorials/tf_hub_film_example) | [YouTube](https://www.youtube.com/watch?v=OAD-BieIjH4) \u003cbr\u003e\n\nThe official Tensorflow 2 implementation of our high quality frame interpolation neural network. We present a unified single-network approach that doesn't use additional pre-trained networks, like optical flow or depth, and yet achieve state-of-the-art results. We use a multi-scale feature extractor that shares the same convolution weights across the scales. Our model is trainable from frame triplets alone. \u003cbr\u003e\n\n[FILM: Frame Interpolation for Large Motion](https://arxiv.org/abs/2202.04901) \u003cbr /\u003e\n[Fitsum Reda](https://fitsumreda.github.io/)\u003csup\u003e1\u003c/sup\u003e, [Janne Kontkanen](https://scholar.google.com/citations?user=MnXc4JQAAAAJ\u0026hl=en)\u003csup\u003e1\u003c/sup\u003e, [Eric Tabellion](http://www.tabellion.org/et/)\u003csup\u003e1\u003c/sup\u003e, [Deqing Sun](https://deqings.github.io/)\u003csup\u003e1\u003c/sup\u003e, [Caroline Pantofaru](https://scholar.google.com/citations?user=vKAKE1gAAAAJ\u0026hl=en)\u003csup\u003e1\u003c/sup\u003e, [Brian Curless](https://homes.cs.washington.edu/~curless/)\u003csup\u003e1,2\u003c/sup\u003e\u003cbr /\u003e\n\u003csup\u003e1\u003c/sup\u003eGoogle Research, \u003csup\u003e2\u003c/sup\u003eUniversity of Washington\u003cbr /\u003e\nIn ECCV 2022.\n\n![A sample 2 seconds moment.](https://github.com/googlestaging/frame-interpolation/blob/main/moment.gif)\nFILM transforms near-duplicate photos into a slow motion footage that look like it is shot with a video camera.\n\n## Web Demo\n\nIntegrated into [Hugging Face Spaces 🤗](https://huggingface.co/spaces) using [Gradio](https://github.com/gradio-app/gradio). Try out the Web Demo: [![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/johngoad/frame-interpolation)\n\nTry the interpolation model with the replicate web demo at \n[![Replicate](https://replicate.com/google-research/frame-interpolation/badge)](https://replicate.com/google-research/frame-interpolation)\n\nTry FILM to interpolate between two or more images with the PyTTI-Tools at [![PyTTI-Tools:FILM](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.sandbox.google.com/github/pytti-tools/frame-interpolation/blob/main/PyTTI_Tools_FiLM-colab.ipynb#scrollTo=-7TD7YZJbsy_)\n\nAn alternative Colab for running FILM on arbitrarily more input images, not just on two images, [![FILM-Gdrive](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1NuaPPSvUhYafymUf2mEkvhnEtpD5oihs)\n\n## Change Log\n* **Nov 28, 2022**: Upgrade `eval.interpolator_cli` for **high resolution frame interpolation**. `--block_height` and `--block_width` determine the total number of patches (`block_height*block_width`) to subdivide the input images. By default, both arguments are set to 1, and so no subdivision will be done.\n* **Mar 12, 2022**: Support for Windows, see [WINDOWS_INSTALLATION.md](https://github.com/google-research/frame-interpolation/blob/main/WINDOWS_INSTALLATION.md).\n* **Mar 09, 2022**: Support for **high resolution frame interpolation**. Set `--block_height` and `--block_width` in `eval.interpolator_test` to extract patches from the inputs, and reconstruct the interpolated frame from the iteratively interpolated patches.\n\n## Installation\n\n*   Get Frame Interpolation source codes\n\n```\ngit clone https://github.com/google-research/frame-interpolation\ncd frame-interpolation\n```\n\n*   Optionally, pull the recommended Docker base image\n\n```\ndocker pull gcr.io/deeplearning-platform-release/tf2-gpu.2-6:latest\n```\n\n* If you do not use Docker, set up your NVIDIA GPU environment with:\n    * [Anaconda Python 3.9](https://www.anaconda.com/products/individual)\n    * [CUDA Toolkit 11.2.1](https://developer.nvidia.com/cuda-11.2.1-download-archive)\n    * [cuDNN 8.1.0](https://developer.nvidia.com/rdp/cudnn-download)\n\n*   Install frame interpolation dependencies\n\n```\npip3 install -r requirements.txt\nsudo apt-get install -y ffmpeg\n```\n\n### See [WINDOWS_INSTALLATION](https://github.com/google-research/frame-interpolation/blob/main/WINDOWS_INSTALLATION.md) for Windows Support\n\n## Pre-trained Models\n\n*   Create a directory where you can keep large files. Ideally, not in this\n    directory.\n\n```\nmkdir -p \u003cpretrained_models\u003e\n```\n\n*   Download pre-trained TF2 Saved Models from\n    [google drive](https://drive.google.com/drive/folders/1q8110-qp225asX3DQvZnfLfJPkCHmDpy?usp=sharing)\n    and put into `\u003cpretrained_models\u003e`.\n\nThe downloaded folder should have the following structure:\n\n```\n\u003cpretrained_models\u003e/\n├── film_net/\n│   ├── L1/\n│   ├── Style/\n│   ├── VGG/\n├── vgg/\n│   ├── imagenet-vgg-verydeep-19.mat\n```\n\n## Running the Codes\n\nThe following instructions run the interpolator on the photos provided in\n'frame-interpolation/photos'.\n\n### One mid-frame interpolation\n\nTo generate an intermediate photo from the input near-duplicate photos, simply run:\n\n```\npython3 -m eval.interpolator_test \\\n   --frame1 photos/one.png \\\n   --frame2 photos/two.png \\\n   --model_path \u003cpretrained_models\u003e/film_net/Style/saved_model \\\n   --output_frame photos/output_middle.png\n```\n\nThis will produce the sub-frame at `t=0.5` and save as 'photos/output_middle.png'.\n\n### Many in-between frames interpolation\n\nIt takes in a set of directories identified by a glob (--pattern). Each directory\nis expected to contain at least two input frames, with each contiguous frame\npair treated as an input to generate in-between frames. Frames should be named such that when sorted (naturally) with `natsort`, their desired order is unchanged.\n\n```\npython3 -m eval.interpolator_cli \\\n   --pattern \"photos\" \\\n   --model_path \u003cpretrained_models\u003e/film_net/Style/saved_model \\\n   --times_to_interpolate 6 \\\n   --output_video\n```\n\nYou will find the interpolated frames (including the input frames) in\n'photos/interpolated_frames/', and the interpolated video at\n'photos/interpolated.mp4'.\n\nThe number of frames is determined by `--times_to_interpolate`, which controls\nthe number of times the frame interpolator is invoked. When the number of frames\nin a directory is `num_frames`, the number of output frames will be\n`(2^times_to_interpolate+1)*(num_frames-1)`.\n\n## Datasets\n\nWe use [Vimeo-90K](http://data.csail.mit.edu/tofu/dataset/vimeo_triplet.zip) as\nour main training dataset. For quantitative evaluations, we rely on commonly\nused benchmark datasets, specifically:\n\n*   [Vimeo-90K](http://data.csail.mit.edu/tofu/testset/vimeo_interp_test.zip)\n*   [Middlebury-Other](https://vision.middlebury.edu/flow/data)\n*   [UCF101](https://people.cs.umass.edu/~hzjiang/projects/superslomo/UCF101_results.zip)\n*   [Xiph](https://github.com/sniklaus/softmax-splatting/blob/master/benchmark.py)\n\n### Creating a TFRecord\n\nThe training and benchmark evaluation scripts expect the frame triplets in the\n[TFRecord](https://www.tensorflow.org/tutorials/load_data/tfrecord) storage format. \u003cbr /\u003e\n\nWe have included scripts that encode the relevant frame triplets into a\n[tf.train.Example](https://www.tensorflow.org/api_docs/python/tf/train/Example)\ndata format, and export to a TFRecord file. \u003cbr /\u003e\n\nYou can use the commands `python3 -m\ndatasets.create_\u003cdataset_name\u003e_tfrecord --help` for more information.\n\nFor example, run the command below to create a TFRecord for the Middlebury-other\ndataset. Download the [images](https://vision.middlebury.edu/flow/data) and point `--input_dir` to the unzipped folder path.\n\n```\npython3 -m datasets.create_middlebury_tfrecord \\\n  --input_dir=\u003croot folder of middlebury-other\u003e \\\n  --output_tfrecord_filepath=\u003coutput tfrecord filepath\u003e \\\n  --num_shards=3\n```\n\nThe above command will output a TFRecord file with 3 shards as `\u003coutput tfrecord filepath\u003e@3`.\n\n## Training\n\nBelow are our training gin configuration files for the different loss function:\n\n```\ntraining/\n├── config/\n│   ├── film_net-L1.gin\n│   ├── film_net-VGG.gin\n│   ├── film_net-Style.gin\n```\n\nTo launch a training, simply pass the configuration filepath to the desired\nexperiment. \u003cbr /\u003e\nBy default, it uses all visible GPUs for training. To debug or train\non a CPU, append `--mode cpu`.\n\n```\npython3 -m training.train \\\n   --gin_config training/config/\u003cconfig filename\u003e.gin \\\n   --base_folder \u003cbase folder for all training runs\u003e \\\n   --label \u003cdescriptive label for the run\u003e\n```\n\n*   When training finishes, the folder structure will look like this:\n\n```\n\u003cbase_folder\u003e/\n├── \u003clabel\u003e/\n│   ├── config.gin\n│   ├── eval/\n│   ├── train/\n│   ├── saved_model/\n```\n\n### Build a SavedModel\n\nOptionally, to build a\n[SavedModel](https://www.tensorflow.org/guide/saved_model) format from a trained\ncheckpoints folder, you can use this command:\n\n```\npython3 -m training.build_saved_model_cli \\\n   --base_folder \u003cbase folder of training sessions\u003e \\\n   --label \u003cthe name of the run\u003e\n```\n\n*   By default, a SavedModel is created when the training loop ends, and it will be saved at\n    `\u003cbase_folder\u003e/\u003clabel\u003e/saved_model`.\n\n## Evaluation on Benchmarks\n\nBelow, we provided the evaluation gin configuration files for the benchmarks we\nhave considered:\n\n```\neval/\n├── config/\n│   ├── middlebury.gin\n│   ├── ucf101.gin\n│   ├── vimeo_90K.gin\n│   ├── xiph_2K.gin\n│   ├── xiph_4K.gin\n```\n\nTo run an evaluation, simply pass the configuration file of the desired evaluation dataset. \u003cbr /\u003e\nIf a GPU is visible, it runs on it.\n\n```\npython3 -m eval.eval_cli \\\n   --gin_config eval/config/\u003ceval_dataset\u003e.gin \\\n   --model_path \u003cpretrained_models\u003e/film_net/L1/saved_model\n```\n\nThe above command will produce the PSNR and SSIM scores presented in the paper.\n\n## Citation\n\nIf you find this implementation useful in your works, please acknowledge it\nappropriately by citing:\n\n```\n@inproceedings{reda2022film,\n title = {FILM: Frame Interpolation for Large Motion},\n author = {Fitsum Reda and Janne Kontkanen and Eric Tabellion and Deqing Sun and Caroline Pantofaru and Brian Curless},\n booktitle = {European Conference on Computer Vision (ECCV)},\n year = {2022}\n}\n```\n\n```\n@misc{film-tf,\n  title = {Tensorflow 2 Implementation of \"FILM: Frame Interpolation for Large Motion\"},\n  author = {Fitsum Reda and Janne Kontkanen and Eric Tabellion and Deqing Sun and Caroline Pantofaru and Brian Curless},\n  year = {2022},\n  publisher = {GitHub},\n  journal = {GitHub repository},\n  howpublished = {\\url{https://github.com/google-research/frame-interpolation}}\n}\n```\n\n## Acknowledgments\n\nWe would like to thank Richard Tucker, Jason Lai and David Minnen. We would also\nlike to thank Jamie Aspinall for the imagery included in this repository.\n\n## Coding style\n\n*   2 spaces for indentation\n*   80 character line length\n*   PEP8 formatting\n\n## Disclaimer\n\nThis is not an officially supported Google product.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Fframe-interpolation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle-research%2Fframe-interpolation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Fframe-interpolation/lists"}