{"id":24297725,"url":"https://github.com/ant-research/codef","last_synced_at":"2025-05-14T15:06:51.488Z","repository":{"id":188483101,"uuid":"678805749","full_name":"ant-research/CoDeF","owner":"ant-research","description":"[CVPR'24 Highlight] Official PyTorch implementation of CoDeF: Content Deformation Fields for Temporally Consistent Video Processing","archived":false,"fork":false,"pushed_at":"2024-04-07T03:27:29.000Z","size":55595,"stargazers_count":4861,"open_issues_count":30,"forks_count":382,"subscribers_count":70,"default_branch":"main","last_synced_at":"2025-04-06T08:02:22.275Z","etag":null,"topics":["editing","video"],"latest_commit_sha":null,"homepage":"https://qiuyu96.github.io/CoDeF/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ant-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-15T12:25:31.000Z","updated_at":"2025-04-03T13:38:24.000Z","dependencies_parsed_at":"2023-08-15T15:08:26.573Z","dependency_job_id":"80f2b825-4616-4d79-9643-a18fc8864516","html_url":"https://github.com/ant-research/CoDeF","commit_stats":{"total_commits":24,"total_committers":7,"mean_commits":"3.4285714285714284","dds":0.5833333333333333,"last_synced_commit":"b1d6f3677561675cf7672e9c60b865319847d254"},"previous_names":["qiuyu96/codef","ant-research/codef"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ant-research%2FCoDeF","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ant-research%2FCoDeF/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ant-research%2FCoDeF/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ant-research%2FCoDeF/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ant-research","download_url":"https://codeload.github.com/ant-research/CoDeF/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248708460,"owners_count":21149002,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["editing","video"],"created_at":"2025-01-16T20:04:22.691Z","updated_at":"2025-04-13T11:38:16.319Z","avatar_url":"https://github.com/ant-research.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CoDeF: Content Deformation Fields for Temporally Consistent Video Processing\n\n\u003cimg src='docs/teaser.gif'\u003e\u003c/img\u003e\n\n[Hao Ouyang](https://ken-ouyang.github.io/)\\*, [Qiuyu Wang](https://github.com/qiuyu96/)\\*, [Yuxi Xiao](https://henry123-boy.github.io/)\\*, [Qingyan Bai](https://scholar.google.com/citations?user=xUMjxi4AAAAJ\u0026hl=en), [Juntao Zhang](https://github.com/JordanZh), [Kecheng Zheng](https://scholar.google.com/citations?user=hMDQifQAAAAJ), [Xiaowei Zhou](https://xzhou.me/),\n[Qifeng Chen](https://cqf.io/)\u0026#8224;, [Yujun Shen](https://shenyujun.github.io/)\u0026#8224; (*equal contribution, \u0026#8224;corresponding author)\n\n**CVPR 2024 Highlight**\n\n#### [Project Page](https://qiuyu96.github.io/CoDeF/) | [Paper](https://arxiv.org/abs/2308.07926) | [High-Res Translation Demo](https://ezioby.github.io/CoDeF_Demo/) | [Colab](https://colab.research.google.com/github/camenduru/CoDeF-colab/blob/main/CoDeF_colab.ipynb)\n\n\u003c!-- Abstract: *This work presents the content deformation field **CoDeF** as a new type of video representation, which consists of a canonical content field aggregating the static contents in the entire video and a temporal deformation field recording the transformations from the canonical image (i.e., rendered from the canonical content field) to each individual frame along the time axis. Given a target video, these two fields are jointly optimized to reconstruct it through a carefully tailored rendering pipeline. We also introduce some decent regularizations into the optimization process, urging the canonical content field to inherit semantics (e.g., the object shape) from the video. With such a design, **CoDeF** naturally supports lifting image algorithms to videos, in the sense that one can apply an image algorithm to the canonical image and effortlessly propagate the outcomes to the entire video with the aid of the temporal deformation field. We experimentally show that **CoDeF** is able to lift image-to-image translation to video-to-video translation and lift keypoint detection to keypoint tracking without any training. More importantly, thanks to our lifting strategy that deploys the algorithms on only one image, we achieve superior cross-frame consistency in translated videos compared to existing video-to-video translation approaches, and even manage to track non-rigid objects like water and smog.* --\u003e\n\n## Requirements\n\nThe codebase is tested on\n\n* Ubuntu 20.04\n* Python 3.10\n* [PyTorch](https://pytorch.org/) 2.0.0\n* [PyTorch Lightning](https://www.pytorchlightning.ai/index.html) 2.0.2\n* 1 NVIDIA GPU (RTX A6000) with CUDA version 11.7. (Other GPUs are also suitable, and 10GB GPU memory is sufficient to run our code.)\n\nTo use video visualizer, please install `ffmpeg` via\n\n```shell\nsudo apt-get install ffmpeg\n```\n\nFor additional Python libraries, please install with\n\n```shell\npip install -r requirements.txt\n```\n\nOur code also depends on [tiny-cuda-nn](https://github.com/NVlabs/tiny-cuda-nn).\nSee [this repository](https://github.com/NVlabs/tiny-cuda-nn#pytorch-extension)\nfor Pytorch extension install instructions.\n\n## Data\n\n### Provided data\n\nWe have provided some videos [here](https://drive.google.com/file/d/1cKZF6ILeokCjsSAGBmummcQh0uRGaC_F/view?usp=sharing) for quick test. Please download and unzip the data and put them in the root directory. More videos can be downloaded [here](https://drive.google.com/file/d/10Msz37MpjZQFPXlDWCZqrcQjhxpQSvCI/view?usp=sharing).\n\n### Customize your own data\n\nWe segement video sequences using [SAM-Track](https://github.com/z-x-yang/Segment-and-Track-Anything). Once you obtain the mask files, place them in the folder `all_sequences/{YOUR_SEQUENCE_NAME}/{YOUR_SEQUENCE_NAME}_masks`. Next, execute the following command:\n\n```shell\ncd data_preprocessing\npython preproc_mask.py\n```\n\nWe extract optical flows of video sequences using [RAFT](https://github.com/princeton-vl/RAFT). To get started, please follow the instructions provided [here](https://github.com/princeton-vl/RAFT#demos) to download their pretrained model. Once downloaded, place the model in the `data_preprocessing/RAFT/models` folder. After that, you can execute the following command:\n\n```shell\ncd data_preprocessing/RAFT\n./run_raft.sh\n```\n\nRemember to update the sequence name and root directory in both `data_preprocessing/preproc_mask.py` and `data_preprocessing/RAFT/run_raft.sh` accordingly.\n\nAfter obtaining the files, please organize your own data as follows:\n\n```\nCoDeF\n│\n└─── all_sequences\n    │\n    └─── NAME1\n           └─ NAME1\n           └─ NAME1_masks_0 (optional)\n           └─ NAME1_masks_1 (optional)\n           └─ NAME1_flow (optional)\n           └─ NAME1_flow_confidence (optional)\n    │\n    └─── NAME2\n           └─ NAME2\n           └─ NAME2_masks_0 (optional)\n           └─ NAME2_masks_1 (optional)\n           └─ NAME2_flow (optional)\n           └─ NAME2_flow_confidence (optional)\n    │\n    └─── ...\n```\n\n## Pretrained checkpoints\n\nYou can download checkpoints pre-trained on the provided videos via\n\n| Sequence Name | Config |                           Download                           | OpenXLab | \n| :-------- | :----: | :----------------------------------------------------------: | :---------:| \n| beauty_0 | configs/beauty_0/base.yaml |  [Google drive link](https://drive.google.com/file/d/11SWfnfDct8bE16802PyqYJqsU4x6ACn8/view?usp=sharing) |[![Open in OpenXLab](https://cdn-static.openxlab.org.cn/header/openxlab_models.svg)](https://openxlab.org.cn/models/detail/HaoOuyang/CoDeF)|\n| beauty_1 | configs/beauty_1/base.yaml |  [Google drive link](https://drive.google.com/file/d/1bSK0ChbPdURWGLdtc9CPLkN4Tfnng51k/view?usp=sharing) | [![Open in OpenXLab](https://cdn-static.openxlab.org.cn/header/openxlab_models.svg)](https://openxlab.org.cn/models/detail/HaoOuyang/CoDeF) |\n| white_smoke      | configs/white_smoke/base.yaml |  [Google drive link](https://drive.google.com/file/d/1QOBCDGV2hHwxq4eL1E_45z5zhZ-wTJR7/view?usp=sharing) | [![Open in OpenXLab](https://cdn-static.openxlab.org.cn/header/openxlab_models.svg)](https://openxlab.org.cn/models/detail/HaoOuyang/CoDeF) |\n| lemon_hit      | configs/lemon_hit/base.yaml |  [Google drive link](https://drive.google.com/file/d/140ctcLbv7JTIiy53MuCYtI4_zpIvRXzq/view?usp=sharing) | [![Open in OpenXLab](https://cdn-static.openxlab.org.cn/header/openxlab_models.svg)](https://openxlab.org.cn/models/detail/HaoOuyang/CoDeF)|\n| scene_0      | configs/scene_0/base.yaml |  [Google drive link](https://drive.google.com/file/d/1abOdREarfw1DGscahOJd2gZf1Xn_zN-F/view?usp=sharing) |[![Open in OpenXLab](https://cdn-static.openxlab.org.cn/header/openxlab_models.svg)](https://openxlab.org.cn/models/detail/HaoOuyang/CoDeF)|\n\nAnd organize files as follows\n\n```\nCoDeF\n│\n└─── ckpts/all_sequences\n    │\n    └─── NAME1\n        │\n        └─── EXP_NAME (base)\n            │\n            └─── NAME1.ckpt\n    │\n    └─── NAME2\n        │\n        └─── EXP_NAME (base)\n            │\n            └─── NAME2.ckpt\n    |\n    └─── ...\n```\n\n## Train a new model\n\n```shell\n./scripts/train_multi.sh\n```\n\nwhere\n* `GPU`: Decide which GPU to train on;\n* `NAME`: Name of the video sequence;\n* `EXP_NAME`: Name of the experiment;\n* `ROOT_DIRECTORY`: Directory of the input video sequence;\n* `MODEL_SAVE_PATH`: Path to save the checkpoints;\n* `LOG_SAVE_PATH`: Path to save the logs;\n* `MASK_DIRECTORY`: Directory of the preprocessed masks (optional);\n* `FLOW_DIRECTORY`: Directory of the preprocessed optical flows (optional);\n\nPlease check configuration files in ``configs/``, and you can always add your own model config.\n\n## Test reconstruction \u003ca id=\"anchor\"\u003e\u003c/a\u003e\n\n```shell\n./scripts/test_multi.sh\n```\nAfter running the script, the reconstructed videos can be found in `results/all_sequences/{NAME}/{EXP_NAME}`, along with the canonical image.\n\n## Test video translation\n\nAfter obtaining the canonical image through [this step](#anchor), use your preferred text prompts to transfer it using [ControlNet](https://github.com/lllyasviel/ControlNet).\nOnce you have the transferred canonical image, place it in `all_sequences/${NAME}/${EXP_NAME}_control` (i.e. `CANONICAL_DIR` in `scripts/test_canonical.sh`).\n\nThen run\n\n```shell\n./scripts/test_canonical.sh\n```\n\nThe transferred results can be seen in `results/all_sequences/{NAME}/{EXP_NAME}_transformed`.\n\n*Note*: The `canonical_wh` option in the configuration file should be set with caution, usually a little larger than `img_wh`, as it determines the field of view of the canonical image.\n\n### BibTeX\n\n```bibtex\n@article{ouyang2023codef,\n      title={CoDeF: Content Deformation Fields for Temporally Consistent Video Processing},\n      author={Hao Ouyang and Qiuyu Wang and Yuxi Xiao and Qingyan Bai and Juntao Zhang and Kecheng Zheng and Xiaowei Zhou and Qifeng Chen and Yujun Shen},\n      journal={arXiv preprint arXiv:2308.07926},\n      year={2023}\n}\n```\n\n### Acknowledgements\nWe thank [camenduru](https://github.com/camenduru) for providing the [colab demo](https://github.com/camenduru/CoDeF-colab).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fant-research%2Fcodef","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fant-research%2Fcodef","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fant-research%2Fcodef/lists"}