{"id":17597684,"url":"https://github.com/trentonom0r3/ezsynth","last_synced_at":"2025-10-10T03:30:43.205Z","repository":{"id":186493574,"uuid":"675262903","full_name":"Trentonom0r3/Ezsynth","owner":"Trentonom0r3","description":"An Implementation of Ebsynth for video stylization, and the original ebsynth for image stylization as an importable python library!","archived":false,"fork":false,"pushed_at":"2024-07-31T09:15:14.000Z","size":288409,"stargazers_count":118,"open_issues_count":2,"forks_count":14,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-03T14:51:25.633Z","etag":null,"topics":["ebsynth","library","python","video","video-style-transfer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Trentonom0r3.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-06T10:41:17.000Z","updated_at":"2025-03-24T11:14:00.000Z","dependencies_parsed_at":"2023-09-26T12:22:31.409Z","dependency_job_id":"c31bb343-785e-49ad-9d53-9f5c57992fed","html_url":"https://github.com/Trentonom0r3/Ezsynth","commit_stats":null,"previous_names":["trentonom0r3/ebsynth.py","trentonom0r3/ezsynth"],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/Trentonom0r3/Ezsynth","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Trentonom0r3%2FEzsynth","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Trentonom0r3%2FEzsynth/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Trentonom0r3%2FEzsynth/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Trentonom0r3%2FEzsynth/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Trentonom0r3","download_url":"https://codeload.github.com/Trentonom0r3/Ezsynth/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Trentonom0r3%2FEzsynth/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279002631,"owners_count":26083425,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-10T02:00:06.843Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ebsynth","library","python","video","video-style-transfer"],"created_at":"2024-10-22T09:34:39.617Z","updated_at":"2025-10-10T03:30:38.191Z","avatar_url":"https://github.com/Trentonom0r3.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Ezsynth - Ebsynth Python Library\n\nReworked version, courtesy of [FuouM](https://github.com/FuouM), with masking support and some visual bug fixes. Aims to be easy to use and maintain. \n\nPerform things like style transfer, color transfer, inpainting, superimposition, video stylization and more!\nThis implementation makes use of advanced physics based edge detection and RAFT optical flow, which leads to more accurate results during synthesis.\n\n:warning: **This is not intended to be used as an installable module.**\n\nCurrently tested on:\n```\nWindows 10 - Python 3.11 - RTX3060\nUbuntu 24 - Python 3.12 - RTX4070(Laptop)\n```\n\n## Get started\n\n### Windows\n```cmd\nrem Clone this repo\ngit clone https://github.com/Trentonom0r3/Ezsynth.git \ncd Ezsynth\n\nrem (Optional) create and activate venv\npython -m venv venv\nvenv\\Scripts\\activate.bat\n\nrem Install requirements\npip install -r requirements.txt\n\nrem A precompiled ebsynth.dll is included. \nrem If don't want to rebuild, you are ready to go and can skip the following steps.  \n\nrem Clone ebsynth\ngit clone https://github.com/Trentonom0r3/ebsynth.git\n\nrem build ebsynth as lib\ncopy .\\build_ebs-win64-cpu+cuda.bat .\\ebsynth\ncd ebsynth \u0026\u0026 .\\build_ebs-win64-cpu+cuda.bat\n\nrem copy lib\ncp .\\bin\\ebsynth.so ..\\ezsynth\\utils\\ebsynth.so\n\nrem cleanup\ncd .. \u0026\u0026 rmdir /s /q .\\ebsynth\n```\n\n### Linux\n```bash\n# clone this repo\ngit clone https://github.com/Trentonom0r3/Ezsynth.git \ncd Ezsynth\n\n# (optional) create and activate venv\npython -m venv venv\nsource ./venv/bin/activate\n\n# install requirements\npip install -r requirements.txt\n\n# clone ebsynth\ngit clone https://github.com/Trentonom0r3/ebsynth.git\n\n# build ebsynth as lib\ncp ./build_ebs-linux-cpu+cuda.sh ./ebsynth\ncd ebsynth \u0026\u0026 ./build_ebs-linux-cpu+cuda.sh\n\n# copy lib\ncp ./bin/ebsynth.so ../ezsynth/utils/ebsynth.so\n\n# cleanup\ncd .. \u0026\u0026 rm -rf ./ebsynth\n```\n\n### All\nYou may also install Cupy and Cupyx to use GPU for some other operations.\n\n## Examples\n\n* To get started, see `test_redux.py` for an example of generating a full video.\n* To generate image style transfer, see `test_imgsynth.py` for all examples from the original `Ebsynth`.\n\n## Example outputs\n\n| Face style | Stylit | Retarget |\n|:-:|:-:|:-:|\n| \u003cimg src=\"output_synth/facestyle_out.png\" height=\"250\"\u003e | \u003cimg src=\"output_synth/stylit_out.png\" height=\"250\"\u003e | \u003cimg src=\"output_synth/retarget_out.png\" height=\"250\"\u003e |\n\nhttps://github.com/user-attachments/assets/aa3cd191-4eb2-4dc0-8213-2c763f1b3316\n\nhttps://github.com/user-attachments/assets/63e50272-aa5c-42a1-a5ec-46178cdf2981\n\nComparison of Edge methods\n\n## Notable things\n\n**Updates:**\n1. [Ef-RAFT](https://github.com/n3slami/Ef-RAFT) is added \n   \n   To use, download models from [the original repo](https://github.com/n3slami/Ef-RAFT/tree/master/models) and place them in `/ezsynth/utils/flow_utils/ef_raft_models`\n   ```\n   .gitkeep\n   25000_ours-sintel.pth\n   ours-things.pth\n   ours_sintel.pth\n   ```\n\n2. [FlowDiffuser](https://github.com/LA30/FlowDiffuser) is added. \n\n   To use, download the model from [the original repo](https://github.com/LA30/FlowDiffuser?tab=readme-ov-file#usage) and place it in `/ezsynth/utils/flow_utils/flow_diffusion_models/FlowDiffuser-things.pth`. \n\n   You will also need to install PyTorch Image Models to run it: `pip install timm`.   On first run, it will download 2 models ~470MB `twins_svt_large (378 MB)` and `twins_svt_small (92 MB)`.\n   \n   This increases the VRAM usage significantly when run along with `EbSynth Run` (~15GB, but may not OOM. Tested on 12GB VRAM).\n\n   In that case, It will throw `CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR` error, but shouldn't be fatal, and instead takes ~3x as long to run.\n\nhttps://github.com/user-attachments/assets/7f43630f-c7c9-40d0-8745-58d1f7c84d4f\n\nComparison of Optical Flow models\n\nOptical Flow directly affects Flow position warping and Style image warping, controlled by `pos_wgt` and `wrp_wgt` respectively.\n\n**Changes:**\n1. Flow is calculated on a frame by frame basis, with correct time orientation, instead of pre-computing only a forward-flow.\n2. Padding is applied to Edge detection and Warping to remove border visual distortion.\n\n\n**Observations:**\n1. Edge detection models return NaN if input tensor has too many zeros(?).\n2. Pre-masked inputs take twice as long to run Ebsynth\n\n## API Overview\n\n### ImageSynth\nFor image-to-image style transfer, via file paths: `test_imgsynth.py`\n```python\nezsynner = ImageSynth(\n    style_path=\"source_style.png\",\n    src_path=\"source_fullgi.png\",\n    tgt_path=\"target_fullgi.png\",\n    cfg=RunConfig(img_wgt=0.66),\n)\n\nresult = ezsynner.run(\n    guides=[\n        load_guide(\n            \"source_dirdif.png\",\n            \"target_dirdif.png\",\n            0.66,\n        ),\n        load_guide(\n            \"source_indirb.png\",\n            \"target_indirb.png\",\n            0.66,\n        ),\n    ]\n)\n\nsave_to_folder(output_folder, \"stylit_out.png\", result[0]) # Styled image\nsave_to_folder(output_folder, \"stylit_err.png\", result[1]) # Error image\n```\n\n### Ezsynth\n\n**edge_method**\n\nEdge detection method. Choose from `PST`, `Classic`, or `PAGE`.\n* `PST` (Phase Stretch Transform): Good overall structure, but not very detailed.\n* `Classic`: A good balance between structure and detail.\n* `PAGE` (Phase and Gradient Estimation): Great detail, great structure, but slow.\n\n**video stylization**\n\nVia file paths (see `test_redux.py`):\n\n```python\nstyle_paths = [\n    \"style000.png\",\n    \"style006.png\"\n]\n\nezrunner = Ezsynth(\n    style_paths=style_paths,\n    image_folder=image_folder,\n    cfg=RunConfig(pre_mask=False, feather=5, return_masked_only=False),\n    edge_method=\"PAGE\",\n    raft_flow_model_name=\"sintel\",\n    mask_folder=mask_folder,\n    do_mask=True\n)\n\nonly_mode = None\nstylized_frames, err_frames  = ezrunner.run_sequences(only_mode)\n\nsave_seq(stylized_frames, \"output\")\n```\n\nVia Numpy ndarrays:\n\n```python\nclass EzsynthBase:\n    def __init__(\n        self,\n        style_frs: list[np.ndarray],\n        style_idxes: list[int],\n        img_frs_seq: list[np.ndarray],\n        cfg: RunConfig = RunConfig(),\n        edge_method=\"Classic\",\n        raft_flow_model_name=\"sintel\",\n        do_mask=False,\n        msk_frs_seq: list[np.ndarray] | None = None,\n    ):\n    pass\n```\n\n### RunConfig\n#### Ebsynth gen params\n* `uniformity (float)`: Uniformity weight for the style transfer.  Reasonable values are between `500-15000`. Defaults to `3500.0`.\n     \n* `patchsize (int)`: Size of the patches [NxN]. Must be an odd number `\u003e= 3`.  Defaults to `7`.\n     \n* `pyramidlevels (int)`: Number of pyramid levels. Larger values useful for things like color transfer.  Defaults to `6`.\n     \n* `searchvoteiters (int)`: Number of search/vote iterations. Defaults to `12`.\n* `patchmatchiters (int)`: Number of Patch-Match iterations. The larger, the longer it takes.  Defaults to `6`.\n     \n* `extrapass3x3 (bool)`: Perform additional polishing pass with 3x3 patches at the finest level.  Defaults to `True`.\n        \n#### Ebsynth guide weights params    \n* `edg_wgt (float)`: Edge detect weights. Defaults to `1.0`.\n* `img_wgt (float)`: Original image weights. Defaults to `6.0`.\n* `pos_wgt (float)`: Flow position warping weights. Defaults to `2.0`.\n* `wrp_wgt (float)`: Warped style image weight. Defaults to `0.5`.\n    \n#### Blending params\n* `use_gpu (bool)`: Use GPU for Histogram Blending (Only affect Blend mode). Faster than CPU. Defaults to `False`.\n     \n* `use_lsqr (bool)`: Use LSQR (Least-squares solver) instead of LSMR (Iterative solver for least-squares) for Poisson blending step. LSQR often yield better results. May change to LSMR for speed (depends).  Defaults to `True`.\n     \n* `use_poisson_cupy (bool)`: Use Cupy GPU acceleration for Poisson blending step. Uses LSMR (overrides `use_lsqr`). May not yield better speed. Defaults to `False`.\n     \n* `poisson_maxiter (int | None)`: Max iteration to calculate Poisson Least-squares (only affect LSMR mode). Expect positive integers. Defaults to `None`.\n     \n* `only_mode (str)`: Skip blending, only run one pass per sequence. Valid values:\n  * `MODE_FWD = \"forward\"` (Will only run forward mode if `sequence.mode` is blend)\n  \n  * `MODE_REV = \"reverse\"` (Will only run reverse mode if `sequence.mode` is blend)\n  \n  * Defaults to `MODE_NON = \"none\"`.\n    \n#### Masking params\n* `do_mask (bool)`: Whether to apply mask. Defaults to `False`.\n \n* `pre_mask (bool)`: Whether to mask the inputs and styles before `RUN` or after. Pre-mask takes ~2x time to run per frame. Could be due to Ebsynth.dll implementation. Defaults to `False`.     \n  \n* `feather (int)`: Feather Gaussian radius to apply on the mask results. Only affect if `return_masked_only == False`. Expects integers. Defaults to `0`.\n\n## Credits\n\njamriska - https://github.com/jamriska/ebsynth\n\n```\n@misc{Jamriska2018,\n  author = {Jamriska, Ondrej},\n  title = {Ebsynth: Fast Example-based Image Synthesis and Style Transfer},\n  year = {2018},\n  publisher = {GitHub},\n  journal = {GitHub repository},\n  howpublished = {\\url{https://github.com/jamriska/ebsynth}},\n}\n```\n```\nOndřej Jamriška, Šárka Sochorová, Ondřej Texler, Michal Lukáč, Jakub Fišer, Jingwan Lu, Eli Shechtman, and Daniel Sýkora. 2019. Stylizing Video by Example. ACM Trans. Graph. 38, 4, Article 107 (July 2019), 11 pages. https://doi.org/10.1145/3306346.3323006\n```\n\nFuouM - https://github.com/FuouM\npravdomil - https://github.com/pravdomil\nxy-gao - https://github.com/xy-gao\n\nhttps://github.com/princeton-vl/RAFT\n\n```\nRAFT: Recurrent All Pairs Field Transforms for Optical Flow\nECCV 2020\nZachary Teed and Jia Deng\n```\n\nhttps://github.com/n3slami/Ef-RAFT\n\n```\n@inproceedings{eslami2024rethinking,\n  title={Rethinking RAFT for efficient optical flow},\n  author={Eslami, Navid and Arefi, Farnoosh and Mansourian, Amir M and Kasaei, Shohreh},\n  booktitle={2024 13th Iranian/3rd International Machine Vision and Image Processing Conference (MVIP)},\n  pages={1--7},\n  year={2024},\n  organization={IEEE}\n}\n```\n\nhttps://github.com/LA30/FlowDiffuser\n\n```\n@inproceedings{luo2024flowdiffuser,\n  title={FlowDiffuser: Advancing Optical Flow Estimation with Diffusion Models},\n  author={Luo, Ao and Li, Xin and Yang, Fan and Liu, Jiangyu and Fan, Haoqiang and Liu, Shuaicheng},\n  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\n  pages={19167--19176},\n  year={2024}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrentonom0r3%2Fezsynth","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftrentonom0r3%2Fezsynth","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrentonom0r3%2Fezsynth/lists"}