{"id":23437708,"url":"https://github.com/whatphilipcodes/sam-aging","last_synced_at":"2025-04-09T18:54:29.517Z","repository":{"id":268580406,"uuid":"904742683","full_name":"whatphilipcodes/sam-aging","owner":"whatphilipcodes","description":"All outputs in full resolution are available on Google Drive:","archived":false,"fork":false,"pushed_at":"2025-01-23T17:33:15.000Z","size":9876,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-15T11:38:08.134Z","etag":null,"topics":["image-processing"],"latest_commit_sha":null,"homepage":"https://drive.google.com/file/d/1kVN8lA6VCZjAq6EupbsEfB7ZTp46gSoV/view?usp=drive_link","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/whatphilipcodes.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-17T13:20:12.000Z","updated_at":"2025-01-23T17:33:19.000Z","dependencies_parsed_at":"2025-02-15T11:31:53.775Z","dependency_job_id":"d136bce7-73df-461f-a046-ba94515f9e3f","html_url":"https://github.com/whatphilipcodes/sam-aging","commit_stats":null,"previous_names":["whatphilipcodes/sam-aging"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whatphilipcodes%2Fsam-aging","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whatphilipcodes%2Fsam-aging/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whatphilipcodes%2Fsam-aging/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whatphilipcodes%2Fsam-aging/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/whatphilipcodes","download_url":"https://codeload.github.com/whatphilipcodes/sam-aging/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248093831,"owners_count":21046758,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["image-processing"],"created_at":"2024-12-23T14:34:16.193Z","updated_at":"2025-04-09T18:54:29.497Z","avatar_url":"https://github.com/whatphilipcodes.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# sam-aging\n\n### About\n\nI was approached by Sina Rohde (MA Production), as she was preparing a presentation about machine learning based methods for age manipulation in movies. As a result, I built this repository as a complement to the [replicate implementation](https://replicate.com/yuval-alaluf/sam) of the paper [Only a Matter of Style: Age Transformation Using a Style-Based Regression Model](https://doi.org/10.48550/arXiv.2102.02754).\n\n### Usage\n\nIf you want to try out the model yourself, please refer to the [replicate implementation](https://replicate.com/yuval-alaluf/sam?input=docker). Once the model is running on your machine, you can use the script `./py/main.py` to feed all of the images from a specified folder to the model and generate multiple target age transformations for each image.\n\n```bash\n# requires poetry to be installed on your machine\ncd to/project/root\npoetry install\npoetry run python ./py/main.py\n```\n\n### Considerations\n\nThe following models were considered for this project:\n\n- [SAM](https://github.com/yuval-alaluf/SAM) (Final Selection)\n\n  - high quality results\n  - model weights are freely available\n  - open-source implementation\n  - robust docker container available\n\n- [Fast-AgingGAN](https://github.com/HasnainRaz/Fast-AgingGAN)\n\n  - medium quality results\n  - model weights are freely available\n  - open-source implementation\n\n- [Production-Ready Face Re-Aging for Visual Effects](https://studios.disneyresearch.com/2022/11/30/production-ready-face-re-aging-for-visual-effects/)\n\n  - highest quality results\n  - model weights are not freely available\n  - no open-source implementation available\n\n### Understanding\n\nThe SAM model is designed to transform facial images to show how a person might look at different ages. Under the hood, it is composed of two neural networks:\n\n- A StyleGAN-based `generator` to create realistic faces\n- An age `regression network` to guide the transformation\n\nWhen an input image enters the system, it first goes through an encoder that breaks down the image into different levels of detail. This encoder then transforms those details into a special representation that can be manipulated. The original image is also encoded separately. The two encodings are then combined, with a specific focus on transforming the image to the desired age. This combined representation is fed into the pre-trained image generation model, which produces the final age-transformed image.\n\n### Goals\n\nEven though the model outputs images, the main goal was to create a `short video` clip to demonstrate on custom footage what the model is capable of. To achieve this, the following steps were taken:\n\n- [x] Running docker image\n\n  - setup und configure docker\n  - run model\n\n- [x] Frame processing pipeline\n\n  - interface with api via python script\n  - create automatic frame processing workflow\n\n- [x] Result generation\n\n  - extract frames from video\n  - process frames with model\n  - reassemble frames into video\n\n### Input Footage\n\n![](./img/input.gif)\n\n\u003e 4 second clip (individual 225 Frames)\n\n### Output\n\n![](./img/output.gif)\n\n\u003e 4 second clip with a target age of 60 years\n\n![](./img/1-30.jpg)\n\n\u003e from left to right ages `01, 10, 20, 30`\n\n![](./img/40-70.jpg)\n\n\u003e from left to right ages `40, 50, 60, 70`\n\n![](./img/80-100.jpg)\n\n\u003e from left to right ages `80, 90, 100`\n\nAll full resolution output images and videos are available on [Google Drive](https://drive.google.com/file/d/1kVN8lA6VCZjAq6EupbsEfB7ZTp46gSoV/view?usp=drive_link)\n\n### Analysis\n\n- Image generation takes around 10 seconds per frame\n- Individual frames have a high quality when looking at the facial features exclusively\n\n---\n\n- Head rotation and facial expressions are not translated well with this model\n- The model is not intended for video processing (temporal coherence issues)\n- Areas around the face (Hair, Ears, Neck etc.) are particularly true to the input\n\n### Conclusion\n\nAlthough the output is fascinating in its own regard, real world application of this exact model for Re- and De-Aging in the VFX pipeline is not feasible. For a convincing result, a lot of manual labor is still involved with this approach. Another major issue with this technique is the loss of facial expressions. The resulting output images all share a very similar facial expression which is not ideal when trying to convey an actor's performance in a shot.\n\nThe closed-source [approach](https://studios.disneyresearch.com/2022/11/30/production-ready-face-re-aging-for-visual-effects/) by Disney is much more capable and even takes care of the whole processing pipeline from video to video automatically. Unfortunately, at the time of writing this, to my knowledge no open-source implementation of this model is available.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwhatphilipcodes%2Fsam-aging","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwhatphilipcodes%2Fsam-aging","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwhatphilipcodes%2Fsam-aging/lists"}