{"id":24665742,"url":"https://github.com/foivospar/Arc2Face","last_synced_at":"2025-10-08T03:30:20.789Z","repository":{"id":227674212,"uuid":"772047488","full_name":"foivospar/Arc2Face","owner":"foivospar","description":"[ECCV 2024 Oral🔥] Arc2Face: A Foundation Model for ID-Consistent Human Faces","archived":false,"fork":false,"pushed_at":"2024-12-03T19:08:44.000Z","size":29768,"stargazers_count":612,"open_issues_count":25,"forks_count":45,"subscribers_count":18,"default_branch":"main","last_synced_at":"2024-12-03T20:22:20.534Z","etag":null,"topics":["face","face-generation","id-embedding","personalization","stable-diffusion","subject-driven-generation"],"latest_commit_sha":null,"homepage":"https://arc2face.github.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/foivospar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-14T12:42:46.000Z","updated_at":"2024-12-03T19:08:47.000Z","dependencies_parsed_at":"2024-09-18T02:55:46.553Z","dependency_job_id":null,"html_url":"https://github.com/foivospar/Arc2Face","commit_stats":null,"previous_names":["foivospar/arc2face"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foivospar%2FArc2Face","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foivospar%2FArc2Face/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foivospar%2FArc2Face/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foivospar%2FArc2Face/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/foivospar","download_url":"https://codeload.github.com/foivospar/Arc2Face/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235674136,"owners_count":19027526,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["face","face-generation","id-embedding","personalization","stable-diffusion","subject-driven-generation"],"created_at":"2025-01-26T07:13:28.238Z","updated_at":"2025-10-08T03:30:20.783Z","avatar_url":"https://github.com/foivospar.png","language":"Python","funding_links":[],"categories":["Papers"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n## 🚀 NEW (2025): ID-Consistent, Precise Expression Generation with Blendshape-Guided Diffusion\n\n📄 **Check out our latest extension!**  \nWe introduce a fine-grained **Expression Adapter**, enabling Arc2Face to generate any subject under any facial expression (even rare, asymmetric, subtle, or extreme ones). See details [below](#arc2face--expression-adapter).\n\n\u003ca href='http://arxiv.org/abs/2510.04706'\u003e\u003cimg src='https://img.shields.io/badge/Paper-arXiv-red'\u003e\u003c/a\u003e\n\n\u003cimg src='assets/exp_teaser.jpg'\u003e\n\n---\n\n# Arc2Face: A Foundation Model for ID-Consistent Human Faces\n\n[Foivos Paraperas Papantoniou](https://foivospar.github.io/)\u003csup\u003e1\u003c/sup\u003e \u0026emsp; [Alexandros Lattas](https://alexlattas.com/)\u003csup\u003e1\u003c/sup\u003e \u0026emsp; [Stylianos Moschoglou](https://moschoglou.com/)\u003csup\u003e1\u003c/sup\u003e   \n\n[Jiankang Deng](https://jiankangdeng.github.io/)\u003csup\u003e1\u003c/sup\u003e \u0026emsp; [Bernhard Kainz](https://bernhard-kainz.com/)\u003csup\u003e1,2\u003c/sup\u003e \u0026emsp; [Stefanos Zafeiriou](https://www.imperial.ac.uk/people/s.zafeiriou)\u003csup\u003e1\u003c/sup\u003e  \n\n\u003csup\u003e1\u003c/sup\u003eImperial College London, UK \u003cbr\u003e\n\u003csup\u003e2\u003c/sup\u003eFAU Erlangen-Nürnberg, Germany\n\n\u003ca href='https://arc2face.github.io/'\u003e\u003cimg src='https://img.shields.io/badge/Project-Page-blue'\u003e\u003c/a\u003e\n\u003ca href='https://arxiv.org/abs/2403.11641'\u003e\u003cimg src='https://img.shields.io/badge/Paper-arXiv-red'\u003e\u003c/a\u003e\n\u003ca href='https://huggingface.co/spaces/FoivosPar/Arc2Face'\u003e\u003cimg src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Demo-green'\u003e\u003c/a\u003e\n\u003ca href='https://huggingface.co/FoivosPar/Arc2Face'\u003e\u003cimg src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model-orange'\u003e\u003c/a\u003e\n\u003ca href='https://huggingface.co/datasets/FoivosPar/Arc2Face'\u003e\u003cimg src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Data-8A2BE2'\u003e\u003c/a\u003e\n\n\u003c/div\u003e\n\nThis is the official implementation of **[Arc2Face](https://arc2face.github.io/)**, an ID-conditioned face model:\n\n\u0026emsp;✅ that generates high-quality images of any subject given only its ArcFace embedding, within a few seconds\u003cbr\u003e\n\u0026emsp;✅ trained on the large-scale WebFace42M dataset offers superior ID similarity compared to existing models\u003cbr\u003e\n\u0026emsp;✅ built on top of Stable Diffusion, can be extended to different input modalities, e.g. with ControlNet\u003cbr\u003e\n\n\u003cimg src='assets/teaser.gif'\u003e\n\n# News/Updates\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/arc2face-a-foundation-model-of-human-faces/diffusion-personalization-tuning-free-on)](https://paperswithcode.com/sota/diffusion-personalization-tuning-free-on?p=arc2face-a-foundation-model-of-human-faces)\n\n- [2025/10/07] 🔥 We release an extension for accurate and ID-consistent facial expression transfer. See details [below](#arc2face--expression-adapter)!\n- [2024/08/16] 🔥 Accepted to ECCV24 as an **oral**!\n- [2024/08/06] 🔥 ComfyUI support available at [caleboleary/ComfyUI-Arc2Face](https://github.com/caleboleary/ComfyUI-Arc2Face)!\n- [2024/04/12] 🔥 We add LCM-LoRA support for even faster inference (check the details [below](#lcm-lora-acceleration)).\n- [2024/04/11] 🔥 We release the training dataset on [HuggingFace Datasets](https://huggingface.co/datasets/FoivosPar/Arc2Face).\n- [2024/03/31] 🔥 We release our demo for pose control using Arc2Face + ControlNet (see instructions [below](#arc2face--controlnet-pose)).\n- [2024/03/28] 🔥 We release our Gradio [demo](https://huggingface.co/spaces/FoivosPar/Arc2Face) on HuggingFace Spaces (thanks to the HF team for their free GPU support)!\n- [2024/03/14] 🔥 We release Arc2Face.\n\n# Installation\n```bash\nconda create -n arc2face python=3.10\nconda activate arc2face\n\n# Install requirements\npip install -r requirements.txt\n```\n\n# Download Models\n1) The models can be downloaded manually from [HuggingFace](https://huggingface.co/FoivosPar/Arc2Face) or using python:\n```python\nfrom huggingface_hub import hf_hub_download\n\nhf_hub_download(repo_id=\"FoivosPar/Arc2Face\", filename=\"arc2face/config.json\", local_dir=\"./models\")\nhf_hub_download(repo_id=\"FoivosPar/Arc2Face\", filename=\"arc2face/diffusion_pytorch_model.safetensors\", local_dir=\"./models\")\nhf_hub_download(repo_id=\"FoivosPar/Arc2Face\", filename=\"encoder/config.json\", local_dir=\"./models\")\nhf_hub_download(repo_id=\"FoivosPar/Arc2Face\", filename=\"encoder/pytorch_model.bin\", local_dir=\"./models\")\n```\n\n2) For face detection and ID-embedding extraction, manually download the [antelopev2](https://github.com/deepinsight/insightface/tree/master/python-package#model-zoo) package ([direct link](https://drive.google.com/file/d/18wEUfMNohBJ4K3Ly5wpTejPfDzp-8fI8/view)) and place the checkpoints under `models/antelopev2`. \n\n3) We use an ArcFace recognition model trained on WebFace42M. Download `arcface.onnx` from [HuggingFace](https://huggingface.co/FoivosPar/Arc2Face) and put it in `models/antelopev2` or using python:\n```python\nhf_hub_download(repo_id=\"FoivosPar/Arc2Face\", filename=\"arcface.onnx\", local_dir=\"./models/antelopev2\")\n```\n4) Then **delete** `glintr100.onnx` (the default backbone from insightface).\n\nThe `models` folder structure should finally be:\n```\n  . ── models ──┌── antelopev2\n                ├── arc2face\n                └── encoder\n```\n\n# Usage\n\nLoad pipeline using [diffusers](https://huggingface.co/docs/diffusers/index):\n```python\nfrom diffusers import (\n    StableDiffusionPipeline,\n    UNet2DConditionModel,\n    DPMSolverMultistepScheduler,\n)\n\nfrom arc2face import CLIPTextModelWrapper, project_face_embs\n\nimport torch\nfrom insightface.app import FaceAnalysis\nfrom PIL import Image\nimport numpy as np\n\n# Arc2Face is built upon SD1.5\n# The repo below can be used instead of the now deprecated 'runwayml/stable-diffusion-v1-5'\nbase_model = 'stable-diffusion-v1-5/stable-diffusion-v1-5'\n\nencoder = CLIPTextModelWrapper.from_pretrained(\n    'models', subfolder=\"encoder\", torch_dtype=torch.float16\n)\n\nunet = UNet2DConditionModel.from_pretrained(\n    'models', subfolder=\"arc2face\", torch_dtype=torch.float16\n)\n\npipeline = StableDiffusionPipeline.from_pretrained(\n        base_model,\n        text_encoder=encoder,\n        unet=unet,\n        torch_dtype=torch.float16,\n        safety_checker=None\n    )\n```\nYou can use any SD-compatible schedulers and steps, just like with Stable Diffusion. By default, we use `DPMSolverMultistepScheduler` with 25 steps, which produces very good results in just a few seconds.\n```python\npipeline.scheduler = DPMSolverMultistepScheduler.from_config(pipeline.scheduler.config)\npipeline = pipeline.to('cuda')\n```\nPick an image and extract the ID-embedding:\n```python\napp = FaceAnalysis(name='antelopev2', root='./', providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])\napp.prepare(ctx_id=0, det_size=(640, 640))\n\nimg = np.array(Image.open('assets/examples/joacquin.png'))[:,:,::-1]\n\nfaces = app.get(img)\nfaces = sorted(faces, key=lambda x:(x['bbox'][2]-x['bbox'][0])*(x['bbox'][3]-x['bbox'][1]))[-1]  # select largest face (if more than one detected)\nid_emb = torch.tensor(faces['embedding'], dtype=torch.float16)[None].cuda()\nid_emb = id_emb/torch.norm(id_emb, dim=1, keepdim=True)   # normalize embedding\nid_emb = project_face_embs(pipeline, id_emb)    # pass through the encoder\n```\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src='assets/examples/joacquin.png' style='width:25%;'\u003e\n\u003c/div\u003e\n\nGenerate images:\n```python\nnum_images = 4\nimages = pipeline(prompt_embeds=id_emb, num_inference_steps=25, guidance_scale=3.0, num_images_per_prompt=num_images).images\n```\n\u003cdiv align=\"center\"\u003e\n\u003cimg src='assets/samples.jpg'\u003e\n\u003c/div\u003e\n\n# LCM-LoRA acceleration\n\n[LCM-LoRA](https://arxiv.org/abs/2311.05556) allows you to reduce the sampling steps to as few as 2-4 for super-fast inference. Just plug in the pre-trained distillation adapter for SD v1.5 and switch to `LCMScheduler`:\n```python\nfrom diffusers import LCMScheduler\n\npipeline.load_lora_weights(\"latent-consistency/lcm-lora-sdv1-5\")\npipeline.scheduler = LCMScheduler.from_config(pipeline.scheduler.config)\n```\nThen, you can sample with as few as 2 steps (and disable `guidance_scale` by using a value of 1.0, as LCM is very sensitive to it and even small values lead to oversaturation):\n```python\nimages = pipeline(prompt_embeds=id_emb, num_inference_steps=2, guidance_scale=1.0, num_images_per_prompt=num_images).images\n```\nNote that this technique accelerates sampling in exchange for a slight drop in quality.\n\n# Start a local gradio demo\nYou can start a local demo for inference by running:\n```bash\npython gradio_demo/app.py\n```\n\n# Arc2Face + ControlNet (pose)\n\u003cdiv align=\"center\"\u003e\n\u003cimg src='assets/controlnet.jpg'\u003e\n\u003c/div\u003e\n\nWe provide a ControlNet model trained on top of Arc2Face for pose control. We use [EMOCA](https://github.com/radekd91/emoca) for 3D pose extraction. To run our demo, follow the steps below:\n### 1) Download Model\nDownload the ControlNet checkpoint manually from [HuggingFace](https://huggingface.co/FoivosPar/Arc2Face) or using python:\n```python\nfrom huggingface_hub import hf_hub_download\n\nhf_hub_download(repo_id=\"FoivosPar/Arc2Face\", filename=\"controlnet/config.json\", local_dir=\"./models\")\nhf_hub_download(repo_id=\"FoivosPar/Arc2Face\", filename=\"controlnet/diffusion_pytorch_model.safetensors\", local_dir=\"./models\")\n```\n### 2) Pull EMOCA\n```bash\ngit submodule update --init external/emoca\n```\n### 3) Installation\nThis is the most tricky part. You will need PyTorch3D to run EMOCA. As its installation may cause conflicts, we suggest to follow the process below:\n1) Create a new environment and start by installing PyTorch3D with GPU support first (follow the official [instructions](https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.md)).\n2) Add Arc2Face + EMOCA requirements with:\n```bash\npip install -r requirements_controlnet.txt\n```\n3) Install EMOCA code:\n```bash\npip install -e external/emoca\n```\n4) Finally, you need to download the EMOCA/FLAME assets. Run the following and follow the instructions in the terminal:\n```bash\ncd external/emoca/gdl_apps/EMOCA/demos \nbash download_assets.sh\ncd ../../../../..\n```\n### 4) Start a local gradio demo\nYou can start a local ControlNet demo by running:\n```bash\npython gradio_demo/app_controlnet.py\n```\n\n# Arc2Face + Expression Adapter\n\nOur extension [\"ID-Consistent, Precise Expression Generation with Blendshape-Guided Diffusion\"](http://arxiv.org/abs/2510.04706) combines Arc2Face with a custom IP-Adapter designed for generating ID-consistent images with precise expression control based on FLAME blendshape parameters. We also provide an optional Reference Adapter which can be used to condition the output directly on the input image, i.e. preserving the subject's appearance and background (to an extent). You can find more details in the report.\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src='assets/arc2face_exp.jpg'\u003e\n\u003c/div\u003e\n\n\u003cbr\u003e\nHere's how to run it:\n\n### 1) Download Model\nDownload the Expression and Reference Adapters manually from [HuggingFace](https://huggingface.co/FoivosPar/Arc2Face) or using python:\n```python\nfrom huggingface_hub import hf_hub_download\n\nhf_hub_download(repo_id=\"FoivosPar/Arc2Face\", filename=\"exp_adapter/exp_adapter.bin\", local_dir=\"./models\")\nhf_hub_download(repo_id=\"FoivosPar/Arc2Face\", filename=\"ref_adapter/pytorch_lora_weights.safetensors\", local_dir=\"./models\")\n```\n### 2) Download third-party models (SMIRK)\nWe use the [SMIRK](https://github.com/georgeretsi/smirk) method to extract FLAME expression parameters from the target image. Download the required checkpoints **face_landmarker.task** and **SMIRK_em1.pt** and put them under `models/smirk`:\n```bash\nmkdir models/smirk\nwget https://storage.googleapis.com/mediapipe-models/face_landmarker/face_landmarker/float16/latest/face_landmarker.task --directory-prefix models/smirk\npip install gdown\ngdown --id 1T65uEd9dVLHgVw5KiUYL66NUee-MCzoE -O models/smirk/\n```\n### 3) Start a local gradio demo\nThen, just run the demo and follow the instructions:\n```bash\npython gradio_demo/app_exp_adapter.py\n```\n\n# Test Data\nThe test images used for comparisons in the paper (Synth-500, AgeDB) are available [here](https://drive.google.com/drive/folders/1exnvCECmqWcqNIFCck2EQD-hkE42Ayjc?usp=sharing). Please use them only for evaluation purposes and make sure to cite the corresponding [sources](https://ibug.doc.ic.ac.uk/resources/agedb/) when using them.\n\n# Community Resources\n\n### Replicate Demo\n- [Demo link](https://replicate.com/camenduru/arc2face) by [@camenduru](https://github.com/camenduru).\n\n### ComfyUI\n- [caleboleary/ComfyUI-Arc2Face](https://github.com/caleboleary/ComfyUI-Arc2Face) by [@caleboleary](https://github.com/caleboleary).\n  \n### Pinokio\n- Pinokio [implementation](https://pinokio.computer/item?uri=https://github.com/cocktailpeanutlabs/arc2face) by [@cocktailpeanut](https://github.com/cocktailpeanut) (runs locally on all OS - Windows, Mac, Linux).\n\n# Acknowledgements\n- Thanks to the creators of Stable Diffusion and the HuggingFace [diffusers](https://github.com/huggingface/diffusers) team for the awesome work ❤️.\n- Thanks to the WebFace42M creators for providing such a million-scale facial dataset ❤️.\n- Thanks to the HuggingFace team for their generous support through the community GPU grant for our demo ❤️.\n- We also acknowledge the invaluable support of the HPC resources provided by the Erlangen National High Performance Computing Center (NHR@FAU) of the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), which made the training of Arc2Face possible.\n\n# Citation\nIf you find Arc2Face useful for your research, please consider citing us:\n\n```bibtex\n@inproceedings{paraperas2024arc2face,\n      title={Arc2Face: A Foundation Model for ID-Consistent Human Faces}, \n      author={Paraperas Papantoniou, Foivos and Lattas, Alexandros and Moschoglou, Stylianos and Deng, Jiankang and Kainz, Bernhard and Zafeiriou, Stefanos},\n      booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},\n      year={2024}\n}\n```\nAdditionally, if you use the Expression Adapter, please also cite the extension:\n\n```bibtex\n@inproceedings{paraperas2025arc2face_exp,\n      title={ID-Consistent, Precise Expression Generation with Blendshape-Guided Diffusion}, \n      author={Paraperas Papantoniou, Foivos and Zafeiriou, Stefanos},\n      booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops},\n      year={2025}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffoivospar%2FArc2Face","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffoivospar%2FArc2Face","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffoivospar%2FArc2Face/lists"}