{"id":18488290,"url":"https://github.com/choyingw/cross-modal-perceptionist","last_synced_at":"2025-04-05T18:08:25.972Z","repository":{"id":37778117,"uuid":"358786708","full_name":"choyingw/Cross-Modal-Perceptionist","owner":"choyingw","description":"CVPR 2022: Cross-Modal Perceptionist: Can Face Geometry be Gleaned from Voices?","archived":false,"fork":false,"pushed_at":"2024-12-11T09:52:13.000Z","size":8228,"stargazers_count":129,"open_issues_count":4,"forks_count":15,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-03-29T17:09:20.575Z","etag":null,"topics":["3d","3d-models","3dmm","biometrics","cognitive-science","computer-vision","cross-modal-learning","cvpr","cvpr2022","deep-learning","machine-learning","pytorch","speech","speech-synthesis","speech-to-face"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/choyingw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-17T04:45:22.000Z","updated_at":"2025-02-26T08:36:51.000Z","dependencies_parsed_at":"2025-03-14T18:12:14.703Z","dependency_job_id":"37e672d9-b05e-463f-9eec-f61f9ea8afbb","html_url":"https://github.com/choyingw/Cross-Modal-Perceptionist","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/choyingw%2FCross-Modal-Perceptionist","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/choyingw%2FCross-Modal-Perceptionist/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/choyingw%2FCross-Modal-Perceptionist/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/choyingw%2FCross-Modal-Perceptionist/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/choyingw","download_url":"https://codeload.github.com/choyingw/Cross-Modal-Perceptionist/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247378144,"owners_count":20929296,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d","3d-models","3dmm","biometrics","cognitive-science","computer-vision","cross-modal-learning","cvpr","cvpr2022","deep-learning","machine-learning","pytorch","speech","speech-synthesis","speech-to-face"],"created_at":"2024-11-06T12:51:31.319Z","updated_at":"2025-04-05T18:08:25.941Z","avatar_url":"https://github.com/choyingw.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"#  \u003cdiv align=\"center\"\u003eCross-Modal Perceptionist\u003c/div\u003e\nCode Repository for CVPR 2022 \"Cross-Modal Perceptionist: Can Face Geometry be Gleaned from Voices?\"\n\nCho-Ying Wu, Chin-Cheng Hsu, Ulrich Neumann, University of Southern California\n\n[\u003ca href=\"https://arxiv.org/abs/2203.09824\"\u003ePaper\u003c/a\u003e] [\u003ca href=\"https://choyingw.github.io/works/Voice2Mesh/index.html\"\u003eProject page\u003c/a\u003e] [\u003ca href=\"https://drive.google.com/drive/folders/1tT36oDujNXBw5SpwhY3PiBnGIE0FbvCs?usp=sharing\"\u003eVoxceleb-3D Data\u003c/a\u003e]\n\n\u003cb\u003eCheck the project page for the introduction of this cool work!\u003c/b\u003e\n\n\u003ci\u003eUpdate: 2022/12/01\u003c/i\u003e Added Evaluation code, pretained model, and execution script for supervised framework. Organized data structure of Voxceleb-3D\n\n\u003ci\u003eVoxceleb-3D\u003c/i\u003e:\n\n(1) [\u003ca href=\"https://drive.google.com/file/d/1xBjyP5BiwLL9LCETy0-UJt5oI4bROZVD/view?usp=share_link\"\u003eHere\u003c/a\u003e] contains data with names starting from F-Z as the \u003cb\u003etraining set\u003c/b\u003e. 100G zipped file, ~250G after unzip. This set contains pointcloud (.xyz), reconstructed mesh overlapped on images from VGGFace (_b.jpg), and 199-dim 3DMM parameters using BFM Face 2009 basis. This is in contrast to simplified 3DMM basis for first 40-dim shape + next 10-dim expression. (\u003cb\u003eSynergyNet follows 3DDFA-v2 and uses the first 40-dim out of 199-dim as the shape code. Thus, the first 40 in 199 correspond to first 40-dim in SynergyNet\u003c/b\u003e.)You can donwload full basis from BFM-2009 \u003ca href=\"https://faces.dmi.unibas.ch/bfm/index.php?nav=1-1-0\u0026id=details\"\u003eofficial website\u003c/a\u003e. There are multiple 3D faces for an identity. \n\n(2) [\u003ca href=\"https://drive.google.com/file/d/1BzuNOYN_8USpp6r8uPxpSezD5GOhf1XI/view?usp=share_link\"\u003eHere\u003c/a\u003e] contains data with names starting from A-E as the \u003cb\u003evalidation set\u003c/b\u003e. 300M. The format is the same except there is only one 3D face for an identity as groundtruth.\n\n(3) [\u003ca href=\"https://drive.google.com/file/d/1jKOCjIsMlfPln8Vkpjx_wCRC20oVigxT/view?usp=share_link\"\u003eHere\u003c/a\u003e] contains images from VGGFace we used to reconstruct 3D faces for (1) and (2)\n\n(4) [\u003ca href=\"https://drive.google.com/file/d/1Ac6b3oDB1Ot274u4CU-OaeSlUT5r6bZK/view?usp=share_link\"\u003eHere\u003c/a\u003e] contains preprocessed voice data (MFCC features) from Voxceleb for all the identities. 38G zipped file. Refer to this [\u003ca href=\"https://drive.google.com/file/d/1h26fBbqeYwgjjwPOZrsshIL0xRcfM6eI/view?usp=share_link\"\u003emeta file\u003c/a\u003e] to map id to name.\n\n(5) [\u003ca href=\"https://drive.google.com/file/d/1CTDhjZfDOTyUyfknpGKkkz0xXc756Ua0/view?usp=share_link\"\u003eHere\u003c/a\u003e] contains preprocessed voice data (MFCC features) from Voxceleb for the testing subset (A-E). You can download it for inference purpose. See later section.\n\n\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"demo/overall_purpose.png\" style=\"width:70%;\" align=\"centering\"\u003e\n\u003c/p\u003e\n\nWe study the cross-modal learning and analyze the correlation between voices and 3D face geometry. Unlike previous methods for studying this correlation between voices and faces and only work on  the 2D domain, we choose 3D representation that can better validate the supportive evidence from the physiology of the correlation between voices and skeletal and articulator structures, which potentially affect facial geometry.\n\n\u003cins\u003eComparison of recovered 3D face meshes with the baseline.\u003c/ins\u003e\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"demo/supervised_comp.png\" style=\"width:70%;\" align=\"centering\"\u003e\n\u003c/p\u003e\n\n\u003cins\u003eConsistency for the same identity using different utterances.\u003c/ins\u003e\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"demo/coherence.png\" style=\"width:70%;\" align=\"centering\"\u003e \n\u003c/p\u003e\n    \n##  \u003cdiv align=\"center\"\u003eDemo: Preprocessed fbank\u003c/div\u003e\n\nWe test on Ubuntu 16.04 LTS, NVIDIA 2080 Ti (only GPU is supported), and use anaconda for installing packages\n\nInstall packages\n\n1. `conda create --name CMP python=3.8`\n2. Install PyTorch compatible to your computer, we test on PyTorch v1.9 (should be compatible with other 1.0+ versions)\n3. install other dependency: opencv-python, scipy, PIL, Cython, pyaudio\n\n    Or use the environment.yml we provide instead: \n    - `conda env create -f environment.yml`\n    - `conda activate CMP`\n\n4. Build the rendering toolkit (by c++ and cython) for overlapping 3D meshes on images with configurations\n\n    ```\n    cd Sim3DR\n    bash build_sim3dr.sh\n    cd ..\n    ```\n\nDownload pretrained models and 3DMM configuration data\n\n5. Download from [\u003ca href=\"https://drive.google.com/file/d/1tqTSDrVVL3LkOWN-hduELm3YkWJ2ZUqu/view?usp=sharing\"\u003ehere\u003c/a\u003e] (~160M) and unzip under the root folder. This will create 'pretrained_models' (trained by unsupervised CMP) and 'train.configs' (3DMM config data) under the root folder. \n\nRead the preprocessed fbank for inference \n\n6. `python demo.py` (This will fetch the preprocessed MFCC and use them as network inputs)\n7. Results will be generated under `data/results/` (pre-generated references are under `data/results_reference`)\n\nMore preprocessed MFCC and 3D mesh (3DMM params) pairs can be downloaded: [\u003ca href=\"https://drive.google.com/drive/folders/1tT36oDujNXBw5SpwhY3PiBnGIE0FbvCs?usp=sharing\"\u003eVoxceleb-3D Data\u003c/a\u003e] (about 100G). \n\n##  \u003cdiv align=\"center\"\u003eDemo: :laughing: Try it! Use device mic input \u003c/div\u003e\n\n1. Do the above 1-5 step. Plus, download the \u003ca href=\"https://drive.google.com/file/d/1pNx8wS5O9QA3nzaToYIKRuUovzFPKkZj/view?usp=sharing\"\u003eface type meshes\u003c/a\u003e and extract under ./face_types\n\n2. `python demo_mic.py` The demo will take 5 seconds recording from your device and predict the face mesh. \n\nWe perform unsupervised gender classfication based on mean male and female shape and calculate the statistics between the predicted face and mean shape. Also we calculate the distance between the four types of faces (Regular, Slim, Skinny, Wide)and indicate which type the voice is closer to.\n\n3. Results will be generated under data/results\n\n##  \u003cdiv align=\"center\"\u003eInference from supervised framework \u003c/div\u003e\n\n1. Do the 1-5 step in Demo. Download pretrained supervised model [\u003ca href=\"https://drive.google.com/file/d/1dWDwtLpt7wW7VNCJCUKscUEP-mIkRpPo/view?usp=share_link\"\u003ehere\u003c/a\u003e]. Download voice data (A-E) for inference [\u003ca href=\"https://drive.google.com/file/d/1CTDhjZfDOTyUyfknpGKkkz0xXc756Ua0/view?usp=share_link\"\u003ehere\u003c/a\u003e], [\u003ca href=\"https://drive.google.com/file/d/1h26fBbqeYwgjjwPOZrsshIL0xRcfM6eI/view?usp=share_link\"\u003emeta file\u003c/a\u003e], and [\u003ca href=\"https://drive.google.com/file/d/1NdkqlCPhl-mvPU9TYlPgHE_FaNJjAysf/view?usp=share_link\"\u003egroundtruth\u003c/a\u003e]. Put the pretrained model under './pretrained_models/supervised_64'. Put the vocie data and meta file under './data'. Put the groundtruth under './data' and extract.\n\n\n2. Edit config.py Line 6: change to 'pretrained_models/supervised_64'\n\n3. \n    ```\n    python eval_sup.py\n    ```\n\nThis will match identity from voiceID and available 3D faces reconstructed from VGGFace via the meta file. Only predict 3D faces for those matched ID. Then it will save all the mesh obj files under './data/supervised_output/'\n\n##  \u003cdiv align=\"center\"\u003eEvaluation \u003c/div\u003e\n\n1. Do the 1-5 step in Demo. Download generated and saved mesh for validation set (name starting from A-E in Voxceleb-3D). From supervised CMP: https://drive.google.com/file/d/1_xobyRM-abjfrvzjbF7uwMVPFPfeKZC9/view?usp=share_link; \n\n(The same as groundtruth in the supervised inference) Voxceleb-3D validation set: https://drive.google.com/file/d/1NdkqlCPhl-mvPU9TYlPgHE_FaNJjAysf/view?usp=share_link. Put them under './data' and extract.\n\nThe valiation set for each identity contains image (.jpg), mesh (.obj), pointcloud (.xyz), image overlapped with mesh (_b.jpg), 3DMM parameters (.npy) (199-dim for shape and 29-dim for expression. This is in contrast to simplified 3DMM basis for 40-dim shape and 10-dim expression. You can donwload full basis from BFM-2009 \u003ca href=\"https://faces.dmi.unibas.ch/bfm/index.php?nav=1-1-0\u0026id=details\"\u003eofficial website\u003c/a\u003e. Otherwise, we provided already reconstructed mesh in obj extension)\n\n2. \n    ```\n    bash cal_size.sh\n    ```\n\nThis will run and report ARE metrics and keypoint error metrics.\n\n##  \u003cdiv align=\"center\"\u003eTraining \u003c/div\u003e\n\n1. Train the unsupervised framework\n\n-- Download 'Voxceleb-3D' data (2), (3), and (4). They are validation set, training images, and training voice banks. Extract and put them under './data'\n\n-- Download a much smaller set \u003ca href=\"https://drive.google.com/file/d/1TkGCd2aACqUXVYsELnd6JTPM6ErRveQC/view?usp=sharing\"\u003e[here]\u003c/a\u003e for fast online validation\n\n-- python gan_train_cascade.py\n\n##  \u003cdiv align=\"center\"\u003eCitation\u003c/div\u003e\nIf you find our work useful, please consider to cite us.\n\n    @inproceedings{wu2022cross,\n    title={Cross-Modal Perceptionist: Can Face Geometry be Gleaned from Voices?},\n    author={Wu, Cho-Ying and Hsu, Chin-Cheng and Neumann, Ulrich},\n    booktitle={CVPR},\n    year={2022}\n    }\n\n\nThis project is developed on [\u003ca href=\"https://github.com/choyingw/SynergyNet\"\u003eSynergyNet\u003c/a\u003e], [\u003ca href=\"https://github.com/cleardusk/3DDFA_V2\"\u003e3DDFA-V2\u003c/a\u003e] and [\u003ca href=\"https://github.com/cmu-mlsp/reconstructing_faces_from_voices\"\u003ereconstruction-faces-from-voice\u003c/a\u003e]\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchoyingw%2Fcross-modal-perceptionist","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchoyingw%2Fcross-modal-perceptionist","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchoyingw%2Fcross-modal-perceptionist/lists"}