{"id":38676563,"url":"https://github.com/wglab/gestaltmml","last_synced_at":"2026-01-17T10:01:16.125Z","repository":{"id":214596196,"uuid":"682849265","full_name":"WGLab/GestaltMML","owner":"WGLab","description":null,"archived":false,"fork":false,"pushed_at":"2024-08-12T02:38:18.000Z","size":488,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-08-13T02:55:44.884Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/WGLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-25T02:46:56.000Z","updated_at":"2024-08-12T02:38:21.000Z","dependencies_parsed_at":null,"dependency_job_id":"98256c78-7df2-4893-8212-94cd64f94e33","html_url":"https://github.com/WGLab/GestaltMML","commit_stats":null,"previous_names":["wglab/gestaltmml-gestaltgpt","wglab/gestaltmml"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/WGLab/GestaltMML","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WGLab%2FGestaltMML","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WGLab%2FGestaltMML/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WGLab%2FGestaltMML/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WGLab%2FGestaltMML/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/WGLab","download_url":"https://codeload.github.com/WGLab/GestaltMML/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WGLab%2FGestaltMML/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28505570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T06:57:29.758Z","status":"ssl_error","status_checked_at":"2026-01-17T06:56:03.931Z","response_time":85,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-17T10:01:15.889Z","updated_at":"2026-01-17T10:01:16.114Z","avatar_url":"https://github.com/WGLab.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GestaltMML\nThe GestaltMML is a cutting-edge multimodal machine learning model integrating frontal facial images, demographic information of patients and clinical text data for the diagnosis of rare genetic disorders. It leverages the power of fine-tuning the Vision-and-Language Transformer (ViLT) (see [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://arxiv.org/abs/2102.03334)) using data from the [GestaltMatcher Database (GMDB)](https://db.gestaltmatcher.org). To gain access to the GMDB, interested individuals need to submit an application [here](https://db.gestaltmatcher.org/documents).\n\nGestaltMML is distributed under [MIT License by Wang Genmoics Lab](https://wglab.mit-license.org).\n\n## Package Installation\nWe need to first install the following required packages (5-10 mins typically) for model training and inference.\n```\nconda create -n gestaltmml python=3.11\nconda activate gestaltmml\nconda install pytorch==2.2.0 torchvision==0.17.0 torchaudio==2.2.0 pytorch-cuda=11.8 -c pytorch -c nvidia\npip install transformers dataset\npip install pandas\nconda install -c anaconda ipykernel\npython -m ipykernel install --user --name=gestaltmml\n```\n## Training\nOnce the package has been installed and access to the GMDB has been granted, the following [GestaltMML_Training.ipynb](https://github.com/WGLab/GestaltMML/blob/main/GestaltMML_Training.ipynb) can be executed to train the GestaltMML model. Please be aware that it is necessary to download the hp.json (HPO dictionary) from [https://hpo.jax.org/app/data/ontology](https://hpo.jax.org/app/data/ontology), and the omim_summarized_1.0.9.json (summarized OMIM texts for data augmentation) is available for public access in this GitHub repository. The model weights will be made available upon request to those who have gained access to GMDB. Additionally, the notebook can be modified to conduct your own experiments. Please be aware that the current version of GestaltMML has been developed using GMDB (v1.0.9). The training script is designed for easy adaptation to subsequent versions of the GMDB.\n\n## Inference\nFor inference using GestaltMML, please see the notebook [GestaltMML_Inference.ipynb](https://github.com/WGLab/GestaltMML/blob/main/GestaltMML_Inference.ipynb) for detailed instruction. \nIf you want to simply load GestaltMML on your local machine for inference, the model weights GestaltMML_model.pt are saved in [GestaltMML_model_weights.zip](https://github.com/WGLab/GestaltMML/releases/download/v1.0.9/GestaltMML_model_weights.zip). In addition, disease_dict.json, three sample test images and sample texts (in GestaltMML_input.csv) are provided in the folder [inference](inference) in this Github page. Please use the following command:\n```\npython inference.py --csv_file path/to/GestaltMML_input.csv --base_image_path path/to/test_images --model_path path/to/GestaltMML_model.pt --disease_dict_path path/to/disease_dict.json --top_n 5\n```\nIt is strongly advised to adhere to the guidelines provided [here](https://github.com/igsb/GestaltMatcher-Arc/tree/service?tab=readme-ov-file#crop-and-align-faces) for cropping and aligning frontal facial images prior to conducting the inference. The typical running time for inference script (inference.py) is \u003c1 mins with CPU.\n## Citation\nWu, D., Yang, J., Liu, C., Hsieh, T.C., Marchi, E., Krawitz, P., Weng, C., Chung, W., Lyon, G.J., Krantz, I.D., Kalish, J.M. and Wang, K., 2024. GestaltMML: Enhancing Rare Genetic Disease Diagnosis through Multimodal Machine Learning Combining Facial Images and Clinical Texts. arXiv preprint [arXiv:2312.15320](https://arxiv.org/abs/2312.15320).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwglab%2Fgestaltmml","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwglab%2Fgestaltmml","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwglab%2Fgestaltmml/lists"}