{"id":38676250,"url":"https://github.com/wglab/mardd","last_synced_at":"2026-01-17T10:01:05.467Z","repository":{"id":327677849,"uuid":"1071183675","full_name":"WGLab/MARDD","owner":"WGLab","description":null,"archived":false,"fork":false,"pushed_at":"2025-12-05T03:04:44.000Z","size":26253,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-08T10:51:47.683Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/WGLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-07T01:45:26.000Z","updated_at":"2025-12-05T03:04:48.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/WGLab/MARDD","commit_stats":null,"previous_names":["wglab/mardd"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/WGLab/MARDD","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WGLab%2FMARDD","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WGLab%2FMARDD/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WGLab%2FMARDD/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WGLab%2FMARDD/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/WGLab","download_url":"https://codeload.github.com/WGLab/MARDD/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WGLab%2FMARDD/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28505570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T06:57:29.758Z","status":"ssl_error","status_checked_at":"2026-01-17T06:56:03.931Z","response_time":85,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-17T10:00:42.724Z","updated_at":"2026-01-17T10:01:05.457Z","avatar_url":"https://github.com/WGLab.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# MARDD: Multimodal AI for Rare Disease Diagnosis with Vision–Language Integration\n\n\u003e **MARDD** (Gestalt Multimodal Model for Mendelian Disease Diagnosis with Vision–Language Integration)  \n\u003e A **diagnosis-first** multimodal model that leverages **facial images** and **HPO-encoded clinical text** for **Mendelian rare disease diagnosis**.  \n\u003e Additionally, we use **synthetic facial images** (the PDIDB code is available [here](https://github.com/WGLab/PDIDB)) for **qualitative evaluation and analysis** and for **oversampling rare classes**.\n\n---\n\n### Overall pipeline (Diagnosis-centric)\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/figures/pipeline.png\" alt=\"Overall diagnosis-centric pipeline of GestaltM3D-VL\" width=\"800\"\u003e\n\u003c/p\u003e\n\n### Example faces \u0026 HPO text\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/figures/examples.png\" alt=\"Example GMDB cases with facial images and HPO descriptions\" width=\"800\"\u003e\n\u003c/p\u003e\n\n---\n\n## Table of Contents\n\n1. [Project Overview](#1-project-overview)\n2. [Dataset: GMDB Ten-Disease Subset](#2-dataset-gmdb-ten-disease-subset)\n3. [Data Preprocessing Pipeline](#3-data-preprocessing-pipeline)\n4. [Part I – GestaltM³D-VL for Multimodal Diagnosis](#4-part-i--gestaltm³d-vl-for-multimodal-diagnosis)\n5. [Part II – StyleGAN3 Synthesis for Evaluation](#5-part-ii--stylegan3-synthesis-for-evaluation)\n6. [Repository Structure](#6-repository-structure)\n7. [Installation](#7-installation)\n8. [Data Preparation](#8-data-preparation)\n9. [Training \u0026 Evaluation](#9-training--evaluation)\n10. [Figures to Include](#10-figures-to-include)\n11. [Citation](#11-citation)\n12. [License \u0026 Disclaimer](#12-license--disclaimer)\n\n---\n\n## 1. Project Overview\n\nThis repository implements a **disease-diagnosis–centric** pipeline on the **GMDB ten-disease subset** (and optionally CHOP cohorts):\n\n1. **Part I – GestaltM³D-VL (Main Contribution: Multimodal Diagnosis)**\n   - A **vision–language model** based on **Qwen-VL** that ingests:\n     - Preprocessed **facial images**.\n     - **HPO-encoded clinical text** (with or without demographics).\n   - Outputs **Mendelian rare disease labels** under a **long-tailed** label space.\n   - Uses **Class-Balanced Focal Loss** to focus on **rare diseases**.\n\n2. **Part II – StyleGAN3-based Synthesis (Support: Evaluation \u0026 Visualization)**\n   - Train **class-conditional StyleGAN3** on GMDB faces.\n   - Generate **synthetic facial images**:\n     - For **qualitative evaluation** of disease-specific gestalt.\n     - For **visual case studies** comparing predicted diseases and synthetic exemplars.\n---\n\n## 2. Dataset: GMDB Ten-Disease Subset\n\nWe use the **GestaltMatcher Database (GMDB)**, which pairs:\n\n- **Facial images**\n- **HPO-encoded clinical text**\n- **Demographic information** (ethnicity, age, sex)\n\nOverall label space: **528 syndromes/disorders** (244 frequent, 284 rare).  \nIn this repository we focus on the **ten diseases with the most distinctive facial phenotypes**, curated by clinical geneticists.\n\nKey stats for the **ten-disease subset**:\n\n- **1,847 cases** with patient-level splits.\n- **Test set**: only **HPO-annotated cases** (≈ 818 cases).\n- Non-HPO cases are used **only as data augmentation** during training.\n\n\u003e ⚠️ **Dataset access**  \n\u003e GMDB (and CHOP, if used) are **not included** in this repository.  \n\u003e You must obtain data access following your **institutional, IRB, and data-use agreements**, then place the files under `data/raw/`.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/figures/data_distribution.svg\" alt=\"Label distribution and cohort statistics for the GMDB ten-disease subset\" width=\"800\"\u003e\n\u003c/p\u003e\n\n---\n\n## 3. Data Preprocessing Pipeline\n\nOriginal facial images in GMDB/CHOP often suffer from:\n\n- Low resolution, blur\n- Grayscale or washed-out colors\n- Cluttered backgrounds\n\nWe build a **three-stage preprocessing pipeline** shared by both the diagnosis model and StyleGAN3:\n\n1. **Color Restoration – DDColor**\n   - Colorize grayscale / washed-out facial images.\n\n2. **Face Restoration \u0026 Super-Resolution – GFPGAN**\n   - Enhance facial details and upsample low-resolution faces.\n\n3. **Face Detection \u0026 Normalization – MediaPipe Face Detection**\n   - Detect faces and perform **tight cropping**.\n   - Remove background and resize to **224×224** RGB.\n   - Normalize input space and control token budget for the multimodal LLM.\n\nThe script `src/data/preprocess_faces.py` orchestrates DDColor, GFPGAN, and MediaPipe and writes processed images into `data/processed/images`.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/figures/data_preprocess.png\" alt=\"DDColor + GFPGAN + MediaPipe facial preprocessing pipeline\" width=\"800\"\u003e\n\u003c/p\u003e\n\n---\n\n## 4. Part I – GestaltM³D-VL for Multimodal Diagnosis\n\n\u003e ✅ **This is the main part of the project.**  \n\u003e All other components (e.g., StyleGAN3) are designed to support or analyze this diagnosis model.\n\n### 4.1 Motivation\n\nWe target **Mendelian rare disease diagnosis** from:\n\n- **Facial gestalt** (front-view facial images).\n- **HPO-encoded clinical text** (+/- demographics).\n\nChallenges:\n\n- GMDB is **highly long-tailed** (many rare diseases with few examples).\n- Data is **noisy** (missing HPO terms, variable image quality).\n- Standard cross-entropy tends to overfit **head classes** and underperform on rare diseases.\n\n### 4.2 Backbone: Qwen-VL Family\n\nGestaltM³D-VL is built on **Qwen-VL** models:\n\n- **Qwen-2-VL-7B-Instruct**\n- **Qwen-2.5-VL-7B-Instruct** (default backbone)\n- (Optional) **Qwen-3-VL-8B-Instruct**\n\nAdvantages:\n\n- Strong **vision–language alignment**.\n- Good **parameter–compute trade-off** (7–8B).\n- Handles **long, noisy clinical text**.\n\n### 4.3 Model Architecture (Sequence Classification)\n\nWe adapt Qwen-VL into a **multimodal sequence classifier**:\n\n1. **Inputs**:\n   - `image`: 224×224 preprocessed facial image.\n   - `text`:\n     - HPO-encoded phenotypes (primary signal).\n     - Optional demographics (often **dropped** to reduce bias).\n\n2. **Multimodal encoding**:\n   - Construct an instruction-style prompt that embeds HPO terms into natural language.\n   - Feed image + text into Qwen-VL.\n   - Obtain the hidden state at a special **`[CLS]` token** (or equivalent pooled representation).\n\n3. **Classifier head**:\n   - Apply a small **MLP/Linear(D → #classes)** head to predict disease logits.\n\n4. **Loss**:\n   - Use **Class-Balanced Focal Loss** to handle severe imbalance.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/figures/mm_llm_architecture.png\" alt=\"GestaltM3D-VL multimodal sequence classification architecture\" width=\"800\"\u003e\n\u003c/p\u003e\n\n### 4.4 Loss: Class-Balanced Focal Loss\n\nLet \\( n_y \\) be the number of samples of class \\( y \\).  \nWe compute effective class weights \\( \\alpha_y \\) based on the **effective number of samples** (e.g., Cui et al., CVPR 2019) and adopt **focal loss** with focusing parameter \\( \\gamma \\):\n\n- Increases the importance of **rare diseases**.\n- Focuses training on **hard examples** instead of easy head-class samples.\n\nImplementation:\n\n- `src/models/mm_llm/losses.py`\n\n### 4.5 Training Strategy\n\nTypical strategy:\n\n- Freeze a large portion of the **text backbone**.\n- Optionally partially freeze the **vision backbone**.\n- Train:\n  - Multimodal projector layers.\n  - Top transformer blocks (via LoRA or full finetuning).\n  - Final classifier head.\n\nTraining entry point:\n\n- `src/models/mm_llm/train_mm_llm.py`\n- `scripts/train_mm_llm.sh`\n\nEvaluation:\n\n- `src/models/mm_llm/evaluate_mm_llm.py`\n- `scripts/eval_mm_llm.sh`\n\n### 4.6 Empirical Observations (internal)\n\n- **Multimodal (image + text) \u003e image-only**.\n- Dropping demographics can **maintain or improve** performance and reduce explicit reliance on ethnicity/age/sex.\n- Upgrading from Qwen-2-VL to **Qwen-2.5-VL** improves overall diagnosis accuracy and especially rare-class performance.\n\n---\n\n## 5. Part II – StyleGAN3 Synthesis for Evaluation\n\n\u003e ℹ️ **StyleGAN3 is auxiliary** in this repository.  \n\u003e Its role is to **support evaluation and visualization** of rare disease facial gestalt, not to perform diagnosis itself.\n\n### 5.1 Goal\n\nWe use **StyleGAN3** as a **class-conditional face generator** trained on GMDB ten-disease faces. Synthetic images are used to:\n\n- Visualize **disease-specific facial patterns** learned from GMDB.\n- Provide **synthetic exemplars** for:\n  - Teaching materials\n  - Qualitative comparison with GestaltM³D-VL predictions\n- Optionally stress-test robustness (e.g., more varied poses, ages, or subtle changes).\n\n### 5.2 StyleGAN3 Overview (Short)\n\n- Alias-free generation for better geometric consistency.\n- Class-conditional setup across ten disease classes (+ optional unaffected class).\n- Trained on the **same preprocessed 224×224 faces** used by GestaltM³D-VL.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/figures/stylegan3_overview.png\" alt=\"StyleGAN3 training overview\" width=\"800\"\u003e\n\u003c/p\u003e\n\n### 5.3 Training \u0026 Sampling\n\nConfiguration (see `configs/stylegan3/ten_disease_default.yaml`):\n\n- Dataset: preprocessed GMDB faces.\n- Classes: 10 diseases (+/- unaffected).\n\nRun:\n\n```bash\nbash scripts/train_stylegan3.sh\nbash scripts/sample_stylegan3.sh\n```\n\nThis will populate:\n\n```text\ndata/processed/stylegan3_samples/\n  disease_0/\n  disease_1/\n  ...\n```\n\nwith class-conditional synthetic faces.\n\n### 5.4 Using Synthetic Images\n\nYou can use the synthetic images to:\n\n- Build **grids of synthetic faces per disease**.\n- Compare **real patient images \u0026 model predictions** with:\n  - Synthetic faces from the predicted disease.\n- Support **case studies** and **clinical teaching** materials.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/figures/synthetic_1.png\" alt=\"Synthetic samples per disease class generated by StyleGAN3_1\" width=\"800\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/figures/synthetic_2.png\" alt=\"Synthetic samples per disease class generated by StyleGAN3_2\" width=\"800\"\u003e\n\u003c/p\u003e\n\n### 5.5 Interactive Synthetic Image Gallery\n\nWe provide an interactive web-based gallery to explore the synthetic facial images generated by our StyleGAN3 model:\n\n**[🌐 Phenotype-Disease Synthetic Image Database](https://hongzhuochen.github.io/Phenotype-Disease-Image-Database-demo/)**\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/figures/web_gallery_screenshot.png\" alt=\"Interactive web gallery showing StyleGAN3 synthetic faces\" width=\"800\"\u003e\n\u003c/p\u003e\n\nThis gallery features:\n- **StyleGAN3-generated synthetic faces** for all ten disease classes\n- **Interactive filtering** by disease type, age, gender, and ethnicity\n- **Metadata annotations** corresponding to the generation conditions\n- **Download functionality** for research and educational purposes\n\n\u003e ⚠️ **Disclaimer**: All images displayed are **fully synthetic** and generated by StyleGAN3. They do not represent real patients and are intended for research and educational purposes only.\n\n---\n\n## 6. Installation\n\n### 6.1 Conda environment\n\n```bash\nconda env create -f environment.yml\nconda activate gestaltm3d-vl\n```\n\n### 6.2 Pip\n\n```bash\npython -m venv .venv\nsource .venv/bin/activate  # on Windows: .venv\\Scripts\\activate\npip install -r requirements.txt\n```\n\n`requirements.txt` (sketch):\n\n- `torch`, `torchvision`, `torchaudio`\n- `transformers`, `accelerate`, `bitsandbytes` (optional)\n- `timm`\n- `opencv-python`, `mediapipe`\n- `gfpgan`, `ddcolor`\n- `scikit-learn`\n- `pandas`, `numpy`\n- `matplotlib`, `seaborn`\n- `rich`, `pyyaml`\n\n---\n\n## 7. Data Preparation\n\n### 7.1 Place raw data\n\nPlace GMDB (and optional CHOP) metadata and images under:\n\n```text\ndata/raw/\n  gmdb_metadata.csv\n  images/\n    patient_*.jpg\n```\n\n\u003e The exact file naming depends on your internal export.  \n\u003e Adjust in `src/data/build_splits.py` and `src/data/preprocess_faces.py` accordingly.\n\n### 7.2 Build patient-level splits\n\n```bash\npython -m src.data.build_splits \\\n  --input_meta data/raw/gmdb_metadata.csv \\\n  --output_dir data/splits/ten_disease\n```\n\nCreates:\n\n- `train.csv`\n- `val.csv`\n- `test.csv`\n\nwith columns such as:\n\n- `image_path`\n- `disease_label`\n- `hpo_text`\n- `demographics` (optional)\n\n### 7.3 Run facial preprocessing\n\n```bash\npython -m src.data.preprocess_faces \\\n  --input_dir data/raw/images \\\n  --meta_csv data/splits/ten_disease/train.csv \\\n  --output_dir data/processed/images\n```\n\nRepeat or adapt for val/test splits.\n\n---\n\n## 8. Training \u0026 Evaluation\n\n### 8.1 MARDD (Main)\n\n```bash\n# Train GestaltM3D-VL on GMDB ten-disease subset\nbash scripts/train_mm_llm.sh\n\n# Evaluate on held-out test set\nbash scripts/eval_mm_llm.sh\n```\n\nControl hyperparameters and backbone in:\n\n- `configs/mm_llm/qwen2_5_vl_ten_disease.yaml`\n\n\n## Citation\n\n```bibtex\nplaceholder\n```\n\nAlso consider citing:\n\n- GestaltMatcher / GMDB / GestaltMML\n- Qwen2-VL / Qwen2.5-VL / Qwen3-VL technical reports\n- StyleGAN3, DDColor, GFPGAN, MediaPipe\n\n---\n\n## License \u0026 Disclaimer\n\n- **License**: MIT LICENSE.  \n- **Data**: This repository does **not** include patient-identifiable GMDB/CHOP data.  \n- **Usage**:\n  - Any use of real patient data must follow **IRB**, **data-use agreements**, and **local regulations**.\n  - Synthetic images are intended for **research** and **evaluation** only.\n\n\u003e This repository is for **research** and **method development** only.  \n\u003e It is **not** a certified medical device and should not be used directly for clinical decision-making without thorough validation and regulatory approval.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwglab%2Fmardd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwglab%2Fmardd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwglab%2Fmardd/lists"}