{"id":51065595,"url":"https://github.com/marrlab/vlm_cytomorphology_eval","last_synced_at":"2026-06-23T06:30:31.687Z","repository":{"id":366028151,"uuid":"1274801097","full_name":"marrlab/vlm_cytomorphology_eval","owner":"marrlab","description":"Evaluation of large vision language models on cytomorphology tasks","archived":false,"fork":false,"pushed_at":"2026-06-19T23:09:39.000Z","size":227,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-20T01:05:13.250Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/marrlab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-19T22:56:20.000Z","updated_at":"2026-06-19T23:09:43.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/marrlab/vlm_cytomorphology_eval","commit_stats":null,"previous_names":["marrlab/vlm_cytomorphology_eval"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/marrlab/vlm_cytomorphology_eval","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marrlab%2Fvlm_cytomorphology_eval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marrlab%2Fvlm_cytomorphology_eval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marrlab%2Fvlm_cytomorphology_eval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marrlab%2Fvlm_cytomorphology_eval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/marrlab","download_url":"https://codeload.github.com/marrlab/vlm_cytomorphology_eval/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marrlab%2Fvlm_cytomorphology_eval/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34678932,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-23T02:00:07.161Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-23T06:30:29.735Z","updated_at":"2026-06-23T06:30:31.678Z","avatar_url":"https://github.com/marrlab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# VLM Cytomorphology Eval\n\nAutomated evaluation framework for vision-language models (VLMs) on cytology tasks and datasets.\n\n## Repository layout\n\n```\n.\n├── src/vlm_cytomorphology_eval/\n│   ├── config/             # dataset paths, naming conventions\n│   ├── prompts/            # task-specific prompts per dataset\n│   ├── models/             # API wrappers (GPT-4o, Gemini, Llama, DeepSeek, LLaVA-Med, MedFlamingo)\n│   ├── data/               # dataset preparation and fold splitting\n│   ├── inference/          # main entry point for VLM inquiries\n│   ├── finetuning/         # DINO / Llama fine-tuning\n│   ├── evaluation/         # answer review pipelines\n│   ├── features/           # feature extraction\n│   ├── visualization/      # confusion matrices, explainability plots\n│   ├── expert_review/      # selecting images for human review\n│   └── utils/\n├── notebooks/              # BiomedCLIP, CONCH zero-shot experiments\n├── scripts/                # SLURM / shell launchers\n├── tests/\n├── requirements.txt\n├── pyproject.toml\n├── .env.example\n└── LICENSE\n```\n\n## Installation\n\nWe recommend **conda** for this project because two of the supported models pin\nincompatible CUDA / PyTorch / `transformers` versions and must live in their\nown environments. Plain `venv` works only for the commercial-API-only path\n(GPT-4o, Gemini); it cannot manage per-env CUDA toolkits.\n\nYou will typically end up with three environments:\n\n| Env | Python | Purpose |\n|---|---|---|\n| `vlm_eval`  | 3.10 | Default — commercial APIs, Llama, fine-tuning, plotting, notebooks |\n| `llavamed`  | 3.10 | LLaVA-Med only (pins older `transformers`) |\n| `deepseek`  | 3.9  | DeepSeek-VL2 only (requires CUDA 11.8 + PyTorch 2.0.1) |\n\n### Default environment (`vlm_eval`)\n\n```bash\ngit clone https://github.com/\u003cyour-org\u003e/vlm_cytomorphology_eval.git\ncd vlm_cytomorphology_eval\n\nconda create -n vlm_eval python=3.10 -y\nconda activate vlm_eval\n\npip install -e .\n```\n\nThis covers GPT-4o, Gemini, Llama, BiomedCLIP, CONCH, fine-tuning (DINO),\nplotting, and all data-preparation scripts.\n\n### Lightweight alternative (API models only, no GPU)\n\nIf you only need the commercial APIs and don't want conda:\n\n```bash\npython -m venv .venv\nsource .venv/bin/activate\npip install -e .\n```\n\n### LLaVA-Med and DeepSeek-VL2\n\nBoth require dedicated conda environments installed from source — see the\n**Setting up LLaVA-Med** and **Setting up DeepSeek VL2** sections below.\n\n### Setting up LLaVA-Med\n\nLLaVA-Med must be installed from the official Microsoft repository. We recommend a separate conda env because LLaVA-Med pins older versions of `transformers` and `torch`.\n\n```bash\nconda create -n llavamed python=3.10 -y\nconda activate llavamed\n\ngit clone https://github.com/microsoft/LLaVA-Med.git\ncd LLaVA-Med\npip install -e .\n```\n\nPre-trained weights (e.g. `llava-med-v1.5-mistral-7b`) are pulled from HuggingFace at runtime — make sure `HF_TOKEN` is set if the checkpoint is gated.\n\nAfter installing LLaVA-Med, install this repo's requirements into the same env:\n\n```bash\ncd /path/to/vlm_cytomorphology_eval\npip install -e .\n```\n\n### Setting up DeepSeek VL2\n\nDeepSeek VL2 requires CUDA 11.8 and PyTorch 2.0.1, so use a dedicated conda env.\n\n```bash\n# Check available CUDA versions\nls /usr/local/ | grep cuda\n\n# Activate CUDA 11.8 (adjust paths for your system)\nexport CUDA_HOME=/usr/local/cuda-11.8\nexport PATH=/usr/local/cuda-11.8/bin:$PATH\nexport LD_LIBRARY_PATH=/usr/local/cuda-11.8/lib64:$LD_LIBRARY_PATH\nnvcc --version\n\n# Create and activate a Python 3.9 env\nconda create -n deepseek python=3.9 -y\nconda activate deepseek\n\n# Install DeepSeek-VL2 from source\ngit clone https://github.com/deepseek-ai/DeepSeek-VL2.git\ncd DeepSeek-VL2\npip install -e .\n\n# DeepSeek's requirements omit a few packages this repo needs:\npip install numpy==1.26.4 pandas==1.5.3 openpyxl==3.1.2\n\n# Install this repo into the same env\ncd /path/to/vlm_cytomorphology_eval\npip install -e .\n```\n\nDeepSeek VL2 only runs on an A100 with 80 GB memory.\n\n### Setting up MedFlamingo (optional)\n\nMedFlamingo uses `open_flamingo` and requires the LLaMA-7B base weights. Follow the [med-flamingo](https://github.com/snap-stanford/med-flamingo) repo's setup.\n\n## Configuration\n\nCopy `.env.example` to `.env` and fill in your API keys:\n\n```bash\ncp .env.example .env\n```\n\n| Variable | Used for |\n|----------|----------|\n| `OPENAI_API_KEY` | GPT-4o inference and fine-tuning |\n| `GEMINI_API_KEY` | Gemini inference |\n| `HF_TOKEN` | Downloading gated HuggingFace checkpoints |\n| `VLM_ROOT_FOLDER_PATH` | Override default data/results root |\n\n\u003e **Important:** The dataset and results paths inside `src/vlm_cytomorphology_eval/config/dataset_info_and_paths.py` are currently hard-coded to internal cluster locations (`/lustre/...`) and contributor home directories. Edit `LOCAL_ROOT_FOLDER_PATH` and the per-dataset paths to match your setup before running.\n\n## Usage\n\n### Running a VLM evaluation\n\n```bash\npython -m vlm_cytomorphology_eval.inference.run_api_inquiry \\\n    --dataset_name \u003cname\u003e --task_type \u003ctask\u003e --vlm_name \u003cmodel\u003e\n```\n\nMake sure the relevant API key is exported (or present in your `.env`).\n\n### Generating reports\n\n```bash\npython -m vlm_cytomorphology_eval.visualization.plot_conf_mat\n```\n\nThis computes confusion matrices and aggregate metrics across all configured models and datasets.\n\n### Adding a new dataset\n\n1. Add the dataset entry to `get_dataset_info` in `src/vlm_cytomorphology_eval/config/dataset_info_and_paths.py`.\n2. Add the dataset-specific prompts in `src/vlm_cytomorphology_eval/prompts/prompts.py`.\n3. Build the evaluation subset:\n\n   ```bash\n   python -m vlm_cytomorphology_eval.data.prepare_dataset \\\n       --dataset_name \u003cname\u003e --n_per_label \u003cn\u003e\n   ```\n\n### Adding a new VLM model family\n\n1. Add the family name to `available_model_families` and choose a representative model in `recommended_models` (both in `get_global_info`).\n2. Register the review model in `get_review_model`.\n3. Add a new `\u003cfamily\u003e_api.py` under `src/vlm_cytomorphology_eval/models/` exposing `\u003cfamily\u003e_api_visual_inquiry`, `\u003cfamily\u003e_api_text_inquiry`, and any task-specific functions (multi-image, etc.). See `gpt_api.py` for the simplest reference.\n4. Wire it into `import_model` in `src/vlm_cytomorphology_eval/inference/run_api_inquiry.py`.\n\n### Adding a new task\n\n1. Append the task name to `available_task_types` in `get_global_info`.\n2. Add prompts for all datasets and for both reviewed and unreviewed cases in `prompts.py`.\n3. Implement `run_api_\u003ctask\u003e` inside `run_api_inquiry.py`.\n4. Add any per-model task functions in the relevant model file under `models/`.\n\n## Generated folder structure\n\nThe pipeline assumes / creates the following layout under the configured root:\n\n```\n\u003croot\u003e/\n├── Datasets/\u003cdataset_name\u003e/{train,val,test}/...\n├── Results/\n│   └── \u003cdataset_name\u003e/\n│       ├── answers/\n│       └── confusion_matrices/\n└── Plots/\u003cdataset_name\u003e/\n```\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n\n## Citation\n\nIf you use this code, please cite the associated publication (link will be added on release).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarrlab%2Fvlm_cytomorphology_eval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarrlab%2Fvlm_cytomorphology_eval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarrlab%2Fvlm_cytomorphology_eval/lists"}