{"id":13708567,"url":"https://github.com/kaiko-ai/eva","last_synced_at":"2025-12-24T17:59:58.873Z","repository":{"id":228574274,"uuid":"743986791","full_name":"kaiko-ai/eva","owner":"kaiko-ai","description":"Evaluation framework for oncology foundation models (FMs)","archived":false,"fork":false,"pushed_at":"2025-04-29T10:16:04.000Z","size":15069,"stargazers_count":105,"open_issues_count":35,"forks_count":9,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-29T10:31:30.038Z","etag":null,"topics":["evaluation-framework","foundation-models","machine-learning","oncology"],"latest_commit_sha":null,"homepage":"https://kaiko-ai.github.io/eva/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kaiko-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"docs/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-01-16T12:05:03.000Z","updated_at":"2025-04-29T09:33:32.000Z","dependencies_parsed_at":"2024-05-21T08:04:45.656Z","dependency_job_id":"9b41f8d2-527f-4bf0-aabd-f34ab7bd1155","html_url":"https://github.com/kaiko-ai/eva","commit_stats":null,"previous_names":["kaiko-ai/eva"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kaiko-ai%2Feva","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kaiko-ai%2Feva/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kaiko-ai%2Feva/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kaiko-ai%2Feva/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kaiko-ai","download_url":"https://codeload.github.com/kaiko-ai/eva/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252693550,"owners_count":21789712,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["evaluation-framework","foundation-models","machine-learning","oncology"],"created_at":"2024-08-02T23:00:27.240Z","updated_at":"2025-12-24T17:59:58.864Z","avatar_url":"https://github.com/kaiko-ai.png","language":"Python","funding_links":[],"categories":["Software"],"sub_categories":["Machine Learning"],"readme":"\u003cdiv align=\"center\"\u003e\n\n\u003cbr /\u003e\n\n\u003cimg src=\"https://github.com/kaiko-ai/eva/blob/main/docs/images/eva-logo.png?raw=true\" width=\"340\"\u003e\n\n\u003cbr /\u003e\n\u003cbr /\u003e\n\n_Oncology FM Evaluation Framework by kaiko.ai_\n\n[![PyPI](https://img.shields.io/pypi/v/kaiko-eva.svg?logo=python)](https://pypi.python.org/pypi/kaiko-eva)\n[![docs](https://img.shields.io/badge/📚_docs-latest-green)](https://kaiko-ai.github.io/eva/latest)\n[![license](https://img.shields.io/badge/⚖️_License-Apache%202.0-blue.svg?labelColor=gray)](https://github.com/kaiko-ai/eva#license)\u003cbr\u003e\n[![paper](http://img.shields.io/badge/OpenReview-MIDL_2024-B31B1B.svg)](https://openreview.net/forum?id=FNBQOPj18N\u0026noteId=FNBQOPj18N)\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/kaiko-ai/eva#installation\"\u003eInstallation\u003c/a\u003e •\n  \u003ca href=\"https://github.com/kaiko-ai/eva#how-to-use\"\u003eHow To Use\u003c/a\u003e •\n  \u003ca href=\"https://github.com/kaiko-ai/eva#quick-start\"\u003eQuick Start\u003c/a\u003e •\n  \u003ca href=\"https://kaiko-ai.github.io/eva/\"\u003eDocumentation\u003c/a\u003e •\n  \u003ca href=\"https://kaiko-ai.github.io/eva/dev/datasets/\"\u003eDatasets\u003c/a\u003e •\n  \u003ca href=\"https://github.com/kaiko-ai/eva#benchmarks\"\u003eBenchmarks\u003c/a\u003e \u003cbr\u003e\n  \u003ca href=\"https://github.com/kaiko-ai/eva#contributing\"\u003eContribute\u003c/a\u003e •\n  \u003ca href=\"https://github.com/kaiko-ai/eva#acknowledgements\"\u003eAcknowledgements\u003c/a\u003e\n\u003c/p\u003e\n\n\u003c/div\u003e\n\n\u003cbr /\u003e\n\n_`eva`_ is an evaluation framework for oncology foundation models (FMs) by [kaiko.ai](https://kaiko.ai/).\nCheck out the [documentation](https://kaiko-ai.github.io/eva/) for more information.\n\n### Highlights:\n- Easy and reliable benchmark of Oncology FMs\n- Supports patch-level classification, slide-level classification, semantic segmentation, and (visual) question answering tasks.\n- Automatic embedding inference and evaluation of a downstream task\n- Native support of popular medical [datasets](https://kaiko-ai.github.io/eva/main/datasets/) and models\n- Produce statistics over multiple evaluation fits and multiple metrics\n\n## Installation\n\nSimple installation from PyPI:\n```sh\n# to install the core version only\npip install kaiko-eva\n\n# to install the expanded `vision` version\npip install 'kaiko-eva[vision]'\n\n# to install the expanded `language` version\npip install 'kaiko-eva[language]'\n\n# to install the expanded `multimodal` version\npip install 'kaiko-eva[multimodal]'\n\n# to install everything\npip install 'kaiko-eva[all]'\n```\n\nTo install the latest version of the `main` branch:\n```sh\npip install \"kaiko-eva[all] @ git+https://github.com/kaiko-ai/eva.git\"\n```\n\nYou can verify that the installation was successful by executing:\n```sh\neva --version\n```\n\n## How To Use\n\n_`eva`_ can be used directly from the terminal as a CLI tool as follows:\n```sh\neva {fit,predict,predict_fit} --config url/or/path/to/the/config.yaml \n```\n\n_`eva`_ uses [jsonargparse](https://jsonargparse.readthedocs.io/en/v4.31.0/) to\nmake it easily configurable by automatically generating command line interfaces (CLIs),\nwhich allows to call *any* Python object from the command line. Moreover, the configuration structure is always in sync with the code. Thus, _`eva`_ can be used either directly from Python or as a CLI tool (recommended).\n\nFor more information, please refer to the [documentation](https://kaiko-ai.github.io/eva/dev/user-guide/tutorials/offline_vs_online/).\n\n\u003cdetails\u003e\n  \u003csummary\u003eLearn about Configs\u003c/summary\u003e\n\nThe following interfaces are identical:\n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003ePython interface\u003c/th\u003e\n\u003cth\u003eConfiguration file\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\u003csub\u003e\n\n```Python\n# main.py\n# execute with: `python main.py`\n\nfrom torch import nn\n\nfrom eva import core\nfrom eva.vision import datasets, transforms\n\n# initialize trainer\ntrainer = core.Trainer(max_steps=100)\n\n# initialize model\nmodel = core.HeadModule(\n  backbone=nn.Flatten(),\n  head=nn.Linear(150528, 4),\n  criterion=nn.CrossEntropyLoss(),\n)\n\n# initialize data\ndata = core.DataModule(\n  datasets=core.DatasetsSchema(\n    train=datasets.BACH(\n      root=\"data/bach\",\n      split=\"train\",\n      download=True,\n      transforms=transforms.ResizeAndCrop(),\n    ),\n  ),\n  dataloaders=core.DataloadersSchema(\n    train=core.DataLoader(batch_size=32),\n  ),\n)\n\n# perform fit\npipeline = core.Interface()\npipeline.fit(trainer, model=model, data=data)\n```\n\u003c/sub\u003e\n\u003ctd\u003e\n\u003csub\u003e\n\n```yaml\n# main.yaml\n# execute with: `eva fit --config main.yaml`\n\n---\ntrainer:\n  class_path: eva.Trainer\n  init_args:\n    max_steps: 100\nmodel:\n  class_path: eva.HeadModule\n  init_args:\n    backbone: torch.nn.Flatten\n    head:\n      class_path: torch.nn.Linear\n      init_args:\n        in_features: 150528\n        out_features: 4\n    criterion: torch.nn.CrossEntropyLoss\ndata:\n  class_path: eva.DataModule\n  init_args:\n    datasets:\n      train:\n        class_path: eva.vision.datasets.BACH\n        init_args:\n          root: ./data/bach\n          split: train\n          download: true\n          transforms: eva.vision.transforms.ResizeAndCrop\n    dataloaders:\n      train:\n        batch_size: 32\n```\n\u003c/sub\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\nThe `.yaml` file defines the functionality of _`eva`_\nby parsing and translating its content to Python objects directly.\nNative supported configs can be found at the\n[configs](https://github.com/kaiko-ai/eva/tree/main/configs) directory\nof the repo, which can be both locally stored or remote.\n\n\u003c/details\u003e\n\n## Quick Start\n\nWe define two types of evaluations: **online** and **offline**.\nWhile online fit uses the backbone (FM) to perform forward passes\nduring the fitting process, offline fit first generates embeddings\nwith the backbone and then fits the model using these embeddings as\ninput, resulting in a faster evaluation.\n\nHere are some examples to get you started:\n\n- Perform a downstream offline **classification** evaluation of `DINO ViT-S/16`\non the `BACH` dataset with linear probing by first pre-calculating the embeddings:\n  ```sh\n  DOWNLOAD_DATA=true \\\n  MODEL_NAME=universal/vit_small_patch16_224_dino \\\n  eva predict_fit --config https://raw.githubusercontent.com/kaiko-ai/eva/main/configs/vision/pathology/offline/classification/bach.yaml\n  ```\n\n- Perform a downstream online **segmentation** evaluation of `DINO ViT-S/16` on the `MoNuSAC` dataset with the `ConvDecoderWithImage` decoder:\n  ```sh\n  DOWNLOAD_DATA=true \\\n  MODEL_NAME=universal/vit_small_patch16_224_dino \\\n  eva fit --config https://raw.githubusercontent.com/kaiko-ai/eva/main/configs/vision/pathology/online/segmentation/monusac.yaml\n  ```\n\nBy default `eva` will perform 5 evaluation runs using different seeds, however, you can control the number of runs through the `N_RUNS` environment variable or in the configuration file. The results will be saved to `./logs` by default, or to `OUTPUT_ROOT` if specified.\n\nFor more examples, take a look at the [configs](https://github.com/kaiko-ai/eva/tree/main/configs)\nand [tutorials](https://kaiko-ai.github.io/eva/main/user-guide/advanced/replicate_evaluations/).\n\n\u003e [!NOTE]\n\u003e All the datasets that support automatic download in the repo have by default the option to automatically download set to false.\n\u003e For automatic download you have to manually set the environment variable `DOWNLOAD_DATA=true` or in the configuration file `download=true`.\n\n## Leaderboards\n\nThe following table shows the FMs we have evaluated with _`eva`_. For more detailed information about the evaluation process, please refer to our [documentation](https://kaiko-ai.github.io/eva/main/leaderboards/).\n\n### Pathology\n\n\u003cimg src=\"./docs/images/leaderboards/pathology.svg\" alt=\"Pathology Leaderboard\"\u003e\n\n### Radiology\n\n\u003cimg src=\"./docs/images/leaderboards/radiology.svg\" alt=\"Radiology Leaderboard\" width=\"600\"\u003e\n\n\n## Contributing\n\n_`eva`_ is an open source project and welcomes contributions of all kinds. Please checkout the [developer](./docs/DEVELOPER_GUIDE.md)\nand [contributing guide](./docs/CONTRIBUTING.md) for help on how to do so.\n\nAll contributors must follow the [code of conduct](./docs/CODE_OF_CONDUCT.md).\n\n\n## Acknowledgements\n\nOur codebase is built using multiple opensource contributions\n\n\u003cdiv align=\"center\"\u003e\n\n[![python](https://img.shields.io/badge/-Python-blue?logo=python\u0026logoColor=white)](https://github.com/pre-commit/pre-commit)\n[![pytorch](https://img.shields.io/badge/PyTorch-ee4c2c?logo=pytorch\u0026logoColor=white)](https://pytorch.org/get-started/locally/)\n[![lightning](https://img.shields.io/badge/-⚡️_Lightning-792ee5?logo=pytorchlightning\u0026logoColor=white)](https://pytorchlightning.ai/)\u003cbr\u003e\n[![black](https://img.shields.io/badge/Code%20Style-Black-black.svg?labelColor=gray)](https://black.readthedocs.io/en/stable/)\n[![isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat\u0026labelColor=ef8336)](https://pycqa.github.io/isort/)\n[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)\n[![Checked with pyright](https://microsoft.github.io/pyright/img/pyright_badge.svg)](https://microsoft.github.io/pyright/)\u003cbr\u003e\n[![pdm-managed](https://img.shields.io/badge/pdm-managed-blueviolet)](https://pdm-project.org)\n[![Nox](https://img.shields.io/badge/%F0%9F%A6%8A-Nox-D85E00.svg)](https://github.com/wntrblm/nox)\n[![Built with Material for MkDocs](https://img.shields.io/badge/Material_for_MkDocs-526CFE?logo=MaterialForMkDocs\u0026logoColor=white)](https://squidfunk.github.io/mkdocs-material/)\n\n\u003c/div\u003e\n\n\n## Citation\n\nIf you find this repository useful, please consider giving a star ⭐ and adding the following citation:\n\n```bibtex\n@inproceedings{kaiko.ai2024eva,\n    title={eva: Evaluation framework for pathology foundation models},\n    author={kaiko.ai and Ioannis Gatopoulos and Nicolas K{\\\"a}nzig and Roman Moser and Sebastian Ot{\\'a}lora},\n    booktitle={Medical Imaging with Deep Learning},\n    year={2024},\n    url={https://openreview.net/forum?id=FNBQOPj18N}\n}\n```\n\n\u003cbr /\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/kaiko-ai/eva/blob/main/docs/images/kaiko-logo.png?raw=true\" width=\"200\"\u003e\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkaiko-ai%2Feva","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkaiko-ai%2Feva","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkaiko-ai%2Feva/lists"}