{"id":15051520,"url":"https://github.com/karthiksoman/biomixqa","last_synced_at":"2026-01-28T01:08:33.740Z","repository":{"id":255628602,"uuid":"852611970","full_name":"karthiksoman/biomixQA","owner":"karthiksoman","description":"Repository for BiomixQA benchmark dataset","archived":false,"fork":false,"pushed_at":"2024-09-05T06:42:59.000Z","size":14,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-12T14:45:04.650Z","etag":null,"topics":["benchmark-datasets","bioinformatics","bioinformatics-data","biomedical-informatics","gpt","large-language-models","llama","retrieval-augmented-generation"],"latest_commit_sha":null,"homepage":"https://huggingface.co/datasets/kg-rag/BiomixQA","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/karthiksoman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-05T05:37:07.000Z","updated_at":"2024-12-23T12:07:49.000Z","dependencies_parsed_at":"2024-09-06T11:53:41.275Z","dependency_job_id":"ff65c31f-85ec-4871-8efd-66e305451e1a","html_url":"https://github.com/karthiksoman/biomixQA","commit_stats":null,"previous_names":["karthiksoman/biomixqa"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karthiksoman%2FbiomixQA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karthiksoman%2FbiomixQA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karthiksoman%2FbiomixQA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karthiksoman%2FbiomixQA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/karthiksoman","download_url":"https://codeload.github.com/karthiksoman/biomixQA/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235222561,"owners_count":18955330,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark-datasets","bioinformatics","bioinformatics-data","biomedical-informatics","gpt","large-language-models","llama","retrieval-augmented-generation"],"created_at":"2024-09-24T21:36:31.117Z","updated_at":"2025-11-17T03:34:26.887Z","avatar_url":"https://github.com/karthiksoman.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# BiomixQA Dataset\n\n## Overview\n\nBiomixQA is a curated biomedical question-answering dataset comprising two distinct components:\n1. Multiple Choice Questions (MCQ)\n2. True/False Questions\n\nThis dataset has been utilized to validate the Knowledge Graph based Retrieval-Augmented Generation (KG-RAG) framework across different Large Language Models (LLMs). The diverse nature of questions in this dataset, spanning multiple choice and true/false formats, along with its coverage of various biomedical concepts, makes it particularly suitable for assessing the performance of KG-RAG framework. \n\nHence, this dataset is designed to support research and development in biomedical natural language processing, knowledge graph reasoning, and question-answering systems.\n\n## Dataset Description\n\n- **Huggingface Repository:** https://huggingface.co/datasets/kg-rag/BiomixQA\n- **Paper:** [Biomedical knowledge graph-optimized prompt generation for large language models](https://arxiv.org/abs/2311.17330)\n- **Point of Contact:** [Karthik Soman](mailto:karthi.soman@gmail.com)\n\n## Dataset Components\n\n### 1. Multiple Choice Questions (MCQ)\n\n- **File**: `mcq_biomix.csv`\n- **Size**: 306 questions\n- **Format**: Each question has five choices with a single correct answer\n\n### 2. True/False Questions\n\n- **File**: `true_false_biomix.csv`\n- **Size**: 311 questions\n- **Format**: Binary (True/False) questions\n\n## Access data using Hugging Face\n\nFollowing snippet shows how to load data in python\n\n(i) MCQ data\n\n```\nfrom datasets import load_dataset\n\nmcq_data = load_dataset(\"kg-rag/BiomixQA\", \"mcq\")\n```\n\n(ii) True/False data\n\n```\nfrom datasets import load_dataset\n\ntf_data = load_dataset(\"kg-rag/BiomixQA\", \"true_false\")\n```\n\n## Potential Uses\n\n1. Evaluating biomedical question-answering systems\n2. Testing natural language processing models in the biomedical domain\n3. Assessing retrieval capabilities of various RAG (Retrieval-Augmented Generation) frameworks\n4. Supporting research in biomedical ontologies and knowledge graphs\n\n## Performance Analysis\n\nWe conducted a comprehensive analysis of the performance of three Large Language Models (LLMs) - Llama-2-13b, GPT-3.5-Turbo (0613), and GPT-4 - on the BiomixQA dataset. We compared their performance using both a standard prompt-based approach (zero-shot) and our novel Knowledge Graph based Retrieval-Augmented Generation (KG-RAG) framework.\n\n### Performance Summary\n\nTable 1: Performance (accuracy) of LLMs on BiomixQA datasets using prompt-based (zero-shot) and KG-RAG approaches (For more details, refer [this](https://arxiv.org/abs/2311.17330) paper)\n\n| Model | True/False Dataset |    | MCQ Dataset |    |\n|-------|-------------------:|---:|------------:|---:|\n|       | Prompt-based | KG-RAG | Prompt-based | KG-RAG |\n| Llama-2-13b | 0.89 ± 0.02 | 0.94 ± 0.01 | 0.31 ± 0.03 | 0.53 ± 0.03 |\n| GPT-3.5-Turbo (0613) | 0.87 ± 0.02 | 0.95 ± 0.01 | 0.63 ± 0.03 | 0.79 ± 0.02 |\n| GPT-4 | 0.90 ± 0.02 | 0.95 ± 0.01 | 0.68 ± 0.03 | 0.74 ± 0.03 |\n\n### Key Observations\n\n1. **Consistent Performance Enhancement**: We observed a consistent performance enhancement for all LLM models when using the KG-RAG framework on both True/False and MCQ datasets.\n\n2. **Significant Improvement for Llama-2**: The KG-RAG framework significantly elevated the performance of Llama-2-13b, particularly on the more challenging MCQ dataset. We observed an impressive 71% increase in accuracy, from 0.31 ± 0.03 to 0.53 ± 0.03.\n\n3. **GPT-4 vs GPT-3.5-Turbo on MCQ**: Intriguingly, we observed a small but statistically significant drop in the performance of the GPT-4 model (0.74 ± 0.03) compared to the GPT-3.5-Turbo model (0.79 ± 0.02) on the MCQ dataset when using the KG-RAG framework. This difference was not observed in the prompt-based approach.\n   - Statistical significance: T-test, p-value \u003c 0.0001, t-statistic = -47.7, N = 1000\n\n4. **True/False Dataset Performance**: All models showed high performance on the True/False dataset, with the KG-RAG approach yielding slightly better results across all models.\n\n\n## Source Data\n\n1. SPOKE: A large scale biomedical knowledge graph that consists of ~40 million biomedical concepts and ~140 million biologically meaningful relationships (Morris et al.\n2023).\n2. DisGeNET: Consolidates data about genes and genetic variants linked to human diseases from curated repositories, the GWAS catalog, animal models, and scientific literature (Piñero et\nal. 2016).\n3. MONDO: Provides information about the ontological classification of Disease entities in the Open Biomedical Ontologies (OBO) format (Vasilevsky et al. 2022).\n4. SemMedDB: Contains semantic predications extracted from PubMed citations (Kilicoglu et al. 2012).\n5. Monarch Initiative: A platform for disease-gene association data (Mungall et al. 2017).\n6. ROBOKOP: A knowledge graph-based system for biomedical data integration and analysis (Bizon et al. 2019).\n\n## Citation\n\nIf you use this dataset in your research, please cite the following paper:\n```\n@article{soman2023biomedical,\n  title={Biomedical knowledge graph-enhanced prompt generation for large language models},\n  author={Soman, Karthik and Rose, Peter W and Morris, John H and Akbas, Rabia E and Smith, Brett and Peetoom, Braian and Villouta-Reyes, Catalina and Cerono, Gabriel and Shi, Yongmei and Rizk-Jackson, Angela and others},\n  journal={arXiv preprint arXiv:2311.17330},\n  year={2023}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkarthiksoman%2Fbiomixqa","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkarthiksoman%2Fbiomixqa","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkarthiksoman%2Fbiomixqa/lists"}