{"id":37608636,"url":"https://github.com/krishnanlab/obnbench","last_synced_at":"2026-01-16T10:15:05.676Z","repository":{"id":65844516,"uuid":"534172446","full_name":"krishnanlab/obnbench","owner":"krishnanlab","description":"Benchmarking repository for OBNB","archived":false,"fork":false,"pushed_at":"2023-12-07T20:09:22.000Z","size":285581,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-05-15T09:36:46.578Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://github.com/krishnanlab/obnb","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/krishnanlab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-09-08T11:04:35.000Z","updated_at":"2023-06-18T20:01:15.000Z","dependencies_parsed_at":"2023-12-07T19:45:53.367Z","dependency_job_id":null,"html_url":"https://github.com/krishnanlab/obnbench","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/krishnanlab/obnbench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krishnanlab%2Fobnbench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krishnanlab%2Fobnbench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krishnanlab%2Fobnbench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krishnanlab%2Fobnbench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/krishnanlab","download_url":"https://codeload.github.com/krishnanlab/obnbench/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krishnanlab%2Fobnbench/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28478049,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T06:30:42.265Z","status":"ssl_error","status_checked_at":"2026-01-16T06:30:16.248Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-16T10:15:04.999Z","updated_at":"2026-01-16T10:15:05.668Z","avatar_url":"https://github.com/krishnanlab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Benchmarking repository for the Open Biomedical Network Benchmark\n\nThis is a benchmarking repository accompanying the [`obnb`](https://github.com/krishnanlab/obnb) Python package.\n\n## Set up environment\n\n```bash\nconda create -n obnb python=3.8 -y \u0026\u0026 conda activate obnb\n\n# Install PyTorch and PyG with CUDA 11.7\nconda install pytorch=2.0.1 torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia\nconda install pyg=2.3.0 -c pyg -y\n\npip install obnb[ext]==0.1.0  # install obnb with extension modules (PecanPy, GraPE, ...)\npip install -r requirements_extra.txt  # extra dependencies for benchmarking\n\nconda clean --all -y  # clean up\n```\n\nThe extra dependencies are, e.g.,\n\n- [`Hydra`](https://github.com/facebookresearch/hydra) for managing experiments.\n- [`Lightning`](https://lightning.ai/docs/pytorch/latest/) for organizing model training framework.\n- [`WandB`](https://docs.wandb.ai/) for logging metrics.\n\n**Note**: if you do not need to run the benchmarking experiments and only want to play around\nwith our benchmarking results with one of the [notebooks](notebook), you can skip the installation\nfor PyTorch and PyG.\n\n```bash\npip install obnb[ext]==0.1.0\n```\n\n## Set up data (optional)\n\nRun `get_data.py` to download and set up data for all the experiments.\nData will be saved under the `datasets/` directory by default, and will take up approximately 6 GB of space.\n\n```bash\npython get_data.py\n```\n\nThis step is completely optional and directly runing the training script will work fine.\nBut runing `get_data.py` once before training prevents multiple parallel jobs doing the same data preprocessing\nwork if the processed data is not available yet.\n\n## Run experiments\n\nAfter setting up the data, one can run a single experiment by specifying the choices of network, label, and model:\n\n```bash\npython main.py dataset.network=BioGRID dataset.label=DisGeNET model=GCN\n```\n\nCheck out the [`conf/model/`](conf/model) directory for all available model presets.\nThe main model presets are:\n\n- `GCN`\n- `GAT`\n- `GCN+BoT`\n- `GAT+BoT`\n- `LogReg+Adj`\n- `LogReg+Node2vec`\n- `LogReg+Walklets`\n\n### Run batch of parallel jobs\n\n```bash\ncd run\n\n# GNN node feature ablation (example of runing GCN with node2vec features on BioGRID)\nsh run_abl_gnn_feature.sh GCN BioGRID Node2vec\n\n# C\u0026S ablation (example of runing GCN with C\u0026S post processing on BioGRID)\nsh run_abl_cs.sh GCN BioGRID\n\n# GNN label reuse ablation (example of runing GCN with label reuse on BioGRID)\nsh run_abl_gnn_label.sh GCN BioGRID\n\n# GNN label reuse with C\u0026S ablation (example of runing GCN with label reuse with C\u0026S on BioGRID)\nsh run_abl_gnn_cs_label.sh GCN BioGRID\n\n# GNN with bag of tricks, i.e., node2vec node feature + label reuse + C\u0026S\nsh run_gnn_bot.sh GCN BioGRID\n```\n\nTo run all experiments presented in the paper (may take several days):\n\n```bash\nsh run_all.sh\n```\n\n### Tuning with W\u0026B\n\nFirst create a sweep agent, e.g., for BioGRID-DisGeNET-GCN:\n\n```bash\nwandb sweep conf/tune/BioGRID-DisGeNET-GCN.yaml\n```\n\nThen, follow the instruction from the command above to spawn sweep agents to automatically\ntune the model configuration on a particular dataset.\n\n## Results anallysis\n\nTo run the [notebooks](notebook), first download our benchmarking results\n(or you can rerun all the benchmarking experiments yourself using our run scripts described above).\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.8048305.svg)](https://doi.org/10.5281/zenodo.8048305)\n\n```bash\nwget -O results/main.csv.gz https://zenodo.org/record/8048305/files/main.csv.gz\n```\n\n## Data stats (`obnbdata-0.1.0`) [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.8045270.svg)](https://doi.org/10.5281/zenodo.8045270)\n\n### Networks\n\n| Network | Weighted | Num. nodes | Num. edges | Density | Category |\n| :------ | :------: | ---------: | ---------: | ------: | -------: |\n| [HumanBaseTopGlobal](https://humanbase.net/) | :white_check_mark: | 25,689 | 77,807,094 | 0.117908 | Large \u0026 Dense |\n| [HuMAP](http://humap2.proteincomplexes.org/) | :white_check_mark: | 15,433 | 35,052,604 | 0.147180 | Large \u0026 Dense |\n| [STRING](https://string-db.org/) | :white_check_mark: | 18,480 | 11,019,492 | 0.032269 | Large |\n| [ConsensusPathDB](http://cpdb.molgen.mpg.de/) | :white_check_mark: | 17,735 | 10,611,416 | 0.033739 | Large |\n| [FunCoup](https://funcoup.org/) | :white_check_mark: | 17,892 | 10,037,478 | 0.031357 | Large |\n| [PCNet](https://www.ndexbio.org/viewer/networks/f93f402c-86d4-11e7-a10d-0ac135e8bacf) | :x: | 18,544 | 5,365,116 | 0.015603 | Large |\n| [BioGRID](https://thebiogrid.org/) | :x: | 19,765 | 1,554,790 | 0.003980 | Medium |\n| [HumanNet](https://staging2.inetbio.org/humannetv3/) | :white_check_mark: | 18,591 | 2,250,780 | 0.006513 | Medium |\n| [HIPPIE](http://cbdm-01.zdv.uni-mainz.de/~mschaefer/hippie/) | :white_check_mark: | 19,338 | 1,542,044 | 0.004124 | Medium |\n| [ComPPIHumanInt](https://comppi.linkgroup.hu/) | :white_check_mark: | 17,015 | 699,620 | 0.002417 | Medium |\n| [OmniPath](https://omnipathdb.org/) | :x: | 16,325 | 289,134 | 0.001085 | Small |\n| [ProteomeHD](https://www.ndexbio.org/viewer/networks/4cb4b0f3-83da-11e9-848d-0ac135e8bacf) | :x: | 2,471 | 125,172 | 0.020509 | Small |\n| [HuRI](http://www.interactome-atlas.org/) | :x: | 8,100 | 103,188 | 0.001573 | Small |\n| [BioPlex](https://bioplex.hms.harvard.edu/explorer/home) | :x: | 8,108 | 71,004 | 0.001080 | Small |\n| [SIGNOR](https://signor.uniroma2.it/) | :x: | 5,291 | 28,676 | 0.001025 | Small |\n\n### Labels\n\n#### [DISEASES](https://diseases.jensenlab.org/About)\n\n| Network | Num. tasks | Num. pos. avg. | Num. pos. std. | Num. pos. med. |\n| :------ | ---------: | -------------: | -------------: | -------------: |\n| BioGRID | 145 | 178.1 | 137.4 | 127.0 |\n| BioPlex | 72 | 123.8 | 64.4 | 101.5 |\n| ComPPIHumanInt | 145 | 174.6 | 134.5 | 125.0 |\n| ConsensusPathDB | 144 | 177.4 | 137.5 | 126.0 |\n| FunCoup | 145 | 177.1 | 135.1 | 127.0 |\n| HIPPIE | 143 | 178.1 | 137.6 | 127.0 |\n| HuMAP | 123 | 168.0 | 119.2 | 120.0 |\n| HuRI | 50 | 130.3 | 56.7 | 112.5 |\n| HumanBaseTopGlobal | 149 | 178.5 | 137.7 | 129.0 |\n| HumanNet | 142 | 179.0 | 136.9 | 127.0 |\n| OmniPath | 135 | 180.2 | 131.1 | 131.0 |\n| PCNet | 143 | 171.8 | 130.6 | 122.0 |\n| ProteomeHD | 15 | 76.9 | 22.4 | 70.0 |\n| SIGNOR | 89 | 144.6 | 89.4 | 117.0 |\n| STRING | 146 | 175.4 | 135.6 | 126.0 |\n\n#### [DisGeNET](https://www.disgenet.org/)\n\n| Network | Num. tasks | Num. pos. avg. | Num. pos. std. | Num. pos. med. |\n| :------ | ---------: | -------------: | -------------: | -------------: |\n| BioGRID | 305 | 208.3 | 143.1 | 159.0 |\n| BioPlex | 189 | 138.6 | 71.4 | 111.0 |\n| ComPPIHumanInt | 301 | 204.1 | 138.7 | 159.0 |\n| ConsensusPathDB | 298 | 207.4 | 140.8 | 161.5 |\n| FunCoup | 299 | 204.7 | 139.4 | 158.0 |\n| HIPPIE | 306 | 208.1 | 142.9 | 159.5 |\n| HuMAP | 279 | 194.3 | 126.7 | 155.0 |\n| HuRI | 152 | 122.9 | 54.7 | 108.0 |\n| HumanBaseTopGlobal | 287 | 219.7 | 145.7 | 173.0 |\n| HumanNet | 302 | 204.2 | 140.3 | 158.5 |\n| OmniPath | 298 | 199.6 | 136.0 | 153.5 |\n| PCNet | 292 | 202.1 | 135.5 | 159.0 |\n| ProteomeHD | 56 | 78.0 | 24.8 | 71.0 |\n| SIGNOR | 219 | 147.3 | 81.9 | 124.0 |\n| STRING | 296 | 208.0 | 140.6 | 162.0 |\n\n#### [GOBP](http://geneontology.org/)\n\n| Network | Num. tasks | Num. pos. avg. | Num. pos. std. | Num. pos. med. |\n| :------ | ---------: | -------------: | -------------: | -------------: |\n| BioGRID | 114 | 89.5 | 37.1 | 76.0 |\n| BioPlex | 38 | 77.6 | 22.6 | 76.0 |\n| ComPPIHumanInt | 104 | 91.8 | 37.0 | 77.5 |\n| ConsensusPathDB | 112 | 90.1 | 37.0 | 76.5 |\n| FunCoup | 114 | 87.8 | 36.7 | 74.0 |\n| HIPPIE | 111 | 89.2 | 37.1 | 76.0 |\n| HuMAP | 96 | 84.6 | 32.3 | 74.0 |\n| HuRI | 27 | 69.9 | 16.0 | 65.0 |\n| HumanBaseTopGlobal | 115 | 89.2 | 37.3 | 76.0 |\n| HumanNet | 117 | 88.6 | 36.9 | 75.0 |\n| OmniPath | 106 | 88.7 | 36.2 | 74.0 |\n| PCNet | 105 | 89.0 | 36.0 | 77.0 |\n| ProteomeHD | 5 | 80.4 | 22.6 | 70.0 |\n| SIGNOR | 41 | 81.3 | 22.7 | 78.0 |\n| STRING | 116 | 88.9 | 36.6 | 75.0 |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkrishnanlab%2Fobnbench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkrishnanlab%2Fobnbench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkrishnanlab%2Fobnbench/lists"}