{"id":28509467,"url":"https://github.com/snap-stanford/conformalized-gnn","last_synced_at":"2025-07-03T01:31:21.837Z","repository":{"id":196741927,"uuid":"686734041","full_name":"snap-stanford/conformalized-gnn","owner":"snap-stanford","description":"Uncertainty Quantification over Graph with Conformalized Graph Neural Networks (NeurIPS 2023)","archived":false,"fork":false,"pushed_at":"2023-09-27T00:58:55.000Z","size":1560,"stargazers_count":81,"open_issues_count":0,"forks_count":6,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-06-08T22:09:38.618Z","etag":null,"topics":["calibration","conformal-prediction","gnn","graph","graph-neural-networks","uncertainty-quantification"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/snap-stanford.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-09-03T19:07:50.000Z","updated_at":"2025-05-20T02:57:06.000Z","dependencies_parsed_at":"2023-09-27T08:11:36.540Z","dependency_job_id":null,"html_url":"https://github.com/snap-stanford/conformalized-gnn","commit_stats":null,"previous_names":["snap-stanford/conformalized-gnn"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/snap-stanford/conformalized-gnn","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snap-stanford%2Fconformalized-gnn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snap-stanford%2Fconformalized-gnn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snap-stanford%2Fconformalized-gnn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snap-stanford%2Fconformalized-gnn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/snap-stanford","download_url":"https://codeload.github.com/snap-stanford/conformalized-gnn/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snap-stanford%2Fconformalized-gnn/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263243630,"owners_count":23436340,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["calibration","conformal-prediction","gnn","graph","graph-neural-networks","uncertainty-quantification"],"created_at":"2025-06-08T22:08:42.973Z","updated_at":"2025-07-03T01:31:21.820Z","avatar_url":"https://github.com/snap-stanford.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Conformalized Graph Neural Networks\n\nThis repository hosts the code base for the paper\n\n**Uncertainty Quantification over Graph with Conformalized Graph Neural Networks**\\\nKexin Huang, Ying Jin, Emmanuel Candès, Jure Leskovec\\\nNeurIPS 2023, Spotlight \\\n[Link to Paper](https://arxiv.org/abs/2305.14535)\n\n\nIf you find this work useful, please consider cite:\n\n```\n@article{huang2023conformalized_gnn,\n  title={Uncertainty quantification over graph with conformalized graph neural networks},\n  author={Huang, Kexin and Jin, Ying and Candes, Emmanuel and Leskovec, Jure},\n  journal={NeurIPS},\n  year={2023}\n}\n```\n\n\n### Overview\n\nGraph Neural Networks (GNNs) are powerful machine learning prediction models on graph-structured data. However, GNNs lack rigorous uncertainty estimates, limiting their reliable deployment in settings where the cost of errors is significant. We propose conformalized GNN (CF-GNN), extending conformal prediction (CP) to graph-based models for guaranteed uncertainty estimates. Given an entity in the graph, CF-GNN produces a prediction set/interval that provably contains the true label with pre-defined coverage probability (e.g.~90%). We establish a permutation invariance condition that enables the validity of CP on graph data and provide an exact characterization of the test-time coverage. Besides valid coverage, it is crucial to reduce the prediction set size/interval length for practical use. We observe a key connection between non-conformity scores and network structures, which motivates us to develop a topology-aware output correction model that learns to update the prediction and produces more efficient prediction sets/intervals. Extensive experiments show that CF-GNN achieves any pre-defined target marginal coverage while significantly reducing the prediction set/interval size by up to 74% over the baselines. It also empirically achieves satisfactory conditional coverage over various raw and network features. \n\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"./fig/graph_conformal.png\" alt=\"logo\" width=\"800px\" /\u003e\u003c/p\u003e\n\n\n## Installation\n\nInstall Torch and PyG following [here](https://pytorch-geometric.readthedocs.io/en/latest/install/installation.html) and then do\n\n```bash\npip install -r requirements.txt\n```\n\n## Run\n\n\n### Datasets\n\n**Classification datasets are supported in PyG. For regression datasets, download from [this link](https://drive.google.com/file/d/1qHqR4JYc9fMVppOj1K9x89OPh4AtjYQ1/view?usp=sharing) and put the folder under this repository.**\n\nHere are the list of datasets for classification tasks: `Cora_ML_CF`, `DBLP_CF`, `CiteSeer_CF`, `PubMed_CF`, `Amazon-Computers`, `Amazon-Photo`, `Coauthor-CS`, `Coauthor-Physics`\n\nHere are the list of datasets for regression tasks: `Anaheim`, `ChicagoSketch`, `county_education_2012`, `county_election_2016`, `county_income_2012`, `county_unemployment_2012`, `twitch_PTBR`\n\n\n### Pre-trained GNN base models\n\n**To reproduce the paper result, please use the fixed pre-trained GNN base models from [this link](https://drive.google.com/file/d/1-z17AWIkDJ7LoI9OG_qQfbCZ9BqDxGbx/view?usp=sharing).** This makes sure the gain is from the conformal adjustment instead of the noise in the base model training. After downloading and unzipping this link, please put the `model` folder under this repository.\n\nIf you wish to re-train GNN base model, simply remove the base model folder in this repository and the model will train again.\n\n### Key Arguments\n\n- `--model`: base GNN model, select from 'GAT', 'GCN', 'GraphSAGE', 'SGC'\n- `--dataset`: dataset name, select from 'Cora_ML_CF', 'CiteSeer_CF', 'DBLP_CF', 'PubMed_CF', 'Amazon-Computers', 'Amazon-Photo', 'Coauthor-CS', 'Coauthor-Physics', 'Anaheim', 'ChicagoSketch', 'county_education_2012', 'county_election_2016', 'county_income_2012', 'county_unemployment_2012', 'twitch_PTBR'\n- `--device`: cuda device\n- `--alpha`: pre-specified miscoverage rate, default is 0.1\n- `--optimal`: use optimal hyperparameter set\n- `--hyperopt`: conduct a sweep of hyperparameter optimization\n- `--num_runs`: number of runs, default is 10\n- `--wandb`: turn on weight and bias tracking\n- `--verbose`: verbose mode, print out log (incl. training loss)\n- `--optimize_conformal_score`: for classification only, options: aps and raps\n- `--not_save_res`: default is saving the result to the pred folder, by adding this flag, you choose to NOT save the result\n- `--epochs`: number of epochs for conformal correction\n\n\n### Training CF-GNN\n\n```bash\npython train.py --model GCN \\\n                --dataset Cora_ML_CF \\\n                --device cuda \\\n                --alpha 0.1\\\n                --optimal \\\n                --num_runs 1\n```\n\n### Training baseline models\n\nFor classification datasets `Cora_ML_CF`, `DBLP_CF`, `CiteSeer_CF`, `PubMed_CF`, `Amazon-Computers`, `Amazon-Photo`, `Coauthor-CS`, `Coauthor-Physics`:\n\nAll baselines are calibration methods, choose `calibrator` from `TS` `VS` `ETS` `CaGCN` `GATS`.\n\n```bash\npython train.py --model GCN \\\n                --dataset Cora_ML_CF \\\n                --device cuda \\\n                --alpha 0.05 \\\n                --conf_correct_model Calibrate \\\n                --calibrator TS\n```\n\nFor regression datasets `Anaheim`, `ChicagoSketch`, `county_education_2012`, `county_election_2016`, `county_income_2012`, `county_unemployment_2012`, `twitch_PTBR`:\n\nTo use `mcDropout`:\n\n```bash\npython train.py --model GCN \\\n                --dataset Anaheim \\\n                --device cuda \\\n                --alpha 0.05 \\\n                --conf_correct_model mcdropout_std\n```\n\nTo use `BayesianNN`:\n\n```bash\npython train.py --model GCN \\\n                --dataset Anaheim \\\n                --device cuda \\\n                --alpha 0.05 \\\n                --bnn\n```\n\nTo use `QuantileRegression`:\n\n```bash\npython train.py --model GCN \\\n                --dataset Anaheim \\\n                --device cuda \\\n                --alpha 0.05 \\\n                --conf_correct_model QR\n```\n\n### Launching a hyper-parameter search for CF-GNN\n\n```bash\npython train.py --model GCN \\\n                --dataset Cora_ML_CF \\\n                --device cuda \\\n                --alpha 0.1 \\          \n                --hyperopt\n```\n\n### Adjusting pre-specified coverage 1-alpha\n\nThis is the script for Fig 5(1).\n\n```bash\nfor data in Anaheim Cora_ML_CF\ndo\nfor alpha in 0.05 0.1 0.15 0.2 0.25 0.3 \ndo\npython train.py --model GCN --dataset $data --device cuda --optimal --alpha $alpha\ndone\ndone\n```\n\n### Adjusting holdout calibration set fraction\n\nThis is the script for Fig 5(2).\n\n```bash\nfor data in Anaheim Cora_ML_CF\ndo\nfor calib_frac in 0.1 0.3 0.7 0.9\ndo\npython train.py --model GCN --dataset $data --device cuda --optimal --calib_fraction $calib_frac\ndone\ndone\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnap-stanford%2Fconformalized-gnn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsnap-stanford%2Fconformalized-gnn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnap-stanford%2Fconformalized-gnn/lists"}