{"id":21004995,"url":"https://github.com/kempnerinstitute/llm_uncertainty","last_synced_at":"2025-09-04T00:36:59.371Z","repository":{"id":220871377,"uuid":"751619432","full_name":"KempnerInstitute/llm_uncertainty","owner":"KempnerInstitute","description":"Code for the paper \"Distinguishing the Knowable from the Unknowable with Language Models\"","archived":false,"fork":false,"pushed_at":"2024-04-15T19:28:58.000Z","size":25886,"stargazers_count":7,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-26T02:42:40.963Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/KempnerInstitute.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-02T00:51:16.000Z","updated_at":"2025-03-21T09:21:41.000Z","dependencies_parsed_at":"2024-11-19T09:02:20.084Z","dependency_job_id":null,"html_url":"https://github.com/KempnerInstitute/llm_uncertainty","commit_stats":null,"previous_names":["gahdritz/llm_uncertainty","kempnerinstitute/llm_uncertainty"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KempnerInstitute%2Fllm_uncertainty","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KempnerInstitute%2Fllm_uncertainty/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KempnerInstitute%2Fllm_uncertainty/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KempnerInstitute%2Fllm_uncertainty/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/KempnerInstitute","download_url":"https://codeload.github.com/KempnerInstitute/llm_uncertainty/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248532449,"owners_count":21119913,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-19T08:38:48.039Z","updated_at":"2025-04-12T07:25:54.742Z","avatar_url":"https://github.com/KempnerInstitute.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Distinguishing the Knowable from the Unknowable with Language Models\n\nCode for the preprint \"Distinguishing the Knowable from the Unknowable with Language Models,\" submitted to ICML 2024.\n\n**Abstract**: We study the feasibility of identifying *epistemic* uncertainty (reflecting a lack of knowledge), as opposed to *aleatoric* uncertainty (reflecting entropy in the underlying distribution), in the outputs of large language models (LLMs) over free-form text. In the absence of ground-truth probabilities, we explore a setting where, in order to (approximately) disentangle a given LLM's uncertainty, a significantly larger model stands in as a proxy for the ground truth. We show that small linear probes trained on the embeddings of frozen, pretrained models accurately predict when larger models will be confident at the token level and that probes trained on one text domain generalize to others. Going further, we propose a fully unsupervised method that achieves non-trivial accuracy on the same task. Taken together, we interpret these results as evidence that LLMs naturally contain internal representations of different types of uncertainty that could potentially be leveraged to devise more informative indicators of model confidence in diverse practical settings.\n\nCorrespondence to: Gustaf Ahdritz(gahdritz@g.harvard.edu), Tian Qin(tqin@g.harvard.edu), Ben Edelman(bedelman@g.harvard.edu).\n\n## Instructions\n\n### Obtaining model weights\n\nOur scripts support (with some gaps) three models: LLaMA, Pythia, and Llama 2.\n\n* LLaMA: Follow the instructions in the original `lit-llama` repository [here](https://github.com/Lightning-AI/lit-llama/blob/main/howto/download_weights.md).\n* Pythia: These should download automatically from Huggingface upon first use\n* Llama 2: Follow the instructions in the `lit-gpt` repository [here](https://github.com/Lightning-AI/lit-gpt/blob/main/tutorials/download_llama_2.md).\n\n\n### Supervised experiments\n\nFirst, precompute model embeddings with `precompute_logits.py`. Embeddings from different layers can be selected with the `return_after_layer_n` flag. We include our Wikipedia dataset (new Wikipedia articles after the LLaMA training cutoff) under `data/`.\n\nNext, generate a dataset filter for a specific entropy band using `create_dataset_filter.py` (or `create_dataset_filter_no_gap.py` for a gapless filter).\n\nFinally, train probes using `train_head.py.`\n\n### Unsupervised experiments\n\nICLT is implemented in `repetition.py` and `generate_from_repetition.py`.\n\nThe precomputed model embeddings and the dataset filters are created same as above (using `precompute_logits.py` and `create_dataset_filter.py`). You can use `repetition.py` to run the repetiton experiment with sample script as below: \n\n```\npython repetition.py   /path/to/data/set/filter/folder/ \\\n    0 # indicate filter shard number  \\\n    /path/to/prompt.json \\\n    llama # specify model \\\n    7B # specify small model size \\\n    --experiment_name default # specify the name of experiment\n```\nThe repetition results will be saved under ```/path/to/data/set/filter/folder/experiment_name/```. To process the results for classificaiton task, run `generate_from_repetition.py`:\n\n```\npython generate_from_repetition.py  \\\n    /path/to/data/set/filter/folder/experiment_name \\\n    llama # specify model \\\n    7B # specify small model size \n```\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkempnerinstitute%2Fllm_uncertainty","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkempnerinstitute%2Fllm_uncertainty","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkempnerinstitute%2Fllm_uncertainty/lists"}