{"id":13775576,"url":"https://github.com/Bradley-Butcher/Conformers","last_synced_at":"2025-05-11T08:32:23.578Z","repository":{"id":179566934,"uuid":"663711358","full_name":"Bradley-Butcher/Conformers","owner":"Bradley-Butcher","description":"Unofficial implementation of Conformal Language Modeling by Quach et al","archived":false,"fork":false,"pushed_at":"2023-07-15T14:00:04.000Z","size":112,"stargazers_count":28,"open_issues_count":2,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-19T18:09:51.996Z","etag":null,"topics":["alignment","conformal-prediction","language-model","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Bradley-Butcher.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-07-07T23:40:07.000Z","updated_at":"2025-02-27T04:05:53.000Z","dependencies_parsed_at":"2023-07-22T14:30:56.764Z","dependency_job_id":null,"html_url":"https://github.com/Bradley-Butcher/Conformers","commit_stats":null,"previous_names":["bradley-butcher/conformers"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bradley-Butcher%2FConformers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bradley-Butcher%2FConformers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bradley-Butcher%2FConformers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bradley-Butcher%2FConformers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Bradley-Butcher","download_url":"https://codeload.github.com/Bradley-Butcher/Conformers/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253539961,"owners_count":21924517,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alignment","conformal-prediction","language-model","transformers"],"created_at":"2024-08-03T17:01:41.686Z","updated_at":"2025-05-11T08:32:23.308Z","avatar_url":"https://github.com/Bradley-Butcher.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Conformers\n\nThis is an unofficial implementation of the paper [Conformal Language Modelling](https://arxiv.org/abs/2306.10193).\nI found the paper interesting and wanted to play around with it.\nStill in a very early state - the only rigorous, statistical guarantee currently is that there are bugs and misunderstandings.\nPlease excuse the state of the current code - I'll clean it up I promise!\n\n## Status\n\n- [ ] Initial implementation done\n- [ ] Tests\n- [ ] Pareto Testing procedure (Rather than the current grid search + Bonferroni combo)\n- [ ] Component selection\n- [ ] PyPI package\n- [ ] Experiments\n\n## Changes from the paper\n\n- Sampling is no longer greedy - authors claim to use greedy sampling (default transformer sampling), but this will result in the same output for all samples.\n- The selection of the admission function, admission function threshold, and epsilon appears to be very sensitive. In paper the authors select task-dependent admission function thresholds (which I assume they derived experimentally) and try different values of epsilon. In this implementation I will attempt to introduce more generic admission functions.\n\n\n## Installation\n\nNo PyPI package is available yet. To install, clone the repository and run\n\n```bash\npip install poetry\npoetry install\n```\n\n## Usage\n\nThe Python API is not yet set in stone, but the aim is to make it easy to experiment with different admission, group confidence, and rejection functions.\nPotentially some quite interesting combinations with the recent CFG language model paper.\nBelow is an example with GPT2.\n\n```python\nfrom conformer import Calibrator, Sampler, Components\nimport torch\nfrom random import randint\n\n\nx =[\n    \"What is the capital of France?\",\n    \"Which prime-minster of the UK was the biggest nob?\",\n] \n\nfrom transformers import GPT2LMHeadModel, GPT2Tokenizer\n\nmodel_name = \"gpt2\"\nmodel = GPT2LMHeadModel.from_pretrained(model_name).cuda()\ntokenizer = GPT2Tokenizer.from_pretrained(model_name)\ntokenizer.pad_token_id = tokenizer.eos_token_id\n\ncalibrator = Calibrator(\n    model=model,\n    tokenizer=tokenizer,\n    calibration_prompts=x,\n)\n\ncalibrator.set_admission_function(Components.admission.debug)\ncalibrator.set_group_confidence_function(Components.group_confidence.debug, torch.tensor([0.1, 0.5, 1]))\ncalibrator.add_rejection_function(Components.rejection.debug, torch.tensor([0.1, 0.5, 1]))\ncalibrator.set_FWER(Components.FWER.debug)\n\nlambdaz = calibrator.search()\n\nsampler = Sampler.from_calibrator(calibrator)\n\nsampler.sample_with_rejection(\"What is the capital of France?\")\n```\n\nThis uses some of the built-in admission/gf/fwer/rejection functions. Can also just use your own function, e.g.:\n\n```python\ncalibrator.set_group_confidence_function(lambda x: x \u003e 0.5, torch.tensor([0.1, 0.5, 1]))\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FBradley-Butcher%2FConformers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FBradley-Butcher%2FConformers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FBradley-Butcher%2FConformers/lists"}