{"id":17187498,"url":"https://github.com/davidmchan/clair","last_synced_at":"2025-04-13T19:07:59.365Z","repository":{"id":199966380,"uuid":"659555351","full_name":"DavidMChan/clair","owner":"DavidMChan","description":"CLAIR: A (surprisingly) simple semantic text metric with large language models.","archived":false,"fork":false,"pushed_at":"2024-01-28T20:11:09.000Z","size":5442,"stargazers_count":18,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-27T09:46:09.373Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DavidMChan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-06-28T04:59:16.000Z","updated_at":"2025-02-11T13:29:51.000Z","dependencies_parsed_at":"2023-12-06T08:29:50.518Z","dependency_job_id":"d85d4df5-41fa-4ab9-a645-79b7662a569a","html_url":"https://github.com/DavidMChan/clair","commit_stats":null,"previous_names":["davidmchan/clair"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DavidMChan%2Fclair","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DavidMChan%2Fclair/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DavidMChan%2Fclair/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DavidMChan%2Fclair/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DavidMChan","download_url":"https://codeload.github.com/DavidMChan/clair/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248766733,"owners_count":21158301,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-15T01:06:32.450Z","updated_at":"2025-04-13T19:07:59.323Z","avatar_url":"https://github.com/DavidMChan.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CLAIR: A (Surprisingly) simple semantic text metric leveraging large language models.\n\n### [Project](https://davidmchan.github.io/clair/) | [Paper](https://davidmchan.github.io/clair/static/pdfs/_EMNLP_2023__CLAIR__Language_Models_for_Caption_Evaluation.pdf)\n\nOfficial implementation of the paper: \"CLAIR: Evaluating Image Captions with Large Language Models\".\n\u003cbr\u003e\n\nThe evaluation of machine-generated image captions poses an interesting yet persistent challenge. Effective evaluation measures must consider numerous dimensions of similarity, including semantic relevance, visual structure, object interactions, caption diversity, and specificity. Existing highly-engineered measures attempt to capture specific aspects, but fall short in providing a holistic score that aligns closely with human judgments. Here, we propose CLAIR, a novel method that leverages the zero-shot language modeling capabilities of large language models (LLMs) to evaluate candidate captions. In our evaluations, CLAIR demonstrates a stronger correlation with human judgments of caption quality compared to existing measures. Notably, on Flickr8K-Expert, CLAIR achieves relative correlation improvements over SPICE of 39.6% and over image-augmented methods such as RefCLIP-S of 18.3%. Moreover, CLAIR provides noisily interpretable results by allowing the language model to identify the underlying reasoning behind its assigned score.\n\n## Getting started\n\n### Setup\n\n```bash\npip install git+https://github.com/DavidMChan/clair.git\n```\n\n### Usage\n\n```python\nimport os\nfrom clair import clair\n\nos.environ['OPENAI_API_KEY'] = 'YOUR_API_KEY'\n\ncandidates = ['A photo of a cat sitting on a couch', 'A cat sitting on a couch']\nreferences = ['A cat sitting on a couch', 'A cat is sitting on a couch', 'A cat is sitting on a sofa']\n\nclair_score = clair(candidates, references, model='chat-gpt')\n\nprint(clair_score)\n```\n\n### Models\n\nCLAIR relies on the [Grazier](https://github.com/DavidMChan/grazier) library to inteface with large language models. The following models are currently supported (see the linked repository for more information on configuring each model):\n\nFrom OpenAI:\n\n-   GPT-4 (Base, 32K) (Chat and Completion Engines)\n-   GPT-3.5 (ChatGPT) (Chat and Completion Engines)\n-   GPT-3 (Davinci (v2,v3), Ada, Babbage, Curie) (Completion Engine)\n\nFrom Anthropic:\n\n-   Claude 2 (Base) (Chat and Completion Engines)\n-   Claude (Base, 100K) (Chat and Completion Engines)\n-   Claude Instant (Base, 100K) (Chat and Completion Engines)\n\nFrom Google/GCP:\n\n-   PaLM (Chat and Completion Engines)\n\nFrom Huggingface\n\n-   GPT-2 (Base, Medium, Large, XL) (Completion Engine)\n-   GPT-Neo (125M, 1.3B, 2.7B) (Completion Engine)\n-   GPT-J (6B) (Completion Engine)\n-   Falcon (7B, 40B, rw-1B, rw-7B) (Completion Engine)\n-   Dolly (v1 - 6B, v2 - 3B, 7B, 12B) (Chat and Completion Engines)\n-   MPT (Instruct - 7B, 30B) (Chat and Completion Engines)\n\nFrom Facebook (via Huggingface)\n\n-   Llama (7B, 13B, 30B, 65B) (Completion Engine)\n-   Llama 2 (7B, 13B, 70B) (Completion Engine)\n-   OPT (125M, 350M, 1.3B, 2.7B, 6.7B, 13B, 30B, 66B) (Completion Engine)\n\nFrom Stanford (via Huggingface)\n\n-   Alpaca (7B) (Chat and Completion Engines)\n\nFrom Berkeley (via Huggingface)\n\n-   Koala (7B, 13B_v1, 13B_v2) (Chat and Completion Engines)\n-   Vicuna (7B, 13B) (Chat and Completion Engines)\n\nFrom StabilityAI (via Huggingface)\n\n-   StableLM (7B, 13B) (Chat and Completion Engines)\n\nFrom AllenAI (via Huggingface)\n\n-   Tulu (7B, 13B, 30B, 65B) (Chat and Completion Engines)\n-   Open Instruct (ShareGPT) (7B, 13B, 30B, 65B) (Chat and Completion Engines)\n\nFrom AI21\n\n-   Jurassic 2 (Light, Mid, Ultra) (Completion Engines)\n\n## Citation\n\nIf you find this repository useful, please cite our paper:\n\n```bibtex\n@inproceedings{chan-etal-2023-clair-evaluating,\n    title = \"CLAIR: Evaluating Image Captions with Large Language Models\",\n    author = \"Chan, David M and\n        Petryk, Suzanne and\n        Gonzalez, Joseph E and\n        Darrell, Trevor and\n        Canny, John\",\n    booktitle = \"Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing\",\n    month = dec,\n    year = \"2023\",\n    address = \"Singapore, Singapore\",\n    publisher = \"Association for Computational Linguistics\",\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidmchan%2Fclair","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdavidmchan%2Fclair","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidmchan%2Fclair/lists"}