{"id":19932102,"url":"https://github.com/amazon-science/tofueval","last_synced_at":"2026-03-05T12:02:26.680Z","repository":{"id":222888550,"uuid":"758645792","full_name":"amazon-science/tofueval","owner":"amazon-science","description":null,"archived":false,"fork":false,"pushed_at":"2024-05-10T00:06:56.000Z","size":3279,"stargazers_count":31,"open_issues_count":2,"forks_count":3,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-11-28T16:22:32.067Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit-0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/amazon-science.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-16T19:02:45.000Z","updated_at":"2025-06-04T06:22:14.000Z","dependencies_parsed_at":"2024-02-16T20:27:47.279Z","dependency_job_id":"dfb9038a-b283-4627-9e3b-410debe04c1a","html_url":"https://github.com/amazon-science/tofueval","commit_stats":null,"previous_names":["amazon-science/tofueval"],"tags_count":0,"template":false,"template_full_name":"amazon-archives/__template_MIT-0","purl":"pkg:github/amazon-science/tofueval","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2Ftofueval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2Ftofueval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2Ftofueval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2Ftofueval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/amazon-science","download_url":"https://codeload.github.com/amazon-science/tofueval/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2Ftofueval/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30123731,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-05T11:11:57.947Z","status":"ssl_error","status_checked_at":"2026-03-05T11:11:29.001Z","response_time":93,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T23:09:01.849Z","updated_at":"2026-03-05T12:02:26.661Z","avatar_url":"https://github.com/amazon-science.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# \u003cimg src=\"images/tofu-logo.png\" alt=\"Local Image\" width=\"45\"\u003e TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarization\n\nAuthors: Liyan Tang, Igor Shalyminov, Amy Wing-mei Wong, Jon Burnsky, Jake W. Vincent, Yu'an Yang, Siffi Singh, Song Feng, Hwanjun Song, Hang Su, Lijia Sun, Yi Zhang, Saab Mansour, Kathleen McKeown\n\n**Update: Our work has been accepted to NAACL 2024 🎉!**\nPlease check out our work [here](https://arxiv.org/pdf/2402.13249.pdf) 📃\n\nThis repository contains the annotations for the released benchmark dataset TofuEval. **Note that this is an evaluation benchmark. Data in the benchmark should not be used in training NLP models.**\n\nOn 05.03.2024, an identifier 64-character string is added to each instance in TofuEval to assist in future detection of contamination in web-crawl corpora.\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"./images/tofueval_main.png\" width=\"280\"\u003e\n\u003c/p\u003e\n\n\n## Documents in TofuEval\n\nWe provide the **dev/test splits** of TofuEval and **document identifier** in `document_ids_dev_test_split.json`, which can be used to obtain the source documents from MediaSum and MeetingBank. You can extract and preprocess the source documents by your own with the following links to the original data repository.\n\n* Documents from MediaSum can be downloaded from [here](https://github.com/zcgzcgzcg1/MediaSum)\n* Documents from MeetingBank can be downloaded from [here](https://meetingbank.github.io)\n\nOr you can use the following code snippet to extract the documents used in TofuEval:\n\n```python\nfrom datasets import load_dataset\nimport json\nimport pandas as pd\n\ndef obtain_dialogue_mediasum(dialogue_selected):\n    dialogue_df = pd.DataFrame(columns=['doc_id', 'source'])\n    for dialogue in dialogue_selected:\n        dialogue_id = dialogue['id']\n        speakers = dialogue['speaker']\n        utts = dialogue['utt']\n        transcript = ''\n        for speaker, utt in zip(speakers, utts):\n            transcript += f\"{speaker}: {utt}\\n\"\n        transcript = transcript.strip()\n        dialogue_df.loc[len(dialogue_df)] = [dialogue_id, transcript]\n    return dialogue_df\n\nwith open(\"document_ids_dev_test_split.json\") as file:\n    document_mapping = json.load(file)\n\nmeetingbank_dev_ids = document_mapping['dev']['meetingbank']\nmeetingbank_test_ids = document_mapping['test']['meetingbank']\nmediasum_dev_ids = document_mapping['dev']['mediasum']\nmediasum_test_ids = document_mapping['test']['mediasum']\n\nmeetingbank = pd.DataFrame(load_dataset(\"lytang/MeetingBank-transcript\")['test'])\nmeetingbank_dev = meetingbank[meetingbank.meeting_id.isin(meetingbank_dev_ids)][['meeting_id', 'source']].reset_index(drop=True).to_csv(\"meetingbank_dev_doc.csv\", index=False)\nmeetingbank_test = meetingbank[meetingbank.meeting_id.isin(meetingbank_test_ids)][['meeting_id', 'source']].reset_index(drop=True).to_csv(\"meetingbank_test_doc.csv\", index=False)\n\nwith open(\"/path/to/news_dialogue.json\") as file:\n    news_dialogue = json.load(file)\ndialogue_dev = [dialogue for dialogue in news_dialogue if dialogue['id'] in mediasum_dev_ids]\ndialogue_test = [dialogue for dialogue in news_dialogue if dialogue['id'] in mediasum_test_ids]\n\nmediasum_dev = obtain_dialogue_mediasum(dialogue_dev).to_csv(\"mediasum_dev_doc.csv\", index=False)\nmediasum_test = obtain_dialogue_mediasum(dialogue_test).to_csv(\"mediasum_test_doc.csv\", index=False)\n```\n* Note: Please download `news_dialogue.json` from [MediaSum](https://github.com/zcgzcgzcg1/MediaSum) and place it in the appropriate directory before preprocessing documents.\n* Please also cite MediaSum and MeetingBank if you use this benchmark.\n\n## Factual Consistency Annotation\n`factual_consistency/{dataset}_factual_eval_{split}.csv` contains factual consistency evaluations by expert linguistic annotators. The followings are descriptions of column names.\n\n|Col. name | Description |\n|--|--|\n|doc_id|The document id of a source document.|\n|annotation_id | The index of the source document in TofuEval. |\n|topic| Topic used to generate topic-focused summaries.|\n|model_name| Model used to generate the summary. Models are anynomized and the orders of models for all topics are shuffled.|\n|sent_idx|The sentence index in model generated summaries. |\n|summ_sent| {sent_idx}-th summary sentence by {model_name}. A full summary can be aggregated by joining these summary sentences using {sent_idx}.|\n|sent_label| `yes` if the summary sentence is factual consistent, `no` otherwise.|\n|exp| Human written explanation for why {summ_sent} is factually inconsistent.|\n|type| Human annotated error type(s) for {summ_sent}. A sentence can have multiple error types.|\n\n### Update: Extra Annotations\n\nWe have extended TofuEval with factual consistency annotations for one more model (`Model-Extra`). The latest version of TofuEval contains annotations for 6 models (1.8K summaries and 5K summary sentences)!\n\n## Completeness Annotation\n`completeness/{dataset}_completeness_final.csv` contains human written key points for each topic. The followings are descriptions of column names.\n\n|Col. name | Description |\n|--|--|\n|doc_id|the document id of a source document.|\n|annotation_id | The index of the source document in TofuEval. |\n|topic| topic used to generate topic-focused summaries.|\n|key_points| human written key points.|\n\n## Topic Categorization\n\n`topic_category/{dataset}_topic_category.json` categorizes each topic into *main* or *marginal*.\n\n## Citation\n\nIf you found the benchmark useful, please consider citing our work.\n```{bibtex}\n@Inproceedings{Tang2024,\n author = {Liyan Tang and Igor Shalyminov and Amy Wong and Jon Burnsky and Jake Vincent and Yu’an Yang and Siffi Singh and Song Feng and Hwanjun Song and Hang Su and Justin Sun and Yi Zhang and Saab Mansour and Kathleen McKeown},\n title = {TofuEval: Evaluating hallucinations of LLMs on topic-focused dialogue summarization},\n year = {2024},\n url = {https://www.amazon.science/publications/tofueval-evaluating-hallucinations-of-llms-on-topic-focused-dialogue-summarization},\n booktitle = {NAACL 2024},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famazon-science%2Ftofueval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famazon-science%2Ftofueval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famazon-science%2Ftofueval/lists"}