{"id":13600928,"url":"https://github.com/tingofurro/summac","last_synced_at":"2025-04-04T11:08:49.886Z","repository":{"id":89161606,"uuid":"414387767","full_name":"tingofurro/summac","owner":"tingofurro","description":"Codebase, data and models for the SummaC paper in TACL","archived":false,"fork":false,"pushed_at":"2025-01-30T20:15:43.000Z","size":160,"stargazers_count":89,"open_issues_count":15,"forks_count":28,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-28T10:02:13.566Z","etag":null,"topics":["factual-consistency","inconsistency-detection","summarization","text-generation"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2111.09525","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tingofurro.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-06T22:23:19.000Z","updated_at":"2025-03-16T19:30:39.000Z","dependencies_parsed_at":"2024-01-16T23:26:32.499Z","dependency_job_id":"97856c58-fe9e-4a93-b780-068643a75dc3","html_url":"https://github.com/tingofurro/summac","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tingofurro%2Fsummac","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tingofurro%2Fsummac/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tingofurro%2Fsummac/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tingofurro%2Fsummac/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tingofurro","download_url":"https://codeload.github.com/tingofurro/summac/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247160922,"owners_count":20893929,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["factual-consistency","inconsistency-detection","summarization","text-generation"],"created_at":"2024-08-01T18:00:51.008Z","updated_at":"2025-04-04T11:08:49.864Z","avatar_url":"https://github.com/tingofurro.png","language":"Jupyter Notebook","funding_links":[],"categories":["Open Source Models for Measuring Hallucinations"],"sub_categories":["[Retrieval-Based Prompt Selection for Code-Related Few-Shot Learning](https://people.ece.ubc.ca/amesbah/resources/papers/cedar-icse23.pdf)"],"readme":"# SummaC: Summary Consistency Detection\n\nThis repository contains the code for TACL2021 paper: SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization\n\nWe release: (1) the trained SummaC models, (2) the SummaC Benchmark and data loaders, (3) training and evaluation scripts.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"400\" src=\"https://tingofurro.github.io/images/tacl2021_summac.png\"\u003e\n\u003c/p\u003e\n\n## Installing/Using SummaC\n\n[Update] Thanks to @Aktsvigun for the help, we now have a pip package, making it easy to install the SummaC models:\n```\npip install summac\n```\n\nRequirement issues: in v0.0.4, we've reduced package dependencies to facilitate installation. We recommend you install `torch` first and verify it works before installing `summac`.\n\nThe two trained models SummaC-ZS and SummaC-Conv are implemented in `model_summac` ([link](https://github.com/tingofurro/summac/blob/master/model_summac.py)). Once the package is installed, the models can be used like this:\n\n### Example use\n\n```python\nfrom summac.model_summac import SummaCZS, SummaCConv\n\nmodel_zs = SummaCZS(granularity=\"sentence\", model_name=\"vitc\", device=\"cpu\") # If you have a GPU: switch to: device=\"cuda\"\nmodel_conv = SummaCConv(models=[\"vitc\"], bins='percentile', granularity=\"sentence\", nli_labels=\"e\", device=\"cpu\", start_file=\"default\", agg=\"mean\")\n\ndocument = \"\"\"Scientists are studying Mars to learn about the Red Planet and find landing sites for future missions.\nOne possible site, known as Arcadia Planitia, is covered instrange sinuous features.\nThe shapes could be signs that the area is actually made of glaciers, which are large masses of slow-moving ice.\nArcadia Planitia is in Mars' northern lowlands.\"\"\"\n\nsummary1 = \"There are strange shape patterns on Arcadia Planitia. The shapes could indicate the area might be made of glaciers. This makes Arcadia Planitia ideal for future missions.\"\nscore_zs1 = model_zs.score([document], [summary1])\nscore_conv1 = model_conv.score([document], [summary1])\nprint(\"[Summary 1] SummaCZS Score: %.3f; SummacConv score: %.3f\" % (score_zs1[\"scores\"][0], score_conv1[\"scores\"][0])) # [Summary 1] SummaCZS Score: 0.582; SummacConv score: 0.536\n\nsummary2 = \"There are strange shape patterns on Arcadia Planitia. The shapes could indicate the area might be made of glaciers.\"\nscore_zs2 = model_zs.score([document], [summary2])\nscore_conv2 = model_conv.score([document], [summary2])\nprint(\"[Summary 2] SummaCZS Score: %.3f; SummacConv score: %.3f\" % (score_zs2[\"scores\"][0], score_conv2[\"scores\"][0])) # [Summary 2] SummaCZS Score: 0.877; SummacConv score: 0.709\n```\n\nWe recommend using the SummaCConv models, as experiments from the paper show it provides better predictions. Two notebooks provide experimental details: [SummaC - Main Results.ipynb](https://github.com/tingofurro/summac/blob/master/SummaC%20-%20Main%20Results.ipynb) for the main results (Table 2) and [SummaC - Additional Experiments.ipynb](https://github.com/tingofurro/summac/blob/master/SummaC%20-%20Additional%20Experiments.ipynb) for additional experiments (Tables 1, 3, 4, 5, 6) from the paper.\n\n### SummaC Benchmark\n\nThe SummaC Benchmark consists of 6 summary consistency datasets that have been standardized to a binary classification task. The datasets included are:\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"500\" src=\"https://tingofurro.github.io/images/tacl2021_summac_benchmark.png?1\"\u003e\u003cbr /\u003e\n  \u003cb\u003e% Positive\u003c/b\u003e is the percentage of positive (consistent) summaries. IAA is the inter-annotator agreement (Fleiss Kappa). \u003cb\u003eSource\u003c/b\u003e is the dataset used for the source documents (CNN/DM or XSum). \u003cb\u003e# Summarizers\u003c/b\u003e is the number of summarizers (extractive and abstractive) included in the dataset. \u003cb\u003e# Sublabel\u003c/b\u003e is the number of labels in the typology used to label summary errors.\n\u003c/p\u003e\n\nThe data-loaders for the benchmark are included in `benchmark.py` ([link](https://github.com/tingofurro/summac/blob/master/summac/benchmark.py)). Each dataset in the benchmark downloads automatically on first run. To load the benchmark:\n```py\nfrom summac.benchmark import SummaCBenchmark\nbenchmark_val = SummaCBenchmark(benchmark_folder=\"/path/to/summac_benchmark/\", cut=\"val\", hf_datasets_cache_dir = \"/path/to/huggingface_datasets_cache_dir/\")\nfrank_dataset = benchmark_val.get_dataset(\"frank\")\nprint(frank_dataset[300]) # {\"document: \"A Darwin woman has become a TV [...]\", \"claim\": \"natalia moon , 23 , has become a tv sensation [...]\", \"label\": 0, \"cut\": \"val\", \"model_name\": \"s2s\", \"error_type\": \"LinkE\"}\n```\n\n\n\n## Cite the work\n\nIf you make use of the code, models, or algorithm, please cite our paper.\n```\n@article{Laban2022SummaCRN,\n  title={SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization},\n  author={Philippe Laban and Tobias Schnabel and Paul N. Bennett and Marti A. Hearst},\n  journal={Transactions of the Association for Computational Linguistics},\n  year={2022},\n  volume={10},\n  pages={163-177}\n}\n```\n\n## Contributing\n\nIf you'd like to contribute, or have questions or suggestions, you can contact us at phillab@berkeley.edu. All contributions welcome, for example helping make the benchmark more easily downloadable, or improving model performance on the benchmark.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftingofurro%2Fsummac","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftingofurro%2Fsummac","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftingofurro%2Fsummac/lists"}