{"id":37071482,"url":"https://github.com/huffon/factsumm","last_synced_at":"2026-01-14T08:23:14.362Z","repository":{"id":51260429,"uuid":"364991176","full_name":"Huffon/factsumm","owner":"Huffon","description":"FactSumm: Factual Consistency Scorer for Abstractive Summarization","archived":false,"fork":false,"pushed_at":"2024-01-01T16:46:08.000Z","size":852,"stargazers_count":113,"open_issues_count":4,"forks_count":10,"subscribers_count":2,"default_branch":"master","last_synced_at":"2026-01-11T12:27:31.786Z","etag":null,"topics":["abstractive-summarization","factual-consistency","natural-language-processing","summarization"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Huffon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-05-06T17:42:40.000Z","updated_at":"2025-10-16T07:57:06.000Z","dependencies_parsed_at":"2022-09-26T19:13:25.947Z","dependency_job_id":null,"html_url":"https://github.com/Huffon/factsumm","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/Huffon/factsumm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Huffon%2Ffactsumm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Huffon%2Ffactsumm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Huffon%2Ffactsumm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Huffon%2Ffactsumm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Huffon","download_url":"https://codeload.github.com/Huffon/factsumm/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Huffon%2Ffactsumm/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28413839,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T08:16:59.381Z","status":"ssl_error","status_checked_at":"2026-01-14T08:13:45.490Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["abstractive-summarization","factual-consistency","natural-language-processing","summarization"],"created_at":"2026-01-14T08:23:13.803Z","updated_at":"2026-01-14T08:23:14.345Z","avatar_url":"https://github.com/Huffon.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# FactSumm: Factual Consistency Scorer for Abstractive Summarization\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/huffon/factsumm/releases\"\u003e\u003cimg alt=\"GitHub release\" src=\"https://img.shields.io/github/release/huffon/factsumm.svg\" /\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/huffon/factsumm/blob/master/LICENSE\"\u003e\u003cimg alt=\"Apache 2.0\" src=\"https://img.shields.io/badge/license-Apache%202.0-blue.svg\" /\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/huffon/factsumm/issues\"\u003e\u003cimg alt=\"Issues\" src=\"https://img.shields.io/github/issues/huffon/factsumm\" /\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n`FactSumm` is a toolkit that scores _**Factualy Consistency**_ for **Abstract Summarization**\n\nWithout fine-tuning, you can simply apply a variety of downstream tasks to both `the source article` and `the generated abstractive summary`\n\n![](assets/triples.png)\n\nFor example, by extracting **fact triples** from source articles and generated summaries, you can verify that generated summaries correctly reflect source-based facts ( _See image above_ )\n\n\u003cbr\u003e\n\n## Installation\n\nWith and _Python 3_ (\u003e= 3.8), you can install `factsumm` simply using `pip`:\n\n```bash\npip install factsumm\n```\n\nOr you can install `FactSumm` from source repository:\n\n```bash\ngit clone https://github.com/huffon/factsumm\ncd factsumm\npip install .\n```\n\n\u003cbr\u003e\n\n## Usage\n\n```python\n\u003e\u003e\u003e from factsumm import FactSumm\n\u003e\u003e\u003e factsumm = FactSumm()\n\u003e\u003e\u003e article = \"Lionel Andrés Messi (born 24 June 1987) is an Argentine professional footballer who plays as a forward and captains both Spanish club Barcelona and the Argentina national team. Often considered as the best player in the world and widely regarded as one of the greatest players of all time, Messi has won a record six Ballon d'Or awards, a record six European Golden Shoes, and in 2020 was named to the Ballon d'Or Dream Team.\"\n\u003e\u003e\u003e summary = \"Lionel Andrés Messi (born 24 Aug 1997) is an Spanish professional footballer who plays as a forward and captains both Spanish club Barcelona and the Spanish national team.\"\n\u003e\u003e\u003e factsumm(article, summary, verbose=True)\n\u003cSource Entities\u003e\nLine No.1: [[('Lionel Andrés Messi', 'PERSON'), ('24 June 1987', 'DATE'), ('Argentine', 'NORP'), ('Spanish', 'NORP'), ('Barcelona', 'ORG'), ('Argentina', 'GPE')]]\nLine No.2: [[('one', 'CARDINAL'), ('Messi', 'PERSON'), ('six', 'CARDINAL'), (\"Ballon d'Or\", 'WORK_OF_ART'), ('European Golden Shoes', 'WORK_OF_ART'), ('2020', 'DATE'), (\"Ballon d'Or Dream Team\", 'WORK_OF_ART')]]\n\n\u003cSummary Entities\u003e\nLine No.1: [[('Lionel Andrés Messi', 'PERSON'), ('24 Aug 1997', 'DATE'), ('Spanish', 'NORP'), ('Barcelona', 'ORG')]]\n\n\u003cSource Facts\u003e\n('Lionel Andrés Messi', 'per:origin', 'Argentine')\n('Lionel Andrés Messi', 'per:employee_of', 'Barcelona')\n('Barcelona', 'org:country_of_headquarters', 'Spanish')\n('Lionel Andrés Messi', 'per:date_of_birth', '24 June 1987')\n('Lionel Andrés Messi', 'per:countries_of_residence', 'Argentina')\n('Spanish', 'org:top_members/employees', 'Lionel Andrés Messi')\n('Barcelona', 'org:top_members/employees', 'Lionel Andrés Messi')\n\n\u003cSummary Facts\u003e\n('Lionel Andrés Messi', 'per:employee_of', 'Barcelona')\n('Lionel Andrés Messi', 'per:date_of_birth', '24 Aug 1997')\n('Barcelona', 'org:country_of_headquarters', 'Spanish')\n('Spanish', 'org:top_members/employees', 'Lionel Andrés Messi')\n('Barcelona', 'org:top_members/employees', 'Lionel Andrés Messi')\n('Lionel Andrés Messi', 'per:origin', 'Spanish')\n('Lionel Andrés Messi', 'per:countries_of_residence', 'Spanish')\n\n\u003cCommon Facts\u003e\n('Barcelona', 'org:top_members/employees', 'Lionel Andrés Messi')\n('Lionel Andrés Messi', 'per:employee_of', 'Barcelona')\n('Barcelona', 'org:country_of_headquarters', 'Spanish')\n('Spanish', 'org:top_members/employees', 'Lionel Andrés Messi')\n\n\u003cDiff Facts\u003e\n('Lionel Andrés Messi', 'per:date_of_birth', '24 Aug 1997')\n('Lionel Andrés Messi', 'per:origin', 'Spanish')\n('Lionel Andrés Messi', 'per:countries_of_residence', 'Spanish')\n\nFact Score: 0.5714285714285714\nAnswers based on Source (Questions are generated from Summary)\n[Q] Who is the captain of the Spanish national team?\t[Pred] \u003cunanswerable\u003e\n[Q] When was Lionel Andrés Messi born?\t[Pred] 24 June 1987\n[Q] Lionel Andrés Messi is a professional footballer of what nationality?\t[Pred] Argentine\n[Q] Lionel Messi is a captain of which Spanish club?\t[Pred] Barcelona\n\nAnswers based on Summary (Questions are generated from Summary)\n[Q] Who is the captain of the Spanish national team?\t[Pred] Lionel Andrés Messi\n[Q] When was Lionel Andrés Messi born?\t[Pred] 24 Aug 1997\n[Q] Lionel Andrés Messi is a professional footballer of what nationality?\t[Pred] Spanish\n[Q] Lionel Messi is a captain of which Spanish club?\t[Pred] Barcelona\n\nQAGS Score: 0.3333333333333333\n\nAvg. ROUGE-1: 0.4415584415584415\nAvg. ROUGE-2: 0.3287671232876712\nAvg. ROUGE-L: 0.4415584415584415\n\u003cBERTScore Score\u003e\nPrecision: 0.9760397672653198\nRecall: 0.9778039455413818\nF1: 0.9769210815429688\n```\n\nYou can use the GPU with the `device`. If you want to use GPU, pass `cuda` (default is `cpu`)\n\n```python\n\u003e\u003e\u003e factsumm(article, summary, device=\"cuda\")\n```\n\n\u003cbr\u003e\n\n## Sub-modules\n\nFrom [here](https://arxiv.org/pdf/2104.14839.pdf), you can find various way to score **Factual Consistency level** with _Unsupervised methods_\n\n\u003cbr\u003e\n\n### Triple-based Module ( _closed-scheme_ )\n\n```python\n\u003e\u003e\u003e from factsumm import FactSumm\n\u003e\u003e\u003e factsumm = FactSumm()\n\u003e\u003e\u003e factsumm.extract_facts(article, summary, verbose=True)\n\u003cSource Entities\u003e\nLine No.1: [[('Lionel Andrés Messi', 'PERSON'), ('24 June 1987', 'DATE'), ('Argentine', 'NORP'), ('Spanish', 'NORP'), ('Barcelona', 'ORG'), ('Argentina', 'GPE')]]\nLine No.2: [[('one', 'CARDINAL'), ('Messi', 'PERSON'), ('six', 'CARDINAL'), (\"Ballon d'Or\", 'WORK_OF_ART'), ('European Golden Shoes', 'WORK_OF_ART'), ('2020', 'DATE'), (\"Ballon d'Or Dream Team\", 'WORK_OF_ART')]]\n\n\u003cSummary Entities\u003e\nLine No.1: [[('Lionel Andrés Messi', 'PERSON'), ('24 Aug 1997', 'DATE'), ('Spanish', 'NORP'), ('Barcelona', 'ORG')]]\n\n\u003cSource Facts\u003e\n('Lionel Andrés Messi', 'per:origin', 'Argentine')\n('Lionel Andrés Messi', 'per:employee_of', 'Barcelona')\n('Barcelona', 'org:country_of_headquarters', 'Spanish')\n('Lionel Andrés Messi', 'per:date_of_birth', '24 June 1987')\n('Lionel Andrés Messi', 'per:countries_of_residence', 'Argentina')\n('Spanish', 'org:top_members/employees', 'Lionel Andrés Messi')\n('Barcelona', 'org:top_members/employees', 'Lionel Andrés Messi')\n\n\u003cSummary Facts\u003e\n('Lionel Andrés Messi', 'per:employee_of', 'Barcelona')\n('Lionel Andrés Messi', 'per:date_of_birth', '24 Aug 1997')\n('Barcelona', 'org:country_of_headquarters', 'Spanish')\n('Spanish', 'org:top_members/employees', 'Lionel Andrés Messi')\n('Barcelona', 'org:top_members/employees', 'Lionel Andrés Messi')\n('Lionel Andrés Messi', 'per:origin', 'Spanish')\n('Lionel Andrés Messi', 'per:countries_of_residence', 'Spanish')\n\n\u003cCommon Facts\u003e\n('Barcelona', 'org:top_members/employees', 'Lionel Andrés Messi')\n('Lionel Andrés Messi', 'per:employee_of', 'Barcelona')\n('Barcelona', 'org:country_of_headquarters', 'Spanish')\n('Spanish', 'org:top_members/employees', 'Lionel Andrés Messi')\n\n\u003cDiff Facts\u003e\n('Lionel Andrés Messi', 'per:date_of_birth', '24 Aug 1997')\n('Lionel Andrés Messi', 'per:origin', 'Spanish')\n('Lionel Andrés Messi', 'per:countries_of_residence', 'Spanish')\n\nFact Score: 0.5714285714285714\n```\n\nThe triple-based module counts the overlap of fact triples between the generated summary and the source document.\n\n\u003cbr\u003e\n\n### QA-based Module\n\n![](assets/qa.png)\n\nIf you ask questions about the summary and the source document, you will get a similar answer if the summary realistically matches the source document\n\n```python\n\u003e\u003e\u003e from factsumm import FactSumm\n\u003e\u003e\u003e factsumm = FactSumm()\n\u003e\u003e\u003e factsumm.extract_qas(article, summary, verbose=True)\nAnswers based on Source (Questions are generated from Summary)\n[Q] Who is the captain of the Spanish national team?\t[Pred] \u003cunanswerable\u003e\n[Q] When was Lionel Andrés Messi born?\t[Pred] 24 June 1987\n[Q] Lionel Andrés Messi is a professional footballer of what nationality?\t[Pred] Argentine\n[Q] Lionel Messi is a captain of which Spanish club?\t[Pred] Barcelona\n\nAnswers based on Summary (Questions are generated from Summary)\n[Q] Who is the captain of the Spanish national team?\t[Pred] Lionel Andrés Messi\n[Q] When was Lionel Andrés Messi born?\t[Pred] 24 Aug 1997\n[Q] Lionel Andrés Messi is a professional footballer of what nationality?\t[Pred] Spanish\n[Q] Lionel Messi is a captain of which Spanish club?\t[Pred] Barcelona\n\nQAGS Score: 0.3333333333333333\n```\n\n\u003cbr\u003e\n\n### ROUGE-based Module\n\n```python\n\u003e\u003e\u003e from factsumm import FactSumm\n\u003e\u003e\u003e factsumm = FactSumm()\n\u003e\u003e\u003e factsumm.calculate_rouge(article, summary)\nAvg. ROUGE-1: 0.4415584415584415\nAvg. ROUGE-2: 0.3287671232876712\nAvg. ROUGE-L: 0.4415584415584415\n```\n\nSimple but effective word-level overlap ROUGE score\n\n\u003cbr\u003e\n\n### BERTScore Module\n\n```python\n\u003e\u003e\u003e from factsumm import FactSumm\n\u003e\u003e\u003e factsumm = FactSumm()\n\u003e\u003e\u003e factsumm.calculate_bert_score(article, summary)\n\u003cBERTScore Score\u003e\nPrecision: 0.9760397672653198\nRecall: 0.9778039455413818\nF1: 0.9769210815429688\n```\n\n[BERTScore](https://github.com/Tiiiger/bert_score) can be used to calculate the similarity between each source sentence and the summary sentence\n\n\u003cbr\u003e\n\n### Citation\n\nIf you apply this library to any project, please cite:\n\n```\n@misc{factsumm,\n  author       = {Heo, Hoon},\n  title        = {FactSumm: Factual Consistency Scorer for Abstractive Summarization},\n  howpublished = {\\url{https://github.com/Huffon/factsumm}},\n  year         = {2021},\n}\n```\n\n\u003cbr\u003e\n\n## References\n\n- [The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey](https://arxiv.org/abs/2104.14839.pdf)\n- [Assessing The Factual Accuracy of Generated Text](https://arxiv.org/abs/1905.13322.pdf)\n- [Asking and Answering Questions to Evaluate the Factual Consistency of Summaries](https://arxiv.org/abs/2004.04228)\n- [FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization](https://arxiv.org/abs/2005.03754)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuffon%2Ffactsumm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhuffon%2Ffactsumm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuffon%2Ffactsumm/lists"}