{"id":20674681,"url":"https://github.com/hasaniqbal777/openfactcheck","last_synced_at":"2025-09-04T23:32:12.659Z","repository":{"id":250496144,"uuid":"829374815","full_name":"hasaniqbal777/OpenFactCheck","owner":"hasaniqbal777","description":"An Open-source Factuality Evaluation Demo for LLMs","archived":false,"fork":false,"pushed_at":"2025-06-11T14:39:21.000Z","size":10638,"stargazers_count":27,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-12T03:50:26.102Z","etag":null,"topics":["artificial-intelligence","fact-checking","llm","natural-language-processing"],"latest_commit_sha":null,"homepage":"https://openfactcheck.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hasaniqbal777.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-07-16T09:53:08.000Z","updated_at":"2025-06-11T14:28:51.000Z","dependencies_parsed_at":"2024-11-22T23:02:13.460Z","dependency_job_id":"669013bc-7707-4cfc-88ed-942beaf345c4","html_url":"https://github.com/hasaniqbal777/OpenFactCheck","commit_stats":null,"previous_names":["hasaniqbal777/openfactcheck"],"tags_count":58,"template":false,"template_full_name":null,"purl":"pkg:github/hasaniqbal777/OpenFactCheck","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hasaniqbal777%2FOpenFactCheck","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hasaniqbal777%2FOpenFactCheck/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hasaniqbal777%2FOpenFactCheck/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hasaniqbal777%2FOpenFactCheck/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hasaniqbal777","download_url":"https://codeload.github.com/hasaniqbal777/OpenFactCheck/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hasaniqbal777%2FOpenFactCheck/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273688211,"owners_count":25150282,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-04T02:00:08.968Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","fact-checking","llm","natural-language-processing"],"created_at":"2024-11-16T21:06:41.143Z","updated_at":"2025-09-04T23:32:07.640Z","avatar_url":"https://github.com/hasaniqbal777.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!--\ntitle: OpenFactCheck\nemoji: ✅\ncolorFrom: green\ncolorTo: purple\nsdk: streamlit\napp_file: src/openfactcheck/app/app.py\npinned: false\n--\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg alt=\"OpenFactCheck Logo\" src=\"https://raw.githubusercontent.com/hasaniqbal777/OpenFactCheck/main/assets/splash.png\" height=\"120\" /\u003e\n  \u003cp align=\"center\"\u003eAn Open-source Factuality Evaluation Demo for LLMs\n    \u003cbr\u003e\n  \u003c/p\u003e\n\u003c/p\u003e\n\n---\n\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://github.com/hasaniqbal777/OpenFactCheck/actions/workflows/release.yaml\"\u003e\n    \u003cimg src=\"https://img.shields.io/github/actions/workflow/status/hasaniqbal777/openfactcheck/release.yaml?logo=github\u0026label=Release\" alt=\"Release\"\u003e\n\u003c/a\u003e\n\u003ca href=\"https://readthedocs.org/projects/openfactcheck/builds/\"\u003e\n    \u003cimg alt=\"Docs\" src=\"https://img.shields.io/readthedocs/openfactcheck?logo=readthedocs\u0026label=Docs\"\u003e\n\u003c/a\u003e\n\u003cbr\u003e\n\u003ca href=\"https://gnu.org/licenses/gpl-3.0.html\"\u003e\n    \u003cimg src=\"https://img.shields.io/github/license/hasaniqbal777/openfactcheck\" alt=\"License\"\u003e\n\u003c/a\u003e\n\u003ca href=\"https://pypi.org/project/openfactcheck/\"\u003e\n    \u003cimg src=\"https://img.shields.io/pypi/pyversions/openfactcheck.svg\" alt=\"Python Version\"\u003e\n\u003c/a\u003e\n\u003ca href=\"https://pypi.org/project/openfactcheck/\"\u003e\n    \u003cimg src=\"https://img.shields.io/pypi/v/openfactcheck.svg\" alt=\"PyPI Latest Release\"\u003e\n\u003c/a\u003e\n\u003ca href=\"https://arxiv.org/abs/2405.05583\"\u003e\u003cimg src=\"https://img.shields.io/badge/arXiv-2405.05583-B31B1B\" alt=\"arXiv\"\u003e\u003c/a\u003e\n\u003ca href=\"https://zenodo.org/doi/10.5281/zenodo.13358664\"\u003e\u003cimg src=\"https://img.shields.io/badge/DOI-10.5281/zenodo.13358664-blue\" alt=\"DOI\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n---\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"#overview\"\u003eOverview\u003c/a\u003e •\n    \u003ca href=\"#installation\"\u003eInstallation\u003c/a\u003e •\n    \u003ca href=\"#usage\"\u003eUsage\u003c/a\u003e •\n    \u003ca href=\"https://huggingface.co/spaces/hasaniqbal777/OpenFactCheck\"\u003eHuggingFace Demo\u003c/a\u003e •\n    \u003ca href=\"https://openfactcheck.readthedocs.io/\"\u003eDocumentation\u003c/a\u003e\n\u003c/p\u003e\n\n## Overview\n\nOpenFactCheck is an open-source repository designed to facilitate the evaluation and enhancement of factuality in responses generated by large language models (LLMs). This project aims to integrate various fact-checking tools into a unified framework and provide comprehensive evaluation pipelines.\n\n\u003cimg src=\"https://raw.githubusercontent.com/hasaniqbal777/OpenFactCheck/main/assets/architecture.png\" width=\"100%\"\u003e\n\n## Installation\n\nYou can install the package from PyPI using pip:\n\n```bash\npip install openfactcheck\n```\n\n## Usage\n\nFirst, you need to initialize the OpenFactCheckConfig object and then the OpenFactCheck object.\n```python\nfrom openfactcheck import OpenFactCheck, OpenFactCheckConfig\n\n# Initialize the OpenFactCheck object\nconfig = OpenFactCheckConfig()\nofc = OpenFactCheck(config)\n```\n\n### Response Evaluation\n\nYou can evaluate a response using the `ResponseEvaluator` class.\n\n```python\n# Evaluate a response\nresult = ofc.ResponseEvaluator.evaluate(response: str)\n```\n\n### LLM Evaluation\n\nWe provide [FactQA](https://raw.githubusercontent.com/hasaniqbal777/OpenFactCheck/main/src/openfactcheck/templates/llm/questions.csv), a dataset of 6480 questions for evaluating LLMs. Onc you have the responses from the LLM, you can evaluate them using the `LLMEvaluator` class.\n\n```python\n# Evaluate an LLM\nresult = ofc.LLMEvaluator.evaluate(model_name: str,\n                                   input_path: str)\n```\n\n### Checker Evaluation\n\nWe provide [FactBench](https://raw.githubusercontent.com/hasaniqbal777/OpenFactCheck/main/src/openfactcheck/templates/factchecker/claims.jsonl), a dataset of 4507 claims for evaluating fact-checkers. Once you have the responses from the fact-checker, you can evaluate them using the `CheckerEvaluator` class.\n\n```python\n# Evaluate a fact-checker\nresult = ofc.CheckerEvaluator.evaluate(checker_name: str,\n                                       input_path: str)\n```\n\n## Cite\n\nIf you use OpenFactCheck in your research, please cite the following:\n\n```bibtex\n@article{wang2024openfactcheck,\n  title        = {OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMs},\n  author       = {Wang, Yuxia and Wang, Minghan and Iqbal, Hasan and Georgiev, Georgi and Geng, Jiahui and Nakov, Preslav},\n  journal      = {arXiv preprint arXiv:2405.05583},\n  year         = {2024}\n}\n\n@article{iqbal2024openfactcheck,\n  title        = {OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMs},\n  author       = {Iqbal, Hasan and Wang, Yuxia and Wang, Minghan and Georgiev, Georgi and Geng, Jiahui and Gurevych, Iryna and Nakov, Preslav},\n  journal      = {arXiv preprint arXiv:2408.11832},\n  year         = {2024}\n}\n\n@software{hasan_iqbal_2024_13358665,\n  author       = {Hasan Iqbal},\n  title        = {hasaniqbal777/OpenFactCheck: v0.3.0},\n  month        = {aug},\n  year         = {2024},\n  publisher    = {Zenodo},\n  version      = {v0.3.0},\n  doi          = {10.5281/zenodo.13358665},\n  url          = {https://doi.org/10.5281/zenodo.13358665}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhasaniqbal777%2Fopenfactcheck","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhasaniqbal777%2Fopenfactcheck","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhasaniqbal777%2Fopenfactcheck/lists"}