{"id":22382902,"url":"https://github.com/muhtasham/summarization-eval","last_synced_at":"2025-07-31T04:31:27.370Z","repository":{"id":217225926,"uuid":"742866348","full_name":"Muhtasham/summarization-eval","owner":"Muhtasham","description":"📝 Reference-Free automatic summarization evaluation with potential hallucination detection","archived":false,"fork":false,"pushed_at":"2024-01-15T12:31:52.000Z","size":516,"stargazers_count":98,"open_issues_count":0,"forks_count":8,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-11-30T00:31:04.263Z","etag":null,"topics":["evaluation-metrics","summarization"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Muhtasham.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2024-01-13T15:53:40.000Z","updated_at":"2024-11-10T11:28:07.000Z","dependencies_parsed_at":"2024-01-15T05:38:01.679Z","dependency_job_id":"77fdf686-c6d3-4542-b56a-5e0da2bab152","html_url":"https://github.com/Muhtasham/summarization-eval","commit_stats":null,"previous_names":["muhtasham/summarization-eval"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Muhtasham%2Fsummarization-eval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Muhtasham%2Fsummarization-eval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Muhtasham%2Fsummarization-eval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Muhtasham%2Fsummarization-eval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Muhtasham","download_url":"https://codeload.github.com/Muhtasham/summarization-eval/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228214814,"owners_count":17886346,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["evaluation-metrics","summarization"],"created_at":"2024-12-05T00:42:16.837Z","updated_at":"2024-12-05T00:42:17.342Z","avatar_url":"https://github.com/Muhtasham.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Summarization Evaluation\n\nElegantly automate text summarization evaluation in reference-free manner with potential hallucination detection.\n\n![Demo 1](assets/demo1.png)\n![Demo 2](assets/demo2.png)\n\n## Features\n\n- **Easy to Use**: Simply provide a text file containing the summary to be evaluated, and the script will handle the rest.\n- **Custom Metrics**: Utilize metrics such as word overlap and Jaccard similarity for in-depth content analysis.\n- **Hallucination Detection**: Systematically identify hallucinated content in summaries by analyzing semantic discrepancies with the original text, also use new word detection to identify potentially hallucinated summaries.If you came here for this just check `detect_hallucinations` under `src/utils.py`.\n- **GPT-based Evaluation**: Use GPT for nuanced qualitative assessments of summaries which takes care of json enforcement with pydantic, for easier parsing.\n- **Adapted ROUGE \u0026 BERTScore**: Rework traditional metrics for use in a reference-free context, focusing on the intrinsic qualities of summaries, as highlighted in [Eugene Yan's writing](https://eugeneyan.com/writing/abstractive/).\n- **Extensible**: Easily add new metrics and models to the project to expand its capabilities, open an issue or a PR if you want to add something.\n\n## Installation\n\nClone the repository and install dependencies:\n\n```bash\ngit clone https://github.com/Muhtasham/summarization-eval\ncd summarization-eval\npip install -r requirements.txt\n```\n\n### Usage\n\nTo use the `main.py` script under `src`, simply provide a text file containing the text you want to summarize. The script reads the file, generates summary, processes the summary, and outputs the evaluation results in a structured and readable format in the terminal.\n\nExample:\n\n```bash\npython main.py --file-path \"assets/news.txt\"\n```\n\n**Note**: You will need to have an OpenAI API key set up in your environment to run the script.\n\n## Contributing\n\nContributions to enhance and expand this project are welcome. Please see the `CONTRIBUTING.md` file for guidelines on how to contribute.\n\n## License\n\nThis project is licensed under the [MIT License](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmuhtasham%2Fsummarization-eval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmuhtasham%2Fsummarization-eval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmuhtasham%2Fsummarization-eval/lists"}