{"id":35222571,"url":"https://github.com/ufal/factgenie","last_synced_at":"2026-01-16T22:56:13.704Z","repository":{"id":242913083,"uuid":"801535501","full_name":"ufal/factgenie","owner":"ufal","description":"Lightweight self-hosted span annotation tool","archived":false,"fork":false,"pushed_at":"2026-01-15T13:01:36.000Z","size":32958,"stargazers_count":38,"open_issues_count":12,"forks_count":8,"subscribers_count":4,"default_branch":"main","last_synced_at":"2026-01-15T16:46:40.339Z","etag":null,"topics":["annotation","annotation-tool","annotations","data-annotation","llm","span-labeling","token-classification","visualization","web-interface","word-classification"],"latest_commit_sha":null,"homepage":"https://quest.ms.mff.cuni.cz/nlg/d2t-llm/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ufal.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-05-16T12:22:40.000Z","updated_at":"2026-01-15T13:01:40.000Z","dependencies_parsed_at":"2024-08-12T11:45:11.005Z","dependency_job_id":"9d7e0a75-f032-4d3b-8d03-e795b9474548","html_url":"https://github.com/ufal/factgenie","commit_stats":null,"previous_names":["kasnerz/factgenie","ufal/factgenie"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/ufal/factgenie","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ufal%2Ffactgenie","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ufal%2Ffactgenie/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ufal%2Ffactgenie/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ufal%2Ffactgenie/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ufal","download_url":"https://codeload.github.com/ufal/factgenie/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ufal%2Ffactgenie/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28486939,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T22:54:02.790Z","status":"ssl_error","status_checked_at":"2026-01-16T22:50:10.344Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["annotation","annotation-tool","annotations","data-annotation","llm","span-labeling","token-classification","visualization","web-interface","word-classification"],"created_at":"2025-12-30T00:24:31.925Z","updated_at":"2026-01-16T22:56:13.697Z","avatar_url":"https://github.com/ufal.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"factgenie/static/img/favicon.png\" width=80px\" alt=\"logo\" /\u003e\n\n\u003ch1\u003e factgenie \u003c/h1\u003e\n\n\u003c!-- ![Github downloads](https://img.shields.io/github/downloads/kasnerz/factgenie/total) --\u003e\n\n![Pytest](https://github.com/ufal/factgenie/actions/workflows/py311_tests.yml/badge.svg)\n![PyPI](https://img.shields.io/pypi/v/factgenie)\n[![slack](https://img.shields.io/badge/slack-factgenie-04AD90.svg?logo=slack)](https://join.slack.com/t/factgenie/shared_invite/zt-2u180yy81-3zCR7mt8EOy55cxA5zhKyQ)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n![Github stars](https://img.shields.io/github/stars/kasnerz/factgenie?style=social)\n\u003c!-- [![arXiv](https://img.shields.io/badge/arXiv-2407.17863-b31b1b.svg)](https://arxiv.org/abs/2407.17863) --\u003e\n\u003c!-- ![PyPI downloads](https://img.shields.io/pypi/dm/factgenie) --\u003e\n\nAnnotate LLM outputs with a lightweight, self-hosted web application 🌈\n\n![factgenie](https://github.com/user-attachments/assets/1d074588-ada1-4974-a42a-0d2195c65283)\n\n\u003c/div\u003e\n\n## 📢  News\n- **2026-01-15** - Version 1.2.1 is released, see the 👉️ [changelog](https://github.com/ufal/factgenie/releases/tag/v1.2.1).\n- **2025-03-06** - Release 1.1.0 is out and comes with many significant improvements! See the 👉️ [changelog](https://github.com/ufal/factgenie/releases/tag/v1.1.0).\n- **2024-11-13** - We released version 1.0.1: our first official release! 🎉\n\n## 👉️ How can factgenie help you?\nOutputs from large language models (LLMs) may contain errors: semantic, factual, and lexical. \n\nWith factgenie, you can have the error spans annotated:\n- From LLMs through an API.\n- From humans through a crowdsourcing service.\n\nFactgenie can provide you:\n1. **A user-friendly website** for collecting annotations from human crowdworkers.\n2. **API calls** for collecting equivalent annotations from LLM-based evaluators.\n3. **A visualization interface** for visualizing the data and inspecting the annotated outputs.\n\n---\n*What does factgenie **not help with** is collecting the data (we assume that you already have these), starting the crowdsourcing campaign (for that, you need to use a service such as [Prolific.com](https://prolific.com)) or running the LLM evaluators (for that, you need a local framework such as [Ollama](https://ollama.com) or a proprietary API).*\n\n## 🏃 Quickstart\nMake sure you have Python \u003e=3.9 installed.\n\nIf you want to quickly try out factgenie, you can install the package from PyPI:\n```bash\npip install factgenie\n```\n\nHowever, the recommended approach for using factgenie is using an editable package:\n```bash\ngit clone https://github.com/ufal/factgenie.git\ncd factgenie\npip install -e .[dev,deploy]\n```\nThis approach will allow you to manually modify configuration files and write your own data classes.\n\nAfter installing factgenie, use the following command to run the server on your local computer:\n```bash\nfactgenie run --host=127.0.0.1 --port 8890\n```\nMore information on how to set up factgenie is on [Github wiki](../../wiki/Setup).\n\n## 💡 Usage guide\n\n\nSee the following **wiki pages** that that will guide you through various use-cases of factgenie:\n\n| Topic                                                               | Description                                        |\n| ------------------------------------------------------------------- | -------------------------------------------------- |\n| 🔧 [Setup](../../wiki/Setup)                                         | How to install factgenie.                          |\n| 🗂️ [Data Management](../../wiki/Data-Management)                     | How to manage datasets and model outputs.          |\n| 🤖 [LLM Annotations](../../wiki/LLM-Annotations)                     | How to annotate outputs using LLMs.                |\n| 👥 [Crowdsourcing Annotations](../../wiki/Crowdsourcing-Annotations) | How to annotate outputs using human crowdworkers.  |\n| ✍️  [Generating Outputs](../../wiki/Generating-Outputs)              | How to generate outputs using LLMs.                |\n| 📊 [Analyzing Annotations](../../wiki/Analyzing-Annotations)         | How to obtain statistics on collected annotations. |\n| 💻 [Command Line Interface](../../wiki/CLI)                          | How to use factgenie command line interface.       |\n| 🌱 [Contributing](../../wiki/Contributing)                           | How to contribute to factgenie.                    |\n\n## 🔥 Tutorials\nWe also provide step-by-step walkthroughs showing how to employ factgenie on [the dataset from the Shared Task in Evaluating Semantic Accuracy](https://github.com/ehudreiter/accuracySharedTask):\n\n| Tutorial                                                                                                                    | Description                                                                                      |\n| --------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |\n| [🏀 #1: Importing a custom dataset](../../wiki/Tutorials#-tutorial-1-importing-a-custom-dataset)                             | Loading the basketball statistics and model-generated basketball reports into the web interface. |\n| [💬 #2: Generating outputs](../../wiki/Tutorials#-tutorial-2-generating-outputs)                                             | Using Llama 3.1 with Ollama for generating basketball reports.                                   |\n| [📊 #3: Customizing data visualization](../../wiki/Tutorials#-tutorial-3-customizing-data-visualization)                     | Manually creating a custom dataset class for better data visualization.                          |\n| [🤖 #4: Annotating outputs with an LLM](../../wiki/Tutorials#-tutorial-4-annotating-outputs-with-an-llm)                     | Using GPT-4o for annotating errors in the basketball reports.                                    |\n| [👨‍💼 #5: Annotating outputs with human annotators](../../wiki/Tutorials#-tutorial-5-annotating-outputs-with-human-annotators) | Using human annotators for annotating errors in the basketball reports.                          |\n\n## 🔊 Join us on Slack\n\nIf you want to get a quick feedback or actively participate in development of factgenie, join our public **Slack workspace**:\n\n\u003ca href=\"https://join.slack.com/t/factgenie/shared_invite/zt-2u180yy81-3zCR7mt8EOy55cxA5zhKyQ\"\u003e\u003cimg width=\"150px\" alt=\"Join us on Slack\" src=\"./img/slack.png\"\u003e\u003c/a\u003e\n\n## 📸 Try a public preview\nWe used factgenie for [our related research project](https://d2t-llm.github.io/). We host the outputs from the project using a public instance of factgenie.\n\n\u003e [!IMPORTANT]\n\u003e Note that this preview is very limited: it enables only data viewing, not any data collection or management.\n\n**👉️ You can access the preview [here](https://quest.ms.mff.cuni.cz/nlg/d2t-llm/)**.\n\n## 💬 Cite us\n\n[Our paper](https://aclanthology.org/2024.inlg-demos.5/) was published at INLG 2024 System Demonstrations!\n\nYou can also find the paper on [arXiv](https://arxiv.org/abs/2407.17863).\n\nFor citing us, please use the following BibTeX entry:\n```bibtex\n@inproceedings{kasner2024factgenie,\n    title = \"factgenie: A Framework for Span-based Evaluation of Generated Texts\",\n    author = \"Kasner, Zden{\\v{e}}k  and\n      Platek, Ondrej  and\n      Schmidtova, Patricia  and\n      Balloccu, Simone  and\n      Dusek, Ondrej\",\n    editor = \"Mahamood, Saad  and\n      Minh, Nguyen Le  and\n      Ippolito, Daphne\",\n    booktitle = \"Proceedings of the 17th International Natural Language Generation Conference: System Demonstrations\",\n    year = \"2024\",\n    address = \"Tokyo, Japan\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2024.inlg-demos.5\",\n    pages = \"13--15\",\n}\n```\n\n## Acknowledgements\nThe authors acknowledge the support of the National Recovery Plan funded project MPO 60273/24/21300/21000 CEDMO 2.0 NPO.\n\nThis work was co-funded by the European Union (ERC, NG-NLG, 101039303).\n\n\u003cimg src=\"img/LOGO_ERC-FLAG_FP.png\" alt=\"erc-logo\" height=\"150\"/\u003e \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fufal%2Ffactgenie","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fufal%2Ffactgenie","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fufal%2Ffactgenie/lists"}