{"id":46551579,"url":"https://github.com/makefinks/rag-annotator","last_synced_at":"2026-03-07T03:30:48.230Z","repository":{"id":288054117,"uuid":"965772508","full_name":"makefinks/rag-annotator","owner":"makefinks","description":"A PySide6 GUI for curating ground truth data for retrieval systems","archived":false,"fork":false,"pushed_at":"2025-06-04T07:12:34.000Z","size":3268,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-04T13:59:07.754Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/makefinks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-13T21:41:52.000Z","updated_at":"2025-06-04T07:12:36.000Z","dependencies_parsed_at":"2025-06-04T08:23:02.136Z","dependency_job_id":null,"html_url":"https://github.com/makefinks/rag-annotator","commit_stats":null,"previous_names":["makefinks/rag-annotator"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/makefinks/rag-annotator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/makefinks%2Frag-annotator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/makefinks%2Frag-annotator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/makefinks%2Frag-annotator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/makefinks%2Frag-annotator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/makefinks","download_url":"https://codeload.github.com/makefinks/rag-annotator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/makefinks%2Frag-annotator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30206561,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-07T03:24:23.086Z","status":"ssl_error","status_checked_at":"2026-03-07T03:23:11.444Z","response_time":53,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-03-07T03:30:47.798Z","updated_at":"2026-03-07T03:30:48.219Z","avatar_url":"https://github.com/makefinks.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# RAG Annotation Tool for Ground Truth Creation\n\nThe annotation tool can be used to simplify the ground truth creation process for the evaluation of retrieval systems.\n\nIt allows for selecting a set of texts that are relevant to a given text / description and supports the following features:\n- **Interactive selection** of relevant texts\n- Metadata display and **highlighting of keywords and phrases**\n- Built in **BM25 search capabilities** to locate and include additional texts from the corpus \n\n## Demo\n![Gif demo](app/images/demo.gif)\n\n## Installation\n### Clone the repository\n\n```bash\ngit clone https://github.com/makefinks/rag-annotator.git\ncd rag-annotator\n```\n\n### Install requirements (uv is recommended)\n```bash\nuv sync\n```\n\u003e alternatively use: \n\u003e pip install -r requirements.txt\n### Activate the virtual environment\n```bash\nsource .venv/bin/activate\n```\n\n\n## How it works\n\n### Input preperation\nThe usage of this tool requires a specific input format (JSON).\nThe detailed format can be seen in `utils/ground_truth_schema.json`.\n\n#### Ground Truth Schema Explanation\nThe input JSON must contain the following main fields:\n\n- **points**: An array of objects, each representing an evaluation point. Each point contains:\n  - `id`: Integer identifier for the point.\n  - `title`: Title string for the point.\n  - `description`: Description or query string.\n  - `keywords (optional)`: Array of keywords relevant to the points description. Highlighted inside retrieved texts.\n  - `fetched_texts`: Array of text objects fetched for this point. Each has:\n    - `id`: Integer identifier for the text.\n    - `text`: The text content.\n    - `source`: The source string.\n    - `metadata` (optional): Additional metadata (e.g., description).\n    - `highlights` (optional): Array of strings to highlight in the text.\n  - `selected_texts`: Array of selected text objects (same structure as `fetched_texts`).\n  - `evaluated`: Boolean indicating if the point has been evaluated.\n- **all_texts**: An array of all possible text objects in the dataset, each with:\n  - `id`: Integer identifier.\n  - `text`: The text content.\n\nRefer to `app/utils/ground_truth_schema.json` for the complete and up-to-date schema.\n\n### Usage of the tool\n```bash\npython annotation_tool.py\n```\nUpon selecting the file in the prepared format the tool will load the data and display the GUI:\n\n- **Top Panel**: The current index of the evaluation object / point, a title dropdown, and the description / text of the object.\n- **Left Panel**: The list of texts that were fetched and are supposed to be selected as relevant.\n- **Right Panel**: A BM25 seach bar and result display, that allows you to search for a specific texts for all texts in the dataset. \n- **Bottom Panel**: Buttons for Navigation and saving the current state of the annotation.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmakefinks%2Frag-annotator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmakefinks%2Frag-annotator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmakefinks%2Frag-annotator/lists"}