{"id":23237026,"url":"https://github.com/nicolay-r/llm-prompt-checking","last_synced_at":"2026-05-01T09:31:12.887Z","repository":{"id":187682620,"uuid":"677367083","full_name":"nicolay-r/llm-prompt-checking","owner":"nicolay-r","description":"Toolset for checking differences in recognising semantic relation presence by: (1) large language models 🤖  and (2) annotators / experts ✍️","archived":false,"fork":false,"pushed_at":"2024-10-01T16:51:41.000Z","size":2719,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-05T21:18:22.544Z","etag":null,"topics":["alignment","benchmark","chatgpt","gpt","llm","nlp","prompt","reasoning","relationextraction","sentimentanalysis"],"latest_commit_sha":null,"homepage":"https://www.youtube.com/watch?v=zeKg1MAQrYA","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nicolay-r.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-11T11:52:16.000Z","updated_at":"2024-10-01T16:51:44.000Z","dependencies_parsed_at":"2023-08-11T18:35:34.683Z","dependency_job_id":"eca0b345-fec1-4ce3-9518-6f0cfb0a3608","html_url":"https://github.com/nicolay-r/llm-prompt-checking","commit_stats":null,"previous_names":["nicolay-r/llm-prompt-checking"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/nicolay-r/llm-prompt-checking","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicolay-r%2Fllm-prompt-checking","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicolay-r%2Fllm-prompt-checking/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicolay-r%2Fllm-prompt-checking/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicolay-r%2Fllm-prompt-checking/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nicolay-r","download_url":"https://codeload.github.com/nicolay-r/llm-prompt-checking/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicolay-r%2Fllm-prompt-checking/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32492080,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-30T13:12:12.517Z","status":"online","status_checked_at":"2026-05-01T02:00:05.856Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alignment","benchmark","chatgpt","gpt","llm","nlp","prompt","reasoning","relationextraction","sentimentanalysis"],"created_at":"2024-12-19T04:13:24.592Z","updated_at":"2026-05-01T09:31:12.860Z","avatar_url":"https://github.com/nicolay-r.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LLM Reasoning Checking in Semantic Relation Extraction\n![](https://img.shields.io/badge/Python-3.9-brightgreen.svg)\n\n\u003e **Update November 01 2024:** ⭐ Implemented a separated [bulk-chain](https://github.com/nicolay-r/bulk-chain) project for handling massive amount of prompts with CoT. This concept was used in this studies.\n\nThis project aimed at assessing advances in generative large language models in the domain of Information Retrieval and Relation **Extraction in particular**.\n\nPlease refer to the related findings and presentation (RU language) for greater details:\n\n[![](https://markdown-videos-api.jorgenkh.no/youtube/zeKg1MAQrYA)](https://youtu.be/zeKg1MAQrYA)\n\n**Presentation**: https://nicolay-r.github.io/website/data/report_llm2023-nerel.pdf\n\n## Usage\n\nThis project shares three main scripts:\n1. `eval.py` -- dedicated for evaluating accuracies of the binary classification of the manually annotated relations. The output accuaracy shows the alignment with the annotation provided by experts ✍️\n2. `graph.py` -- script that adopts D3JS library for visualizing radial graphs that highlights: alignment 🟢 and misalignment 🔴.\n3. `launch.py` -- launches the application of the LLM model towards the composed input prompts to perform **binary semantic relations classification** of knonw semantic relations (present / absent, i.e. agree or disagree with experts annotaiton)\n\n\u003e **Update 08/08/2024:** ⚠️ This project adopts `RevGPT` API, which is obsolete at present. Please use the other project such as [QuickCoT](https://github.com/nicolay-r/quick_cot) for inferring other LLM models useing the official API.\n\n## Experiment \n\nWe refer to the [NEREL collection](https://github.com/nerel-ds/NEREL) which is used as a sorce for the semantic relations.\n\nWe experiment with `OpenAI/ChatGPT-3.5-0613` model.\n\n### Results\n\nResults are shortly higlighted in [the related presentation](https://nicolay-r.github.io/website/data/report_llm2023-nerel.pdf)\n\nTop 3 relation types are bolded.\n\nHigh: 70-100% agreement\n\n|Relation Type |Accuracy | Number of Relations |\n|-|-|-|\n|**CAUSE_OF_DEATH** |0.93|41|\n|**DATE_OF_BIRTH** |0.92|114|\n|**DATE_OF_DEATH** |0.86|87|\n|PLACE_OF_DEATH |0.86|63|\n|END_TIME |0.86|22|\n|START_TIME |0.82|38|\n|PLACE_OF_BIRTH |0.76|97|\n|DATE_OF_CREATION |0.75|117|\n|SCHOOLS_ATTENDED |0.74|84|\n|PART_OF |0.73|45|\n|MEMBER_OF |0.72|218|\n|WORKS_AS |0.71|3053|\n|LOCATED_IN |0.71|611|\n\nMedium: 70-50% agreement\n\n|Relation Type |Accuracy | Number of Relations |\n|-|-|-|\n|TAKES_PLACE_IN |0.7|1222|\n|SUBEVENT_OF |0.68|212|\n|DATE_FOUNDED_IN |0.68|44|\n|DATE_DEFUNCT_IN |0.67|6|\n|AWARDED_WITH |0.64|401|\n|ORIGINS_FROM |0.61|956|\n|RELIGION_OF |0.61|31|\n|RELATIVE |0.57|30|\n|MEDICAL_CONDITION |0.57|196|\n|OWNER_OF |0.54|94|\n|PENALIZED_AS |0.54|123|\n|WORKPLACE |0.53|804|\n|PARENT_OF |0.5|200|\n\nLow: Less than 40% agreement\n\n|Relation Type |Accuracy | Number of Relations |\n|-|-|-|\n|SIBLING |0.48|86|\n|SPOUSE |0.47|119|\n|PARTICIPANT_IN |0.42|2764|\n|HAS_CAUSE |0.42|481|\n|KNOWS |0.41|264|\n|EXPENDITURE |0.41|29|\n|ALTERNATIVE_NAME |0.4|942|\n|FOUNDED_BY |0.37|86|\n|ORGANIZES |0.3|123|\n|CONVICTED_OF |0.22|286|\n|SUBORDINATE_OF |0.15|105|\n\n\n## References\n\nThe visualization has been taken from the side [ARElight](https://github.com/nicolay-r/ARElight) project\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnicolay-r%2Fllm-prompt-checking","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnicolay-r%2Fllm-prompt-checking","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnicolay-r%2Fllm-prompt-checking/lists"}