{"id":19520144,"url":"https://github.com/osu-nlp-group/llm-cn-eval","last_synced_at":"2026-07-25T16:30:24.289Z","repository":{"id":223148620,"uuid":"759437189","full_name":"OSU-NLP-Group/LLM-CN-Eval","owner":"OSU-NLP-Group","description":"[NAACL'24] A Multi-Aspect Framework for Counter Narrative Evaluation using Large Language Models","archived":false,"fork":false,"pushed_at":"2024-03-28T18:35:08.000Z","size":3526,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-08T13:34:40.758Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OSU-NLP-Group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-18T15:42:04.000Z","updated_at":"2024-04-26T09:58:27.000Z","dependencies_parsed_at":"2024-11-11T00:24:45.876Z","dependency_job_id":null,"html_url":"https://github.com/OSU-NLP-Group/LLM-CN-Eval","commit_stats":null,"previous_names":["osu-nlp-group/llm-cn-eval"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FLLM-CN-Eval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FLLM-CN-Eval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FLLM-CN-Eval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FLLM-CN-Eval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OSU-NLP-Group","download_url":"https://codeload.github.com/OSU-NLP-Group/LLM-CN-Eval/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240768141,"owners_count":19854399,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T00:24:02.772Z","updated_at":"2026-07-25T16:30:24.226Z","avatar_url":"https://github.com/OSU-NLP-Group.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# A Multi-Aspect Framework for Counter Narrative Evaluation using Large Language Models\n\nThis repo contains code for \"[A Multi-Aspect Framework for Counter Narrative Evaluation using Large Language Models](https://arxiv.org/abs/2402.11676)\".\n\n## Introduction\n\nIn this work, we propose a novel counter narrative evaluation framework that prompts LLMs to generate evaluation scores and feedback for generated counter narrative candidates using defined aspects derived from NGO guidelines. We validate this approach by measuring the correlation between LLM and human-generated scores from Amazon Mechanical Turk, demonstrating the potential of LLM-as-a-Judge as a multi-aspect, interpretable counter narrative evaluation strategy.\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"methodology.png\" title=\"Methodology Figure\"\u003e\n\u003c/div\u003e\n\nFigure 2: Validation pipeline for our counter narrative evaluation framework. (Left) Evaluation prompt template including task description, a ChatGPT-generated aspect score rubric, and hate speech/counter narrative pair. (Right) LLM evaluation scores are generated for counter narratives and are compared to AMT-annotated evaluation.\n\n## Evaluation Aspects\n\nWe define key aspects of counter narrative quality for evaluation, specifically drawing inspiration from NGO guidelines that advocate for constructive, focused counter narrative responses that challenge hate speech claims while de-escalating encounters in a non-toxic manner. When using our LLM evaluation framework, we utilize the following aspects:\n\n- Opposition: how much the counter narrative opposes and contradicts the hate speech example through the expression of an opposing sentiment regardless of the argument’s effectiveness or persuasiveness.\n- Relatedness: the association between the counter narrative response and hate speech example based on contextual or semantic similarity.\n- Specificity: how much the counter narrative presents focused and specific arguments that effectively counter the key ideas within the hate speech example through the use of in-depth arguments, nuanced reasoning, and supporting evidence.\n- Toxicity: how rude, unreasonable, or disrespectful a response is; definition from [Tekiroglu et al. (2022)](https://arxiv.org/abs/2204.01440)\n- Fluency: the quality of a response based on whether they are well-written and grammatically correct; definition from [Fu et al. (2023)](https://arxiv.org/abs/2302.04166)\n\nOur LLM evaluation prompts can be found in `evaluation/prompts.csv`.\n\n## Approach\n\n### Generation\n\nIn order to test the performance of LLM-as-a-Judge strategies for counter narrative evaluation, we generate evaluation scores to generated counter narrative candidates and measure the correlation to human scores collected from Amazon Mechanical Turk (AMT). We generated counter narrative responses to 180 test set examples from the [Multitarget-CONAN](https://github.com/marcoguerini/CONAN) dataset using finetuned [DialoGPT](https://huggingface.co/microsoft/DialoGPT-medium), ChatGPT and [Vicuna-v1.3 33b](https://huggingface.co/lmsys/vicuna-33b-v1.3). \n\nAll generation code and generated counter narrative candidates can be found in `generation`.\n\n### Evaluation\n\nWe then evaluate the generated counter narrative candidates using our evaluation framework to generate multi-aspect evaluation scores and interpretable evaluation feedback. We prompt the following LLMs:\n- GPT-3.5-Turbo\n- GPT-4\n- [Vicuna-v1.3 33b](https://huggingface.co/lmsys/vicuna-33b-v1.3)\n- [Prometheus 13b](https://huggingface.co/kaist-ai/prometheus-13b-v1.0)\n\nWe also evaluate the generated counter narrative candidates using the automatic metrics [BLEU](https://www.nltk.org/_modules/nltk/translate/bleu_score.html), [ROUGE](https://pypi.org/project/rouge-score/), [METEOR](https://huggingface.co/spaces/evaluate-metric/meteor), [BERTScore](https://huggingface.co/spaces/evaluate-metric/bertscore), and [BARTScore](https://github.com/neulab/BARTScore) and compare their performance to our counter narrative evaluation framework. \n\nAll evaluation code and prompts can be found in `evaluation`.\n\n### Correlation\n\nWe measure the [Pearson](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.pearsonr.html), [Spearman](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.spearmanr.html), and [Kendall tau](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kendalltau.html) correlations between each of the automatic evaluation metrics and human-annotated evalaution scores from AMT using Scipy. \n\nAll correlation code can be found in `correlation`. \n\n# Citation\n\nIf you find this work useful, please consider citing our work:\n\n```\n@inproceedings{jones2024a,\n    title={A Multi-Aspect Framework for Counter Narrative Evaluation using Large Language Models},\n    author={Jaylen Jones and Lingbo Mo and Eric Fosler-Lussier and Huan Sun},\n    booktitle={2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics},\n    year={2024},\n    url={https://openreview.net/forum?id=l3KoMef1KA}\n}\n```\n\nPlease also kindly cite the original papers, datasets, and evaluation metrics used throughout our study as well.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosu-nlp-group%2Fllm-cn-eval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fosu-nlp-group%2Fllm-cn-eval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosu-nlp-group%2Fllm-cn-eval/lists"}