https://github.com/jxtse/GEC-Metrics-DSGram
Metrics for Grammatical Error Correction models that is closer to human feedback, proposed a novel dynamic weighting evaluation method 一种新颖的语法纠错模型评价无参考指标，采用大语言模型生成动态权重的评价方法
https://github.com/jxtse/GEC-Metrics-DSGram
Last synced: about 1 year ago
JSON representation
Metrics for Grammatical Error Correction models that is closer to human feedback, proposed a novel dynamic weighting evaluation method 一种新颖的语法纠错模型评价无参考指标，采用大语言模型生成动态权重的评价方法
- Host: GitHub
- URL: https://github.com/jxtse/GEC-Metrics-DSGram
- Owner: jxtse
- Created: 2024-05-28T09:17:10.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2024-12-22T06:48:36.000Z (over 1 year ago)
- Last Synced: 2024-12-22T07:28:18.493Z (over 1 year ago)
- Language: Python
- Homepage:
- Size: 2.22 MB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- StarryDivineSky - jxtse/GEC-Metrics-DSGram - Metrics-DSGram项目旨在为语法纠错（GEC）模型提供更贴近人工反馈的评估指标。该项目提出了一种新颖的动态权重评估方法，利用大型语言模型（LLM）生成动态权重，以更准确地反映不同类型语法错误的严重程度。这种无参考指标无需人工标注的参考答案即可进行评估，降低了评估成本。项目核心在于利用LLM的强大语言理解能力，赋予不同错误类型不同的权重，从而使评估结果更符合人类的直觉。通过动态调整权重，DSGram能够更有效地识别和惩罚严重的语法错误，提高GEC模型评估的准确性和可靠性。该项目为GEC领域的研究人员和开发者提供了一种更有效的模型评估工具，有助于推动GEC技术的进步。 (A01_文本生成_文本对话 / 大语言对话模型及数据)
README
# DSGram: Dynamic Weighting Sub-Metrics for Grammatical Error Correction
## Introduction
**DSGram** is a novel evaluation framework designed to enhance the performance evaluation of Grammatical Error Correction (GEC) models, especially in the era of large language models (LLMs). Traditional reference-based evaluation metrics often fall short due to the inherent discrepancies between model-generated corrections and provided gold references. DSGram addresses this issue by introducing a dynamic weighting mechanism that integrates Semantic Coherence, Edit Level, and Fluency.
This repository contains the code and data associated with the paper: **"DSGram: Dynamic Weighting Sub-Metrics for Grammatical Error Correction in the Era of Large Language Models"** by Jinxiang Xie, Yilin Li, Xunjian Yin, and Xiaojun Wan.
## Repository Structure
- `Dataset/`: Contains the datasets used for evaluation, including human-annotated and LLM-simulated sentences.
- `results/`: Directory to store the evaluation results.
- `DSGram/`: Source code for implementing the DSGram evaluation framework.
## Citation
If you use DSGram in your research, please cite our paper:
```
@misc{xie2024dsgramdynamicweightingsubmetrics,
title={DSGram: Dynamic Weighting Sub-Metrics for Grammatical Error Correction in the Era of Large Language Models},
author={Jinxiang Xie and Yilin Li and Xunjian Yin and Xiaojun Wan},
year={2024},
eprint={2412.12832},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2412.12832},
}
```
# Tech Used
 