https://github.com/Alibaba-NLP/CDQA

CDQA: Chinese Dynamic Question Answering Benchmark
https://github.com/Alibaba-NLP/CDQA

Last synced: 12 months ago
JSON representation

CDQA: Chinese Dynamic Question Answering Benchmark

• Host: GitHub
• URL: https://github.com/Alibaba-NLP/CDQA
• Owner: Alibaba-NLP
• Created: 2024-02-29T14:50:15.000Z (about 2 years ago)
• Default Branch: main
• Last Pushed: 2024-12-13T23:52:02.000Z (over 1 year ago)
• Last Synced: 2024-12-14T00:27:28.389Z (over 1 year ago)
• Language: Python
• Size: 3.71 MB
• Stars: 14
• Watchers: 4
• Forks: 0
• Open Issues: 1
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• StarryDivineSky - Alibaba-NLP/CDQA - of-Thought 和 Rephrase-and-Respond）以进行评估。 (A01_文本生成_文本对话 / 大语言对话模型及数据)

README

          
# CDQA: Chinese Dynamic Question Answering Benchmark





Zhikun Xu*, Yinghui Li*, Ruixue Ding†, Xinyu Wang, Boli Chen, Yong Jiang†, Hai-Tao Zheng, Wenlian Lu, Pengjun Xie, Fei Huang





Institute for Intelligent Computing, Alibaba Group





*Equal Contribution; †Corresponding Author



[![paper](https://img.shields.io/badge/arXiv-2402.19248-red)](https://arxiv.org/abs/2402.19248)



We propose a Chinese QA benchmark containing question-answer pairs related to the latest news on the Chinese Internet by the following semi-automatic generation pipeline.



Besides, questions and answers are carefully categorized according to the frequency of answer changes and predefined answer types. Our contribution is for better evaluating Chinese-oriented LLMs, preventing the data contamination during evaluation with periodic updates on answers.

## Dataset Summary

The following tables are evaluation results for different baseline models. For searched results, we use Google by default. For prompts, we use three types: **Vanilla**, **Chain-of-Thought** and **Rephrase-and-Respond**. 







## Citation

If you found this work useful, consider giving this repository a star and citing our paper as followed:

```

@misc{xu2024let,

      title={Let LLMs Take on the Latest Challenges! A Chinese Dynamic Question Answering Benchmark}, 

      author={Zhikun Xu, Yinghui Li, Ruixue Ding, Xinyu Wang, Boli Chen, Yong Jiang, Hai-Tao Zheng, Wenlian Lu, Pengjun Xie, Fei Huang},

      year={2024},

      eprint={2402.19248},

      archivePrefix={arXiv},

      primaryClass={cs.CL}

}

```