https://github.com/clap-purdue/gricean-pragmatics

https://clap-purdue.github.io/gricean-pragmatics/
https://github.com/clap-purdue/gricean-pragmatics

benchmark evaluation large-language-models linguistic-competence llama multilingual pragmatics

Last synced: about 1 month ago
JSON representation

https://clap-purdue.github.io/gricean-pragmatics/

• Host: GitHub
• URL: https://github.com/clap-purdue/gricean-pragmatics
• Owner: clap-purdue
• License: apache-2.0
• Created: 2024-09-05T01:32:11.000Z (2 months ago)
• Default Branch: main
• Last Pushed: 2024-09-24T12:25:13.000Z (about 2 months ago)
• Last Synced: 2024-10-06T18:41:45.104Z (about 1 month ago)
• Topics: benchmark, evaluation, large-language-models, linguistic-competence, llama, multilingual, pragmatics
• Language: Python
• Homepage:
• Size: 2.7 MB
• Stars: 1
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# gricean-pragmatics

A library of five metrics evaluating large language models' pragmatic competence:

1. **Naturalness**: LLMs will generate surprisal scores as a proxy to text naturalness for each sentence in a minimal pair, which reflect how unexpected a sentence is, given the preceding context. Hypothetically, if LLMs show pragmatic sensitivity, LLMs should assign a lower surprisal score to the intended implied meaning in an appropriate context.

2. **Sensitivity to different Shades of Meaning (SSM)**

3. **Pragmatic Reasoning Chains (PRC)**

4. **Implicature Recovery Rate (IRR)**

5. **Pragmatic Sensitivity Index (PSI)**

Benchmark datasets (work-in-progress), examples, and documentation are also provided.