https://github.com/nmalkin/kupper_hafner
Kupper-Hafner inter-rater agreement calculation library
https://github.com/nmalkin/kupper_hafner
agreement inter-rater-agreement interrater-reliability irr kappa python reliability statistics
Last synced: 2 months ago
JSON representation
Kupper-Hafner inter-rater agreement calculation library
- Host: GitHub
- URL: https://github.com/nmalkin/kupper_hafner
- Owner: nmalkin
- License: bsd-3-clause
- Created: 2017-12-31T09:36:29.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2022-04-03T05:30:57.000Z (about 4 years ago)
- Last Synced: 2025-12-15T14:18:48.910Z (6 months ago)
- Topics: agreement, inter-rater-agreement, interrater-reliability, irr, kappa, python, reliability, statistics
- Language: Python
- Homepage:
- Size: 6.84 KB
- Stars: 4
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
Kupper-Hafner inter-rater agreement calculation
================================================
This repository contains code to calculate inter-rater agreement for multiple attribute responses, as described by Lawrence L. Kupper and Kerry B. Hafner in [their article](https://dx.doi.org/10.2307/2531695):
On Assessing Interrater Agreement for Multiple Attribute Responses
------------------------------------------------------------------
Authors: Lawrence L. Kupper and Kerry B. Hafner
Source: Biometrics, Vol. 45, No. 3 (Sep., 1989), pp. 957-967
Published by: International Biometric Society
Stable URL: https://www.jstor.org/stable/2531695
Why Kupper-Hafner agreement?
----------------------------
A common technique in scientific studies is for two or more raters to independently assign attributes to each item in a dataset (such as a response or specimen).
Inter-rater agreement, or [inter-rater reliability](https://en.wikipedia.org/wiki/Inter-rater_reliability), is used to measure the degree to which these raters end up agreeing with each other. Commonly used statistics for this include [Cohen's kappa](https://en.wikipedia.org/wiki/Cohen%27s_kappa) and [Fleiss' kappa](https://en.wikipedia.org/wiki/Fleiss'_kappa).
One assumption of these statistics, however, is that items are assigned to only one of several mutually exclusive categories. But, in certain coding situations, we want to be able to assign multiple codes to the same item. If we do that, the assumptions in the kappa statistics no longer hold.
The paper by Kupper and Hafner provides a statistic that allows the calculation of inter-rater agreement in these situations as well.
Installation
------------
This code implements Kupper-Hafner agreement in Python. It should support both Python 2 and 3.
You can install it using pip:
pip install kupper-hafner
Usage
-----
Let's say we have three items, and two raters (A and B) assigned them the following categories:
- Item 0: A rated it 10; B rated it 10, 20
- Item 1: A rated it 20, 30; B rated it 30
- Item 2: A rated it 40; B rated it 40
We would calculate Kupper-Hafner agreement as follows:
```python
from kupper_hafner import kupper_hafner
A_ratings = [[10], [20, 30], [40]]
B_ratings = [[10, 20], [30], [40]]
kupper_hafner(A_ratings, B_ratings)
```
If the codebook included categories that were not used by either rater, it can be provided as follows:
```python
codebook = [10, 20, 30, 40, 50]
kupper_hafner(A_ratings, B_ratings, codebook)
```