https://github.com/tysonstanley/autoscore
R package for automatically scoring speech perception data 💬
https://github.com/tysonstanley/autoscore
Last synced: 7 months ago
JSON representation
R package for automatically scoring speech perception data 💬
- Host: GitHub
- URL: https://github.com/tysonstanley/autoscore
- Owner: TysonStanley
- License: gpl-3.0
- Created: 2018-07-25T21:38:09.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2018-10-01T19:15:22.000Z (about 7 years ago)
- Last Synced: 2025-01-24T18:12:00.956Z (9 months ago)
- Language: R
- Homepage:
- Size: 1.17 MB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.Rmd
- Changelog: NEWS.md
- License: LICENSE
Awesome Lists containing this project
README
---
output:
md_document:
variant: markdown_github
---
```{r, echo = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "README-"
)
devtools::load_all()
```
[](https://travis-ci.org/TysonStanley/autoscore)
# `autoscore` 
> R Package: `r packageVersion("autoscore")`
Shiny App: temporary location at [https://tysonstanley.shinyapps.io/autoscore/](https://tysonstanley.shinyapps.io/autoscore/)
*Authors:*
- *Tyson S. Barrett*
- *Sarah E. Yoho*
- *Stephanie A. Borrie*
The purpose of `autoscore` is to automatically score word identification in speech perception research, such as studies involving listener understanding of speech in background noise or disordered speech. The program uses a flexible number of rules that determine whether a response set of words (i.e., listener transcriptions) match a target set of words (i.e., speech corpus). At the most basic level, Autoscore counts words in the listener transcript as correct if they match the words in the target phrase exactly (regardless of word order), or match a homophone or common misspelling of the target word. Individual rules can be applied or removed, depending on the needs of researcher and the scoring rules of the research lab. Examples of rules available in Autoscore include the ability to count as correct substitutions of articles (A for The) or differences in plural or tense (adding -s or -ed to a word). Additional rules can be added by the researcher as needed.
The rule options are categorized into either spelling rules or grammar rules.
#### Spelling Rules
1. `acceptable_spell_rule`: Response word counted correct if it is a homophone or common misspelling of the target word, according to a preloaded default acceptable spelling list (contains over 300 common acceptable spellings). User can also download the default acceptable spelling list, add/remove items, and upload for automation. Response word counted correct if is on the acceptable spelling list. Default is `FALSE`.
2. `root_word_rule`: Response word counted correct if the target word (e.g. ‘day’) is embedded at either the beginning (e.g. ‘daybreak’) of the target word. Default is `FALSE`.
3. `double_letter_rule`: Response word counted correct if it omitted a double letter within a word (e.g. ‘atack’ matches ‘attack’) or added an unnecessary double letter (e.g. ‘occassion’ matches ‘occasion’). Default is `FALSE`.
#### Grammar Rules
4. `tense_rule`: Response word counted correct if it differs from the target word only by tense; Default is `FALSE`.
5. `tense_add_rule`: Response word counted correct if it only adds a tense as compared to the target word; Default is `FALSE`. It is also referred to as "Tense+ Rule."
6. `plural_rule`: Response word counted correct if it differs from the target word only by plurality; Default is `FALSE`.
7. `plural_add_rule`: Response word counted correct if it only adds a plural as compared to the target word; Default is `FALSE`. It is also referred to as "Plural+ Rule."
8. `a_the_rule`: Substitutions between “a” and “the” to be scored as correct; Default is `FALSE`.
## Design
The API of the `R` package is simple. A single call to `autoscore()` with the formatted data will run everything for you. This function is a composite of several sub-functions that do various jobs:
- `select_cols()` -- The first function which takes the data and gets it in the right format for analysis.
- `split_clean()` -- Using the cleaned data from `select_cols()`, this uses `stringr` to turn the phrases into individual words.
- `alternate_fun()` -- If a data.frame of alternate spellings is provided, this function will find and normalize all alternate spellings to match the original spelling as defined by the researcher.
- `match_position_basic()` -- This function is the workhorse of the package. It takes the cleaned data and does three main things: 1) applies all the rules except for the `position_rule`, 2) finds the matches between the responses and the targets, and 3) reports how far away the matches are from each other.
- `count_matches()` -- Finally, this function takes the information from `match_position_basic()` and counts the number of matches based on the `position_rule`.
Beyond the main analysis when using `autoscore()`, we can also call `pwc()` to get the percent words correct (based on the number of target words) for each observation.
## Use of the Online Tool
Visit [https://tysonstanley.shinyapps.io/autoscore/](https://tysonstanley.shinyapps.io/autoscore/) to use the online tool. Instructions for its use are found there.
## Use of the R Package
To install the package use the developmental version as it is not yet on CRAN.
```{r, eval = FALSE}
devtools::install_github("tysonstanley/autoscore_package")
```
An example of the use of `autoscore` is below. We will use the example data set provided in the package.
```{r}
library(tidyverse)
library(autoscore)
data("example_data")
example_data
```
First, let's use all the defaults and look at the first 10 rows of the output.
```{r}
example_data %>%
autoscore() %>% ## using all the defaults
as.tibble() ## to shorted output
```
Next, let's change some of the rules.
```{r}
example_data %>%
autoscore(plural_rule = FALSE, tense_rule = FALSE) %>%
as.tibble()
```
We can also change the output type to "none" to get all the data from the computation.
```{r}
example_data %>%
autoscore(output = "none")
```
To use the acceptable spelling rule, let's use the default provided in `autoscore`. . In the data frame below, the `target` spellings are the generally accepted spellings that are in the target list of words while the `acceptable` column are those that should also be counted as correct.
```{r}
autoscore::acceptable_spellings
```
Using this, we can provide it to the `autoscore()` function with the `acceptable_df` argument.
```{r}
example_data %>%
autoscore::autoscore(acceptable_df = autoscore::acceptable_spellings) %>%
as.tibble()
```
In each of these examples, it is clear that the human and "autoscore" agree the majority of the time. The times that they disagree, it is usually predictably a human error or a subjective judgement that the researcher will have to consider (for example by including alternate spellings of words as we just demonstrated).
### Learn More
Peer-reviewed publications and further tutorials are forthcoming. For more information, contact .