{"id":24847556,"url":"https://github.com/stefantaubert/mean-opinion-score","last_synced_at":"2025-07-26T00:35:26.452Z","repository":{"id":77878510,"uuid":"605555868","full_name":"stefantaubert/mean-opinion-score","owner":"stefantaubert","description":"Python library for calculating the mean opinion score and 95% confidence interval of the standard deviation of text-to-speech ratings according to Ribeiro et al. (2011).","archived":false,"fork":false,"pushed_at":"2025-01-31T13:53:29.000Z","size":79,"stargazers_count":24,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-07-06T22:52:55.091Z","etag":null,"topics":["intelligibility","mos","naturalness","speech-synthesis","subjective-evaluation","text-to-speech","tts"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stefantaubert.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-02-23T12:08:42.000Z","updated_at":"2025-04-28T13:32:05.000Z","dependencies_parsed_at":null,"dependency_job_id":"6e6ceb85-9dd6-4a11-9457-55a7ac547a0c","html_url":"https://github.com/stefantaubert/mean-opinion-score","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/stefantaubert/mean-opinion-score","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefantaubert%2Fmean-opinion-score","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefantaubert%2Fmean-opinion-score/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefantaubert%2Fmean-opinion-score/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefantaubert%2Fmean-opinion-score/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stefantaubert","download_url":"https://codeload.github.com/stefantaubert/mean-opinion-score/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefantaubert%2Fmean-opinion-score/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267093872,"owners_count":24034956,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-25T02:00:09.625Z","response_time":70,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["intelligibility","mos","naturalness","speech-synthesis","subjective-evaluation","text-to-speech","tts"],"created_at":"2025-01-31T11:29:58.619Z","updated_at":"2025-07-26T00:35:26.444Z","avatar_url":"https://github.com/stefantaubert.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mean-opinion-score\n\n[![PyPI](https://img.shields.io/pypi/v/mean-opinion-score.svg)](https://pypi.python.org/pypi/mean-opinion-score)\n[![PyPI](https://img.shields.io/pypi/pyversions/mean-opinion-score.svg)](https://pypi.python.org/pypi/mean-opinion-score)\n[![MIT](https://img.shields.io/github/license/stefantaubert/mean-opinion-score.svg)](https://github.com/stefantaubert/mean-opinion-score/blob/master/LICENSE)\n[![PyPI](https://img.shields.io/pypi/wheel/mean-opinion-score.svg)](https://pypi.python.org/pypi/mean-opinion-score/#files)\n![PyPI](https://img.shields.io/pypi/implementation/mean-opinion-score.svg)\n[![PyPI](https://img.shields.io/github/commits-since/stefantaubert/mean-opinion-score/latest/master.svg)](https://github.com/stefantaubert/mean-opinion-score/compare/v0.0.2...master)\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.8238259.svg)](https://doi.org/10.5281/zenodo.8238259)\n\nPython library for calculating the mean opinion score (MOS) and 95% confidence interval (CI) of the standard deviation (SD) of text-to-speech (TTS) ratings according to [\"Ribeiro, F., Florêncio, D., Zhang, C., \u0026 Seltzer, M. (2011). CrowdMOS: An approach for crowdsourcing mean opinion score studies\"](https://doi.org/10.1109/ICASSP.2011.5946971). To determine CIs, the authors used a two-way random effects model with the variables: diversity of intrinsic sentence quality, diversity of rater preference, and subjective uncertainty.\n\n## Installation\n\n```sh\npip install mean-opinion-score --user\n```\n\n## Usage\n\n```py\nimport numpy as np\n\nfrom mean_opinion_score import get_ci95, get_ci95_default, get_mos\n\n_ = np.nan\n\nratings = np.array([\n    # columns represent sentences\n    [4, 5, _, 4, _, 3],  # rater 1\n    [4, 4, 4, 5, _, 4],  # rater 2\n    [_, 3, 5, 4, _, 1],  # rater 3\n    [_, _, _, _, _, _],  # rater 4\n])\n\nmos = get_mos(ratings)\nci = get_ci95(ratings)\nci_default = get_ci95_default(ratings)\n\nprint(f\"MOS: {mos:.2f} ± {ci:.4f}\")\nprint(f\"MOS: {mos:.2f} ± {ci_default:.4f}\")\n# MOS: 3.85 ± 1.3316\n# MOS: 3.85 ± 0.5579\n```\n\n## Dependencies\n\n- `numpy`\n- `scipy`\n\n## Contributing\n\nIf you notice an error, please don't hesitate to open an issue.\n\n### Development setup\n\n```sh\n# update\nsudo apt update\n# install Python 3.6, 3.7, 3.8, 3.9, 3.10 \u0026 3.11 for ensuring that tests can be run\nsudo apt install python3-pip \\\n  python3.6 python3.6-dev python3.6-distutils python3.6-venv \\\n  python3.7 python3.7-dev python3.7-distutils python3.7-venv \\\n  python3.8 python3.8-dev python3.8-distutils python3.8-venv \\\n  python3.9 python3.9-dev python3.9-distutils python3.9-venv \\\n  python3.10 python3.10-dev python3.10-distutils python3.10-venv \\\n  python3.11 python3.11-dev python3.11-distutils python3.11-venv\n# install pipenv for creation of virtual environments\npython3.11 -m pip install pipenv --user\n\n# check out repo\ngit clone https://github.com/stefantaubert/mean-opinion-score.git\ncd mean-opinion-score\n# create virtual environment\npython3.11 -m pipenv install --dev\n```\n\n## Running the tests\n\n```sh\n# first install the tool like in \"Development setup\"\n# then, navigate into the directory of the repo (if not already done)\ncd mean-opinion-score\n# activate environment\npython3.11 -m pipenv shell\n# run tests\ntox\n```\n\nFinal lines of test result output:\n\n```log\n  py36: OK\n  py37: OK\n  py38: OK\n  py39: OK\n  py310: OK\n  py311: OK\n  congratulations :)\n```\n\n## License\n\nMIT License\n\n## Acknowledgments\n\nMOS and CI calculation is taken from:\n\n- Ribeiro, F., Florêncio, D., Zhang, C., \u0026 Seltzer, M. (2011). CrowdMOS: An approach for crowdsourcing mean opinion score studies. 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2416–2419. [https://doi.org/10.1109/ICASSP.2011.5946971](https://doi.org/10.1109/ICASSP.2011.5946971)\n\nFunded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Project-ID 416228727 – CRC 1410.\n\n## Citation\n\nIf you want to cite this repo, you can use this BibTeX-entry generated by GitHub (see *About =\u003e Cite this repository*).\n\n```txt\nTaubert, S. (2023). mean-opinion-score (Version 0.0.2) [Computer software]. https://doi.org/10.5281/zenodo.8238259\n```\n\n## Changelog\n\n- v0.0.2 (2023-08-11)\n  - Added:\n    - commonly used 95% confidence interval calculation\n- v0.0.1 (2023-02-23)\n  - Initial release\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstefantaubert%2Fmean-opinion-score","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstefantaubert%2Fmean-opinion-score","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstefantaubert%2Fmean-opinion-score/lists"}