{"id":27954369,"url":"https://github.com/ebu/benchmarkstt","last_synced_at":"2026-03-11T19:32:06.985Z","repository":{"id":45127033,"uuid":"158224105","full_name":"ebu/benchmarkstt","owner":"ebu","description":"Open Source AI Benchmarking toolkit for benchmarking speech to text services","archived":false,"fork":false,"pushed_at":"2024-04-17T09:59:07.000Z","size":4518,"stargazers_count":58,"open_issues_count":15,"forks_count":8,"subscribers_count":16,"default_branch":"master","last_synced_at":"2026-01-03T06:45:22.377Z","etag":null,"topics":["asr-benchmark","benchmark","benchmarking-suite","machine-learning","speech-to-text","stt-benchmark"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ebu.png","metadata":{"files":{"readme":"README.rst","changelog":"CHANGELOG.rst","contributing":"docs/CONTRIBUTING.rst","funding":null,"license":null,"code_of_conduct":"docs/CODE_OF_CONDUCT.rst","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-11-19T13:04:34.000Z","updated_at":"2025-09-23T18:07:20.000Z","dependencies_parsed_at":"2025-05-07T17:36:59.842Z","dependency_job_id":null,"html_url":"https://github.com/ebu/benchmarkstt","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/ebu/benchmarkstt","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ebu%2Fbenchmarkstt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ebu%2Fbenchmarkstt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ebu%2Fbenchmarkstt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ebu%2Fbenchmarkstt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ebu","download_url":"https://codeload.github.com/ebu/benchmarkstt/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ebu%2Fbenchmarkstt/sbom","scorecard":{"id":364846,"data":{"date":"2025-08-11","repo":{"name":"github.com/ebu/benchmarkstt","commit":"3ecaf44b724e5f27cf73defe05fcfd11e05284f6"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.7,"checks":[{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Code-Review","score":2,"reason":"Found 6/26 approved changesets -- score normalized to 2","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/pythonpackage.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENCE.md:0","Info: FSF or OSI recognized license: MIT License: LICENCE.md:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pythonpackage.yml:30: update your workflow using https://app.stepsecurity.io/secureworkflow/ebu/benchmarkstt/pythonpackage.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pythonpackage.yml:32: update your workflow using https://app.stepsecurity.io/secureworkflow/ebu/benchmarkstt/pythonpackage.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pythonpackage.yml:47: update your workflow using https://app.stepsecurity.io/secureworkflow/ebu/benchmarkstt/pythonpackage.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pythonpackage.yml:49: update your workflow using https://app.stepsecurity.io/secureworkflow/ebu/benchmarkstt/pythonpackage.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pythonpackage.yml:62: update your workflow using https://app.stepsecurity.io/secureworkflow/ebu/benchmarkstt/pythonpackage.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pythonpackage.yml:64: update your workflow using https://app.stepsecurity.io/secureworkflow/ebu/benchmarkstt/pythonpackage.yml/master?enable=pin","Warn: containerImage not pinned by hash: Dockerfile:1: pin your Docker image by updating python:3.7-alpine to python:3.7-alpine@sha256:f3d31c8677d03f0b3c724446077f229a6ce9d3ac430f5c08cd7dff00292048c3","Warn: pipCommand not pinned by hash: Dockerfile:8","Warn: pipCommand not pinned by hash: Dockerfile:13","Warn: pipCommand not pinned by hash: .github/workflows/pythonpackage.yml:37","Warn: pipCommand not pinned by hash: .github/workflows/pythonpackage.yml:38","Warn: pipCommand not pinned by hash: .github/workflows/pythonpackage.yml:41","Warn: pipCommand not pinned by hash: .github/workflows/pythonpackage.yml:54","Warn: pipCommand not pinned by hash: .github/workflows/pythonpackage.yml:55","Warn: pipCommand not pinned by hash: .github/workflows/pythonpackage.yml:69","Info:   0 out of   6 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   1 containerImage dependencies pinned","Info:   0 out of   8 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 18 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":0,"reason":"16 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GHSA-67hx-6x53-jw92","Warn: Project is vulnerable to: GHSA-hqq7-2q2v-82xq","Warn: Project is vulnerable to: GHSA-q8gg-vj6m-hgmj","Warn: Project is vulnerable to: GHSA-v6h2-p8h4-qcjw","Warn: Project is vulnerable to: GHSA-36jr-mh4h-2g58","Warn: Project is vulnerable to: GHSA-gxpj-cx7g-858c","Warn: Project is vulnerable to: GHSA-pfq8-rq6v-vf5m","Warn: Project is vulnerable to: GHSA-4f6x-49g2-99fm","Warn: Project is vulnerable to: GHSA-p3rp-vmj9-gv6v","Warn: Project is vulnerable to: GHSA-x3vm-38hw-55wf","Warn: Project is vulnerable to: GHSA-m4gq-x24j-jpmf","Warn: Project is vulnerable to: GHSA-f8q6-p94x-37v3","Warn: Project is vulnerable to: GHSA-pq67-2wwv-3xjx","Warn: Project is vulnerable to: GHSA-8cj5-5rvv-wf4v","Warn: Project is vulnerable to: GHSA-4wf5-vphf-c2xc","Warn: Project is vulnerable to: GHSA-3h5v-q93c-6h6q"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-18T11:37:49.263Z","repository_id":45127033,"created_at":"2025-08-18T11:37:49.263Z","updated_at":"2025-08-18T11:37:49.263Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30395601,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-11T18:46:22.935Z","status":"ssl_error","status_checked_at":"2026-03-11T18:46:17.045Z","response_time":84,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asr-benchmark","benchmark","benchmarking-suite","machine-learning","speech-to-text","stt-benchmark"],"created_at":"2025-05-07T17:26:23.850Z","updated_at":"2026-03-11T19:32:06.980Z","avatar_url":"https://github.com/ebu.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"============\nBenchmarkSTT\n============\n\n.. image:: https://img.shields.io/github/license/ebu/benchmarkstt.svg\n    :target: https://github.com/ebu/benchmarkstt/blob/master/LICENCE.md\n\n.. image:: https://github.com/ebu/benchmarkstt/actions/workflows/pythonpackage.yml/badge.svg\n    :alt: GitHub Workflow Status (branch)\n    :target: https://github.com/ebu/benchmarkstt/actions/workflows/pythonpackage.yml\n\n.. image:: https://readthedocs.org/projects/benchmarkstt/badge/?version=latest\n    :target: https://benchmarkstt.readthedocs.io/\n    :alt: Documentation Status\n\nAbout\n------\n\nThis is a command line tool for benchmarking Automatic Speech Recognition engines.\n\nIt is designed for non-academic production environments, and prioritises ease of use and relative benchmarking over scientific procedure and high-accuracy absolute scoring.\n\nBecause of the wide range of languages, algorithms and audio characteristics, no single STT engine can be expected to excel in all circumstances. For this reason, this tool places responsibility on the users to design their own benchmarking procedure and to decide, based on the combination of test data and metrics, which engine is best suited for their particular use case.\n\n\n\nUsage examples\n--------------\n\nReturns the number of word insertions, deletions, replacements and matches for the hypothesis transcript compared to the reference::\n\n    benchmarkstt --reference reference.txt --hypothesis hypothesis.txt --diffcounts\n\nReturns the Word Error Rate after lowercasing both reference and hypothesis. This normlization improves the accuracy of the Word Error Rate as it removes diffs that might otherwise be considered errors::\n\n    benchmarkstt -r reference.txt -h hypothesis.txt --wer --lowercase\n\nReturns a visual diff after applying all the normalization rules specified in the config file::\n\n    benchmarkstt -r reference.txt -h hypothesis.txt --worddiffs --config conf\n\n\nFurther information\n-------------------\n\nThis is a collaborative project to create a library for benchmarking AI/ML applications. It was created in response to the needs of broadcasters and providers of Access Services to media organisations, but anyone is welcome to contribute. The group behind this project is the EBU's `AI and Metadata group \u003chttps://tech.ebu.ch/groups/aim\u003e`_.\n\nCurrently the group is focussing on Speech-to-Text, but it will consider creating benchmarking tools for other AI/ML services.\n\nFor general information about this project, including the `motivations \u003chttps://github.com/ebu/benchmarkstt/wiki\u003e`_ and `guiding principles \u003chttps://github.com/ebu/benchmarkstt/wiki/Principles\u003e`_, please see the project `wiki \u003chttps://github.com/ebu/benchmarkstt/wiki\u003e`_\n\nTo install and start using the tool, go to the `documentation \u003chttps://benchmarkstt.readthedocs.io\u003e`_.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Febu%2Fbenchmarkstt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Febu%2Fbenchmarkstt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Febu%2Fbenchmarkstt/lists"}