{"id":21423956,"url":"https://github.com/picovoice/wake-word-benchmark","last_synced_at":"2025-07-14T08:31:40.762Z","repository":{"id":31940165,"uuid":"130927749","full_name":"Picovoice/wake-word-benchmark","owner":"Picovoice","description":"wake word engine benchmark framework","archived":false,"fork":false,"pushed_at":"2021-11-25T18:15:23.000Z","size":101622,"stargazers_count":133,"open_issues_count":0,"forks_count":28,"subscribers_count":18,"default_branch":"master","last_synced_at":"2025-04-08T05:13:31.395Z","etag":null,"topics":["benchmark","benchmark-framework","keyword-spotting","wake-word","wake-word-detection"],"latest_commit_sha":null,"homepage":"https://picovoice.ai/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Picovoice.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-04-25T00:12:46.000Z","updated_at":"2025-03-15T05:36:18.000Z","dependencies_parsed_at":"2022-08-07T17:01:02.318Z","dependency_job_id":null,"html_url":"https://github.com/Picovoice/wake-word-benchmark","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Picovoice/wake-word-benchmark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Picovoice%2Fwake-word-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Picovoice%2Fwake-word-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Picovoice%2Fwake-word-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Picovoice%2Fwake-word-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Picovoice","download_url":"https://codeload.github.com/Picovoice/wake-word-benchmark/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Picovoice%2Fwake-word-benchmark/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265262611,"owners_count":23736428,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","benchmark-framework","keyword-spotting","wake-word","wake-word-detection"],"created_at":"2024-11-22T21:19:01.103Z","updated_at":"2025-07-14T08:31:35.751Z","avatar_url":"https://github.com/Picovoice.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Wake Word Benchmark\n\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/Picovoice/wakeword-benchmark/blob/master/LICENSE)\n\nMade in Vancouver, Canada by [Picovoice](https://picovoice.ai)\n\nThe purpose of this benchmarking framework is to provide a scientific comparison between different wake word detection\nengines in terms of accuracy and runtime metrics. While working on [Porcupine](https://github.com/Picovoice/Porcupine)\nwe noted that there is a need for such a tool to empower customers to make data-driven decisions.\n\n## Table of Contents\n- [Wake Word Benchmark](#wake-word-benchmark)\n  - [Table of Contents](#table-of-contents)\n  - [Results](#results)\n    - [Accuracy](#accuracy)\n    - [Runtime](#runtime)\n  - [Data](#data)\n  - [Engines](#engines)\n  - [How to Reproduce?](#how-to-reproduce)\n\n## Results\n\n### Accuracy\n\nBelow is the result of running the benchmark framework averaged on six different keywords. The plot below shows the miss\nrate of different engines at 1 false alarm per 10 hours. The lower the miss rate the more accurate the engine is.\n\n![](doc/img/summary.png)\n\n### Runtime\n\nBelow is the runtime measurements on a Raspberry Pi 3. For Snowboy the runtime highly-depends on the keyword. Therefore,\nwe measured the CPU usage for each keyword and used the average.\n\n![](doc/img/cpu.png)\n\n## Data\n\n[LibriSpeech](http://www.openslr.org/12/) (test_clean portion) is used as background dataset. It can be downloaded\nfrom [OpenSLR](http://www.openslr.org/resources/12/test-clean.tar.gz).\n\nFurthermore, more than 300 recordings of six keywords (alexa, computer, jarvis, smart mirror, snowboy, and view glass)\nfrom more than 50 distinct speakers are used. The recordings are crowd-sourced. The recordings are stored within the\nrepository [here](audio).\n\nIn order to simulate real-world situations, the data is mixed with noise (at 10 dB SNR). For this purpose, we use\n[DEMAND](https://asa.scitation.org/doi/abs/10.1121/1.4799597) dataset which has noise recording in 18 different\nenvironments (e.g. kitchen, office, traffic, etc.). It can be downloaded from\n[Kaggle](https://www.kaggle.com/aanhari/demand-dataset).\n\n## Engines\n\nThree wake-word engines are used. [PocketSphinx](https://github.com/cmusphinx/pocketsphinx) which can\nbe installed using [PyPI](https://pypi.org/project/pocketsphinx/). [Porcupine](https://github.com/Picovoice/Porcupine)\nand [Snowboy](https://github.com/Kitt-AI/snowboy) which are included as submodules in this repository. The Snowboy engine\nhas a audio frontend component which is not normally a part of wake word engines and is considered a  separate part of\naudio processing chain. The other two engines have not such component in them. We enabled this component in Snowboy engine\nfor this benchmark as this is the optimal way of running it.\n\n## How to Reproduce?\n\n### Prerequisites\n\nThe benchmark has been developed on Ubuntu 20.04 with Python 3.8. Clone the repository using\n\n```console\ngit clone --recurse-submodules git@github.com:Picovoice/wakeword-benchmark.git\n```\n\nMake sure the Python packages in the [requirements.txt](/requirements.txt) are properly installed for your Python\nversion as Python bindings are used for running the engines. The repositories for Porcupine and Snowboy are cloned in\n[engines](/engines). Follow the instructions on their repositories to be able to run their Python demo before proceeding\nto the next step.\n\n### Running the Accuracy Benchmark\n\nUsage information can be retrieved via\n\n```console\npython3 benchmark.py -h\n```\n\nThe benchmark can be run using the following command from the root of the repository\n\n```console\npython3 benchmark.py \\\n--librispeech_dataset_path ${LIBRISPEECH_DATASET_PATH} \\\n--demand_dataset_path ${DEMAND_DATASET_PATH} \\\n--keyword ${KEYWORD} \\\n--access-key ${ACCESS_KEY}\n```\n\n### Running the Runtime Benchmark\n\nRefer to runtime [documentation](/runtime/README.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpicovoice%2Fwake-word-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpicovoice%2Fwake-word-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpicovoice%2Fwake-word-benchmark/lists"}