{"id":24098467,"url":"https://github.com/matlab-deep-learning/speech-based-information-retrieval","last_synced_at":"2025-06-26T16:33:57.619Z","repository":{"id":74608413,"uuid":"588557359","full_name":"matlab-deep-learning/speech-based-information-retrieval","owner":"matlab-deep-learning","description":"Retrieve answers from a knowledge base via speech recognition and information retrieval.","archived":false,"fork":false,"pushed_at":"2023-01-13T12:29:42.000Z","size":915,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-05-07T19:47:39.976Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"MATLAB","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/matlab-deep-learning.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-01-13T12:14:42.000Z","updated_at":"2023-02-18T17:34:33.000Z","dependencies_parsed_at":null,"dependency_job_id":"1b52ec2f-e033-4efb-a3f1-d99e72f01a2b","html_url":"https://github.com/matlab-deep-learning/speech-based-information-retrieval","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/matlab-deep-learning/speech-based-information-retrieval","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matlab-deep-learning%2Fspeech-based-information-retrieval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matlab-deep-learning%2Fspeech-based-information-retrieval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matlab-deep-learning%2Fspeech-based-information-retrieval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matlab-deep-learning%2Fspeech-based-information-retrieval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/matlab-deep-learning","download_url":"https://codeload.github.com/matlab-deep-learning/speech-based-information-retrieval/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matlab-deep-learning%2Fspeech-based-information-retrieval/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262102644,"owners_count":23259302,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-10T14:46:07.537Z","updated_at":"2025-06-26T16:33:57.606Z","avatar_url":"https://github.com/matlab-deep-learning.png","language":"MATLAB","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Speech-Based Information Retrieval for Answering Science Questions\n![](images/SpeechToTextInformationRetrieval.png)\n\n## This example shows how to retrieve answers from a knowledge base via speech recognition and information retrieval:\n* Record and read audio data in MATLAB\u0026reg; using the [Audio Toolbox\u0026trade;](https://www.mathworks.com/products/audio.html)\n* Translate questions to English using Hugging Face\u0026reg; Transformers using co-execution with Python\u0026reg;\n* Retrieve answers for science questions using Information Retrieval techniques via [Text Analytics Toolbox\u0026trade;](https://www.mathworks.com/products/text-analytics.html)\n\n## Requirements \n\n* [MATLAB\u0026reg; R2022a or later](https://www.mathworks.com/products/matlab.html)\n* [Text Analytics Toolbox\u0026trade;](https://www.mathworks.com/products/text-analytics.html)\n* [Audio Toolbox\u0026trade;](https://www.mathworks.com/products/audio.html)\n    * [speech2text](https://www.mathworks.com/matlabcentral/fileexchange/65266-speech2text)\n    * [text2speech](https://www.mathworks.com/matlabcentral/fileexchange/73326-text2speech)\n* [Python\u0026reg; 3.8+](https://www.python.org/downloads/)\n* [PyTorch\u0026reg; 1.11.0](https://pytorch.org/)\n* [Hugging Face\u0026reg; Transformers 4.17.0+](https://huggingface.co/docs/transformers/index)\n\n## Setup\n\nIn this example, you use the train split of the QASC Data Set [1] as a knowledge base for pairs of answers and questions. This dataset contains questions and answers about grade school science, such as \"Fossils are formed when what happens?\".\nWhile the dataset contains questions and multiple alternatives for the correct answer, the preprocessing step filters only the correct answer.\n\nTo download the data set and preprocess the data, run the script `downloadAndPreprocessData`. \n```Matlab:Code(Display)\n\u003e\u003e downloadAndPreprocessData\n```\n\nThe script saves the preprocessed data to the file `questions_answers.mat` inside the `text_data` folder.\n\n## Overview\n\nUsing this example, you can ask questions and retrieve answers with the following steps:\n\n1. Ask your question using an audio input device, such as your microphone, or using a pre-recorded question \n2. Use the speech2text functionality to convert your question to text form\n2. If you want to ask a question in a different language, use a pretrained transformer model to translate your question to English\n3. Compare your question with the questions in the knowledge base and retrieve the most similar question and answer using TF-IDF\n4. If you want your answer in a different language, translate it again using the transformer model\n5. Use the text2speech functionality to convert your answer to audio form and output it\n\n## Asking Questions in English\n\n![](images/EnglishIRExample.png)\n\nIf you want to ask questions in English, you don't have to translate them, since the knowledge base is already in English.\n\nFor asking questions while the script is running, you will have to use an input device and ask your question when you see the message 'Start speaking.' after running the following script:\n\n```Matlab:Code(Display)\n\u003e\u003e englishRetrieval\n```\n\nIf you have a pre-recorded question, you can also use this example. You can find some pre-recorded English questions examples inside `audio_data\\en` and run the example using:\n\n```Matlab:Code(Display)\n\u003e\u003e audioFileEnglishRetrieval\n```\n\nFor using your own files as input, you can change the variable `inputAudioFile` to your file location.\n\n## Asking Questions in Other Languages\n\n![](images/NonEnglishIRExample.png)\n\nIf you want to ask question in another language, other than English, you will need to translate your question to English. This is done in this example using a pretrained transformer model from Hugging Face.\nIn this example, we show how to ask questions and get answers in German, which is a language supported by Text Analytics Toolbox. However, you can change the variable `inputLanguage` to different language codes if you want.\n\nAgain, for asking questions while the script is running, you will have to use an input device and ask your question when you see the message 'Begin recording...' after running the following script:\n\n```Matlab:Code(Display)\n\u003e\u003e nonEnglishRetrieval\n```\n\nSimilarly, if you have a pre-recorded question in a different language than English, you can also use this example. You can find some pre-recorded German questions examples inside `audio_data\\de` and run the example using:\n\n```Matlab:Code(Display)\n\u003e\u003e englishRetrieval\n```\n\n\n## References\n\n[1] Khot, T., Clark, P., Guerquin, M., Jansen, P., \u0026 Sabharwal, A. (2020, April). Qasc: A dataset for question answering via sentence composition. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 05, pp. 8082-8090).\n\nCopyright 2022, The MathWorks, Inc.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatlab-deep-learning%2Fspeech-based-information-retrieval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmatlab-deep-learning%2Fspeech-based-information-retrieval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatlab-deep-learning%2Fspeech-based-information-retrieval/lists"}