https://github.com/lionsoul2014/freeswitch-asr
TTS and ASR module with auto Voice Active Detecting supported for Freeswitch. I build it for Nature sound interactive, With the embedded LUA engine we could easly build a Freeswtich application like this.
https://github.com/lionsoul2014/freeswitch-asr
call-center freeswitch freeswitch-asr freeswitch-tts ivr-application lua-script
Last synced: about 1 year ago
JSON representation
TTS and ASR module with auto Voice Active Detecting supported for Freeswitch. I build it for Nature sound interactive, With the embedded LUA engine we could easly build a Freeswtich application like this.
- Host: GitHub
- URL: https://github.com/lionsoul2014/freeswitch-asr
- Owner: lionsoul2014
- License: apache-2.0
- Created: 2019-07-21T04:49:19.000Z (almost 7 years ago)
- Default Branch: master
- Last Pushed: 2019-07-21T09:46:27.000Z (almost 7 years ago)
- Last Synced: 2025-04-30T14:29:13.499Z (about 1 year ago)
- Topics: call-center, freeswitch, freeswitch-asr, freeswitch-tts, ivr-application, lua-script
- Language: C
- Homepage:
- Size: 22.5 KB
- Stars: 69
- Watchers: 8
- Forks: 38
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE.md
Awesome Lists containing this project
README
# Description
TTS and ASR module with auto Voice Active Detecting supported for Freeswitch.
I build it for Nature sound interactive, With the embedded LUA engine we could easly build a Freeswtich application like this.
# How to use it ?
1. Copy and merge the ${PROJECT_ROOT}/src and the ${PROJECT_ROOT}/conf with the Freeswitch source tree.
2. Add the following two module in the ${FREESWITCH_SOURCE_ROOT}/modules.conf
```
asr_tts/mod_yytts
asr_tts/mod_yyasr
```
3. Re-compile and install the freeswitch to install mod_yytts and mod_yyasr modules.
4. Active the mod_yytts and the mod_yyasr by add the following to lines into the ${FREESWITCH_INSTALLATION_ROOT}/conf/autoload_configs/modules.conf.xml
```
```
5. Copy the lua scripts under ${PROJECT_ROOT}/scripts to ${FREESWITCH_INSTALLATION_ROOT}/scripts/
6. Bind a number to build application by adding the following xml settings to the ${FREESWITCH_INSTALLATION_ROOT}/conf/dialplan/default.xml
```
```
# How to test it ?
Start the Freeswitch and install a Linphone client.
1. Dial 5001 for TTS testing, check ./scripts/yytts_demo.lua for detail usage, module interface overview:
```lua
session:answer(); -- answer the call
-- set to use the yytts TTS with zhilingf speaker
session:set_tts_params("yytts", "zhilingf");
-- fire the speak
session:speak("Hello，我是人工智能电话系统, Bye!");
-- hangup the call
session:hangup();
```
2. Dial 5002 for ASR testing, check ./scripts/yyasr_demo for detail usage, ASR interface overview:
```lua
session:answer(); -- answer the call
-- register the sound input callback
session:setInputCallback(function onInput(s, type, obj) end);
-- start to detect the speech to keep firing the feed interface insite yyasr module
session:execute("detect_speech", "yyasr directory directory");
-- resume the detect_speech
session:execute("detect_speech", "resume");
-- stop the detecting
session:execute("detect_speech", "stop");
-- check ./script/yyasr_demo for details usage.
```
# Secondary development
### TTS module
1. TTS module structure:
```c
*module_interface = switch_loadable_module_create_module_interface(pool, modname);
tts_interface = switch_loadable_module_create_interface(*module_interface, SWITCH_SPEECH_INTERFACE);
tts_interface->interface_name = "yytts";
tts_interface->speech_open = yytts_speech_open;
tts_interface->speech_close = yytts_speech_close;
tts_interface->speech_feed_tts = yytts_speech_feed_tts;
tts_interface->speech_read_tts = yytts_speech_read_tts;
tts_interface->speech_flush_tts = yytts_speech_flush_tts;
tts_interface->speech_text_param_tts = yytts_text_param_tts;
tts_interface->speech_numeric_param_tts = yytts_numeric_param_tts;
tts_interface->speech_float_param_tts = yytts_float_param_tts;
```
2. Invoke process for TTS module in Freeswitch:
```
open
while ( thread running ) do
feed
read
flush
end
close
```
3. Embedded the other TTS engine by override the yytts_speech_feed_tts
```c
static switch_status_t yytts_speech_feed_tts(switch_speech_handle_t *sh, char *text, switch_speech_flag_t *flags)
{
// feed the wave stream to the yytts->audio_buffer buffer
return SWITCH_STATUS_SUCCESS;
}
```
### ASR module
1. ASR module structure:
```
asr_interface = switch_loadable_module_create_interface(*module_interface, SWITCH_ASR_INTERFACE);
asr_interface->interface_name = "yyasr";
asr_interface->asr_open = yyasr_asr_open;
asr_interface->asr_load_grammar = yyasr_asr_load_grammar;
asr_interface->asr_unload_grammar = yyasr_asr_unload_grammar;
asr_interface->asr_close = yyasr_asr_close;
asr_interface->asr_feed = yyasr_asr_feed;
asr_interface->asr_resume = yyasr_asr_resume;
asr_interface->asr_pause = yyasr_asr_pause;
asr_interface->asr_check_results = yyasr_asr_check_results;
asr_interface->asr_get_results = yyasr_asr_get_results;
asr_interface->asr_start_input_timers = yyasr_asr_start_input_timers;
asr_interface->asr_text_param = yyasr_asr_text_param;
asr_interface->asr_numeric_param = yyasr_asr_numeric_param;
asr_interface->asr_float_param = yyasr_asr_float_param;
```
2. Invoke process for ASR module in Freeswitch:
```
open
while ( thread running ) do
feed
check_results
get_results
end
optional pause|resume
close
```
3. Embedded the other ASR engine by override the yyasr_asr_feed
```c
/*! function to feed audio to the ASR */
static switch_status_t yyasr_asr_feed(
switch_asr_handle_t *ah, void *data, unsigned int len, switch_asr_flag_t *flags)
{
// feed the text to the yyasr->text_buffer text buffer
return SWITCH_STATUS_SUCCESS;
}
/*
* Voice Active detecting implementation.
* Optimize the VAD by modify stop_audio_detect function.
*/
static switch_bool_t stop_audio_detect(
switch_asr_handle_t *ah, int16_t *data, unsigned int samples)
{
return SWITCH_FALSE;
}
```