{"id":13729132,"url":"https://github.com/ms-iot/ros_msft_luis","last_synced_at":"2025-05-08T01:32:08.663Z","repository":{"id":46302411,"uuid":"160089214","full_name":"ms-iot/ros_msft_luis","owner":"ms-iot","description":"This ROS node interfaces Robotics with the Microsoft Azure Language Understanding Service. This ROS node works on both Windows and Linux.","archived":true,"fork":false,"pushed_at":"2023-01-17T23:17:15.000Z","size":24548,"stargazers_count":5,"open_issues_count":2,"forks_count":6,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-08-04T02:08:32.689Z","etag":null,"topics":["azure","luis","ros"],"latest_commit_sha":null,"homepage":"http://aka.ms/ros","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ms-iot.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-12-02T20:23:03.000Z","updated_at":"2024-07-22T17:17:39.000Z","dependencies_parsed_at":"2023-02-10T12:30:36.627Z","dependency_job_id":null,"html_url":"https://github.com/ms-iot/ros_msft_luis","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ms-iot%2Fros_msft_luis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ms-iot%2Fros_msft_luis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ms-iot%2Fros_msft_luis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ms-iot%2Fros_msft_luis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ms-iot","download_url":"https://codeload.github.com/ms-iot/ros_msft_luis/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224688932,"owners_count":17353287,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azure","luis","ros"],"created_at":"2024-08-03T02:00:55.452Z","updated_at":"2024-11-14T20:30:35.068Z","avatar_url":"https://github.com/ms-iot.png","language":"C++","funding_links":[],"categories":["Tools for Robotics","Robotics Tools and Frameworks","Tools"],"sub_categories":["In-memory data grids","Winetricks","Objective-C Tools, Libraries, and Frameworks","Mesh networks"],"readme":"# Azure Cognitive Services LUIS ROS Node\nThis ROS node bridges between ROS and the Azure Language Understanding Service. The ROS node can be configured to process audio directly from a microphone, or can subscribe to a ROS audio topic, then processes speech and generates \"intent\" ROS messages which can be processed by another ROS node to generate ROS commands.\n\n\u003e NOTE: This ROS node is in preview.\n\n## CI Status\n![Windows CI](https://github.com/ms-iot/ros_msft_luis/workflows/CI/badge.svg)\n\n## Working with LUIS on ROS1\nThis ROS node works on ROS1 Melodic forward, with Linux and Windows.\n\n\u003eDuring the preview, this ROS node will be provided as sourcecode. \n\n### Building on Windows\nThe Makefile for the LUIS ROS Node will download the LUIS SDK using Nuget.exe when it is present in the path.\n\nPlease install ROS1 per the instructions at [Microsoft's ROS Landing page](http://aka.ms/ros)\n\n``` batch\nmkdir c:\\ws\\luis_ws\\src\ncd c:\\ws\\luis_ws\\src\ngit clone --recursive https://github.com/ms-iot/ros_msft_luis\ngit clone https://github.com/ms-iot/audio_common\n\ncd c:\\ws\\luis_ws\ncatkin_make\ndevel\\setup.bat\n```\n\n### Building on Ubuntu\n\n\n``` batch\nsudo apt-get update\nsudo apt-get install build-essential libssl1.0.0 libasound2 wget\n\nmkdir ~/luis_ws/src\ncd ~/luis_ws/src\ngit clone --recursive https://github.com/ms-iot/ros_msft_luis\n\ncd ~/luis_ws\ncatkin_make\nsource devel/setup.bash\n```\n### Running the LUIS ROS Node\n\nTo get started with LUIS and the Speech Services, check [this guideline](./docs/walkthrough-luis-config.md).\n\n APP ID -  In Manage Tab --\u003e Settings --\u003e App ID\n  \n Primary Key - In the Manage Tab --\u003e Azure Resources --\u003e Prediction Resources --\u003e Primary Key\n  \n Location - In the Manage Tab --\u003e Azure Resources --\u003e Rediction Resources --\u003e Location\n\nThe ROS node requires the following information from the  portal of [speech studio](https://speech.microsoft.com/portal?noredirect=true)\n\n\n Speech Resource Key\n  \n Region information example \"westus\" if it is \"West US\", \"westus2\" if it is \"West US 2\"\n\n\n\nThe LUIS ROS node can be configured two ways - by embedded in the Azure resource keys in the launch file, or setting them in the environment.\n\n\u003e if you are committing launch files to an open source repo, it is best to use the environment method as to not leak your keys.\n\n\nWindows:\n``` batch\nset azure_cs_luis_appid=\u003cEnter your APP ID mentioned above\u003e\nset azure_cs_luis_key=\u003cEnter your Primary Key mentioned above\u003e\nset azure_cs_luis_region=\u003cEnter your location mentioned above\u003e\nset azure_cs_kw_key=\u003cEnter your key mentioned above from the speech studio\u003e\nset azure_cs_kw_region=\u003cEnter your location mentioned above from the speech studio\u003e\nset azure_cs_kw_path=\u003cEnter the location of the of .table file that you downloaded from above\u003e\nset azure_cs_stop_kw_path=\u003cEnter the location of the of .table file of the Stop Keyword that you downloaded from above\u003e\nset azure_cs_kw=\u003cEnter your custom keyword\u003e\nset azure_cs_stop_kw=\u003cEnter your stop custom keyword\u003e\n```\n\u003e NOTE: You can use the command `setx` instead of `set` to save this to the system environment. However, you'll have to recycle your command window. \n\nUbuntu:\n``` bash\nexport azure_cs_luis_appid=\u003cEnter your APP ID mentioned above\u003e\nexport azure_cs_luis_key=\u003cEnter your Primary Key mentioned above\u003e\nexport azure_cs_luis_region=\u003cEnter your location mentioned above\u003e\nexport azure_cs_kw_key=\u003cEnter your key mentioned above from the speech studio\u003e\nexport azure_cs_kw_region=\u003cEnter your location mentioned above from the speech studio\u003e\nexport azure_cs_kw_path=\u003cEnter the location of the of .table file that you downloaded from above\u003e\nexport azure_cs_stop_kw_path=\u003cEnter the location of the of .table file of the Stop Keyword that you downloaded from above\u003e\nexport azure_cs_kw=\u003cEnter your custom keyword\u003e\nexport azure_cs_stop_kw=\u003cEnter your stop custom keyword\u003e\n```\n\u003e NOTE: You may wish to place this in your .shellrc file so it is available in each terminal. \n\n```\nroslaunch ros_msft_luis luis.launch\n```\n\nIf you would like to use a custom microphone, such as the Respeaker which is available on the [Hello Robot](https://hello-robot.com/product), you can set the subscription topic in the launch file:\n\n``` xml\n\u003claunch\u003e\n  \u003cnode name=\"luis_test\" pkg=\"ros_msft_luis\" type=\"ros_msft_luis_node\" output=\"screen\"\u003e\n    \u003carg name=\"mic_topic\" value=\"/audio\"\u003e\n  \u003c/node\u003e\n\u003c/launch\u003e\n```\n\n### Using the `move_base` node\n\nThe `ros_msft_luis_move_base` node is provided to translate the intents into `move_base` navigation goals. It currently understands the following intents:\n\n- \"Move Forward\" / \"Move Backward\", followed by a distance expressed in meters, feet or yards. For example: \"move forward 2 meters\", \"move backward 3 feet\".\n- \"Turn Left\" / \"Turn Right\", followed by an angle expressed in degrees. For example: \"turn left 34 degrees\". If no angle is given, a default of 90 degrees will be used, so you can just say \"turn right\" for example.\n- \"Stop\", which cancels all current navigation.\n\nTo launch the node, you can use the provided launch configuration:\n\n```\nroslaunch ros_msft_luis luis_move_base.launch\n```\n\n### Using the `moveit` node\n\nThe `ros_msft_luis_moveit` node is provided to translate the intents into [MoveIt](https://moveit.ros.org/) movements. It was developed with a simulated [xArm](https://github.com/xArm-Developer/xarm_ros) robotic arm, and thus has some hard-coded values in the code that are specific to this device, in `ros_msft_luis_moveit/src/ros_msft_luis_moveit.cpp`:\n\n```cpp\nstatic const std::string PLANNING_GROUP_ARM = \"xarm7\";\nstatic const std::string PLANNING_GROUP_HAND = \"xarm_gripper\";\n```\n\nIf you are using a different robotic arm, you will need to adjust these values to match the names of the planning groups.\n\nThe node currently implements the following intents:\n\n- Arm movements: \"Forward\", \"Backward\", \"Up\", \"Down\", \"Left\", \"Right\". These directions can be followed by a length expressed in centimeters, feet or inches. For example: \"raise arm one foot\", \"move arm backward 3 inches\".\n- Hand movements: \"Open\" and \"Close\". Note that these two intents are using pre-defined named targets that are probably specific to the [xArm grippper](https://github.com/xArm-Developer/xarm_ros).\n- \"Stop\", which stops all movements.\n\nTo launch the node, you can use the provided launch configuration:\n\n```\nroslaunch ros_msft_luis luis_moveit.launch\n```\n\n### Using containers with ROS\n\nThe Azure Cognitive Services support edge deployments using [containers](https://docs.microsoft.com/en-us/azure/cognitive-services/cognitive-services-container-support), including Speech-to-Text and Language Understanding Service (LUIS). These models can be deployed directly to the robot if there are sufficient resources, or can be deployed to an edge server or Kubernetes cluster.\n\n**Starting the Speech-to-Text container**\n\nYou will first need to run the Speech container. Please refer to the [documentation for using Speech Services containers](https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/speech-container-howto?tabs=stt%2Ccsharp%2Csimple-format).\n\nYou will need to select a local port to expose the Speech service; in this example, 5000.\n\n``` shell\ndocker run --rm -it -p 5000:5000 \\\n--memory 4g --cpus 4 \\\nmcr.microsoft.com/azure-cognitive-services/speechservices/speech-to-text \\\nEula=accept \\\nBilling=\u003cyour endpoint URI\u003e \\\nApiKey=\u003cyour API key\u003e\n\n```\n\n**Starting the LUIS container**\n\nThe language model you have defined is exported from the Azure LUIS portal and injected into the container when started. Please refer to the [documentation for using a LUIS Container](https://docs.microsoft.com/en-us/azure/cognitive-services/LUIS/luis-container-howto?tabs=v3).\n\nYou will find below an example running the LUIS node using Docker. In this case, the exported language model has been downloaded into the ROS workspace root `c:\\ws\\luis_ws`, and mounted into the `/input` and `/output` directory. (the language model is an input, logs are output).\n\nMake sure to use a different local port than the one used for the Speech container: in this example, 5001.\n\n``` batch\ndocker run --rm -it -p 5001:5000 ^\n--memory 4g ^\n--cpus 2 ^\n--mount type=bind,src=c:\\ws\\luis_ws,target=/input ^\n--mount type=bind,src=c:\\ws\\luis_ws,target=/output ^\nmcr.microsoft.com/azure-cognitive-services/luis ^\nEula=accept ^\nBilling=https://westus.api.cognitive.microsoft.com/ ^\nApiKey=\u003cyour API key\u003e ^\nLogging:Console:LogLevel:Default=Debug\n```\n\n**Starting the ROS node**\n\nThe containers expose endpoints in the form of URLs like these:\n\n- LUIS: `http://localhost:5001`\n- Speech: `ws://localhost:5000/speech/recognition/conversation/cognitiveservices/v1`\n\nIn order for the ROS node to use these endpoints, they must be configured via additional environment variables (which are shared by all ROS LUIS instances) or by ladditional parameters in the aunch file (which could allow multiple concurrent language models).\n\nMake sure to use the right local ports, defined when running the containers!\n\nIf you chose to use environment variables, they can be configured in your launch shell or in the system by setting the following environment variables:\n\nWindows:\n\n``` batch\nset azure_cs_luis_endpoint=http://host:port\nset azure_cs_speech_endpoint=ws://host:port/speech/recognition/conversation/cognitiveservices/v1\n```\n\nUbuntu:\n\n``` bash\nexport azure_cs_luis_endpoint=http://host:port\nexport azure_cs_speech_endpoint=ws://host:port/speech/recognition/conversation/cognitiveservices/v1\n```\n\nYou can also specify the endpoint parameters in the XML launch file:\n\n``` xml\n\u003claunch\u003e\n  \u003cnode name=\"luis_test\" pkg=\"ros_msft_luis\" type=\"ros_msft_luis_node\" output=\"screen\"\u003e\n    \u003cparam name=\"luisendpoint\" value=\"\u003cendpoint\u003e\" /\u003e\n    \u003cparam name=\"speechendpoint\" value=\"\u003cendpoint\u003e\" /\u003e\n    \u003c!-- other configuration parameters as defined above --\u003e\n  \u003c/node\u003e\n\u003c/launch\u003e\n```\n\nYou can then run the ROS node as previously.\n\n```\nroslaunch ros_msft_luis luis.launch\n```\n\n### Microphone array support\n\nThe LUIS Ros node supports two types of microphones:\n\n- External microphone topic from another node\n- Embedded microphone - which is [4 channel ReSpeaker MicArray V2.0](https://wiki.seeedstudio.com/ReSpeaker_Mic_Array_v2.0/)\n\nTo enable the support of the Microphone array, you need to specify the `DeviceGeometry` and the `SelectedGeometry` variables in the [azure_cs_luis.cpp](./ros_msft_luis/src/azure_cs_luis.cpp) file.\n\n## Working with LUIS on ROS2\n\n*Coming in Fall 2020*\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fms-iot%2Fros_msft_luis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fms-iot%2Fros_msft_luis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fms-iot%2Fros_msft_luis/lists"}