https://github.com/yasirudex/slam-ros2-bot-working

Explore a project that develops a SLAM-based navigation system using vision-language data inputs. This project integrates natural language vocal instructions and image feeds to guide a differential drive robot equipped with a Kinect V2 sensor through dynamic environments.
https://github.com/yasirudex/slam-ros2-bot-working

bert-model blip huggingface-transformers kinect2-sensor llms nav2 pointclouds ros2 slam urdf vision

Last synced: 3 months ago
JSON representation

Explore a project that develops a SLAM-based navigation system using vision-language data inputs. This project integrates natural language vocal instructions and image feeds to guide a differential drive robot equipped with a Kinect V2 sensor through dynamic environments.

• Host: GitHub
• URL: https://github.com/yasirudex/slam-ros2-bot-working
• Owner: YasiruDEX
• License: apache-2.0
• Created: 2024-06-12T18:42:38.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2024-09-07T09:49:52.000Z (about 1 year ago)
• Last Synced: 2025-04-10T18:05:13.171Z (6 months ago)
• Topics: bert-model, blip, huggingface-transformers, kinect2-sensor, llms, nav2, pointclouds, ros2, slam, urdf, vision
• Language: Python
• Homepage: https://slam-ros2-bot-working.vercel.app
• Size: 207 KB
• Stars: 4
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE.md

Awesome Lists containing this project

README

          
SLAM-based Navigation System with Vision-Language Inputs


Explore the repository for a cutting-edge project aimed at developing a SLAM-based navigation system using vision-language data inputs. The project integrates natural language vocal instructions and image feeds to guide a differential drive robot equipped with a Kinect V2 sensor through dynamic environments.


Key Features:



  


URDF Model: Custom differential drive robot with Kinect V2 sensor.

  


ROS2 Control: Differential drive and position controllers.

  


ROS SLAM Toolbox: Utilizes Xbox Kinect sensor data for effective mapping.

  


Teleoperation: Control via Xbox game controller with seamless multiplexer support.

  


Frontend: Next.js interface for visual and natural language input.



Machine Learning Architecture:



  
Multi-model approach integrating NLP and computer vision.

  
NLP with vocal data using OpenAI Whisper and BLIP for image feeds.

  
Mistral 7B large language model quantized to int8 for robot action prediction.

  
Utilizes Huggingface Transformers and CTransformers for local deployment.



Localization and Navigation:



  
AMCL for localization with prerecorded environment maps.

  
Nav2 stack for precise navigation to goal destinations.

  
ROS 2 control for armature actions.



Tools & Platforms:

Acknowledging the support of ROS 2 Iron Irwini, Ubuntu 22.04 Jammy Jellyfish, Gazebo, HuggingFace Transformers, Salesforce, Whisper, Nav2, and Mistral AI for making this project possible.


Check out the GitHub repository for more details and code implementation.


Stay tuned for LUNA, the broader real-world implementation of this SLAM project!


🔗 Video explanation: Explore Here


#Robotics #AI #ROS2 #MachineLearning #SLAM #VLN #Innovation #TechProjects