{"id":20164708,"url":"https://github.com/nourmorsy/home_service_robot","last_synced_at":"2026-04-29T12:41:15.287Z","repository":{"id":260348319,"uuid":"881025887","full_name":"nourmorsy/Home_Service_Robot","owner":"nourmorsy","description":"Code is designed to assist individuals with limited mobility by autonomously performing everyday tasks in home environments. Utilizing vision models, NLP, and language grounding","archived":false,"fork":false,"pushed_at":"2024-10-31T12:45:20.000Z","size":1674,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-13T14:31:43.541Z","etag":null,"topics":["6dof-pose","human-robot-interaction","llm","manipulation","motion-planning","multi","pddl","planning-algorithms","robotics","ros","vision-and-language"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nourmorsy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-30T19:34:19.000Z","updated_at":"2025-01-12T11:48:27.000Z","dependencies_parsed_at":"2024-10-30T21:19:39.888Z","dependency_job_id":"f29dd80f-1ade-4ba9-b715-b2ae5316768e","html_url":"https://github.com/nourmorsy/Home_Service_Robot","commit_stats":null,"previous_names":["nourmorsy/home_service_robot"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nourmorsy%2FHome_Service_Robot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nourmorsy%2FHome_Service_Robot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nourmorsy%2FHome_Service_Robot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nourmorsy%2FHome_Service_Robot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nourmorsy","download_url":"https://codeload.github.com/nourmorsy/Home_Service_Robot/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241600490,"owners_count":19988715,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["6dof-pose","human-robot-interaction","llm","manipulation","motion-planning","multi","pddl","planning-algorithms","robotics","ros","vision-and-language"],"created_at":"2024-11-14T00:35:31.037Z","updated_at":"2026-04-29T12:41:10.255Z","avatar_url":"https://github.com/nourmorsy.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Home Service Robot Project\n\n## Table of Contents\n- [Project Goal and Overview](#Project-Overview)\n- [System Architecture](#System-Architecture)\n- [Hardware Setup](#Hardware-Setup)\n- [Directory Structure](#Directory-Structure)\n- [Installation](#Installation)\n- [Usage](#Usage)\n- [Future Improvements](#Future-Improvements)\n- [Acknowledgments](#Acknowledgments)\n---\n\n## Project Goal and Overview\n\nThe **Home Service Robot** is a groundbreaking project aimed at assisting the elderly and individuals with limited mobility by autonomously performing daily tasks. This robot integrates cutting-edge **multi-vision models**, **natural language processing (NLP)**, and **language grounding techniques** to ensure seamless human-robot interaction in home environments.\n\n**Key components include:**\n- **Multi-Vision Models** for Enviroment recognition and detection\n- **Natural Language Understanding (NLU)** for command interpretation\n- **PDDL-based Planning** for decision-making and task execution\n- **MoveIt with 6DOF** for robotic arm manipulation\n\n\n---\n\n## System Architecture\n\n- **Human-Robot Interaction**: Communicate with the robot using natural language through a messaging interface.\n- **Integrated Multi-Vision Models**: Combines VQA, DOPE, and DINO models for rich environmental understanding.\n- **Language Grounding**: Utilizes NLU and language models to convert language into actionable tasks.\n- **Autonomous Navigation**: Leverages LiDAR and the ROS navigation stack for safe movement.\n- **Object Manipulation**: Uses MoveIt and a 5DOF robotic arm for complex object handling.\n\n![System Architecture](./System-arch.png)\n\n---\n\n## Hardware Setup\n\n---\n\n\u003cp align=\"left\"\u003e\n  \u003cimg src=\"./robot.png\" alt=\"Home Service Robot Hardware\" width=\"150\" align=\"right\"/\u003e\n\u003c/p\u003e\n\n\n- **RGBD Camera**: Provides color (RGB) and depth information for object detection, and human-robot interaction.\n- **4DOF Arm**: A 4-degree-of-freedom robotic arm that allows for object manipulation and grasping tasks.            \n- **RPLidar**: Used for precise environment mapping and obstacle avoidance, ensuring safe navigation.\n- **Mobile Base**: The mobile platform that enables movement across the environment.\n- **Onboard Computer**: Handles all processing tasks, running the robot's software stack, vision models, and navigation algorithms.  \n\n---\n\n## Directory Structure\n```\nHome_Service_robot/\n├── src/  # Source code directory\n│   ├── Classification   # Classification package\n│   ├── detection        # GroundingDINO package\n│   ├── dope             # Deep Pose estimation package\n│   ├── navigation       # Navigation package\n│   ├── VQA              # Visual question answering package\n│   ├── LLM              # Large Language Model package\n│   ├── planning         # PDDLStream package\n│   ├── manipulation                # Manipulation package\n│   ├── Porject_environment.yml     # List of project dependencies for conda enviroment\n│   └── DOPE.yml                    # List of Dope package dependencies for conda enviroment                 \n└── README.md                 # Project README\n```\n---\n## Installation\n\n### 1. Pre-installation Requirements\n\nEnsure the following software is installed on your system:\n- **ROS (Robot Operating System)**: Required for robot control and navigation.\n  - Installation instructions: [ROS Installation Guide](http://wiki.ros.org/ROS/Installation)\n- **Python 3.x**: The project requires Python 3 for compatibility with various models and libraries.\n- **Anaconda**: Recommended for managing Python environments.\n  - Installation instructions: [Anaconda Installation Guide](https://docs.anaconda.com/anaconda/install/)\n\n### 2. Clone the Repository\n\nStart by cloning this repository to your local machine:\n```bash\ngit clone https://github.com/yourusername/Home_Service_robot.git\ncd Home_Service_robot\ncatkin make \n```\n\n### 3. Set Up the Conda Environments\n\nThe project uses multiple Conda environments to manage dependencies for different modules.\n```bash\nconda env create -f Project_environment.yml #This file includes dependencies for NLP, navigation, and general utilities.\nconda env create -f dope.yml # This environment specifically handles the DOPE (Deep Object Pose Estimation) model dependencies.\n```\n\n### 4.  Download robot packages and AI Models\n\n- **Turtlebot package**: Required for robot control and navigation.\n  - Installation instructions: [Turtlebot Installation Guide](https://github.com/turtlebot)\n  - Place the model in main directory: ``` src/ ```\n- **DOPE Model**: Required for Object Pose Estimation.\n  - Installation instructions: [DOPE Installation Guide](https://github.com/NVlabs/Deep_Object_Pose)\n  - Place the model in ``` src/dope/ ```\n- **VQA Model**: Required for Visual Question Answering.\n  - Installation instructions: [VQA Installation Guide](https://github.com/dandelin/ViLT)\n  - Place the model in ``` src/VQA/ ```\n- **GroundingDINO Model**: Required for Object Detection.\n  - Installation instructions: [GroundingDINO Installation Guide](https://github.com/IDEA-Research/GroundingDINO)\n  - Place the model in ``` src/detection/ ```\n- **LLM Model**: Required for human robot language interaction.\n  - Installation instructions: [LLM Installation Guide](https://github.com/tincans-ai/gazelle)\n  - Place the model in ``` src/LLM/ ```\n- **PPDLStream Model**: Required for decision-making and task execution .\n  - Installation instructions: [LLM Installation Guide](https://github.com/caelan/pddlstream)\n  - Place the package in ``` src/planning/ ```\n-  **Finally**: in the main directory : ```catkin make ```\n\n---\n\n## Usage\n\n1. **LLM and Communication Node:**\n   - This node enables communication and language processing for user commands:\n     ```bash\n     conda activate Home_Service_robot\n     rosrun llm llm.py\n     ```\n2. **NLU (Natural Language Understanding) Node:**\n   - For command parsing and classification:\n     ```bash\n     conda activate Home_Service_robot\n     rosrun grounding rnn_classification.py\n     ```\n3. **Planning Node:**\n   - Use this node for planning tasks and goal management:\n     ```bash\n     conda activate Home_Service_robot\n     rosrun planning main_node.py\n     ```\n4. **Navigation Node:**\n   - For robot navigation, launch the navigation stack:\n     ```bash\n     roslaunch \u003crobot_bringup_package\u003e \u003crobot_bringup_launch\u003e.launch\n     roslaunch \u003crobot_navigation_package\u003e \u003cnavigation_launch\u003e.launch map_file:=\u003cpath_to_map\u003e\n     roslaunch \u003crviz_launchers_package\u003e \u003cview_navigation_launch\u003e.launch\n     rosrun navigation navigation_node.py\n     rosrun navigation move_motors_node.py\n     ```\n5. **Object Detection Node:**\n   - Run this node for camera activation and object detection using DINO:\n     ```bash\n     conda activate Home_Service_robot\n     roslaunch \u003crobot_camera_package\u003e \u003crobot_camera_launch\u003e.launch\n     rosrun detection read_camera.py\n     rosrun detection dino.py\n     ```\n6. **Vision-Language Model (VQA):**\n   - Use this node for visual question answering with the vision-language model:\n     ```bash\n     conda activate Home_Service_robot\n     rosrun vqa vqa_ros.py\n     ```\n7. **Manipulation Setup:**\n   - Set up the robotic arm and camera for manipulation tasks:\n     ```bash\n     roslaunch \u003crobot_arm_bringup_package\u003e arm_with_group.launch\n     roslaunch \u003crobot_arm_bringup_package\u003e moveit_bringup.launch\n     roslaunch astra_camera astra.launch\n     rosrun tf static_transform_publisher x, y, z, yaw, pitch, roll  camera_topic camera_topic_frame period_hz\n     ```\n8. **DOPE for Object Pose Estimation:**\n   - Activate the DOPE environment and launch for pose estimation tasks:\n     ```bash\n     conda activate dope\n     roslaunch dope dope.launch\n     ```\n9. **Manipulation Node:**\n   - Run this node to enable robotic arm manipulation:\n     ```bash\n     rosrun \u003crobot_arm_demos_package\u003e grasp.py\n     ```\n\n---\n\n## Future Improvements\n\n - **Decision Trees for Improved Planning**: We plan to incorporate decision trees to enhance task-planning capabilities, allowing the       robot to make smarter choices when performing tasks.\n - **Additional Multi-Modal Models**: Adding other vision and language models for improved perception and interaction.\n\n---\n\n## Acknowledgments\n\n\n  - This project is inspired by the work of [**Sebastian Castro**](https://github.com/sea-bass). His contributions and insightful blog       post, [2020 Review: Service Robotics – MIT CSAIL](https://roboticseabass.com/2020/12/30/2020-review-service-robotics-mit-csail/),       provided invaluable inspiration and guidance for our team.\n  - We extend our gratitude for his dedication to advancing the field of robotics, which has greatly influenced our approach and the         development of this project.\n\n---\n    \n    \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnourmorsy%2Fhome_service_robot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnourmorsy%2Fhome_service_robot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnourmorsy%2Fhome_service_robot/lists"}