{"id":25761226,"url":"https://github.com/rorosaga/holo-cierge","last_synced_at":"2026-04-30T11:37:20.360Z","repository":{"id":253033889,"uuid":"823808025","full_name":"rorosaga/holo-cierge","owner":"rorosaga","description":"🛎️ Transforming guest experiences with a 3D digital concierge! 🌟 Dive into immersive interactions with holographic displays, real-time speech recognition, lifelike animations, and AI-driven responses. Powered by ChatGPT, Eleven Labs, and GuestHub integration. 🤖💬✨  ","archived":false,"fork":false,"pushed_at":"2024-12-06T21:39:46.000Z","size":118166,"stargazers_count":1,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-26T14:50:54.017Z","etag":null,"topics":["3d-avatar","chatgpt","concierge-bot","digitalconcierge","elevenlabs","guesthub","holographicdisplays","hotelmanagement","looking-glass","lookingglass","mixamo","nodejs","reactjs","rhubarblipsync","threejs","webapp","whisperai"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rorosaga.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-07-03T19:00:05.000Z","updated_at":"2025-04-01T12:27:40.000Z","dependencies_parsed_at":"2025-06-26T14:43:43.715Z","dependency_job_id":null,"html_url":"https://github.com/rorosaga/holo-cierge","commit_stats":null,"previous_names":["rorosaga/holo-cierge"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/rorosaga/holo-cierge","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rorosaga%2Fholo-cierge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rorosaga%2Fholo-cierge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rorosaga%2Fholo-cierge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rorosaga%2Fholo-cierge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rorosaga","download_url":"https://codeload.github.com/rorosaga/holo-cierge/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rorosaga%2Fholo-cierge/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32463892,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-29T22:27:22.272Z","status":"online","status_checked_at":"2026-04-30T02:00:05.929Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-avatar","chatgpt","concierge-bot","digitalconcierge","elevenlabs","guesthub","holographicdisplays","hotelmanagement","looking-glass","lookingglass","mixamo","nodejs","reactjs","rhubarblipsync","threejs","webapp","whisperai"],"created_at":"2025-02-26T18:29:47.719Z","updated_at":"2026-04-30T11:37:15.350Z","avatar_url":"https://github.com/rorosaga.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Holo-cierge 👨‍💼🛎️\n\n\u003cp\u003e\n    \u003cimg alt=\"JavaScript\" src=\"https://img.shields.io/badge/javascript-%23323330.svg?style=for-the-badge\u0026logo=javascript\u0026logoColor=%23F7DF1E\" /\u003e\n    \u003cimg alt=\"React\" src=\"https://img.shields.io/badge/React-20232A?style=for-the-badge\u0026logo=react\u0026logoColor=61DAFB\" /\u003e\n    \u003cimg alt=\"ThreeJS\" src=\"https://img.shields.io/badge/ThreeJs-black?style=for-the-badge\u0026logo=three.js\u0026logoColor=white\" /\u003e\n    \u003cimg alt=\"NodeJS\" src=\"https://img.shields.io/badge/Node%20js-339933?style=for-the-badge\u0026logo=nodedotjs\u0026logoColor=white\" /\u003e\n    \u003cimg alt=\"Yarn\" src=\"https://img.shields.io/badge/Yarn-2C8EBB?style=for-the-badge\u0026logo=yarn\u0026logoColor=white\" /\u003e\n    \u003cimg alt=\"Python\" src=\"https://img.shields.io/badge/Python-3776AB?style=for-the-badge\u0026logo=python\u0026logoColor=yellow\" /\u003e\n    \u003cimg alt=\"OpenAI\" src=\"https://img.shields.io/badge/ChatGPT-74aa9c?style=for-the-badge\u0026logo=openai\u0026logoColor=white\" /\u003e\n    \u003cimg alt=\"FFmpeg\" src=\"https://shields.io/badge/FFmpeg-%23171717.svg?logo=ffmpeg\u0026style=for-the-badge\u0026labelColor=171717\u0026logoColor=5cb85c\" /\u003e\n    \u003cimg alt=\"Docker\" src=\"https://img.shields.io/badge/docker-%230db7ed.svg?style=for-the-badge\u0026logo=docker\u0026logoColor=white\" /\u003e\n\u003c/p\u003e\n\n**Holo-cierge** is an innovative 3D digital concierge that bridges cutting-edge AI, immersive holographic display technology, and advanced animation to redefine guest services in hospitality. The system leverages **`ReactJS`**, **`Three.js`**, and **`Node.js`** to create an interactive, lifelike concierge experience compatible with [**Looking Glass**](https://lookingglassfactory.com/) holographic displays. By integrating [**OpenAI Whisper**](https://github.com/openai/whisper) for speech recognition and [**OpenAI ChatGPT**](https://openai.com/index/chatgpt/) as its core LLM, Holo-cierge can interpret guest inquiries, respond conversationally, and perform real-world actions such as creating tickets for hotel services through its integration with [**GuestHub**](https://guesthub.io/) hotel management application. Used **Docker** for backend containerization and deployment.\n\n\u003cp\u003e\n    \u003cimg src=\"media/demo2.gif\" alt=\"drawing\" width=\"477.5\"/\u003e\n\u003c/p\u003e\n\n## 📚 **Table of Contents**\n1. [Overview](#overview)\n2. [Features](#features)\n3. [Installation](#installation)\n4. [Usage](#usage)\n5. [Tech-Stack](#tech-stack)\n6. [Acknowledgements](#acknowledgements)\n7. [Resources](#resources)\n\n## Overview 🌠\n\nHolo-cierge represents a leap forward in guest experience management by combining the power of AI and immersive 3D environments. Key highlights include:\n\n- A **lifelike digital concierge** with Mixamo animations for fluid, responsive movement. 💃🤖\n- **Speech-to-text integration** powered by **Whisper** for seamless voice interactions. 🎤📝\n- **Dynamic speech generation** via **Eleven Labs**, enabling natural, expressive dialogue. 🗨️🔊\n- **GuestHub integration** for automated service ticket creation and tracking. 🏨🛎️  \n- **Lipsync precision** powered by the **Rhubarb library** with rapid audio conversions using **FFmpeg**. 🎥🎶\n\n## Features 🚀\n\n- **Interactive Holographic Concierge:** Designed for Looking Glass displays, offering an immersive guest interaction experience.\n- **Real-time Speech Recognition:** Locally runs Whisper for accurate and private voice processing.\n- **Dynamic Speech Generation:** Text responses generated by ChatGPT are vocalized using Eleven Labs, creating a unique, customizable concierge voice.\n- **Natural Movement and Expressions:** Incorporates Mixamo animations for responsive life-like movements and Rhubarb lipsync for precise speech synchronization.\n- **GuestHub Integration:** Supports real-world actions like creating and managing hotel service tickets.\n- **Developer-Friendly Architecture:** Modular backend and frontend design with reusable components.\n\n## Installation 🛠️\n\n### Prerequisites 📦\n\n- Node.js and Yarn for dependency management.\n- Python 3.8+ for running Whisper.\n- FFmpeg for audio processing.\n- Rhubarb Library for lipsync. [*Find here*](https://github.com/DanielSWolf/rhubarb-lip-sync)\n\n### Steps 📋\n\n1. Clone the repository and navigate to the project directory.\n\n```bash\ngit clone https://github.com/rorosaga/holo-cierge.git\n```\n```bash\ncd holo-cierge\n```\n\n2. **Backend Setup:**\n\n- Navigate to `concierge-backend` directory and install dependencies.\n\n```bash\ncd concierge-backend\n```\n```bash\nyarn\n```\n\n- Set up the environment variables:\n\u003e- Create a `.env` file based on `.env.example` and add your OpenAI API key, ElevenLabs API key, and Rhubarb executable path.\n\n- Set up **Whisper** env:\n```bash\npython -m venv venv\n```\n- Activate the virtual environment:\n```bash\nsource venv/bin/activate # MacOS/Linux\n```\nor\n```bash\n.\\venv\\Scripts\\activate # Windows\n```\n- Install dependencies:\n```bash\npip install -r requirements.txt\n```\n\n3. **Frontend Setup:**\n- Navigate to `concierge-frontend` directory and install dependencies.\n\n```bash\ncd ../concierge-frontend\n```\n```bash\nyarn\n```\n\n4. Install FFmpeg:\n- For MacOS:\n```bash\nbrew install ffmpeg\n```\n- For Windows:\n\u003e Download the latest build from [here](https://ffmpeg.org/download.html) and add the `bin` directory to your system's PATH.\n\n5. **Start the development servers:**\n- Open two terminal windows, one in `concierge-backend` and the other in `concierge-frontend`.\n- Start the backend server:\n```bash\nyarn dev\n```\n- Start the frontend server:\n```bash\nyarn start\n```\n\n## Usage 🎮\n\n1. **Activate the Avatar:** Place the holographic display in front of the guest.\n2. **Engage in Conversation:** Speak directly to the concierge. Whisper will transcribe the audio, and ChatGPT will generate a contextual response. Guest can also type their queries thorugh the input box.\n3. **Perform Actions:** Trigger actions such as ticket creation through GuestHub.\n\n\n## Acknowledgements 🙌\n\nMassive thanks to [Wawa Sensei](https://www.youtube.com/@WawaSensei) for his amazing tutorials. Don't think twice to check him out, he is an amazing sensei with a great community!! 🥋🎓\n\n- Wawa Sensei frontend repo [here](https://github.com/wass08/r3f-virtual-girlfriend-frontend)\n\n- Wawa Sensei backend repo [here](https://github.com/wass08/r3f-virtual-girlfriend-backend)\n\n\u003e **Note:** This project is a fork of the original project by Wawa Sensei. We have added new features, integrations, and improvements to the existing project. Wawa Sensei doesn't have this licensed, but their work is deeply appreciated!\n\n## Resources 🧾\n\n- [Rhubarb Library](https://github.com/DanielSWolf/rhubarb-lip-sync) - Lip sync library\n- [Ready Player Me](https://readyplayer.me/) - Avatar creator\n- [mixamo](https://www.mixamo.com/) - 3D animations\n- [OpenAI](https://platform.openai.com/) - GPT API\n- [OpenAI Whisper](https://github.com/openai/whisper) - speech-to-text model\n- [Three.js](https://threejs.org/) - 3D library\n- [Eleven Labs](https://elevenlabs.io/) - text-to-speech API\n- [Looking Glass WebXR Library](https://github.com/Looking-Glass/looking-glass-webxr) - WebXR library for Looking Glass devices\n- [Looking Glass Bridge](https://lookingglassfactory.com/software/looking-glass-bridge) - software to connect to the Looking Glass device\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frorosaga%2Fholo-cierge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frorosaga%2Fholo-cierge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frorosaga%2Fholo-cierge/lists"}