https://github.com/frederickroman/navlead
The client-side of Navlead. Navlead is an deep-learning assistant chatbot for navigation in 3D enviroments.
https://github.com/frederickroman/navlead
3d-simulation blender chatbot client-side first-person-game navigation nextjs pwa reactjs unity unity3d wasm webgl webgl-game
Last synced: 3 months ago
JSON representation
The client-side of Navlead. Navlead is an deep-learning assistant chatbot for navigation in 3D enviroments.
- Host: GitHub
- URL: https://github.com/frederickroman/navlead
- Owner: FrederickRoman
- License: mit
- Created: 2022-03-08T03:02:22.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2022-10-18T00:52:55.000Z (over 3 years ago)
- Last Synced: 2025-01-17T19:55:06.959Z (over 1 year ago)
- Topics: 3d-simulation, blender, chatbot, client-side, first-person-game, navigation, nextjs, pwa, reactjs, unity, unity3d, wasm, webgl, webgl-game
- Language: TypeScript
- Homepage: https://navlead.vercel.app
- Size: 137 MB
- Stars: 3
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Navlead
## AI-powered assistant chatbot for navigation in 3D enviroments.
The chatbot was trained on real-world navigation assistance dialogs to give natural responses.
The 3D enviroments were reconstructed form real-world houses.
## Overview of NavLead
Navlead is an AI-powered assistant chatbot for navigation in 3D enviroments. The simulation, the chatbot, and their deployment online are all part of this project. (Note: this repo contains the client-side section only)
The simulation has custom 3D assets, a chat dialog box, motion controls and game logic. The chatbot has a custom ML NLP model that has a seq2seq architecture and was trained on CVDN dataset. The client-host loads the simulation and mediates its interactions with the API server. The API server authenticates the request and runs the ML NLP model.
### Motivation
Inspired by the rise of VR and AI, this project aims to combine both into a navigation assistance system that can run through the browser.
## Tech stack of Navlead
- Client-side
- Simulation
- Blender (3.1)
- Unity (2020.3 LTS)
- UI
- React (Next ts)
- Server-side (not on this repo)
- API server
- Django
- ML service
- Pytorch
## In-depth explanation
For an in-depth explanation of this project, please, see [Navlead's About page](https://navlead.vercel.app/about)
- [TLDR;](https://navlead.vercel.app/about#about_tldr)
- [Motivation](https://navlead.vercel.app/about#about_motivation)
- [3D simulation](https://navlead.vercel.app/about#about_simulation-3d-modeling)
- [Modeling 3D enviroment in Blender](https://navlead.vercel.app/about#about_modeling-3d-enviroment)
- [Adding game controls in Unity](https://navlead.vercel.app/about#about_adding-game-controls)
- [Bridging simulation to browser (Unity-side)](https://navlead.vercel.app/about#about_sim-to-browser-js)
- [Client-side integration](https://navlead.vercel.app/about#about_client-side)
- [Loading Unity Script](https://navlead.vercel.app/about#about_loading-unity-script)
- [Bridging simulation to browser (browser-side)](https://navlead.vercel.app/about#about_sim-to-browser-js)
- [Answer simulation question](https://navlead.vercel.app/about#about_answer-sim-question)
- [Server-side API](https://navlead.vercel.app/about#about_server-side)
- [Request handling](https://navlead.vercel.app/about#about_req-handling)
- [Machine Learning](https://navlead.vercel.app/about#about_machine-learning)
- [Data](https://navlead.vercel.app/about#about_ml-data)
- [Training](https://navlead.vercel.app/about#about_ml-training)
- [Results](https://navlead.vercel.app/about#about_ml-results)
- [References](https://navlead.vercel.app/about#about_references)
- [Authors](https://navlead.vercel.app/about#about_authors)
- [Frederick Roman](https://navlead.vercel.app/about#about_author-frederick-roman)
- [Homero Roman](https://navlead.vercel.app/about#about_author-homero-roman)
## Live website
See [Navlead](https://navlead.vercel.app).
## Project setup
```
npm install
```
### Compiles and hot-reloads
```
npm run dev
```
### Compiles and minifies for production
```
npm run build
```
### Lints and fixes files
```
npm run lint
```
## Acknowledgements
### Vision-and-Dialog Navigation
https://arxiv.org/abs/1907.04957
https://github.com/mmurray/cvdn