https://github.com/gabrieljuan349/tracexr
Meta Quest application for Vitol challenge to track and recognize objects and patterns in Mixed Reality. LAUZHACK
https://github.com/gabrieljuan349/tracexr
computer-vision deep-learning drawing object-detection object-tracking unity xr xr-multiplayer
Last synced: 5 months ago
JSON representation
Meta Quest application for Vitol challenge to track and recognize objects and patterns in Mixed Reality. LAUZHACK
- Host: GitHub
- URL: https://github.com/gabrieljuan349/tracexr
- Owner: GabrielJuan349
- License: mit
- Created: 2024-11-30T12:54:00.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-12-02T09:44:37.000Z (over 1 year ago)
- Last Synced: 2025-02-20T21:52:25.526Z (over 1 year ago)
- Topics: computer-vision, deep-learning, drawing, object-detection, object-tracking, unity, xr, xr-multiplayer
- Language: Jupyter Notebook
- Homepage:
- Size: 7.37 MB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
TraceXR
Meta Quest application for Vitol challenge to track and recognize objects and patterns in Mixed Reality.
Report bug
·
Request feature
## Table of contents
- [Quick start](#quick-start)
- [About this project](#about-this-project)
- [Status](#status)
- [What's included](#whats-included)
- [Bugs and feature requests](#bugs-and-feature-requests)
- [Creators](#creators)
- [Copyright and license](#copyright-and-license)
## Quick start
### Model download - ONNX format
You can find our ONNX model for EfficientNet B7 trained on TU-Berlin Sketch dataset in [Google Drive](https://drive.google.com/file/d/1s6j8zwpggz0hqwEiRSArXn19FGD4639y/view?usp=sharing).
## About this project
This project combines multiple challenges from [LauzHack](https://lauzhack.com/) of [EPFL, Switzerland](https://www.epfl.ch/en/), which are proposed by companies such as [AXA Group](https://axa.com/) (an Artificial Intelligence model that can run on a laptop, mobile device, or immersive device), [Logitech](https://www.logitech.com/) (using the [MX Ink](https://www.logitech.com/es-es/products/vr/mx-ink.html) together with the [Meta Quest 3/3S](https://www.meta.com/ch/en/quest/quest-3/) to create a Mixed Reality (XR) application), and primarily [Vitol](https://www.vitol.com/) (creating an AI service for recognizing static and moving objects and/or a chatbot capable of interacting with the user).
As shown in the image below, this project is a multi-agent AI system combining Speech-To-Text with [OpenAI Whisper](https://openai.com/index/whisper/) for multi-agent routing and generating written responses when necessary using [Qwen2.5-0.5b](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF). It also utilizes [YoLo11](https://github.com/ultralytics/ultralytics) for object detection in images, an [EfficientNet-B7](https://pytorch.org/vision/main/models/efficientnet.html) ([Arxiv](https://arxiv.org/pdf/1905.11946)) for recognizing patterns or drawings made with the MX Ink, and finally, [OpenAI TTS](https://platform.openai.com/docs/guides/text-to-speech) for Text-to-Speech.
The implementation within the Meta Quest has been done using WebXR. For more information... [WebXR]( https://github.com/JG03dev/WebXR)
## Status
During the LauzHack is in development
## What's included
```text
agent-src/
│ ├── agent/
│ │ ├── image_prepos.py
│ │ ├── router.py
│ │ └── main.py
│ └── data-models/
│ ├── label_mapping.pkl
│ └── efficientnet_b7.onnx
├── models/
│ ├──efficient_net_b7.ipynb
│ ├──mobile_net.ipynb
│ └──yolov8.ipynb
├── assets/
├── examples/
├── .env.example
├── requirements.txt
```
## Bugs and feature requests
Have a bug or a feature request? Please first read the [issue guidelines](https://reponame/blob/master/CONTRIBUTING.md) and search for existing and closed issues. If your problem or idea is not addressed yet, [please open a new issue](https://reponame/issues/new).
## Creators
**Gabriel Juan**
- GitHub: [@GabrielJuan349](https://github.com/GabrielJuan349)
- LinkedIn: [in/gabi-juan](https://www.linkedin.com/in/gabi-juan)
**Jan Gras**
- GitHub: [@JG03dev](https://github.com/JG03dev)
- LinkedIn: [in/jangras](https://www.linkedin.com/in/jangras/)
**Yeray Cordero**
- GitHub: [@yeray142](https://github.com/yeray142)
- LinkedIn: [in/yeray142](https://www.linkedin.com/in/yeray142/)
**Nikalas Boyanov**
- GitHub: [@finnithegamer](https://github.com/finnithegamer)
- LinkedIn: [in/nikalas-boyanov-nunev](https://www.linkedin.com/in/nikalas-boyanov-nunev)
## Copyright and license
Code and documentation copyright 2024-2036 the authors. Code released under the [MIT License](https://reponame/blob/master/LICENSE).
Enjoy :metal: