{"id":23752371,"url":"https://github.com/gabrieljuan349/tracexr","last_synced_at":"2026-02-03T01:03:41.681Z","repository":{"id":265714642,"uuid":"896468750","full_name":"GabrielJuan349/TraceXR","owner":"GabrielJuan349","description":"Meta Quest application for Vitol challenge to track and recognize objects and patterns in Mixed Reality. LAUZHACK","archived":false,"fork":false,"pushed_at":"2024-12-02T09:44:37.000Z","size":7731,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-20T21:52:25.526Z","etag":null,"topics":["computer-vision","deep-learning","drawing","object-detection","object-tracking","unity","xr","xr-multiplayer"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GabrielJuan349.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-30T12:54:00.000Z","updated_at":"2024-12-02T19:54:35.000Z","dependencies_parsed_at":"2024-11-30T17:26:49.522Z","dependency_job_id":"a0c5584d-c418-4204-b3bd-935e46f92808","html_url":"https://github.com/GabrielJuan349/TraceXR","commit_stats":null,"previous_names":["gabrieljuan349/tracexr"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GabrielJuan349%2FTraceXR","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GabrielJuan349%2FTraceXR/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GabrielJuan349%2FTraceXR/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GabrielJuan349%2FTraceXR/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GabrielJuan349","download_url":"https://codeload.github.com/GabrielJuan349/TraceXR/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239921877,"owners_count":19718842,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","deep-learning","drawing","object-detection","object-tracking","unity","xr","xr-multiplayer"],"created_at":"2024-12-31T17:28:49.678Z","updated_at":"2026-02-03T01:03:41.641Z","avatar_url":"https://github.com/GabrielJuan349.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/GabrielJuan349/TraceXR/\"\u003e\n    \u003cimg src=\"./assets/traceXR-logo.png\" alt=\"Logo\" width=150 height=150\u003e\n  \u003c/a\u003e\n\n  \u003ch3 align=\"center\"\u003eTraceXR\u003c/h3\u003e\n\n  \u003cp align=\"center\"\u003e\n    Meta Quest application for Vitol challenge to track and recognize objects and patterns in Mixed Reality.\n    \u003cbr\u003e\n    \u003ca href=\"https://github.com/GabrielJuan349/TraceXR/issues/new?template=bug.md\"\u003eReport bug\u003c/a\u003e\n    ·\n    \u003ca href=\"https://github.com/GabrielJuan349/TraceXR/issues/new?template=feature.md\u0026labels=feature\"\u003eRequest feature\u003c/a\u003e\n  \u003c/p\u003e\n\u003c/p\u003e\n\n\n## Table of contents\n\n- [Quick start](#quick-start)\n- [About this project](#about-this-project)\n- [Status](#status)\n- [What's included](#whats-included)\n- [Bugs and feature requests](#bugs-and-feature-requests)\n- [Creators](#creators)\n- [Copyright and license](#copyright-and-license)\n\n\n## Quick start\n\n### Model download - ONNX format\nYou can find our ONNX model for EfficientNet B7 trained on TU-Berlin Sketch dataset in [Google Drive](https://drive.google.com/file/d/1s6j8zwpggz0hqwEiRSArXn19FGD4639y/view?usp=sharing).\n\n## About this project\n\nThis project combines multiple challenges from [LauzHack](https://lauzhack.com/) of [EPFL, Switzerland](https://www.epfl.ch/en/), which are proposed by companies such as [AXA Group](https://axa.com/) (an Artificial Intelligence model that can run on a laptop, mobile device, or immersive device), [Logitech](https://www.logitech.com/) (using the [MX Ink](https://www.logitech.com/es-es/products/vr/mx-ink.html) together with the [Meta Quest 3/3S](https://www.meta.com/ch/en/quest/quest-3/) to create a Mixed Reality (XR) application), and primarily [Vitol](https://www.vitol.com/) (creating an AI service for recognizing static and moving objects and/or a chatbot capable of interacting with the user).\n\nAs shown in the image below, this project is a multi-agent AI system combining Speech-To-Text with [OpenAI Whisper](https://openai.com/index/whisper/) for multi-agent routing and generating written responses when necessary using [Qwen2.5-0.5b](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF). It also utilizes [YoLo11](https://github.com/ultralytics/ultralytics) for object detection in images, an [EfficientNet-B7](https://pytorch.org/vision/main/models/efficientnet.html) ([Arxiv](https://arxiv.org/pdf/1905.11946)) for recognizing patterns or drawings made with the MX Ink, and finally, [OpenAI TTS](https://platform.openai.com/docs/guides/text-to-speech) for Text-to-Speech.\n\n\u003cimg src=\"./assets/Multi-Agent Graph.png\" alt=\"Multi Agent Architecture\" width=100% height=225\u003e\n\nThe implementation within the Meta Quest has been done using WebXR. For more information... [WebXR]( https://github.com/JG03dev/WebXR)\n\n## Status\n\nDuring the LauzHack is in development\n\n## What's included\n\n\n```text\nagent-src/\n│  ├── agent/\n│  │    ├── image_prepos.py\n│  │    ├── router.py\n│  │    └── main.py\n│  └── data-models/\n│       ├── label_mapping.pkl\n│       └── efficientnet_b7.onnx\n├── models/\n│    ├──efficient_net_b7.ipynb\n│    ├──mobile_net.ipynb\n│    └──yolov8.ipynb\n├── assets/\n├── examples/\n├── .env.example\n├── requirements.txt\n```\n\n## Bugs and feature requests\n\nHave a bug or a feature request? Please first read the [issue guidelines](https://reponame/blob/master/CONTRIBUTING.md) and search for existing and closed issues. If your problem or idea is not addressed yet, [please open a new issue](https://reponame/issues/new).\n\n\n## Creators\n\n **Gabriel Juan**\n  - GitHub: [@GabrielJuan349](https://github.com/GabrielJuan349)\n  - LinkedIn: [in/gabi-juan](https://www.linkedin.com/in/gabi-juan)\n\n**Jan Gras**\n  - GitHub: [@JG03dev](https://github.com/JG03dev)\n  - LinkedIn: [in/jangras](https://www.linkedin.com/in/jangras/)\n\n**Yeray Cordero**\n  - GitHub: [@yeray142](https://github.com/yeray142)\n  - LinkedIn: [in/yeray142](https://www.linkedin.com/in/yeray142/)\n\n**Nikalas Boyanov**\n  - GitHub: [@finnithegamer](https://github.com/finnithegamer)\n  - LinkedIn: [in/nikalas-boyanov-nunev](https://www.linkedin.com/in/nikalas-boyanov-nunev)\n\n## Copyright and license\n\nCode and documentation copyright 2024-2036 the authors. Code released under the [MIT License](https://reponame/blob/master/LICENSE).\n\nEnjoy :metal:\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgabrieljuan349%2Ftracexr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgabrieljuan349%2Ftracexr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgabrieljuan349%2Ftracexr/lists"}