https://github.com/mikeroyal/AR-VR-Guide

A guide covering Augmented Reality (AR), Virtual Reality (VR), Mixed Reality(MR), and Extended Reality(XR). Including headsets such as Apple Vision Pro, HoloLens, PlayStation VR, and Quest 3.
https://github.com/mikeroyal/AR-VR-Guide

ar arkit augmented-reality camera-tracking mesh-processing mesh-simplification photogrammetry pixar-usd polygon real-time-rendering reality-capture realitykit scenekit ue5 universal-scene-description usd usdz virtual-reality virtualreality xr

Last synced: 3 months ago
JSON representation

A guide covering Augmented Reality (AR), Virtual Reality (VR), Mixed Reality(MR), and Extended Reality(XR). Including headsets such as Apple Vision Pro, HoloLens, PlayStation VR, and Quest 3.

Host: GitHub
URL: https://github.com/mikeroyal/AR-VR-Guide
Owner: mikeroyal
Created: 2021-08-22T19:54:13.000Z (over 4 years ago)
Default Branch: main
Last Pushed: 2024-01-04T22:38:42.000Z (over 2 years ago)
Last Synced: 2025-05-25T16:09:25.420Z (12 months ago)
Topics: ar, arkit, augmented-reality, camera-tracking, mesh-processing, mesh-simplification, photogrammetry, pixar-usd, polygon, real-time-rendering, reality-capture, realitykit, scenekit, ue5, universal-scene-description, usd, usdz, virtual-reality, virtualreality, xr
Language: C++
Homepage:
Size: 2.13 MB
Stars: 447
Watchers: 14
Forks: 46
Open Issues: 2
Metadata Files:
- Readme: README.md
- Contributing: Contributing.md

Awesome Lists containing this project

README

AR/VR Guide

![Maintenance](https://img.shields.io/maintenance/yes/2024?style=for-the-badge)
![Last-Commit](https://img.shields.io/github/last-commit/mikeroyal/ar-vr-guide?style=for-the-badge)

#### A guide covering Augmented Reality (AR) & Virtual Reality (VR) including the applications, libraries and tools that will make you a better and more efficient Augmented Reality (AR) & Virtual Reality (VR) development.

**Note: You can easily convert this markdown file to a PDF in [VSCode](https://code.visualstudio.com/) using this handy extension [Markdown PDF](https://marketplace.visualstudio.com/items?itemName=yzane.markdown-pdf).**

# Table of Contents

1. [Getting Started with AR/VR](https://github.com/mikeroyal/AR-VR-Guide#getting-started-with-arvr)

* [Developer Resources](https://github.com/mikeroyal/AR-VR-Guide#developer-resources)
* [AR/VR Training Courses](https://github.com/mikeroyal/AR-VR-Guide#arvr-training-courses)

2. [AR/VR Tools, Frameworks, and Hardware](https://github.com/mikeroyal/AR-VR-Guide#arvr-tools-frameworks-and-hardware)

* [Tools & Frameworks](https://github.com/mikeroyal/AR-VR-Guide#tools--frameworks)

* [Unreal Engine 5 Development](https://github.com/mikeroyal/AR-VR-Guide#unreal-engine-5-development)

* [Unity Development](https://github.com/mikeroyal/AR-VR-Guide#unity-development)

* [AMD FidelityFX Super Resolution (FSR)](https://github.com/mikeroyal/AR-VR-Guide#amd-fidelityfx-super-resolution-fsr)

* [NVIDIA DLSS](https://github.com/mikeroyal/AR-VR-Guide#nvidia-dlss)

* [Microsoft HoloLens Headset](https://github.com/mikeroyal/AR-VR-Guide#Microsoft-HoloLens-Headset)

* [PlayStation® VR Headset](https://github.com/mikeroyal/AR-VR-Guide#PlayStation-VR-Headset)

* [PlayStation®VR 2 Headset](https://github.com/mikeroyal/AR-VR-Guide#PlayStation-VR-2-Headset)

* [Oculus Quest Headsets](https://github.com/mikeroyal/AR-VR-Guide#Oculus-Quest-headsets)

* [Oculus Developer Tools](https://github.com/mikeroyal/AR-VR-Guide#Oculus-Developer-Tools)

* [Meta Quest 3](#meta-quest-3)

* [Meta Quest 2](https://github.com/mikeroyal/AR-VR-Guide#Quest-2)

* [Meta Quest Pro](https://github.com/mikeroyal/AR-VR-Guide#Quest-Pro)

* [Apple Vision Pro Headset](https://github.com/mikeroyal/AR-VR-Guide#apple-vision-pro-headset)

* [VisionOS](https://github.com/mikeroyal/AR-VR-Guide#visionOS)
* [Apple AR Toolkits and Frameworks](https://github.com/mikeroyal/AR-VR-Guide#apple-ar-toolkits-and-frameworks)

* [HTC Vive Headset](https://github.com/mikeroyal/AR-VR-Guide#HTC-Vive-Headset)

* [SteamVR](https://github.com/mikeroyal/AR-VR-Guide#SteamVR)

* [Nreal Air AR Glasses](#Nreal-Air-AR-Glasses)

* [Perception Neuron Motion Capture Suit](https://github.com/mikeroyal/AR-VR-Guide#Perception-Neuron-Motion-Capture-Suit)

3. [Vulkan Development](https://github.com/mikeroyal/AR-VR-Guide#vulkan-development)

4. [Metal Development](https://github.com/mikeroyal/AR-VR-Guide#metal-development)

5. [DirectX Development](https://github.com/mikeroyal/AR-VR-Guide#directx-development)

6. [Computer Vision Development](https://github.com/mikeroyal/AR-VR-Guide#computer-vision-development)

7. [Photogrammetry Development](https://github.com/mikeroyal/AR-VR-Guide#photogrammetry-development)

8. [Autodesk Development](https://github.com/mikeroyal/AR-VR-Guide#autodesk-development)

9. [LiDAR Development](https://github.com/mikeroyal/AR-VR-Guide#lidar-development)

10. [Game Development](https://github.com/mikeroyal/AR-VR-Guide#game-development)

11. [Linear Algebra](https://github.com/mikeroyal/AR-VR-Guide#linear-algebra)

12. [Algorithms](https://github.com/mikeroyal/AR-VR-Guide#algorithms)

13. [Machine Learning](https://github.com/mikeroyal/AR-VR-Guide#machine-learning)

14. [CUDA Development](https://github.com/mikeroyal/AR-VR-Guide#cuda-development)

15. [MATLAB Development](https://github.com/mikeroyal/AR-VR-Guide#matlab-development)

16. [C/C++ Development](https://github.com/mikeroyal/AR-VR-Guide#cc-development)

# Getting Started with AR/VR
[Back to the Top](#table-of-contents)

[Augmented Reality (AR)](https://en.wikipedia.org/wiki/Augmented_reality) is an interactive experience of a real-world environment where the objects that reside in the real world are enhanced by computer-generated perceptual information.

[Virtual Reality (VR)](https://en.wikipedia.org/wiki/Virtual_reality) is a simulated experience that can be similar to or completely different from the real world. The applications of virtual reality include entertainment (video games), education (medical or military training) and business (virtual meetings).

[Mixed Reality (MR)](https://en.wikipedia.org/wiki/Mixed_reality) is the merging of real and virtual worlds to produce new environments and visualizations, where physical and digital objects co-exist and interact in real time.

[Extended Reality (XR)](https://en.wikipedia.org/wiki/Extended_reality) is a concept referring to all real-and-virtual combined environments and human-machine interactions generated by computer technology and wearables. Including augmented reality (AR), mixed reality (MR) and virtual reality (VR).

### Developer Resources
[Back to the Top](#table-of-contents)

* [Virtual Reality - Experiences and Devices](https://www.microsoft.com/en-us/mixed-reality/windows-mixed-reality)

* [Oculus | VR Headsets, Games & Equipment](https://www.oculus.com/)

* [Virtual Reality on Steam](https://store.steampowered.com/vr/)

* [Valve Index® VR Headset](https://store.steampowered.com/valveindex)

* [PlayStation VR](https://www.playstation.com/explore/playstation-vr/)

* [HTC Vive VR Headset](https://www.vive.com/us/)

* [Augmented Reality applications - Apple](https://www.apple.com/augmented-reality/)

* [iOS, iPadOS, and tvOS Development on Unreal Engine](https://docs.unrealengine.com/5.0/en-US/ios-ipados-and-tvos-support-for-unreal-engine/)

* [Android Development on Unreal Engine](https://docs.unrealengine.com/5.0/en-US/android-support-for-unreal-engine)

* [XR Development on Unreal Engine](https://docs.unrealengine.com/5.0/en-US/developing-for-xr-experiences-in-unreal-engine)

* [Setting Up Visual Studio for Unreal Engine](https://docs.unrealengine.com/en-us/Programming/Development/VisualStudioSetup)

* [Unreal Engine Performance and Profiling](https://docs.unrealengine.com/5.0/en-US/TestingAndOptimization/PerformanceAndProfiling/)

### AR/VR Training Courses
[Back to the Top](#table-of-contents)

* [Unity Learn Training Program](https://learn.unity.com)

* [Unity Manual: XR](https://docs.unity3d.com/Manual/XR.html)

* [Intro to XR: VR, AR, and MR Foundations - Unity Learn](https://learn.unity.com/course/introduction-to-xr-vr-ar-and-mr-foundations)

* [Unity XR: Build VR and AR Apps](https://unity3d.com/learn/unity-xr-apps)

* [Top Virtual Reality Courses Online | Udemy](https://www.udemy.com/topic/virtual-reality/)

* [Top Augmented Reality Courses Online | Udemy](https://www.udemy.com/topic/augmented-reality/)

* [Google AR & VR Online Courses | Coursera](https://www.coursera.org/googlearvr)

* [Top Augmented Reality Courses | Coursera](https://www.coursera.org/courses?query=augmented%20reality)

* [Learn Augmented Reality with Online Courses and Lessons | edX](https://www.edx.org/learn/augmented-reality)

# AR/VR Tools, Frameworks, and Hardware
[Back to the Top](https://github.com/mikeroyal/AR-VR-Guide#table-of-contents)

### Tools & Frameworks

[RealityCapture](https://www.capturingreality.com/) is a state-of-the-art photogrammetry software solution that creates virtual reality scenes, textured 3D meshes, orthographic projections, geo-referenced maps and much more from images and/or laser scans completely automatically.

[Reality Composer](https://developer.apple.com/augmented-reality/tools/) is a powerful tool that makes it easy for you to create interactive augmented reality experiences with no prior 3D experience. The Reality Converter quickly converts your existing 3D models to [USDZ](https://graphics.pixar.com/usd/files/USDZFileFormatSpecification.pdf) so it works seamlessly in our tools and on all AR-enabled iPhone and iPad devices.

[ARKit](https://developer.apple.com/augmented-reality/arkit/) is a set set of software development tools to enable developers to build augmented-reality apps for iOS developed by Apple. The latest version ARKit 3.5 takes advantage of the new LiDAR Scanner and depth sensing system on iPad Pro(2020) to support a new generation of AR apps that use Scene Geometry for enhanced scene understanding and object occlusion.

[RealityKit](https://developer.apple.com/documentation/realitykit) is a framework to implement high-performance 3D simulation and rendering with information provided by the ARKit framework to seamlessly integrate virtual objects into the real world.

[RealityUI](https://github.com/maxxfrazer/RealityUI) is a Swift Package for creating familiar UI Elements and animations in a RealityKit rendered Augmented Reality or Virtual Reality scene.

[SceneKit](https://developer.apple.com/scenekit/) is a high-level 3D graphics framework that helps you create 3D animated scenes and effects in your iOS apps.

[GPUImage framework](https://github.com/BradLarson/GPUImage) is a BSD-licensed iOS library that lets you apply GPU-accelerated filters and other effects to images, live camera video, and movies. In comparison to Core Image (part of iOS 5.0), GPUImage allows you to write your own custom filters, supports deployment to iOS 4.0, and has a simpler interface. However, it currently lacks some of the more advanced features of Core Image, such as facial detection.

[GPUImage3](https://github.com/BradLarson/GPUImage3) is the third generation of the [GPUImage framework](https://github.com/BradLarson/GPUImage), an open source project for performing GPU-accelerated image and video processing on Mac and iOS. The original GPUImage framework was written in Objective-C and targeted Mac and iOS, the second iteration rewritten in Swift using OpenGL to target Mac, iOS, and Linux, and now this third generation is redesigned to use [Apple's Metal](https://developer.apple.com/metal/) in place of OpenGL.

[ARCore](https://developers.google.com/ar/) is a software development kit developed by Google that allows for augmented reality applications in the real world. These tools include environmental understanding, which allows devices to detect horizontal and vertical surfaces and planes. It also includes motion tracking, which lets phones understand and track their positions relative to the world. Also ARCore’s Light Estimation API lets your digital objects appear realistically as if they’re actually part of the physical world.

[Adobe Scantastic](https://labs.adobe.com/projects/scantastic/) is a tool that makes the creation of 3D assets accessible to everyone. It can be used with just a mobile device (combined with Adobe's server-based photogrammetry pipeline), users can easily scan objects in their physical environment and turn them into 3D models which can then be imported into tools like Adobe Dimension and Adobe Aero.

[Adobe Aero](https://www.adobe.com/products/aero.html) is a tool that helps you build, view, and share immersive AR experiences. Simply build a scene by bringing in 2D images from Adobe Photoshop and Illustrator, or 3D models from Adobe Dimension, Substance, third-party apps like Cinema 4D, or asset libraries like Adobe Stock and TurboSquid. Aero optimizes a wide array of assets, including OBJ, GLB, and glTF files, for AR, so you can visualize them in real time.

[Vuforia](https://developer.vuforia.com/) is a comprehensive, scalable enterprise AR platform. Our wide-ranging solution suite ensures that we can provide the right AR technology to every customer based on their business needs.

[Vuforia Studio](https://www.ptc.com/en/products/vuforia/vuforia-studio) is a tool that allows anyone to create beautiful 3D augmented reality (AR) experiences in a matter of minutes with no coding required.

[SceneView](https://github.com/SceneView/sceneview-android) is a 3D and AR Android Composable and View with Google Filament and ARCore. This is a Sceneform replacement in Kotlin.

[Polycam](https://poly.cam/) is the world's most popular LiDAR 3D scanning app for iOS(iPhone/iPad), Web, and Android. It let's you scan the world around you with your mobile device, DSLR camera, or drone to get beautiful, accurate 3D models.

[3D Scanner App for Mac](https://apps.apple.com/us/app/3d-scanner-app/id1419913995) is a desktop tool for processing photos and videos into 3D models using the power of Photogrammetry; which makes it perfect for 3D Design, CAD, Architecture, Games Assets, AR, VR, XR. It shares USDZ models via iMessage to let friends and family see your models in Augmented Reality.

[OpenVX™](https://www.khronos.org/openvx/) is an open-source, royalty-free standard for cross platform acceleration of computer vision applications. OpenVX enables performance and power-optimized computer vision processing, especially important in embedded and real-time use cases such as face, body and gesture tracking, smart video surveillance, advanced driver assistance systems (ADAS), object and scene reconstruction, augmented reality, visual inspection, robotics and more.

[OpenXR](https://www.khronos.org/OpenXR/) is an open, royalty-free standard for access to virtual reality and augmented reality platforms and devices.

[OpenXR™ SDK](https://github.com/KhronosGroup/OpenXR-SDK) is a Software Development Kit that contains OpenXR headers, as well as source code and build scripts for the OpenXR loader.

[NVIDIA Flex](https://github.com/NVIDIAGameWorks/FleX) is a particle-based simulation library designed for real-time applications.

[ARToolKit](https://artoolkit.org) is a fast and modern open source tracking and recognition SDK which enables computers to see and understand more in the environment around them. It's built from the ground up using modern computer vision techniques.

[ALVR(Air Light VR)](https://github.com/alvr-org/ALVR) is a Stream VR games from your PC to your headset via Wi-Fi.
ALVR uses technologies like [Asynchronous Timewarp](https://developer.oculus.com/documentation/native/android/mobile-timewarp-overview) and [Fixed Foveated Rendering](https://developer.oculus.com/documentation/native/android/mobile-ffr) for a smoother experience.

[ARmedia](https://www.inglobetechnologies.com/ar-media/) is a plugin tool that makes it easy to develop and create AR experiences without coding.

[Wikitude](https://www.wikitude.com/products/wikitude-sdk/) is a Cross-platform augmented reality SDK to create high-performing object tracking AR experiences using 3D models as an input method (CAD, glTF 2.0 and more) for multiple devices and operating systems.

[Kudan AR SDK](https://www.xlsoft.com/en/products/kudan/index.html) is a Professional AR SDK to develop compelling applications for mobile devices.

[Kundan](https://www.kudan.io/) is a tool that provides proprietary Artificial Perception technologies based on SLAM to enable use cases with significant market potential and impact on our lives such as autonomous driving, robotics, AR/VR and smart cities.

[OVR Toolkit](https://store.steampowered.com/app/1068820/OVR_Toolkit/) is a utility application designed to make viewing the desktop in VR simple and fast, it allows for viewing the desktop within VR, placing desktop windows around the world, mouse input, typing with a virtual keyboard, and quickly switching between windows.

[OpenVR](https://github.com/ValveSoftware/openvr) is an API and runtime that allows access to VR hardware(Steam Index, HTC Vive, and Oculus Rift) from multiple vendors without requiring that applications have specific knowledge of the hardware they are targeting.

[OpenVR Benchmark on Steam](https://store.steampowered.com/app/955610/OpenVR_Benchmark/) is the first benchmark tool for reproducibly testing your real VR performance, rendering inside of your VR headset.

[OpenHMD](http://www.openhmd.net/) is open source API and drivers that supports a wide range of HMD(head-mounted display) devices such as Oculus Rift, HTC Vive, Sony PSVR, and others.

[StereoKit](https://github.com/StereoKit/StereoKit) is an easy-to-use open source mixed reality library for building HoloLens and VR applications with C# and OpenXR.

[Monado](https://monado.dev/) is the first OpenXR™ runtime for GNU/Linux. Monado aims to jump-start development of an open source XR ecosystem and provide the fundamental building blocks for device vendors to target the GNU/Linux platform.

[Libsurvive](https://github.com/cntools/libsurvive) is a set of tools and libraries that enable 6 dof tracking on lighthouse and vive based systems that is completely open source and can run on any device. It currently supports both SteamVR 1.0 and SteamVR 2.0 generation of devices and should support any tracked object commercially available.

[Simula](https://github.com/SimulaVR/Simula) is a VR window manager for Linux that runs on top of Godot. It takes less than 1 minute to install. Simula is officially compatible with SteamVR headsets equipped with Linux drivers (e.g. HTC Vive, HTC Vive Pro, & Valve Index). We have also added experimental support to OpenXR headsets that have Monado drivers (e.g. North Star, OSVR HDK, and PSVR). Some people have gotten the Oculus Rift S to run Simula via OpenHMD ([see here](https://github.com/OpenHMD/OpenHMD/issues/225#issuecomment-638454156)).

### Unreal Engine 5 Development

[Back to the Top](#table-of-contents)

[Unreal Engine 5](https://www.unrealengine.com/unreal-engine-5) is a game engine developed by Epic Games with the world's most open and advanced real-time 3D creation tool. Continuously evolving to serve not only its original purpose as a state-of-the-art game engine, today it gives creators across industries the freedom and control to deliver cutting-edge content, interactive experiences, and immersive virtual worlds.

Unreal Engine 5 with Big City Sample Project

Unreal Engine Twinmotion. Source: [Unreal Engine](https://www.unrealengine.com/en-US/blog/twinmotion-2021-1-is-here)

Unreal Engine 5 Project browser

Unreal Engine 5 Vehicle Project Demo

Unreal Engine Marketplace

#### Unreal Engine Tools
[Back to the Top](#table-of-contents)

[Blueprint Visual Scripting](https://docs.unrealengine.com/en-US/Engine/Blueprints/index.html) is a complete gameplay scripting system in Unreal Engine based on the concept of using a node-based interface to create gameplay elements from within Unreal Editor. As with many common scripting languages, it is used to define object-oriented (OO) classes or objects in the engine.

Blueprint Visual Scripting UI

[Datasmith](https://www.unrealengine.com/en-US/datasmith) is a collection of tools and plugins that bring entire pre-constructed scenes and complex assets created in a variety of industry-standard design applications into Unreal Engine.

[Lumen](https://docs.unrealengine.com/5.0/en-US/RenderingFeatures/Lumen/TechOverview/) is an Unreal Engine 5 feature that uses multiple ray tracing methods to solve Global Illumination and Reflections. Screen traces are done first, followed by a more reliable method. It uses Software Ray Tracing through Signed Distance Fields by default, but can achieve higher quality on supporting video cards when Hardware Ray Tracing is enabled.

[Nanite](https://docs.unrealengine.com/5.0/en-US/RenderingFeatures/Nanite/) is Unreal Engine 5's new virtualized geometry system which uses a new internal mesh format and rendering technology to render pixel scale detail and high object counts. It intelligently does work on only the detail that can be perceived and no more.

[Niagara VFX System](https://docs.unrealengine.com/5.0/en-US/RenderingAndGraphics/Niagara/) is one of two tools you can use to create and adjust visual effects (VFX) inside Unreal Engine 5 (UE5). Before Niagara, the primary way to create and edit visual effects in UE4 was to use [Cascade](https://docs.unrealengine.com/4.27/en-US/RenderingAndGraphics/ParticleSystems). It has many of the same of particle manipulation methods that Cascade offers, the way you interact and build visual effects with Niagara is vastly different.

[Chaos Physics](https://docs.unrealengine.com/5.0/en-US/InteractiveExperiences/Physics/ChaosPhysics/Overview/) is a Beta feature that is the light-weight physics simulation solution used in Fortnite, and it includes the following major features.

* RBAN (Rigid Body Animation Nodes)
* Destruction
* Cloth
* Ragdoll
* Vehicles

[MetaHuman Creator](https://www.unrealengine.com/en-US/metahuman-creator) is a free cloud-based app that empowers anyone to create photorealistic digital humans, complete with hair and clothing, in minutes. MetaHumans come fully rigged and ready to animate in your Unreal Engine projects.

[Twinmotion](https://www.twinmotion.com/en-US) is a real-time 3D immersion software that produces high-quality images, panoramas and standard or 360° VR videos in seconds. Developed for architecture, construction, urban planning and landscaping professionals.

[Bridgew by Quixel](https://www.unrealengine.com/en-US/bridge) is a gateway to create a world of 3D content right inside the Unreal Engine. it includes Megascans—the world's largest library of AAA, cinema-quality assets based on real-world scan data.

[Unreal Engine Marketplace](https://unrealengine.com/marketplace/en-US/store) is Unreal Engine's Store that has a library Of Textures, Models, Animations, Tutorials, and More for creating amazing 3D projects and games.

[UnrealBuildTool (UBT)](https://docs.unrealengine.com/5.0/en-US/ProductionPipelines/BuildTools/UnrealBuildTool) is a tool that manages the process of building UE4 source code across a variety of build configurations.

[UnrealHeaderTool (UHT)](https://docs.unrealengine.com/5.0/en-US/ProductionPipelines/BuildTools/UnrealHeaderTool) is a custom parsing and code-generation tool supporting the UObject system.

[AutomationTool](https://docs.unrealengine.com/4.27/en-US/ProductionPipelines/BuildTools/AutomationTool) is a generic system used to automate processes, including testing and building games.

[Proxy Geometry Tool](https://docs.unrealengine.com/5.0/en-US/TestingAndOptimization/ProxyGeoTool/) is a tool-set that was developed as a way to increase your Unreal Engine 4 (UE4) project's performance while keeping the visual quality of your project uneffected.

[Replay System](https://docs.unrealengine.com/5.0/en-US/TestingAndOptimization/ReplaySystem/) is a tool which can record gameplay for later viewing. This feature is available in all games, from live, multiplayer games played on dedicated servers, to single-player games, and even including Play-In-Editor sessions. At a high level, the Replay system works by using a DemoNetDriver to read data drawn from the built-in replication system, similar to how a NetDriver operates in a live, networked gameplay environment.

#### Developer Resources
[Back to the Top](#table-of-contents)

**Note:** you will need to create an Epic Games account to download Unreal Engine.

* [Sign-up for Epic Games Acount](https://www.epicgames.com/account/password)

* [Sign-up for Epic Games GitHub](https://github.com/EpicGames/Signup)

* [How to migrate your Unreal Engine 4 projects to Unreal Engine 5 Early Access quickly and smoothly](https://docs.unrealengine.com/5.0/en-US/MigrationGuide/)

* [Multi-platform development - Unreal Engine](https://www.unrealengine.com/en-US/features/multi-platform-development)

* [Unreal Engine 5 Documentation](https://docs.unrealengine.com/5.0/)

* [Unreal Engine Forums](https://forums.unrealengine.com/)

* [Windows Development Requirements for Unreal Engine](https://docs.unrealengine.com/5.0/en-US/hardware-and-software-specifications-for-unreal-engine/)

* [macOS Development Requirements for Unreal Engine](https://docs.unrealengine.com/5.0/en-US/hardware-and-software-specifications-for-unreal-engine/)

* [Linux Development Requirements for Unreal Engine](https://docs.unrealengine.com/5.0/en-US/linux-development-requirements-for-unreal-engine/)

* [iOS, iPadOS, and tvOS Development on Unreal Engine](https://docs.unrealengine.com/5.0/en-US/ios-ipados-and-tvos-support-for-unreal-engine/)

* [Android Development on Unreal Engine](https://docs.unrealengine.com/5.0/en-US/android-support-for-unreal-engine)

* [XR Development on Unreal Engine](https://docs.unrealengine.com/5.0/en-US/developing-for-xr-experiences-in-unreal-engine)

* [Setting Up Visual Studio for Unreal Engine](https://docs.unrealengine.com/en-us/Programming/Development/VisualStudioSetup)

* [Unreal Engine Performance and Profiling](https://docs.unrealengine.com/5.0/en-US/TestingAndOptimization/PerformanceAndProfiling/)

* [Unreal Engine C++ API Reference](https://docs.unrealengine.com/5.0/en-US/API/index.html)

* [Unreal Engine Blueprint API Reference](https://docs.unrealengine.com/5.0/en-US/BlueprintAPI/index.html)

* [Unreal Engine Python API Reference](https://docs.unrealengine.com/5.0/en-US/PythonAPI/index.html)

* [Unreal Online Learning](https://www.unrealengine.com/en-US/onlinelearning-courses)

* [Unreal Engine Authorized Training Program](https://www.unrealengine.com/en-US/training-partners)

* [Unreal Engine for education](https://www.unrealengine.com/en-US/education/)

* [Unreal Engine Training & Simulation](https://www.unrealengine.com/en-US/industry/training-simulation)

* [Unreal Engine | NVIDIA Developer](https://developer.nvidia.com/unrealengine)

* [Autodesk for Games](https://www.autodesk.com/campaigns/autodesk-for-games)

* [Getting Started with DirectX 12 Ultimate](https://devblogs.microsoft.com/directx/directx-12-ultimate-getting-started-guide/)

* [Getting Started with Vulkan](https://www.khronos.org/vulkan/)

* [Getting Started with Apple Metal](https://developer.apple.com/metal/)

### Unity Development

[Back to the Top](#table-of-contents)

[Unity](https://unity.com) is a cross-platform game development platform. Unity can be used to build high-quality 3D and 2D games, deploy them across mobile, desktop, VR/AR, consoles or the Web, and connect with loyal and enthusiastic players and customers. Checkout the [Unity Manual](https://docs.unity3d.com/Manual/UnityOverview.html) to get started on your Unity projects.

Unity Terrain project. Source: [Unity](https://blog.unity.com/technology/evolving-the-unity-editor-ux)

Unity Hub Installs

Unity Asset Store

#### Unity Tools
[Back to the Top](#table-of-contents)

[Unity Hub](https://unity.com/unity-hub) is a tool that helps manage multiple installations of the Unity Editor, create new projects, and access your work.

[Unity Asset Store](https://assetstore.unity.com)is Unity's Store that has a library Of Textures, Models, Animations, Tutorials, and More for creating amazing 3D projects and games.

[Unity Plus](https://store.unity.com/products/unity-plus) is a version of Unity for small businesses and serious hobbyists get more functionality and training resources to power up their projects.

[Unity Pro](https://store.unity.com/products/unity-pro) is a version of Unity to create and operate amazing applications and immersive experiences with a complete solution for professionals across industries.

[Unity Enterprise](https://store.unity.com/products/unity-enterprise) is a version of Unity for large teams creating at scale, Unity Enterprise delivers comprehensive technology, resources, and support that fuel innovation, reduce risk, and power your success.

[Unity XR](https://docs.unity3d.com/Manual/XR.html) is a plug-in framework that enables XR providers to integrate with the Unity engine and make full use of its features.

[Unity MARS](https://unity.com/products/unity-mars) is a tool that helps you create augmented reality (AR) apps with better workflows and purpose built authoring tools.

[Unity Build Server](https://unity.com/products/unity-build-server) is a tool that offloads your Unity project builds to network hardware, empowering your creative team to iterate more, produce higher quality products, and ship on time.

[Unity Teams](https://unity.com/products/unity-teams) is a tool that helps you and your team create together, faster. It makes it simple to save, share and sync your Unity projects with anyone.

[Unity Automated QA](https://unity.com/products/automated-qa) is a tool that shortens the development feedback loop with on-demand tests that anyone on your team can create and run without writing code.

[Unity Simulation Pro](https://unity.com/products/unity-simulation-pro) is a tool that unlocks the true power of scalable simulations through a simulation-optimized version of the Unity runtime.

[Unity Forma](https://unity.com/products/unity-forma) is a tool that brings the power of real-time 3D to your marketing. It turns 3D product data into interactive 3D configurators, stunning images and more.

[Unity Reflect](https://unity.com/products/unity-reflect) is a tool that helps create an experience immersive and collaborative real-time 3D environments, including in AR and VR, for better decision-making across your entire project lifecycle.

[Unity Reflect Develop](https://unity.com/products/unity-reflect-develop) is a tool that builds your own custom applications to address any problem across the entire building and infrastructure lifecycle for internal or commercial deployment.

[Unity ArtEngine](https://unity.com/products/unity-artengine) is Unity’s material authoring tool. Harnessing the power of assisted creation, the tool helps creators accelerate production pipelines.

[Unity Machine Learning Agents](https://unity.com/products/machine-learning-agents) is a tol that helps train and embed intelligent agents by leveraging state-of-the-art deep learning technology.

[Unity Essential Success ](https://unity.com/products/unity-essential-success) is a tool that is your dedicated Support Coordinator and experienced Unity engineers are ready to help.

[Bolt Visual scripting](https://unity.com/products/unity-visual-scripting) is a tool in Unity empowers creators to develop gameplay mechanics or interaction logic with a visual, graph-based system, instead of writing traditional lines of code.

[VisualLive](https://unity.com/products/visuallive) is a tool for construction professionals can visualize designs and collaborate in real-time by overlaying large BIM and CAD files onto jobsites using AR.

[Professional Services: Optimization Accelerator](https://unity.com/products/optimization-accelerator) is atool that boosts your project and your productivity in a four-day sprint with hands-on guidance from a Unity engineer. It will help you analyze performance problems, test solutions, and implement improvements.

[Odin Inspector Enterprise](https://unity.com/products/odin) is a tool that optimizes your project workflows and save thousands of development hours.

[Multiplay](https://unity.com/products/multiplay) is a resilient, multicloud hybrid server hosting and matchmaking platform without having to build and maintain your own infrastructure. Proven at scale in some of the world’s biggest games.

[Vivox](https://unity.com/products/vivox) is a voice and text comms tool that boosts player engagement, retention rates and create immersive multiplayer experiences by leveraging an easy-to-implement, feature-rich voice and text chat service.

[Relayᴮᴱᵀᴬ](https://unity.com/products/relay) is a tool that provides a great multiplayer gaming experiences with Unity’s Relay service that's now in open beta.

[Lobbyᴮᴱᵀᴬ](https://unity.com/products/lobby) is a tool that connects your players for great multiplayer gaming experiences with Lobby that's now in Open Beta.

[Cloud Codeᴮᴱᵀᴬ](https://unity.com/products/cloud-code) is a tool that runs your game logic in the cloud as serverless functions and interact with other backend services.

[Cloud Saveᴮᴱᵀᴬ](https://unity.com/products/cloud-save) is a tool that builds better player experiences by storing game data to the cloud.

[Furioos](https://unity.com/products/furioos) is a tool that streams fully interactive 3D experiences from Unity, other real-time 3D platforms, and Windows applications with peak visual quality in any web browser.

[OctaneRender® for Unity](https://unity.com/products/otoy-octanerender) is a tool that lets you harness physically accurate rendering directly in Unity and compose gorgeous, cinematic-quality scenes with OctaneRender materials and lighting.

[Pacelab WEAVR](https://unity.com/products/weavr) is a tool that provides a complete XR platform that enables any company, in any industry, to develop and manage enterprise-scale immersive training.

[Interact](https://unity.com/products/interact) is a tool that creates advanced real-time, human-centric simulations on any VR configuration for training, visualization, and safety.

[Prespective](https://unity.com/products/prespective) is a tool that helps you design and simulate your machine, factory, or system in Unity real-time 3D. Create common ground among stakeholders and speed up decision processes with improved visualization tools.

[Pixyz](https://unity.com/products/pixyz) is a tool that lets you can quickly import, prep, and optimize your large CAD, mesh and point cloud models for real-time visualizations in Unity.

[Backtrace](https://unity.com/products/backtrace) is a game crash management platform that delivers uninterrupted game experiences by automating your response to errors.

[SpeedTree®](https://store.speedtree.com/) is a group of vegetation programming and modeling software products developed and sold by Interactive Data Visualization, Inc. that generates virtual foliage for animations, architecture and in real time for video games and demanding real time simulations. [Unity Technologies acquired SpeedTree in July 2021](https://investors.unity.com/news/news-details/2021/Unity-Acquires-Interactive-Data-Visualization-Inc.-IDV-Creators-of-SpeedTree-Environment-Creation-Suite/default.aspx).

[SyncSketch](https://syncsketch.com) is a collaboration software for visual artists that allows a team to view, review, and provide feedback on each other’s work. It includes support for both 2D and 3D assets as well as video. As of [December 2021 SyncSketch has been acquired by Unity Technologies](https://blog.syncsketch.com/news/sycncsketch-acquired-by-unity/).

#### Unity Developer Resources
[Back to the Top](#table-of-contents)

* [Unity Knowledge Base](https://support.unity.com/hc/en-us) is a library of articles that helps you troubleshoot common problems, whether it’s creating an account, importing assets, or baking a scene.

* [Unity Forum](https://forum.unity.com/)

* [Unity Issue Tracker](https://issuetracker.unity3d.com/)

* [Unity Certifications](https://unity.com/products/unity-certifications)

* [Learn game development with Unity](https://learn.unity.com/courses)

* [Autodesk for Games](https://www.autodesk.com/campaigns/autodesk-for-games)

* [Getting Started with DirectX 12 Ultimate](https://devblogs.microsoft.com/directx/directx-12-ultimate-getting-started-guide/)

* [Getting Started with Vulkan](https://www.khronos.org/vulkan/)

* [Getting Started with Apple Metal](https://developer.apple.com/metal/)

### AMD FidelityFX Super Resolution (FSR)
[Back to the Top](#table-of-contents)

* **[AMD FidelityFX Super Resolution (FSR)](https://www.amd.com/en/technologies/radeon-software-fidelityfx)** is an open source, high-quality solution for producing high resolution frames from lower resolution inputs. FSR enables “practical performance” for costly render operations, such as hardware ray tracing for the AMD RDNA™ and AMD RDNA™ 2 architectures.

* **[AMD FidelityFX Super Resolution (FSR) 2.0](https://github.com/GPUOpen-Effects/FidelityFX-FSR2)** is an open source, high-quality solution for producing high resolution frames from lower resolution inputs. It uses temporal data and optimized anti-aliasing to boost framerates in supported games while delivering similar or better image quality than native resolution.

* **[AMD Open Source Driver for Vulkan®](https://github.com/GPUOpen-Drivers/AMDVLK)** is an open-source Vulkan driver for Radeon™ graphics adapters on Linux®. It is built on top of AMD's Platform Abstraction Library (PAL), a shared component that is designed to encapsulate certain hardware and OS-specific programming details for many of AMD's 3D and compute drivers.

* **[Vulkan® Memory Allocator (VMA)](https://gpuopen.com/vulkan-memory-allocator/)** is a library provides a simple and easy to integrate API to help you allocate memory for Vulkan® buffer and image storage.

* **[Radeon™ Raytracing Analyzer (RRA)](https://gpuopen.com/radeon-raytracing-analyzer/)** is a tool that investigates the performance of your raytracing applications and highlight potential bottlenecks.

* **[Radeon™ GPU Profiler](https://gpuopen.com/rgp/)** is a low-level optimization tool that provides detailed information on Radeon™ GPUs.

* **[Radeon™ GPU Analyzer](https://gpuopen.com/rga/)** is an offline compiler and performance analysis tool for DirectX®, Vulkan®, SPIR-V™, OpenGL®, and OpenCL™. It can be used together with [RGP](https://gpuopen.com/rgp/), [RMV](https://gpuopen.com/rmv/), and [RDP](https://gpuopen.com/rdp/).

* **[Radeon™ Developer Panel (RDP)](https://gpuopen.com/rdp/)** is an essential part of the Radeon™ Developer Tool Suite. It provides the communication channel that delivers requests to, and receives data from, the AMD Radeon™ driver.

### NVIDIA DLSS
[Back to the Top](#table-of-contents)

* **[NVIDIA DLSS (Deep Learning Super Sampling)](https://developer.nvidia.com/dlss)** is a temporal image upscaling AI rendering technology that increases graphics performance using dedicated Tensor Core AI processors on GeForce RTX™ GPUs. DLSS uses the power of a deep learning neural network to boost frame rates and generate beautiful, sharp images for your games.

* **[NVIDIA Image Scaling (NIS)](https://developer.nvidia.com/rtx/image-scaling)** is an open-source spatial upscaler and sharpening algorithm that works cross-platform on all GPUs. It's similar to AMD's FidelityFX Super Resolution (FSR) solution.

AMID EVIL NVIDIA DLSS demo. Credit: [NVIDIA](https://www.nvidia.com/en-us/geforce/news/may-2021-rtx-dlss-game-update/) and [AMID EVIL](https://store.steampowered.com/app/673130/AMID_EVIL/)

### Microsoft HoloLens Headset
[Back to the Top](#table-of-contents)

[Microsoft HoloLens Headset](https://www.microsoft.com/en-us/hololens/) is a pair of mixed reality smartglasses developed and manufactured by Microsoft. It's also the first fully self-contained holographic computer to run Windows 10.

Microsoft HoloLens Headset. Source: [Microsoft](https://www.microsoft.com/en-us/hololens/)

### PlayStation® VR Headset
[Back to the Top](#table-of-contents)

[PlayStation® VR Headset](https://www.playstation.com/en-us/ps-vr/) is a virtual reality headset developed by Sony Interactive Entertainment. It gives the user an amazing gaming experience with stunning High Dynamic Range visuals and up to 120FPS, as two 1920 x 1080 OLED displays deliver a vast 100º field of view. PSVR is fully functional with the PlayStation 4™ and is forward compatible with the PlayStation 5™ home video game console.

PlayStation VR Headset. Source: [PlayStation](https://www.playstation.com/en-us/ps-vr/)

### PlayStation® VR 2 Headset
[Back to the Top](#table-of-contents)

[PlayStation®VR 2](https://www.playstation.com/en-us/ps-vr2/) is the next generation of virtual reality gaming by PlayStation. It gives the user an amazing gaming experience with new stunning 4K High Dynamic Range visuals and up to 120fps, as two 2000 x 2040 OLED displays deliver over four times the resolution generated by the original PlayStation®VR headset. A vast 110º field of view and a light, well-balanced Fresnel lens. The lens adjustment dial allows you to customize spacing between lenses to match your eye position for an optimal onscreen image. Along with ground-breaking new controllers and genre-defining games, all made possible by the power of PS5™.

[psvr2

PlayStation VR 2 Headset. Source: [PlayStation](https://www.playstation.com/en-us/ps-vr2/)

### Oculus Quest Headsets
[Back to the Top](#table-of-contents)

#### Oculus Developer Tools

* [Getting Started with Oculus](https://developer.oculus.com/quest/)

[Oculus Developer Hub (ODH) for Windows](https://developer.oculus.com/downloads/package/oculus-developer-hub-win) is a desktop companion tool that streamlines app development for the Oculus Quest.

Oculus Developer Hub (ODH)

[Oculus Developer Hub (ODH) for macOS](https://developer.oculus.com/downloads/package/oculus-developer-hub-mac) is a desktop companion tool that streamlines app development for the Oculus Quest.

[Oculus Integration SDK](https://developer.oculus.com/downloads/package/unity-integration) is a toolkit that provides tools for Unity packages OVRPlugin, Interaction SDK (Experimental), Platform SDK, Voice SDK, Audio SDK, and Lipsync SDK.

[OVR Metrics Tool](https://developer.oculus.com/downloads/package/ovr-metrics-tool) is a tool that provides performance metrics for Quest and Quest 2, including frame rate, heat, GPU & CPU throttling values, tears and stale frames per second.

[GPU Systrace](https://developer.oculus.com/downloads/package/gpu-systrace) is a utility that contains required files to modify the Android SDK tool Systrace in order to provide low-level GPU pipeline data for apps running on Quest and Quest 2.

[Meta Presence Platform](https://developer.oculus.com/presence-platform/) is a suite of tools that let's you build the future of spatial computing with state-of-the-art AI, machine perception, graphics, and audio technologies, so you can invent experiences that blend the physical and virtual worlds.

[Arkio](https://www.oculus.com/experiences/quest/2280319701979278) is a collaborative design tool that enables you to sketch buildings, interiors or urban plans with others using VR, desktops, tablets and phones.

[Gravity Sketch](https://www.oculus.com/experiences/quest/1587090851394426) is a 3D design and collaboration tool that enables you to create vehicles, sneakers, furniture, characters, art, and more. Express your ideas with 3D strokes and geometry; solve complex design challenges working directly in 3D at any scale, all with a fun and intuitive toolset that makes working in 3D easy.

[Immersed](https://www.oculus.com/experiences/quest/2849273531812512) is a free app for multiple virtual screens in VR for Mac/PC/Linux (with no extra hardware) in stunning virtual worlds. It's great for solo work with multiple screens in a virtual coffee shop, or for collaborating with your team around even more screens and whiteboards.

#### Meta Quest 3

[Back to the Top](#table-of-contents)

[Checkout the Meta Quest 3 (Available October 10th)](https://www.meta.com/quest/quest-3/)

Meta Quest 3 Mixed Reality headset

**Hardware Specs:**

* **CPU:** Snapdragon XR2 "Gen 2" chipset (double the GPU processing power of the Meta Quest 2).
* **RAM:** 8GB.
* **Display Resolution:** 2064x2208 pixels per eye nearly 30% better than the Meta Quest 2.
* **Field of View (FOV):** 110 degrees horizontal and 96 degrees vertical.
* **Refresh Rate:** 90Hz, 120Hz (experimental).
* **Wi-Fi:** Wi-Fi 6E enabled (region permitting).

**Price:**

* $499 for a 128GB model.
* $649 for a 512GB model.

**Battery Life:**

* Up to 2.2 hours of usage on average.
* Gaming: 2.4 hours of usage on average.
* Social: 2.2 hours of usage on average.
* Productivity: 1.5 hours of usage on average.
* Media: 2.9 hours of usage on average.
* Charging Time: Approximately 2.3 hours with the included 18W power adapter (headset only).

**Privacy Options**

* **External LED:** Signal notifies bystanders that the external cameras are in use.
* **Data Protection:** Advanced encryption plus customized settings include data sharing opt-in/opt-out, electronic mic/camera disconnect and password/PIN protection to let multiple users explore safely.

#### Quest 2
[Back to the Top](#table-of-contents)

[Oculus Quest 2](https://www.oculus.com/quest-2/) is a virtual reality headset developed by Facebook AKA Meta Technologies, LLC. The Quest 2 provides multiplayer games to unique social experiences to joining up with friends at a live show or instructor-led workout. With High Dynamic Range visuals and up to 120fps, as two 1832x1920 per-eye and vast 97° field of view horizontal. Oculus Quest 2 can be connected to a PC via Oculus Link, which can be performed using the official Link cable or a third party USB-C cable. This allows the headset to be used to run content from SteamVR and the desktop Oculus store. Note that a high-bandwidth USB specification (USB 3.0 at a minimum) is required to achieve a high-quality and smooth picture.

[oculus-quest

Oculus Quest 2 Headset. Source: [Oculus](https://www.oculus.com/quest-2/)

#### Quest Pro
[Back to the Top](#table-of-contents)

**Meta Quest Pro($1500 USD, out October 25)** is the first-ever device powered by the new Qualcomm Snapdragon XR2+ platform, which is optimized for VR to run at 50% more power than Meta Quest 2 with better thermal dissipation. Each Meta Quest Pro comes with the following specs:

* Snapdragon XR2+ processor
* 12GB of memory(RAM)
* Up to 256GB of storage.
* 90 Hz refresh rate.
* 50% more performance than Quest 2.
* 1.3x larger color gamut than Quest 2.
* 1800 by 1920 pixels per display, which is 37% more pixels per inch and 10% greater pixels per degree than Meta Quest 2.
* 10 high-res sensors (five inside the headset and five outside).
* New sensor architecture with high-res outward-facing cameras that capture **4x** the pixels as Meta Quest 2 cameras.
* Charging dock and 45W USB-C power adapter.

Oculus Quest Pro Headset. Credit: [Meta](https://www.meta.com/quest/)

#### Quest Pro Accessories:

* [Meta Quest Pro Compact Charging Dock ($79.99 USD, out October 25)]()
* [Meta Quest Pro VR Earphones ($49.99 USD, out October 25)]()
* [Meta Quest Pro Carry Case from Incase ($119.95 USD, out October 25)]()
* [Meta Quest Pro Full Light Blocker ($49.99 USD, out November 22)]()

Oculus Quest Pro Headset & Accessories. Credit: [Meta](https://www.meta.com/quest/)

Oculus Quest Pro Controllers. Credit: [Meta](https://www.meta.com/quest/)

### Apple Vision Pro Headset

[Back to the Top](#table-of-contents)

Vision Pro Headset

Vision Pro Headset with battery pack

R1 & M2 chips in the Vision Pro Headset

VisionOS

**Patents:**

* [Mixed Reality Headset GUI's, how to control menus with Eye Tracking & in-air Micro Gesturing](https://www.patentlyapple.com/patently-apple/2021/04/apple-reveals-more-about-mixed-reality-headset-guis-how-to-control-menus-with-eye-tracking-in-air-micro-gesturing.html)
* [Apple VR Headset Input System that Detects Various Hand Gestures using Multiple Apple Rings (Patent US 20220365598 A1)](https://www.patentlyapple.com/2022/11/apple-invents-a-vr-headset-input-system-that-detects-various-hand-gestures-using-multiple-apple-rings.html)
* [Apple 3D Mapping Technology with 'Multi-Sensor' Longer-Range Depth Mapping ((Patent US 20220364849 A1)](https://www.patentlyapple.com/2022/11/apple-advances-their-3d-mapping-technology-with-multi-sensor-longer-range-depth-mapping.html)
* [Apple XR Headset Self-Correcting Face Camera System](https://patentscope.wipo.int/search/en/detail.jsf?docId=EP392329757&_cid=P20-LEOVXX-31432-1#detailMainForm:MyTabViewId:NATIONALDOCUMENTS)
* [Euro Patent Filing for Apple 'Continuity' Feature being extended to their Future XR Headset](https://patentscope.wipo.int/search/en/detail.jsf?docId=EP392329787&_cid=P20-LEOVXX-31432-1#detailMainForm:MyTabViewId:NATIONALDOCUMENTS)
* [Apple XR Headset Finger Devices that include sensors with submicron resolution that detect the tiniest of movements (Patent US 20230073039 A1)](https://www.patentlyapple.com/2023/03/apple-invents-xr-headset-finger-devices-that-include-sensors-with-submicron-resolution-that-detect-the-tiniest-of-movements.html)
* [Apple Finger-Mounted Device and Ring Accessories to Assist future MR Headset users maneuver and work in 3D spaces. (Patent US 20220382376 A1.)](https://www.patentlyapple.com/2022/12/apple-reveals-advanced-finger-mounted-device-and-ring-accessories-to-assist-future-mr-headset-users-maneuver-and-work-in-3d-s.html)
* [Apple Ring that uses a Self-Mixing Interferometry Sensor-based Gesture System (Patent US 20220382384 A1.)](https://www.patentlyapple.com/2022/12/the-us-patent-office-published-a-2nd-apple-ring-patent-today-that-uses-a-self-mixing-interferometry-sensor-based-gesture-sy.html)
* [Apple Voltage Standing Wave Radio (VSWR) Sensors in their Future MR Headset, Smartglasses, iPhones+ (Patent US 20220381895 A1.)](https://www.patentlyapple.com/2022/12/apple-working-on-possibly-using-voltage-standing-wave-ration-vswr-sensors-in-their-future-mr-headset-smartglasses-iphones.html)
* [Apple MR Headset Cooling System (Patent US 20220394889 A1)](https://www.patentlyapple.com/2022/12/apple-invents-a-quiet-mr-headset-cooling-system-designed-to-make-it-comfortable-for-users-to-wear-ov.html)
* [Apple Clip-On Prescription Lenses System for a Future HMD (Eyeglasses and/or MR Headset) (Patent US 20220391608 A1)](https://www.patentlyapple.com/2022/12/apple-patent-reveals-a-clip-on-prescription-lenses-system-for-a-future-hmd-eyeglasses-andor-mr-headset.html)

**Apple Vision Pro Headset Specs:**

* Powered by an M2 chip and **R1 chip(dedicated toward real-time sensor processing for the 12 cameras and 5 sensors on the Vision Pro headset)**.
* High-resolution 4K micro OLED displays.
* Display refresh rate of [90Hz and supports a special 96Hz mode for 24fps video](https://developer.apple.com/videos/play/wwdc2023/10071/?time=143).
* Display Pixel Size will be between 3000 ppi-4000 ppi.
* Display offers a brightness of 5,000 nits.
* The color gamut is quoted as DCI > 97% DCI.
* 96W USB-C power adapter.
* A USB-C port for data transfer.
* A proprietary magnetic port(MagSafe) to attach the [battery pack](https://www.amazon.com/Apple-MJWY3AM-A-MagSafe-Battery/dp/B099BWY7WT).
* Expected battery life of about **2 hours per pack**.
* 12 optical cameras and 5 sensors for tracking movements, mapping the environment, and projecting visual experiences.
* WiFi 6E, which adds 6GHz spectrum to the 2.4GHz & 5GHz bands for increased bandwidth and less device interference.

OLEDoS (OLED on Silicon) is a display panel that typically has a diagonal length of less than 1 inch and meets the 3000 ppi-4000 ppi resolution criteria of AR/VR device displays. Existing OLED displays use Low-Temperature-Poly-Silicon (LTPS) or Oxide TFT based on glass substrates. OLEDoS uses silicon-wafer-based CMOS substrates. Using silicon substrates, ultra-fine circuit structures typically used in semiconductor processes can be reproduced, which in turn lead to the creation of ultra-high-resolution OLEDs when organic matter is deposited on them.

**LG OLEDoS Display Specs:**

* High-resolution 4K micro OLED displays.
* Display Pixel Size will be between 3000 ppi-4000 ppi.
* Display offers a brightness of 5,000 nits.
* The color gamut is quoted as DCI > 97% DCI.

OLEDoS: Solution for ultra-high resolution

Credit: [LG](https://www.lgdisplay.com/eng/technology/oled)

OLEDoS is also called Micro OLED in the market and features high efficiency, high luminance, infinite contrast, fast response and long LED life compared to OLED. The size is smaller than 1 inch, the user does not see the panel directly but sees the enlarged image through the optical lens. When used on AR/VR equipment, it shows high resolution in a small, lightweight wearable device.

Specs for OLED and OLEDoS

Credit: [Sony](https://www.sony.com/en/SonyInfo/research/technologies/OLED_microdisplay/)

#### VisionOS
[Back to the Top](#table-of-contents)

* [VisionOS](https://www.apple.com/newsroom/2023/06/introducing-apple-vision-pro/) is Apple's iOS-like operating system for their upcoming AR/VR (Augmented Reality/Virtual Reality) headset powered by Apple's Silicon M2 and R1 chips. It will have an App Store with apps that include mixed-reality versions of their core Apple apps like **[Messages](https://support.apple.com/messages), [FaceTime](https://support.apple.com/facetime), [Maps](https://www.apple.com/maps/), and AR/VR games from [Apple Arcade](https://www.apple.com/apple-arcade/)**. Along with features such as [Memojis](https://apps.apple.com/us/story/id1445637997) and [SharePlay](https://support.apple.com/guide/iphone/use-shareplay-to-watch-and-listen-together-iphb657eb791/ios) that could be central to the user experience.

VisionOS. Image Credit: Apple

**visionOS Resources**

* [visionOS SDK](https://developer.apple.com/visionos/)
* [Learn about visionOS](https://developer.apple.com/visionos/learn/)
* [Apple Vision Pro developer kit](https://developer.apple.com/visionos/developer-kit/)
* [Apple Vision Pro developer Labs](https://developer.apple.com/visionos/labs/)
* [Get started with building apps for spatial computing](https://developer.apple.com/videos/play/wwdc2023/10260/)
* [Apple Vision Pro compatibility evaluations](https://developer.apple.com/visionos/compatibility-evaluations/)
* [Checking whether your existing app is compatible with visionOS](https://developer.apple.com/documentation/visionOS/checking-whether-your-app-is-compatible-with-visionos/)
* [Making your existing app compatible with visionOS](https://developer.apple.com/documentation/visionos/making-your-app-compatible-with-visionos/)
* [Bringing your existing apps to visionOS](https://developer.apple.com/documentation/visionos/bringing-your-app-to-visionos/)
* [Interacting with your app in the visionOS simulator](https://developer.apple.com/documentation/visionOS/interacting-with-your-app-in-the-visionos-simulator/)
* [Diagnosing and resolving bugs in your running app](https://developer.apple.com/documentation/xcode/diagnosing-and-resolving-bugs-in-your-running-app/)
* [Running your app in Simulator or on a device](https://developer.apple.com/documentation/xcode/running-your-app-in-simulator-or-on-a-device/)
* [Diagnosing and resolving bugs in your running app](https://developer.apple.com/documentation/xcode/diagnosing-and-resolving-bugs-in-your-running-app/)
* [Improving your app’s performance](https://developer.apple.com/documentation/metrickit/improving_your_app_s_performance/)
* [Analyzing the performance of your Metal app](https://developer.apple.com/documentation/xcode/analyzing-the-performance-of-your-metal-app/)
* [Establishing UIRequiredDeviceCapabilities](https://developer.apple.com/documentation/bundleresources/information_property_list/uirequireddevicecapabilities/)
* [Game controllers (GCSupportsControllerUserInteraction)](https://developer.apple.com/design/human-interface-guidelines/game-controllers/)
* [Vision Pro Human Interface Guidelines](https://developer.apple.com/design/human-interface-guidelines/designing-for-visionos/)

#### Apple AR Toolkits and Frameworks

[Back to the Top](#table-of-contents)

* [Reality Composer Pro](https://developer.apple.com/videos/play/wwdc2023/10083/) is a tool that let's you discover how to easily compose, edit, and preview 3D content with Reality Composer Pro. Follow along as we explore this developer tool by setting up a new project, composing scenes, adding particle emitters and audio, and even previewing content on device.
* [Reality Composer](https://developer.apple.com/augmented-reality/tools/) is a powerful tool that makes it easy for you to create interactive augmented reality experiences with no prior 3D experience. The Reality Converter quickly converts your existing 3D models to [USDZ](https://graphics.pixar.com/usd/files/USDZFileFormatSpecification.pdf) so it works seamlessly in our tools and on all AR-enabled iPhone and iPad devices.
* [Apple Vision](https://developer.apple.com/documentation/vision) is a framework that performs face and face landmark detection, text detection, barcode recognition, image registration, and general feature tracking. Vision also allows the use of custom Core ML models for tasks like classification or object detection.
* [USDZ](https://graphics.pixar.com/usd/release/spec_usdz.html) is a file format that contains a 3D scene or object saved in the USDZ Universal format, which is developed by Apple and Pixar Animation Studios. It is an uncompressed and unencrypted .ZIP archive that stores a Universal Scene Description (.USD, USDA, or USDC) file, which includes 3D geometry and shading data. USDZ files may also contain .PNG and .JPEG image textures and .M4A, .MP3, or .WAV audio files utilized in the 3D object or scene.
* [ARKit](https://developer.apple.com/augmented-reality/arkit/) is a set set of software development tools to enable developers to build augmented-reality apps for iOS developed by Apple. The latest version ARKit 3.5 takes advantage of the new LiDAR Scanner and depth sensing system on iPad Pro(2020) to support a new generation of AR apps that use Scene Geometry for enhanced scene understanding and object occlusion.
* [RealityKit](https://developer.apple.com/documentation/realitykit) is a framework to implement high-performance 3D simulation and rendering with information provided by the ARKit framework to seamlessly integrate virtual objects into the real world.
* [RealityUI](https://github.com/maxxfrazer/RealityUI) is a Swift Package for creating familiar UI Elements and animations in a RealityKit rendered Augmented Reality or Virtual Reality scene.
* [SceneKit](https://developer.apple.com/scenekit/) is a high-level 3D graphics framework that helps you create 3D animated scenes and effects in your iOS apps.
* [UIKit](https://developer.apple.com/documentation/uikit) is a framework provides the required infrastructure for your iOS or tvOS apps. It provides the window and view architecture for implementing your interface, the event handling infrastructure for delivering Multi-Touch and other types of input to your app, and the main run loop needed to manage interactions among the user, the system, and your app.
* [SwiftUI](https://developer.apple.com/documentation/swiftui) is a user interface toolkit that provides views, controls, and layout structures for declaring your app's user interface. The SwiftUI framework provides event handlers for delivering taps, gestures, and other types of input to your application.
* [MetalFX](https://developer.apple.com/videos/play/wwdc2022/10103/) is a new API that provides platform optimized graphics effects for Metal applications. With MetalFX Upscaling, your application can now render frames at a lower resolution, reducing rendering time, without compromising rendering quality. We'll also show you how and when to use its two effects: spatial upscaling, which delivers substantial performance gains, and temporal AA and upscaling, which delivers the highest quality rendering.

MetalFX Rendering. Image Credit: Apple

* [Apple Core Animation Framework](https://developer.apple.com/documentation/quartzcore) is a graphics rendering and animation infrastructure that provides high frame rates and smooth animations without burdening the CPU and slowing down your app.
* [Apple Core Graphics Framework](https://developer.apple.com/documentation/coregraphics) is a framework based on the Quartz advanced drawing engine. It provides low-level, lightweight 2D rendering with unmatched output fidelity.
* [GPUImage3](https://github.com/BradLarson/GPUImage3) is the third generation of the [GPUImage framework](https://github.com/BradLarson/GPUImage), an open source project for performing GPU-accelerated image and video processing on Mac and iOS. The original GPUImage framework was written in Objective-C and targeted Mac and iOS, the second iteration rewritten in Swift using OpenGL to target Mac, iOS, and Linux, and now this third generation is redesigned to use [Apple's Metal](https://developer.apple.com/metal/) in place of OpenGL.
* [Unity AR Foundation](https://unity.com/unity/features/arfoundation) is a framework purpose-built for augmented reality development that allows you to build rich experiences once, then deploy across multiple mobile and wearable AR devices.
* [Universal Render Pipeline (URP)](https://docs.unity3d.com/Manual/universal-render-pipeline.html) is a prebuilt Scriptable Render Pipeline, made by Unity. It provides artist-friendly workflows that let you quickly and easily create optimized graphics across a range of platforms, from mobile to high-end consoles and PCs.

### HTC Vive Headset
[Back to the Top](#table-of-contents)

[HTC Vive Headset](https://www.vive.com/) is a virtual reality headset developed by HTC.

* [Getting Started with OpenVR in Unity](https://docs.unity3d.com/Manual/VRDevices-OpenVR.html)

* [SteamVR Plugin](https://assetstore.unity.com/packages/templates/systems/steamvr-plugin-32647)

* [Vive Studios](https://developer.vive.com/us/vive-studios/)

* [Vive Tracker](https://developer.vive.com/us/vive-tracker-for-developer/)

* [Vive Community](https://community.viveport.com/)

[htc-vive

### SteamVR
[Back to the Top](#table-of-contents)

[SteamVR](https://store.steampowered.com/steamvr) is the ultimate tool for experiencing VR content on the hardware of your choice. SteamVR supports the Valve Index, HTC Vive, Oculus Rift, Windows Mixed Reality headsets, and others.

SteamVR Home

Valve Index VR Headset. Source: [Steam](https://store.steampowered.com/valveindex)

### Nreal Air AR Glasses

[Back to the Top](#table-of-contents)

[Nreal Air AR Glasses & Nreal Adapter](https://www.amazon.com/dp/B0BM4NGK93?th=1) is a pair AR Glasses that you Watch, Stream, and Game on PC, Android, iOS, Nintendo Switch, Steam Deck, Playstation 4|5, Xbox Series X|S, and Cloud Gaming.

### Samsung Gear VR Headset

[Samsung Gear VR Headset](https://www.samsung.com/global/galaxy/gear-vr/) is a virtual reality headset developed by Samsung along with collaboration from Oculus VR.

* [Mobile Development with Unity and Unreal](https://developer.oculus.com/documentation/mobilesdk/latest/concepts/mobile-game-engine/#mobile-game-engine)

* [Preparing for Mobile Development](https://developer.oculus.com/documentation/unity/latest/concepts/unity-mobileprep/)

* [Gear VR Device Setup](https://developer.oculus.com/documentation/mobilesdk/latest/concepts/mobile-device-setup/)

* [Building Gear VR apps in Unity](https://developer.oculus.com/documentation/unity/latest/concepts/unity-build-android/#unity-build-android-store)

[gear-vr

### Perception Neuron Motion Capture Suit

[Back to the Top](#table-of-contents)

[Perception Neuron Motion Capture Suit](https://neuronmocap.com/products-fullwidth?field_category_value=PN) is a series of industry-leading motion capture solutions designed for every type of creator, from the amateur 3D animator to the VFX pro.

[axis neuron

[perceptionneuron

# Vulkan Development
[Back to the Top](https://github.com/mikeroyal/AR-VR-Guide#table-of-contents)

## Vulkan Learning Resources

[Vulkan®](https://www.khronos.org/vulkan/) is a modern cross-platform graphics and compute API that provides high-efficiency, cross-platform access to modern GPUs used in a wide variety of devices from PCs and consoles to mobile phones and embedded platforms. Vulkan is currently in development by the Khronos consortium.

[Khronos Group GitHub](https://github.com/KhronosGroup)

[Vulkan Documentation](https://github.com/KhronosGroup/Vulkan-Docs)

[HLSL to SPIR-V Feature Mapping Manual](https://github.com/microsoft/DirectXShaderCompiler/blob/master/docs/SPIR-V.rst)

[Vulkan GLSL Ray Tracing Emulator Tutorial](https://www.gsn-lib.org/docs/nodes/raytracing.php)

[Getting Started with Vulkan](https://vulkan-tutorial.com/)

[Vulkan Samples](https://github.com/KhronosGroup/Vulkan-Samples)

[Khronos Community Forums](https://community.khronos.org/)

## Vulkan Tools, Libraries, and Frameworks

[Vulkan SDK](https://vulkan.lunarg.com) is a set of tools that enables Vulkan developers to develop Vulkan applications.

[SPIR-V](https://www.khronos.org/spir/) is a set of tools that enables high-level language front-ends to emit programs in a standardized intermediate form to be ingested by Vulkan, OpenGL or OpenCL drivers. It eliminates the need for high-level language front-end compilers in device drivers, significantly reducing driver complexity, enables a broad range of language and framework front-ends to run on diverse hardware architectures and encourages a vibrant ecosystem of open source analysis, porting, debug and optimization tools.

[SPIRV-Reflect](https://github.com/KhronosGroup/SPIRV-Reflect) is a lightweight library that provides a C/C++ reflection API for SPIR-V shader bytecode in Vulkan applications.

[Vulkan® Tools](https://github.com/KhronosGroup/Vulkan-Tools) is a project that provides Khronos official Vulkan Tools and Utilities for Windows, Linux, Android, and macOS.

[Vulkan-Hpp](https://github.com/KhronosGroup/Vulkan-Hpp) is a API that provides a header only C++ bindings for the Vulkan C API to improve the developers Vulkan experience without introducing CPU runtime cost. It adds features like type safety for enums and bitfields, STL container support, exceptions and simple enumerations.

[Vulkan® Memory Allocator (VMA)](https://gpuopen.com/vulkan-memory-allocator/) is a library that provides a simple and easy to integrate API to help you allocate memory for Vulkan® buffer and image storage.

[AMD Open Source Driver for Vulkan®](https://gpuopen.com/amd-open-source-driver-for-vulkan/) is an open-source Vulkan driver for AMD Radeon™ graphics adapters on Linux®.

[NVIDIA® Nsight™ Visual Studio Edition](https://developer.nvidia.com/nsight-visual-studio-edition) is an application development environment for heterogeneous platforms which brings GPU computing into Microsoft Visual Studio. NVIDIA Nsight™ VSE allows you to build and debug integrated GPU kernels and native CPU code as well as inspect the state of the GPU and memory.

[Radeon™ GPU Profiler](https://gpuopen.com/rgp/) is a performance tool that can be used by developers to optimize DirectX®12, Vulkan® and OpenCL™ applications for AMD RDNA™ and GCN hardware.

[Radeon™ GPU Analyzer](https://gpuopen.com/rga/) is a compiler and code analysis tool for Vulkan®, DirectX®, OpenGL® and OpenCL™.

[Radeon™ Memory Visualizer (RMV)](https://gpuopen.com/rmv/) is a tool provided by AMD for use by game engine developers. It allows engineers to examine, diagnose, and understand the GPU memory management within their projects.

[DXVK](https://github.com/doitsujin/dxvk) is a Vulkan-based translation layer for Direct3D 9/10/11 which allows running 3D applications on Linux using Wine.

[MoltenVK](https://moltengl.com/moltenvk) is an implementation of Vulkan running on iOS and macOS using Apple's [Metal](https://developer.apple.com/metal/) graphics framework.

[RenderDoc](https://renderdoc.org) is a stand-alone graphics debugger that allows quick and easy single-frame capture and detailed introspection of any application using Vulkan, D3D11, OpenGL & OpenGL ES or D3D12 across Windows, Linux, Android, Stadia, or Nintendo Switch™.

[PerfDoc](https://github.com/ARM-software/perfdoc) is a cross-platform Vulkan layer which checks Vulkan applications for [best practices on Arm Mali](https://developer.arm.com/graphics/developer-guides/mali-gpu-best-practices) devices.

[GLFW](https://www.glfw.org/) is an Open Source, multi-platform library for OpenGL, OpenGL ES and Vulkan application development. It provides a simple, platform-independent API for creating windows, contexts and surfaces, reading input, handling events, etc. GLFW natively supports Windows, macOS and Linux and other Unix-like systems. On Linux both X11 and Wayland are supported.

[VulkanSharp](https://github.com/mono/VulkanSharp) is a project provides a .NET binding for the Vulkan API.

[Vortice.Vulkan](https://github.com/amerkoleci/Vortice.Vulkan) is a .NET Standard 2.0 and .NET5 low-level bindings for Vulkan API.

[VKD3D-Proton](https://github.com/HansKristian-Work/vkd3d-proton) is a fork of VKD3D, which aims to implement the full Direct3D 12 API on top of Vulkan.

[ImGui](https://github.com/ocornut/imgui) is a bloat-free graphical user interface library for C++. It outputs optimized vertex buffers that you can render anytime in your 3D-pipeline enabled application. It is fast, portable, renderer agnostic and self-contained (no external dependencies).

[Ash](https://github.com/MaikKlein/ash) is a very lightweight wrapper around Vulkan.

[gfx-rs](https://github.com/gfx-rs/gfx) is a low-level, cross-platform graphics and compute abstraction library in Rust.

[Vulkan.jl](https://github.com/JuliaGPU/Vulkan.jl) is a lightweight wrapper around the Vulkan graphics and compute library. It exposes abstractions over the underlying C interface, primarily geared towards developers looking for a more natural way to work with Vulkan with minimal overhead.

# Metal Development
[Back to the Top](https://github.com/mikeroyal/AR-VR-Guide#table-of-contents)

## Metal Learning Resources

[Metal](https://developer.apple.com/metal/) is a low-level API that provides a platform-optimized, low-overhead API for developing the latest 3D pro applications and amazing games using a rich shading language with tighter integration between graphics and compute programs. To help you do more while managing ever more complex shader code, Metal adds an unparalleled suite of advanced GPU debugging tools to help you realize the full potential of your graphics code.

[Apple Developer Documentation](https://developer.apple.com/documentation)

[MetalKit](https://developer.apple.com/documentation/metalkit/)

[Metal Shading Language Specification](https://developer.apple.com/metal/Metal-Shading-Language-Specification.pdf)

[Using Metal Feature Set Tables](https://developer.apple.com/documentation/metal/gpu_features/using_metal_feature_set_tables/)

[Metal Performance Shaders](https://developer.apple.com/documentation/metalperformanceshaders/)

[Optimizing Performance with the GPU Counters Instrument](https://developer.apple.com/documentation/metal/optimizing_performance_with_the_gpu_counters_instrument?language=objc)

[Enabling Frame Capture](https://developer.apple.com/documentation/metal/frame_capture_debugging_tools/enabling_frame_capture?language=objc)

[Reducing the Memory Footprint of Metal Apps](https://developer.apple.com/documentation/metal/reducing_the_memory_footprint_of_metal_apps)

[Metal Developer Tools for Windows](https://developer.apple.com/download/release/)

[Metal Sample code](https://developer.apple.com/metal/sample-code/)

[Metal plugin for TensorFlow](https://developer.apple.com/metal/tensorflow-plugin/)

[Metal Developer discussions](https://developer.apple.com/forums/tags/metal/)

## Metal Tools, Libraries, and Frameworks

[Apple Foundation Framework](https://developer.apple.com/documentation/foundation) is a framework provides a base layer of functionality for apps and frameworks, including data storage and persistence, text processing, date and time calculations, sorting and filtering, and networking. The classes, protocols, and data types defined by Foundation are used throughout the macOS, iOS, watchOS, and tvOS SDKs.

[Apple Core Animation Framework](https://developer.apple.com/documentation/quartzcore) is a graphics rendering and animation infrastructure that provides high frame rates and smooth animations without burdening the CPU and slowing down your app.

[Apple Core Graphics Framework](https://developer.apple.com/documentation/coregraphics)is a framework based on the Quartz advanced drawing engine. It provides low-level, lightweight 2D rendering with unmatched output fidelity.

[Paravirtualized Graphics Framework](https://developer.apple.com/documentation/paravirtualizedgraphics) is a framework that implements hardware-accelerated graphics for macOS running in a virtual machine, hereafter known as the guest. The macOS operating system provides a graphics driver that runs inside the guest, communicating with the framework in the host operating system to take advantage of Metal-accelerated graphics.

[Xcode](https://developer.apple.com/xcode/) includes everything developers need to create great applications for Mac, iPhone, iPad, Apple TV, and Apple Watch. Xcode provides developers a unified workflow for user interface design, coding, testing, and debugging. Xcode 12 is built as an Universal app that runs 100% natively on Intel-based CPUs and Apple Silicon. It includes a unified macOS SDK that features all the frameworks, compilers, debuggers, and other tools you need to build apps that run natively on Apple Silicon and the Intel x86_64 CPU.

[SwiftUI](https://developer.apple.com/documentation/swiftui) is a user interface toolkit that provides views, controls, and layout structures for declaring your app's user interface. The SwiftUI framework provides event handlers for delivering taps, gestures, and other types of input to your application.

[UIKit](https://developer.apple.com/documentation/uikit) is a framework provides the required infrastructure for your iOS or tvOS apps. It provides the window and view architecture for implementing your interface, the event handling infrastructure for delivering Multi-Touch and other types of input to your app, and the main run loop needed to manage interactions among the user, the system, and your app.

[AppKit](https://developer.apple.com/documentation/appkit) is a graphical user interface toolkit that contains all the objects you need to implement the user interface for a macOS app such as windows, panels, buttons, menus, scrollers, and text fields, and it handles all the details for you as it efficiently draws on the screen, communicates with hardware devices and screen buffers, clears areas of the screen before drawing, and clips views.

[SceneKit](https://developer.apple.com/scenekit/) is a high-level 3D graphics framework that helps you create 3D animated scenes and effects in your iOS apps.

[Instruments](https://help.apple.com/instruments/mac/current/#/dev7b09c84f5) is a powerful and flexible performance-analysis and testing tool that’s part of the Xcode tool set. It’s designed to help you profile your iOS, watchOS, tvOS, and macOS apps, processes, and devices in order to better understand and optimize their behavior and performance.

[Cocoapods](https://cocoapods.org/) is a dependency manager for Swift and Objective-C used in Xcode projects by specifying the dependencies for your project in a simple text file. CocoaPods then recursively resolves dependencies between libraries, fetches source code for all dependencies, and creates and maintains an Xcode workspace to build your project.

[AppCode](https://www.jetbrains.com/objc/) is constantly monitoring the quality of your code. It warns you of errors and smells and suggests quick-fixes to resolve them automatically. AppCode provides lots of code inspections for Objective-C, Swift, C/C++, and a number of code inspections for other supported languages.

[MoltenVK](https://moltengl.com/moltenvk) is an implementation of Vulkan running on iOS and macOS using Apple's [Metal](https://developer.apple.com/metal/) graphics framework.

# DirectX Development
[Back to the Top](https://github.com/mikeroyal/AR-VR-Guide#table-of-contents)

## DirectX Learning Resources

[Microsoft DirectX®](https://support.microsoft.com/en-us/topic/how-to-install-the-latest-version-of-directx-d1f5ffa5-dae2-246c-91b1-ee1e973ed8c2) is a low-level API that handles tasks related to multimedia for game programming and video on Microsoft platforms(Windows & Xbox).

[Getting Started with DirectX 12 Ultimate](https://devblogs.microsoft.com/directx/directx-12-ultimate-getting-started-guide/)

[Getting Started with the DirectX 12 Agility SDK](https://devblogs.microsoft.com/directx/gettingstarted-dx12agility/)

[DirectX 12 and Graphics Education | YouTube](https://www.youtube.com/channel/UCiaX2B8XiXR70jaN7NK-FpA)

[DirectX— Feature Level 12_2](https://devblogs.microsoft.com/directx/new-in-directx-feature-level-12_2/)

[DirectX 12 Technology | NVIDIA](https://www.nvidia.com/en-us/geforce/technologies/dx12/)

[AMD DirectX® 12 (DX12) Technology | AMD](https://www.amd.com/en/technologies/directx12)

[Top Microsoft DirectX Courses Online | Udemy](https://www.udemy.com/topic/microsoft-directx/)

[DirectX - Learn Microsoft DirectX from Scratch Course | Udemy](https://www.udemy.com/course/directx-learn-microsoft-directx-from-scratch/)

[DirectX 11 Programming Course | Udemy](https://www.udemy.com/course/directx11/)

## DirectX Tools, Libraries, and Frameworks

[Visual Studio](https://visualstudio.microsoft.com/) is an integrated development environment (IDE) from Microsoft; which is a feature-rich application that can be used for many aspects of software development. Visual Studio makes it easy to edit, debug, build, and publish your app. By using Microsoft software development platforms such as Windows API, Windows Forms, Windows Presentation Foundation, and Windows Store.

[Visual Studio Code](https://code.visualstudio.com/) is a code editor redefined and optimized for building and debugging modern web and cloud applications.

[DirectX-Graphics-Samples](https://github.com/Microsoft/DirectX-Graphics-Samples) is a project that contains the DirectX 12 Graphics samples that demonstrate how to build graphics intensive applications for Windows 10.

[PIX on Windows](https://devblogs.microsoft.com/pix/documentation/) is a performance tuning and debugging tool for DirectX 12 games on Windows.

[DirectStorage API](https://devblogs.microsoft.com/directx/directstorage-is-coming-to-pc/) is an API in the DirectX family originally designed for the [Velocity Architecture](https://news.xbox.com/en-us/2020/07/14/a-closer-look-at-xbox-velocity-architecture/) to Windows. The DirectX API is architected in a way that takes all this into account and maximizes performance throughout the entire pipeline from NVMe drive all the way to the GPU. It does this in several ways: by reducing per-request NVMe overhead, enabling batched many-at-a-time parallel IO requests which can be efficiently fed to the GPU, and giving games finer grain control over when they get notified of IO request completion instead of having to react to every tiny IO completion. The DirectStorage API will be available on [Windows 11](https://www.microsoft.com/en-us/windows/windows-11) PCs with NVMe SSDs, but will also be support in [Windows 10](https://www.microsoft.com/software-download/windows10) version 1909 and newer.

[NVRHI (NVIDIA Rendering Hardware Interface)](https://github.com/NVIDIAGameWorks/nvrhi) is a library that implements a common abstraction layer over multiple graphics APIs (GAPIs): Direct3D 11, Direct3D 12, and Vulkan 1.2. It works on Windows (x64 only) and Linux (x64 and ARM64).

[RTXMU - RTX Memory Utility SDK](https://github.com/NVIDIAGameWorks/RTXMU) is an SDK tool that batchs up all of the acceleration structure build inputs and pass them to RTXMU which in turn will perform all the suballocation memory requests and build details including compaction. Then post build info is abstracted away by the SDK in order to do compaction under the hood. RTXMU returns acceleration structure handle ids that are used to reference the underlying memory buffers. These handle ids are passed into RTXMU to create compaction copy workloads, deallocate unused build resources or remove all memory associated with an acceleration structure.

[Radeon™ GPU Profiler](https://gpuopen.com/rgp/) is a performance tool that can be used by developers to optimize DirectX®12, Vulkan® and OpenCL™ applications for AMD RDNA™ and GCN hardware.

[Radeon™ GPU Analyzer](https://gpuopen.com/rga/) is a compiler and code analysis tool for Vulkan®, DirectX®, OpenGL® and OpenCL™.

[FNA](https://fna-xna.github.io/) is an XNA4 reimplementation that focuses solely on developing a fully accurate XNA4 runtime for the desktop.

[FAudio](https://fna-xna.github.io/) is an XAudio reimplementation that focuses solely on developing fully accurate DirectX Audio runtime libraries for the FNA project, including [XAudio2](https://docs.microsoft.com/en-us/windows/win32/xaudio2/xaudio2-introduction), [X3DAudio](https://docs.microsoft.com/en-us/windows/win32/xaudio2/x3daudio-overview), [XAPO](https://docs.microsoft.com/en-us/windows/win32/xaudio2/xapo-overview), and [XACT3](https://en.wikipedia.org/wiki/Cross-platform_Audio_Creation_Tool).

[Simple DirectMedia Layer](https://www.libsdl.org/) is a cross-platform development library designed to provide low level access to audio, keyboard, mouse, joystick, and graphics hardware via OpenGL and Direct3D. It is used by video playback software, emulators, and popular games including Valve's award winning catalog.

[DXVK](https://github.com/doitsujin/dxvk) is a Vulkan-based translation layer for Direct3D 9/10/11 which allows running 3D applications on Linux using Wine.

[VKD3D-Proton](https://github.com/HansKristian-Work/vkd3d-proton) is a fork of VKD3D, which aims to implement the full Direct3D 12 API on top of Vulkan.

# Computer Vision Development
[Back to the Top](https://github.com/mikeroyal/AR-VR-Guide#table-of-contents)

## Computer Vision Learning Resources

[Computer Vision](https://azure.microsoft.com/en-us/overview/what-is-computer-vision/) is a field of Artificial Intelligence (AI) that focuses on enabling computers to identify and understand objects and people in images and videos.

[OpenCV Courses](https://opencv.org/courses/)

[Exploring Computer Vision in Microsoft Azure](https://docs.microsoft.com/en-us/learn/paths/explore-computer-vision-microsoft-azure/)

[Top Computer Vision Courses Online | Coursera](https://www.coursera.org/courses?languages=en&query=computer%20vision)

[Top Computer Vision Courses Online | Udemy](https://www.udemy.com/topic/computer-vision/)

[Learn Computer Vision with Online Courses and Lessons | edX](https://www.edx.org/learn/computer-vision)

[Computer Vision and Image Processing Fundamentals | edX](https://www.edx.org/course/computer-vision-and-image-processing-fundamentals)

[Introduction to Computer Vision Courses | Udacity](https://www.udacity.com/course/introduction-to-computer-vision--ud810)

[Computer Vision Nanodegree program | Udacity](https://www.udacity.com/course/computer-vision-nanodegree--nd891)

[Machine Vision Course |MIT Open Courseware ](https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-801-machine-vision-fall-2004/)

[Computer Vision Training Courses | NobleProg](https://www.nobleprog.com/computer-vision-training)

[Visual Computing Graduate Program | Stanford Online](https://online.stanford.edu/programs/visual-computing-graduate-program)

## Computer Vision Tools, Libraries, and Frameworks

[OpenCV](https://opencv.org) is a highly optimized library with focus on real-time computer vision applications. The C++, Python, and Java interfaces support Linux, MacOS, Windows, iOS, and Android.

[Microsoft Cognitive Toolkit (CNTK)](https://docs.microsoft.com/en-us/cognitive-toolkit/) is an open-source toolkit for commercial-grade distributed deep learning. It describes neural networks as a series of computational steps via a directed graph. CNTK allows the user to easily realize and combine popular model types such as feed-forward DNNs, convolutional neural networks (CNNs) and recurrent neural networks (RNNs/LSTMs). CNTK implements stochastic gradient descent (SGD, error backpropagation) learning with automatic differentiation and parallelization across multiple GPUs and servers.

[Scikit-Learn](https://scikit-learn.org/stable/index.html) is a Python module for machine learning built on top of SciPy, NumPy, and matplotlib, making it easier to apply robust and simple implementations of many popular machine learning algorithms.

[NVIDIA cuDNN](https://developer.nvidia.com/cudnn) is a GPU-accelerated library of primitives for [deep neural networks](https://developer.nvidia.com/deep-learning). cuDNN provides highly tuned implementations for standard routines such as forward and backward convolution, pooling, normalization, and activation layers. cuDNN accelerates widely used deep learning frameworks, including [Caffe2](https://caffe2.ai/), [Chainer](https://chainer.org/), [Keras](https://keras.io/), [MATLAB](https://www.mathworks.com/solutions/deep-learning.html), [MxNet](https://mxnet.incubator.apache.org/), [PyTorch](https://pytorch.org/), and [TensorFlow](https://www.tensorflow.org/).

[Automated Driving Toolbox™](https://www.mathworks.com/products/automated-driving.html) is a MATLAB tool that provides algorithms and tools for designing, simulating, and testing ADAS and autonomous driving systems. You can design and test vision and lidar perception systems, as well as sensor fusion, path planning, and vehicle controllers. Visualization tools include a bird’s-eye-view plot and scope for sensor coverage, detections and tracks, and displays for video, lidar, and maps. The toolbox lets you import and work with HERE HD Live Map data and OpenDRIVE® road networks. It also provides reference application examples for common ADAS and automated driving features, including FCW, AEB, ACC, LKA, and parking valet. The toolbox supports C/C++ code generation for rapid prototyping and HIL testing, with support for sensor fusion, tracking, path planning, and vehicle controller algorithms.

[LRSLibrary](https://github.com/andrewssobral/lrslibrary) is a Low-Rank and Sparse Tools for Background Modeling and Subtraction in Videos. The library was designed for moving object detection in videos, but it can be also used for other computer vision and machine learning problems.

[Image Processing Toolbox™](https://www.mathworks.com/products/image.html) is a tool that provides a comprehensive set of reference-standard algorithms and workflow apps for image processing, analysis, visualization, and algorithm development. You can perform image segmentation, image enhancement, noise reduction, geometric transformations, image registration, and 3D image processing.

[Computer Vision Toolbox™](https://www.mathworks.com/products/computer-vision.html) is a tool that provides algorithms, functions, and apps for designing and testing computer vision, 3D vision, and video processing systems. You can perform object detection and tracking, as well as feature detection, extraction, and matching. You can automate calibration workflows for single, stereo, and fisheye cameras. For 3D vision, the toolbox supports visual and point cloud SLAM, stereo vision, structure from motion, and point cloud processing.

[Statistics and Machine Learning Toolbox™](https://www.mathworks.com/products/statistics.html) is a tool that provides functions and apps to describe, analyze, and model data. You can use descriptive statistics, visualizations, and clustering for exploratory data analysis; fit probability distributions to data; generate random numbers for Monte Carlo simulations, and perform hypothesis tests. Regression and classification algorithms let you draw inferences from data and build predictive models either interactively, using the Classification and Regression Learner apps, or programmatically, using AutoML.

[Lidar Toolbox™](https://www.mathworks.com/products/lidar.html) is a tool that provides algorithms, functions, and apps for designing, analyzing, and testing lidar processing systems. You can perform object detection and tracking, semantic segmentation, shape fitting, lidar registration, and obstacle detection. Lidar Toolbox supports lidar-camera cross calibration for workflows that combine computer vision and lidar processing.

[Mapping Toolbox™](https://www.mathworks.com/products/mapping.html) is a tool that provides algorithms and functions for transforming geographic data and creating map displays. You can visualize your data in a geographic context, build map displays from more than 60 map projections, and transform data from a variety of sources into a consistent geographic coordinate system.

[UAV Toolbox](https://www.mathworks.com/products/uav.html) is an application that provides tools and reference applications for designing, simulating, testing, and deploying unmanned aerial vehicle (UAV) and drone applications. You can design autonomous flight algorithms, UAV missions, and flight controllers. The Flight Log Analyzer app lets you interactively analyze 3D flight paths, telemetry information, and sensor readings from common flight log formats.

[Parallel Computing Toolbox™](https://www.mathworks.com/products/matlab-parallel-server.html) is a tool that lets you solve computationally and data-intensive problems using multicore processors, GPUs, and computer clusters. High-level constructs such as parallel for-loops, special array types, and parallelized numerical algorithms enable you to parallelize MATLAB® applications without CUDA or MPI programming. The toolbox lets you use parallel-enabled functions in MATLAB and other toolboxes. You can use the toolbox with Simulink® to run multiple simulations of a model in parallel. Programs and models can run in both interactive and batch modes.

[Partial Differential Equation Toolbox™](https://www.mathworks.com/products/pde.html) is a tool that provides functions for solving structural mechanics, heat transfer, and general partial differential equations (PDEs) using finite element analysis.

[ROS Toolbox](https://www.mathworks.com/products/ros.html) is a tool that provides an interface connecting MATLAB® and Simulink® with the Robot Operating System (ROS and ROS 2), enabling you to create a network of ROS nodes. The toolbox includes MATLAB functions and Simulink blocks to import, analyze, and play back ROS data recorded in rosbag files. You can also connect to a live ROS network to access ROS messages.

[Robotics Toolbox™](https://www.mathworks.com/products/robotics.html) provides a toolbox that brings robotics specific functionality(designing, simulating, and testing manipulators, mobile robots, and humanoid robots) to MATLAB, exploiting the native capabilities of MATLAB (linear algebra, portability, graphics). The toolbox also supports mobile robots with functions for robot motion models (bicycle), path planning algorithms (bug, distance transform, D*, PRM), kinodynamic planning (lattice, RRT), localization (EKF, particle filter), map building (EKF) and simultaneous localization and mapping (EKF), and a Simulink model a of non-holonomic vehicle. The Toolbox also including a detailed Simulink model for a quadrotor flying robot.

[Deep Learning Toolbox™](https://www.mathworks.com/products/deep-learning.html) is a tool that provides a framework for designing and implementing deep neural networks with algorithms, pretrained models, and

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/mikeroyal/AR-VR-Guide

Awesome Lists containing this project

README

AR/VR Guide