{"id":16533402,"url":"https://github.com/joshperry/ovrly","last_synced_at":"2026-06-21T01:01:18.637Z","repository":{"id":137663483,"uuid":"254489138","full_name":"joshperry/ovrly","owner":"joshperry","description":"OpenVR Webtech Overlay Engine","archived":false,"fork":false,"pushed_at":"2023-12-24T00:13:38.000Z","size":2846,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"linix","last_synced_at":"2025-03-03T12:33:11.701Z","etag":null,"topics":["cef","openvr","openvr-overlay","vr"],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/joshperry.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-09T22:13:21.000Z","updated_at":"2023-12-19T19:28:22.000Z","dependencies_parsed_at":"2023-12-24T01:24:16.945Z","dependency_job_id":null,"html_url":"https://github.com/joshperry/ovrly","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/joshperry/ovrly","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joshperry%2Fovrly","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joshperry%2Fovrly/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joshperry%2Fovrly/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joshperry%2Fovrly/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/joshperry","download_url":"https://codeload.github.com/joshperry/ovrly/tar.gz/refs/heads/linix","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joshperry%2Fovrly/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34590216,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-20T02:00:06.407Z","response_time":98,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cef","openvr","openvr-overlay","vr"],"created_at":"2024-10-11T18:14:45.048Z","updated_at":"2026-06-21T01:01:18.573Z","avatar_url":"https://github.com/joshperry.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ovrly\n\nNB: this `linix` branch from master will probably persist for a bit (today: 4/21/23) until I get time to do windows integration testing.\n\n    ovrly manages vr overlays and runs browser overlies\n\nThe purpose of this project is to compose [chromium embedded](https://bitbucket.org/chromiumembedded/cef/src/master/)\nwith [OpenVR](https://github.com/ValveSoftware/openvr) to allow rapid creation of, and experimentation with,\nVR overlays whose logic and rendering are handled by web technologies.\n\nWhy Though? Well, though they are anemic feature-wise, up to this point they are the only HCI in the VR\nworld that can persist across different VR app executions. To use an anachronism, they're like the\n[TSRs](https://en.wikipedia.org/wiki/Terminate_and_stay_resident_program) of the DOS days, which brought\nthe first instance of persistent multitasking into a single-tasking non-reentrant world.\n\nThis is the state of VR today.\n\nBesides using cef to create the content of overlays placed in the VR world, an OS-windowed browser instance\nwill be shown as the primary desktop UI and hosts the root context where the ovrly core javascript-side\nlogic is executed.\n\n## ovrlies\n\n    Extensibility points should be rich and experimentally compelling.\n\nAn ovrlie is an \"app\" for ovrly.\n\nThe interface for integrating a custom ovrlie is meant to be simple while supporting the widest possible\nrange of target development and runtime environments. Because ovrly employs a fully-featured browser runtime\nand HTTP server libraries are ubiquitous, it makes sense to use HTTP for content delivery and IPC.\n\nEach ovrlie implementation must provide an http server executable (written in any OS executable format)\nwith a `--port` argument.\n\nWhen loading an overlie, the server process is started as a child daemon and passed a random available local\nhighport, a root browser context is then spun up and pointed at `http://localhost:\u003crandomport\u003e/`.\nThis URL should serve the HTML/JS/CSS assets that implement the ovrlie client logic and UI.\n\n- TODO: Do we need to synchronize browser load with the server process startup somehow?\n\nThe content area of the root browser context is rendered into a root VR overlay that is also created\nat ovrlie start time, the ovrly javascript APIs are bound in the browser javascript context where\ncode can manipulate them at will.\n\nA primary benefit offered by the daemon model is that overlies can directly access OS resources\nthat are not available through the ovrly browser environment. This access can be shared with the browser\ncontext via HTTP requests or websockets back to the daemon.\n\n- TODO: Expose desktop UI for ovrlies?\n\n- TODO: Securing access to the server daemon\n- TODO: Consider security and other aspects related to loading ovrlies directly from public https URIs\n  - Any ovrlie without the need for a daemon could deliver assets over the internet.\n  - (pro) Zero install or update\n  - (con) Scripts loaded from the internet have access to the ovrly javascript API\n\n## Javascript API\n\nTo facilitate ease of development and updates it is desirable to have as much logic implemented in javascript as possible.\nWhile logic in the input-\u003erender loop is time sensitive and must be implemented in native code, a large majority is not.\n\n    Some experiments need to be conducted as to what kind of latency is introduced by shifting logic into js.\n    The primary use-case which may need to be done in native code, but possibly could be done in js, is what\n    effectively could be considered the ovrly \"window manager\".\n\n    Will js be able to track input controller pose and button updates quickly enough to detect intersections\n    and interactions, execute logic, and then update overlay state with low enough latency that the\n    interaction experience will be comfortable?\n\nAll javascript interface will be exposed beneath the global `ovrly` object, its existence can be used by\ncode to detect it is running in the ovrly runtime environment.\n\n*ovrly.initialized* (bool): Whether everything was successfully initialized.\nIf this is false, only the string property `ovrly.error` will exist with a description of the init error.\n\n### Overlay\n\nThe primary purpose of ovrly is to expose and manage one or more overlays into the OpenVR environment.\n\n*ovrly.Overlay(options)*: Constructor function to create an overlay.\n\n```js\n{\n  id: 'is.invr.ovrly-main',\n  name: 'ovrly main',\n  matrix: [3][4],\n}\n```\n\n*ovrly.root* (Overlay): The root overlay instance created for an overlie. Does not exist in the ovrly root js context.\n\nTODO: Hash out the overlay JS interface\n\n### OpenVR\n\n\nInformation about the VR environment is available through the OpenVR API which is bound globally in\neach js context at `ovrly.vr`.\n\n- TODO: chaperone bounds/playarea info\n\n- TODO: tracked device info\n\nTODO: Hash out the openvr JS interface\n\n### Facade\n\nOwing to the archaic nature of the native interface exposed by v8, the API exported from native code into the\njs context is rudimentary. To make this API more idiomatic javascript, a facade will sit on top\nof the native exports to bridge that gap.\n\nThe interface described above is this ideomatic interface.\n\n### Low-level JS Interface\n\nThis describes the low-level JS interface that the ideomatic facade described above uses under the covers.\n\n### Native to JS Event Dispatch\n\nThere will be single-source events and APIs exposed in the browser process, for openvr and other modules,\nto 1..n javascript contexts running separately in each render process.\nFor events, not every context will be interested in every type, so a method is needed for subscribing and\nthen dispatching specific events between processes.\n\nBecause of its high-performance and low-latency, [ZeroMQ](zeromq.org) was chosen for communications\nbetween the browser process and the render processes. The cef interprocess messaging system was explored for\nthis task and was found to be lacking in a number of places for the efficient delivery of realtime data.\n\nBecause zmq supports zero-copy transport of opaque binary messages, and both the sending and receiving\nsides are guaranteed to be processes of the same executable on the same machine, this should allow the use\nof extremely high-performance methods of serializing data (see `serralize.hpp`).\n\nThe browser process sets up an xpub socket listening on localhost, to which each render process connects\nusing a sub socket and sets up needed subscriptions. Because zmq's IPC transport only supports unix\ndomain sockets, a localhost TCP connection is the only cross-platform method of IPC that it offers.\n\nTODO: Implement xpub subscriptions to filter message topics on the server side instead of the client.\n\n## C++ Architecture\n\nThe ovrly C++ architecture is primarily wrappers and specializations interconnected\nvia event dispatch.\n\nA bulk of the modules in the project expose an interface which calls into the OpenVR or cef\ndependency libraries. The modules are organized primarily in a data-oriented fashion.\n\nThe API exposed by each module, which facilitate interaction between the multiple modules, each\nhave both a functional interface and an event-driven interface. Event subscription is accomplished\nwith Observables to which many target listeners can subscribe.\n\n### App\n\nImplements the CefApp interface and is in charge of dispatching logic between code\nintented to run in the browser or render process on startup.\n\nIt exposes two observables, `OnBrowser` and `OnRender` which are dispatched in\ntheir respective process types.\n\nIt also exposes a task dispatch function `runOnMain` which will run a provided\ncode on the process' main thread.\n\n### GFX\n\nThis is defines and implements access to the the graphics system like OpenGL,\nVulkan, or DirectX. It's primary purpose is to provide an abstract interface\nover these different techs to provide for xplat.\n\nThis is accomplished through duck-typing; a platform-specific header is\nincluded with conditional ifdefs, and the build system will compile/link only\nthe OS-specific source module.\n\nOvrly has fairly simple needs as far as the graphics subsystem is concerned. It\nonly needs to create and blit into a texture which is given to the OpenVR\noverlay for rendering.\n\nIt's possible that OpenGL or Vulkan may eventually be the only needed backend.\n\n### JS\n\nThis contains the logic that mediates between OpenVR and browser JS instances\n(one in each render process). This acts similarly to other cef interfaces which\nuse IPC to single-source certain logic (like the gpu-process) so that we only\nhave a single connection to the OpenVR API.\n\nOn the browser process it uses the OpenVR API to enumerate the interface\n(hardware, playarea, etc.) and receive events for changes to the environment\n(input events, positional events, hardware state changes, etc), it also handles\nrequests for changes to the OpenVR state.\n\nFor delivering events it creates a listening ZMQ PUB socket, for taking\nstate-change requests it creates a listening ZMQ REP socket.\n\nOn the render process it connects to the browser process' ZMQ PUB socket to\nrecieve VR state-change events. It marshals this event data into the JS object\nmodel using its V8 engine interface.\n\nWhen JS functions are called that should mutate the VR state, it marshals this\ndata and makes a call to the browser process using the ZMQ REP socket and\nmarshals any response data back to the V8 callsite.\n\n### Logging\n\nA simple logging module based on spdlog. It's job is to initialize and make the\nlogger available to other modules.\n\n### Main\n\nThis is the primary startup code for ovrly, it composes all the modules and\nthen calls cef functions to initialize the app. The rest of the application\nlogic branches from cef-triggered observables like `OnBrowser`.\n\n### Manager\n\nThis is where code for managing the native instances of browsers and overlay\ninstances.\n\nIt currently does very little, but may eventually be where the\nwindow-manageresque logic will reside.\n\n### UI\n\nThis is mainly a CefClient implementation that handles the desktop UI for\novrly. It's fairly xplat already as it uses chromium's views for creating and\nmanaging most of the browser UI/UX.\n\n### VR\n\nThis module handles most logic for interacting and marshaling data and events\nbetween OpenVR, abstracting much of the interface to ovrly's needs so other\nmodules don't need to know much of anything about OpenVR.\n\nIn the future, this may be where support for other VR tech like OpenXR is\nimplemented (OpenXR doesn't yet have the concept of overlays yet).\n\nOne of the primary interfaces provided by this module is the `Overlay` abstract\nclass which handles base logic that can be used to implement overlays which\nderive their visual surface from any source (e.g. a cef browser, a PNG, a\nwebcam, etc), and provides virtual functions like `onLayout` for delivering\nevents overlay shape changes.\n\n### Web\n\nSimilar to UI, this is a CefClient implementation, though its implementation is\nspecific to the offscreen browser instances behind each VR overlay. It also\nprovides an implementation of the VR module's Overlay class.\n\nIt's primary task is to recieve the buffer and dirty rects provided by the\nrender process and hand them to the Overlay render logic, . It currently doesn't\nimplement much UX, but this is also where things like virtual keyboard and\nmouse input handling would occur.\n\n### CEF integration\n\nBecause cef wraps chromium to act as the ovrly user interface, the logic necessarily\nis split between processes; architecturally, cef exposes an event-driven interface to\nwhich a single handler can be provided from user code.\n\nWhen interfacing with cef, a common pattern is used to map the cef handler interface to an ovrly Observable interface.\n\nAs an example of using the Observable module interface, any module that needs to run in the cef\nbrowser process would subscribe to the `process::OnBrowser` event to be notified,\nin the browser process, when it is about to come up. This observable also delivers a\n`process::Browser` object which has additional process lifetime events that can be observed\n(e.g. `OnContextInitialized`).\n\nThere is also a `process::OnRender` event for code that needs to run at render process\nstartup. These are found in the app module and are only a few among a number of other events\nacross different modules.\n\n### Conventions\n\n- Some modules expose a `registerHooks()` function that should be called at app start to allow the module to setup inter-module event listeners.\n- Primary modules in ovrly are in files that end with `ovrly` (e.g. uiovrly.cc, or vrovrly.h).\n- Because the observable system is meant for composing application code there is no facility made for removing event subscriptions.\n- Module local code is wrapped in an anonymous namespace to prevent symbol exports.\n\n\n## Browser to Overlay Rendering\n\nThe cef lib is in charge of creating the composited view of the off-screen browser, and it is OpenVR's job to render\nthat view into the overlay presented by the VR compositor. Shuffling this data between the two is ovrly's task.\n\nPage rendering is currently implemented using the [standard off-screen rendering technique](https://bitbucket.org/chromiumembedded/cef/wiki/GeneralUsage#markdown-header-off-screen-rendering)\ndemonstrated in the cefclient test application.\nThis provides a pixel buffer to the `OnPaint` function of the [CefRenderHandler interface](https://magpcss.org/ceforum/apidocs3/projects/(default)/CefRenderHandler.html).\n\nFor each frame rendered to `OnPaint`, the buffer is copied into the DirectX/opengl texture that is bound to the openvr overlay.\nThe framerate is driven by cef and is configured by the `windowless_frame_rate` setting, this value can\nbe set for each overlay to balance the needs of the content with desired performance.\n\nIn order to keep the desktop UI hardware accelerated, ovrly will have cef use GPU rendering\n(though it does support software rendering with some caveats).\nFor overlay UIs, cef will be pulling the rendered page frame from the GPU and then ovrly will copy it through the CPU,\nand back to the GPU to the texture bound to the OpenVR overlay.\n\nThis is not optimally performant, but for the type of content meant for overlays, this limit on perf shouldn't be a huge limitation for now.\n\n    texture = glCreateTexture()\n    vrSetOverlayTexture(texture)\n    cef osr onpaint(pixel buffer) { copyToTexture(texture, pixel buffer) }\n\n### Shared Texture Optimization\n\nThere have been patches to cef in the past to support shared GPU textures so that the data need not travel through the CPU\nwhen the consumer is also the GPU, but this functionality is currently broken as chromium recently implemented\na new compositing layer called [Viz](https://chromium.googlesource.com/chromium/src/+/master/services/viz/).\n\nWhen the [new shared texture patch](https://bitbucket.org/chromiumembedded/cef/pull-requests/285/reimplement-shared-texture-support-for-viz/diff)\nis merged into cef, ovrly will be able to mediate rendering pages into OpenVR at native VR framerates (if necessary for the overlay content)\nby sharing a texture between the cef and openvr rendering pipelines.\n\n    Using this method will allow ovrly to take over the main rendering loop by telling cef when it should render each frame with `SendExternalBeginFrame`.\n\n\n## CEF Notes\n\nAs cef is a wrapper over chromium, it also shares chromiums multi-process design.\n\nIn a cef app there is one primary browser process with multiple threads and is\ngenerally responsible for rendering to the UI, processing OS input events,\ndoing disk and network IO, and handling general orchestration of browser frame\noperations. This is the application startup process.\n\nEach browser frame renders and executes javascript inside of a child process\ncalled the render process to sandbox untrusted code and data using the process\nboundary (many sandboxing techs at the OS-level target processes).\n\nThere are also a number of utility processes, such as the gpu-process where\naccess to the GPU is mediated since access to the GPU is a privileged operation\n(e.g. it's possible to scrape the user's screen). For simplicity, the rest of\nthe text will just be written as if the render process was the only other\nchild because our code doesn't touch the others.\n\n### Process Dispatch\n\nWhen cef needs to fork a new child process, it executes its own executable\nagain but with special cli flags. When the cef init code sees these flags, it\nwill internally coopt the startup thread and only return when the render\nprocess shuts down.\n\nWhen the init function returns in this case, it returns `-1` so that the code\nknows this was a terminating child render process and can `exit()` instead of\nsetting up the browser process singleton.\n\nAs little work as possible should be done in the process before cef init\nfunction is called to minimize child process load time.\n\n### CefApp\n\nThis is an embedder-implemented collection of event dispatch interfaces used by\ncef to find handlers for the browser process and render processes. These\nhandlers are primarily for lifetime events, threads, IPC, and V8 contexts of\nthese processes.\n\n### CefBrowserProcessHandler\n\nEvent dispatch interface that contains functions cef code calls from the\nbrowser process.\n\n*Important Handlers*\n\n*OnContextInitialized:* Called when the browser process is bootstrapped and cef\nis fully ready to do things. This is a good place to trigger initial browser\ndisplay and loading other modules that require access to a ready-to-go cef.\n\n### CefRenderProcessHandler\n\nEvent interface with functions that cef calls in the render process.\n\n*Important Handlers*\n\n*OnContextCreated:* When the javascript context for the render process is\ncreated. This is where extensions to the JS environment should be hooked in.\n\n*OnProcessMessageReceived:* When a render process gets a message from another\nprocess.\n\n### CefClient\n\nAnother embedder-implemented event dispatch interface collection with functions\nused by cef that define operational logic for browser frames.\n\nThis interface is implemented by cef as an interprocess proxy instances in the\nbrowser process for orchestration logic to interact with the browser frame and\nv8 context running in the render processes.\n\nThese handlers are for communicating things like display, dialog, download,\nrender, request, focus, find, drag-n-drop, and loading events. This forms the primary\nabstraction for UI and OS-specific logic; there is a separate impl for overlays\nand for the desktop window as the UX differs quite a bit between the two.\n\nWhen creating each new browser instance (render process), a unique instance of\nthis and/or each each of the dispatch interface impls can be created, or a\nsingle instance can be used if the logic between all are absolutely identical\n(i.e. no local state in the dispatch logic).\n\n*Important Handlers*\n\n- *OnProcesMessageReceived:* When a render process sends the browser process a message.\n\n### CefRenderHandler\n\nAnother event dispatch interface with functions cef runs from the browser\nprocess to respond to events related to rendering happening in the render\nprocess for a particular browser frame.\n\nThis is a primary integration point for hooking render data when using cef to\nrender browser views in off-screen mode.\n\n*Important Handlers*\n\n*OnPaint:* Provides a pixel buffer of the rendered browser view to the browser\nprocess each time the render process finishes a render.\n\n*OnAcceleratedPaint:* When using shared textures (currently broken) provides\nnotification to the browser process after the render process updates the\ntexture.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoshperry%2Fovrly","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoshperry%2Fovrly","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoshperry%2Fovrly/lists"}