{"id":18652976,"url":"https://github.com/chromeos/low-latency-stylus","last_synced_at":"2025-04-11T16:32:06.039Z","repository":{"id":48234986,"uuid":"366761710","full_name":"chromeos/low-latency-stylus","owner":"chromeos","description":null,"archived":false,"fork":false,"pushed_at":"2022-07-20T11:04:31.000Z","size":3666,"stargazers_count":16,"open_issues_count":4,"forks_count":8,"subscribers_count":5,"default_branch":"main","last_synced_at":"2023-03-04T08:43:14.355Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chromeos.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-05-12T15:23:04.000Z","updated_at":"2023-01-12T04:29:55.000Z","dependencies_parsed_at":"2022-09-05T10:30:50.940Z","dependency_job_id":null,"html_url":"https://github.com/chromeos/low-latency-stylus","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chromeos%2Flow-latency-stylus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chromeos%2Flow-latency-stylus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chromeos%2Flow-latency-stylus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chromeos%2Flow-latency-stylus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chromeos","download_url":"https://codeload.github.com/chromeos/low-latency-stylus/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223472780,"owners_count":17150745,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T07:09:37.887Z","updated_at":"2024-11-07T07:09:39.644Z","avatar_url":"https://github.com/chromeos.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Low-Latency Stylus for Chrome OS\n\nAlpha 1.0.3\n\nUpdated 5 August 2021\n\nA low-latency stylus library for Android apps on Chrome OS. This library\nprovides mechanisms to reduce the touch-to-draw latency on Chrome\nOS devices by using direct rendering and prediction.\n\nA simple demo app is provided showing both the CPU and GPU-driven\nlow-latency implementations as well as a side-by-side comparison for\neach implementation with a regular (not low-latency) canvas.\n\nPlease file bugs and features requests to help us improve the library as\nwe move toward launch.\n\n\n## How it works\n\nThe low-latency stylus library achieves fast stylus-to-screen response\nby leveraging two mechanisms.\n\n\n1. **Direct-rendering** **-** by rendering pen strokes directly through\n   the hardware compositor, delays due to OS compositing can be avoided\n2. **Prediction** **-** there will always be some latency in drawing\n   response to screen response time, hardware limitations, and required\n   OS functionality. The library attempts to compensate for this\n   remaining latency using prediction - guessing where the next part of\n   the stroke will be drawn based on the current speed and direction of\n   the stroke - and can achieve what feels like “zero-latency”.\n\n\n## Getting the library\n\n\n1. In your top-level **build.gradle** file, be sure you have google() as\n   one of your maven repositories:\n\n   ```\n   repositories { \n       // low-latency libraries are stored in google's maven repository\n       google() \n       mavenCentral() \n   }\n   ```\n2. Put the following in the dependencies section of the app-level\n   **build.gradle** file:\n\n   ```\n   implementation 'com.google.chromeos:chromeos-lowlatencystylus:1.0.1\n   ```\n\n\n## Using the library\n\nFirst, decide if you would like to use the CPU-based version of the\nlibrary or the GPU-based version.\n\n\n* **CPU:** The CPU version is easier to implement but may not be\n  suitable for apps with complex brushes and graphics requirements.\n* **GPU:** The GPU version requires some OpenGL knowledge to use and\n  offers some performance improvements over the CPU version and allows\n  for more advanced brushes.\n\nDemo applications and instructions are provided for both versions of the\nlibrary.\n\n\n\u003e**NOTE:** To use the library, you will need Chrome OS version M91 or\n\u003egreater and an Android runtime (ARC) version greater\n\u003ethan 7316937. To check this, go to chrome://version in the browser and\n\u003eread the ARC line. You may need to change your device's update channel\n\u003eto the beta channel or dev channel to get this version.\n\n\n## CPU Library\n\nThe best way to get started using the CPU library is to examine the\n`LowLatencyStylusDemo` app. Import it into Android Studio and use the\n`cpu` package and the `LowLatencyStylusActivityCpu` activity as your\nreference. You may also want to reference the API documentation provided\nhere.\n\n\n### Create your main canvas\n\nCreate a drawing canvas for your main drawing surface that will receive\n`MotionEvents` and show the completed drawing gestures. In the demo app,\na reference class is provided called `SampleInkView`.\n\n\n### Create an InkOverlayView\n\nCreate an `InkOverlayView` that will sit on top of your main canvas.\nThis is the view that will show the current, incomplete, drawing gesture\nas well as the predicted path of the current gesture. Correctly tuned\nprediction can give the appearance of zero-latency drawing.\n\n\n### Extend InkDelegate\n\nDrawing gestures passed to an `InkOverlayView` will be processed via an\n`InkDelegate`. Extending the `InkDelegate` class allows you to control\nhow the current drawing gesture and predicted gestures will be drawn. An\nexample of how this is shown in the `SampleInkDelegate` class of the\ndemo app.\n\n\n### Pass current gestures to the InkOverlayView\n\nYour main canvas should receive `MotionEvents`, keep track of them, and\nshow any completed drawing gestures. The `MotionEvents` for the current,\nincomplete, drawing gesture should be passed to the `InkOverlayView`\nusing `InkOverlayView.onTouch`.\n\n\n### Commit to main canvas\n\nWhen a stroke is finished or cancelled (like due to\n[palm rejection](https://developer.android.com/topic/arc/input-compatibility#palm_rejection)),\nand lines representing the real completed strokes have been drawn to the\nmain canvas, the `InkOverlayView` should be cleared. In the demo app,\nthis is done in the `onTouchEvent` method of the `SampleInkView` class.\n\n\n### Using/disabling prediction\n\nSet latency compensation level for the prediction algorithm in ms using\n`setPredictionTarget `on your `InkOverlayView.` You can disable\nprediction by setting this to **0**. Use the `Prediction` slider in the\ndemo app to tune the prediction value you want. Note: setting this value\ntoo high will result in noticeable over-prediction.\n\n\n### Testing the library\n\nYou should notice a decrease in latency on Chrome OS devices when using\nthe library. Test it out for a given device using the demo application.\nYou can test the difference with and without low-latency by using the\nside-by-side comparison Activity provided in the demo application.\n\nNote: pressing `[SPACE]` will clear the canvas in the demo application.\n\n\n## GPU Library\n\nThe best way to get started using the GPU library is to examine the\n`LowLatencyStylusDemo` app. Import it into Android Studio and use the\n`gpu` package and the `LowLatencyStylusActivityGpu` activity as your\nreference. You may also want to reference the API documentation provided\nhere.\n\n\n### Create your main canvas\n\nThe main low-latency canvas will be subclassed from `GLInkSurfaceView`.\nTo set up this view:\n\n\n1. Set the prediction target in milliseconds (or disable prediction by\n   setting target to 0)\n2. Set the `GLInkRenderer` to handle the rendering\n3. Receive input events and queue them correctly to the `GLInkRenderer`\n\nInput events are generally handled by looking for\n[ACTION\\_DOWN](https://developer.android.com/reference/android/view/MotionEvent#ACTION_DOWN),\n[ACTION\\_MOVE](https://developer.android.com/reference/android/view/MotionEvent#ACTION_MOVE),\nand\n[ACTION\\_UP](https://developer.android.com/reference/android/view/MotionEvent#ACTION_UP)\nevents in the\n[onTouchEvent](https://developer.android.com/reference/android/view/View#onTouchEvent(android.view.MotionEvent))\ncallback. Note that\n[ACTION\\_MOVE](https://developer.android.com/reference/android/view/MotionEvent#ACTION_MOVE)\ncan contain a *list* of intermediate points that have been batched by\nthe OS’s input logic. See\n[Batching](https://developer.android.com/reference/android/view/MotionEvent#batching)\nin the Android documentation for more details.\n\nTo draw on the low-latency surface,\n[MotionEvents](https://developer.android.com/reference/android/view/MotionEvent)\nmust be passed down to the `GLInkSurfaceView.onTouch` method. This\nmethod will automatically request unbuffered dispatch of input events\nfor you to ensure that data is received as fast as possible.\n\n\u003e**Note:** If you are not directly passing down MotionEvents received\n\u003efrom the system - for example if you are interpreting the events and\n\u003econstructing new ones before sending them to `GLInkSurfaceView.onTouch`\n\u003e\\- you should call\n\u003e[View.requestUnbufferedDispatch](https://developer.android.com/reference/android/view/View#requestUnbufferedDispatch(android.view.MotionEvent))\n\u003emanually every time you receive a system\n\u003e[ACTION\\_DOWN](https://developer.android.com/reference/android/view/MotionEvent#ACTION_DOWN)\n\u003eevent, in order to get the lowest input latency.\n\n\n### Set your GLInkRenderer\n\nYour `GLInkRenderer` should handle your brushes/shaders as well as\nmanage incremental damage regions to provide the best drawing\nperformance. Reference the included `SampleInkRenderer` as a starting\npoint.\n\nTo configure the renderer:\n\n1. Manage OpenGL shaders for drawing brush strokes\n2. Keep track of damaged regions\n3. Handle surface related callbacks (clear, onSurfaceCreated,\n   onSurfaceChanged)\n4. Override `beforeDraw` to handle input events received from the\n   `GLInkSurfaceView` to be correctly passed to shaders and to return\n   the damage area\n5. Override `onDraw` to execute the draw\n\n\u003e**Note:** the damage Rect returned from `beforeDraw` will be passed to\n\u003ean internal\n\u003e[glScissor](https://www.khronos.org/registry/OpenGL-Refpages/es2.0/xhtml/glScissor.xml)\n\u003ecall before the render call is made.\n\n\n### Brushes/Shaders\n\nYou will need to create OpenGL shaders for your different brushes to\nhandle the actual rendering of brush strokes received from the\n`GLInkRenderer`. The provided `BrushShader` class gives an example of a\nsimple line brush (GL_LINES) as well as a bitmap-based brush shader.\n\nTo try your own bitmap files with the demo, create a 100x100px bitmap\nfile with 0% alpha for blank areas and non-0% alpha for areas to paint.\n\n\u003e**Note:** currently, changing between the line shader and the brush\n\u003e shader will re-draw all previous strokes with the newly selected\n\u003e shader.\n\n### Using/disabling prediction\n\nAs with the CPU library, the `setPredictionTargetMs` method in\n`GLInkSurfaceView` allows you to set your desired level of\nprediction. You can disable prediction by setting this\nto **0**.\n\n\n### View and projection matrices\n\nIn the demo app, the view and projection matrices are set to the\nidentity matrix. The API allows you to change this. If you are using\ndifferent matrices in your application, please let us know. We would\nlike feedback on this part of the API.\n\n\n### Testing the library\n\nYou should notice a decrease in latency on Chrome OS devices when using\nthe library. Test it out for a given device using the demo application.\nYou can test the difference with and without low-latency by using the\nside-by-side comparison Activity provided in the demo application.\n\nNote: pressing `[SPACE]` will clear the canvas in the demo application.\n\n\n## Known Issues\n\n* Prior to Chrome OS version M93 and Android runtime version 7434780,\n  the GPU prediction target was fixed at 25ms. Adjusting the slider in\n  the demo app or using `setPredictionTargetMs` will have no effect on\n  these versions. To check your OS and Android runtime (ARC) versions,\n  go to chrome://version in the browser. GPU prediction should work\n  correctly on Chrome OS M93+ and ARC versions \u003e= 7434780.\n\n* Display scaling: in order to leverage direct compositing, the user\n  needs to have their Chrome OS display resolution set to the “default”\n  value. With other settings, additional GPU calculations may be needed\n  to scale the output which makes direct rendering not possible.\n\n## Feedback and Bugs\n\nWe appreciate you exercising this library and strongly value your\nfeedback. Please file bugs and feature requests here on the issue\ntracker.\n\n\u003cimg alt=\"Screenshot of LowLatencyStylusDemo on a Chromebook\" src=\"https://github.com/chromeos/low-latency-stylus/blob/main/images/screenshot.png\" /\u003e\n\n\n## This is not an officially supported Google product\n\n## LICENSE\n***\n\nCopyright 2021 Google LLC\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    https://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchromeos%2Flow-latency-stylus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchromeos%2Flow-latency-stylus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchromeos%2Flow-latency-stylus/lists"}