{"id":18034624,"url":"https://github.com/jcparkyn/dpoint","last_synced_at":"2025-04-12T17:40:13.278Z","repository":{"id":196347253,"uuid":"640127889","full_name":"Jcparkyn/dpoint","owner":"Jcparkyn","description":"Open-source digital stylus using camera tracking and inertial measurements","archived":false,"fork":false,"pushed_at":"2023-11-27T07:51:37.000Z","size":23035,"stargazers_count":1034,"open_issues_count":2,"forks_count":50,"subscribers_count":18,"default_branch":"main","last_synced_at":"2025-04-03T19:13:52.904Z","etag":null,"topics":["6dof","aruco","ekf","stylus"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Jcparkyn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-13T04:41:46.000Z","updated_at":"2025-03-26T14:19:00.000Z","dependencies_parsed_at":"2023-09-24T17:28:11.055Z","dependency_job_id":"444a6548-473a-4d11-85fd-abafa0d6ce40","html_url":"https://github.com/Jcparkyn/dpoint","commit_stats":null,"previous_names":["jcparkyn/dpoint"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jcparkyn%2Fdpoint","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jcparkyn%2Fdpoint/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jcparkyn%2Fdpoint/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jcparkyn%2Fdpoint/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Jcparkyn","download_url":"https://codeload.github.com/Jcparkyn/dpoint/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248607214,"owners_count":21132495,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["6dof","aruco","ekf","stylus"],"created_at":"2024-10-30T11:12:55.256Z","updated_at":"2025-04-12T17:40:13.254Z","avatar_url":"https://github.com/Jcparkyn.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# D-POINT: Digital Pen with Optical-Inertial Tracking\n\n**D-POINT** is an open-source digital stylus that uses camera tracking and inertial measurements to achieve 6DoF (six degrees of freedom) inputs, with low latency, pressure sensitivity, and sub-millimetre accuracy.\nThe stylus can be used on any flat surface, and works with consumer-grade webcams.\n\n\u003cvideo src=\"https://github.com/Jcparkyn/dpoint/assets/51850908/21d00ce3-6456-4fba-8c24-4bc8e6642a0d\" width=\"1920\" height=\"1080\" autoplay loop muted\u003e\u003c/video\u003e\n\nThis project was part of my undergraduate thesis for electrical engineering. I've open-sourced the code and design files in the hopes that they might be useful to somebody, but it's not intended to be a \"plug and play\" DIY project. If you want to try building it anyway, follow the [setup guide](./setup-guide.md).\n\n## Design\n\nThis is a very brief overview of how the system works. For all the details, plus literature review and lots of evaluation, read the full thesis (note: I haven't published this online yet).\n\n![Block diagram showing how the system works](assets/block-diagram.png)\n\n### Hardware\n\nThe main body of the stylus was 3D printed as two halves, shown below. The stylus contains a force sensor, a Li-ion battery which charges over USB-C, and an Arduino-based development board for logic and Bluetooth. Eight printed [ArUco](https://www.uco.es/investiga/grupos/ava/portfolio/aruco/) markers are glued to the back of the stylus, for visual pose estimation.\n\n![CAD drawing showing the hardware design of the stylus](assets/cad-drawing.png)\n\n### Visual pose estimation (VPE)\n\nThe VPE process involves the four main steps:\n1. **Marker detection:** First, we use OpenCV to detect the corners of each visible ArUco marker on the stylus.\n1. **Rolling shutter correction:** We use a simple 2D motion model to estimate and correct for the effects of [rolling shutter](https://en.wikipedia.org/wiki/Rolling_shutter) on the observed corner locations.\n1. **Perspective-n-Point (PnP):** From these corner positions, we use a [PnP](https://en.wikipedia.org/wiki/Perspective-n-Point) algorithm to estimate the pose of the stylus relative to the camera. When possible, we use the pose from the previous frame as a starting point to refine with virtual visual servoing (VVS), otherwise we fall back to SQPnP.\n1. **Coordinate conversion:** Using the calibrated pose of the stylus and the drawing surface relative to the camera, we calculate the position and orientation of the stylus tip relative to the drawing surface.\n\n### Inertial fusion\n\nWe use an Extended Kalman Filter (EKF) to fuse the VPE estimates with the inertial data from the accelerometer and gyroscope, and refine the estimates in real-time using the Rauch-Tung-Striebel (RTS) algorithm. To account for time delay from the camera frames, we use a negative-time measurement update algorithm. The EKF is implemented using NumPy and [Numba](https://numba.pydata.org/).\n\nUsing inertial measurements allows us to dramatically reduce latency compared to a camera-only implementation, while also improving accuracy and report rate for fast movements.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjcparkyn%2Fdpoint","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjcparkyn%2Fdpoint","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjcparkyn%2Fdpoint/lists"}