{"id":21885298,"url":"https://github.com/dorisoy/dorisoy.handtracking","last_synced_at":"2026-02-12T10:32:23.087Z","repository":{"id":259214228,"uuid":"852056204","full_name":"dorisoy/Dorisoy.HandTracking","owner":"dorisoy","description":"基于卷积神经网络(CNN)，线性回归和向量机(SVM)算法，通过python脚本和C#脚本组合来实现一个具有手部跟踪和手势识别功能的Unity目标检测应用程序","archived":false,"fork":false,"pushed_at":"2024-10-22T09:22:00.000Z","size":65861,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-07T19:39:10.501Z","etag":null,"topics":["cnn-handtracking","hand-tracking","handtracking","handtracking-arm","r-cnn","unity-arm"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dorisoy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-04T06:36:25.000Z","updated_at":"2025-01-06T14:21:28.000Z","dependencies_parsed_at":"2024-10-28T04:07:52.736Z","dependency_job_id":null,"html_url":"https://github.com/dorisoy/Dorisoy.HandTracking","commit_stats":null,"previous_names":["dorisoy/dorisoy.handtracking"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dorisoy/Dorisoy.HandTracking","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dorisoy%2FDorisoy.HandTracking","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dorisoy%2FDorisoy.HandTracking/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dorisoy%2FDorisoy.HandTracking/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dorisoy%2FDorisoy.HandTracking/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dorisoy","download_url":"https://codeload.github.com/dorisoy/Dorisoy.HandTracking/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dorisoy%2FDorisoy.HandTracking/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29363153,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-12T08:51:36.827Z","status":"ssl_error","status_checked_at":"2026-02-12T08:51:26.849Z","response_time":55,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cnn-handtracking","hand-tracking","handtracking","handtracking-arm","r-cnn","unity-arm"],"created_at":"2024-11-28T10:22:10.010Z","updated_at":"2026-02-12T10:32:23.082Z","avatar_url":"https://github.com/dorisoy.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dorisoy.HandTracking\n\n使用 python 脚本和 C#脚本的组合来创建一个具有手部跟踪和手势识别功能的 Unity 应用程序。使用 python 脚本来检测手部运动和相对手势。\n\n## 要求\n\n项目需要[Vuforia](https://developer.vuforia.com/)如果你想自己安装，下载 Asset 文件夹（在 Unity Project 文件夹内）。\n\n## 手检测\n\n1.使用3x3高斯滤波器去噪图像。\n\n2.转换图像到HSV颜色空间。\n\n3.应用阈值操作以获得二进制图像（肤色像素变为白色，其他为黑色）。\n\n4.应用膨胀和腐蚀填充可能的孔洞。\n\n5.再次应用高斯滤波器以平滑边缘。\n\n6.使用OpenCV的findContours()函数从二进制图像获取轮廓。\n\n7.使用Suzuky和Abe开发的算法并通过OpenCV的convexHull()实现Sklansky算法找到凸包。\n\n## 手跟踪（左侧）\n\n1.将点放置在之前找到的轮廓的最高点上（y值最小）。\n\n2.实现稳定器例程以避免点围绕指尖摆动。\n\n## 手势识别（右侧）\n\n1.基于右手每帧的手指数量进行手势识别。\n\n2.使用余弦定理从凸包和轮廓中找到手的缺陷。\n\n3.为每个缺陷形成三角形，并计算红点（对应于缺陷）处的角度：\n\n\n[\\theta = \\arccos\\left(\\frac{b^2 + c^2 - a^2}{2bc}\\right) \\times \\frac{180}{\\pi}]\n\n其中 ( \\theta ) 是度数形式的角度，a、b、c是图中三角形的边。\n\n\n## 使用RCNN进行手跟踪\n\n1.使用Fast-RCNN进行手部追踪，训练了40个epoch。使用了迁移学习（仅训练最后一层）。\n\n## 使用CNN进行手指计数\n\n1.开发了一个接受手部图像作为输入并输出手指数量的简单CNN。\n\n2.网络结构包括4个卷积层、每层后跟一个最大池化层和最后的两个全连接层，激活函数选择ReLU。\n\n3.在200个图像上训练了250个epoch，使用Cross Entropy作为损失函数和AdaDelta作为优化器。\n\n\n## Python\n\n```python\nimport numpy as np\nimport cv2\nimport math\nimport socket\nimport time\n\nfrom hand_detector_utils import *\n\nUDP_IP = \"127.0.0.1\"\nUDP_PORT = 5065\n\nsock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)\n\nlast = []\n\ngood_condition = False\ndrawing_box = True\nfull_frame = False\nstabilize_highest_point = True\n\nold_highest_point = (-1, -1)\n\nx1_crop = 0\ny1_crop = 60\nx2_crop = 320\ny2_crop = 420\n\n# Open Camera\ntry:\n    default = 1 # Try Changing it to 1 if webcam not found\n    capture = cv2.VideoCapture(default)\nexcept:\n    print(\"No Camera Source Found!\")\n\nwhile capture.isOpened():\n\n    # Capture frames from the camera\n    ret, frame = capture.read()\n\n    width = frame.shape[1]\n\n    img_left = frame[y1_crop:y2_crop, 0:int(width/3)]\n    img_right = frame[y1_crop:y2_crop, int(width/3 * 2): int(width)]\n\n    try:\n\n        contour_left = detectHand(img_left)\n        contour_right = detectHand(img_right)\n\n        defects_left, drawing_left = findDefects(img_left, contour_left)\n        defects_right, drawing_right = findDefects(img_right, contour_right)\n\n        # Count defects (in the right image)\n        count_defects = countDefects(defects_right, contour_right, img_right)\n\n        # Track highest point (in the left image)\n        highest_point = trackHighestPoint(defects_left, contour_left)\n\n        if(stabilize_highest_point):\n            if( old_highest_point == (-1, -1)): old_highest_point = highest_point\n            else:\n                # Evaluate the magnitude of the difference\n                diag_difference = np.linalg.norm(np.asarray(old_highest_point) - np.asarray(highest_point))\n\n                # If the difference is bigger than a threshold then I actually moved my finger\n                if(diag_difference \u003e= 9.5):\n                    # print(\"diag_difference = \", diag_difference)\n                    old_highest_point = highest_point\n                else: highest_point = old_highest_point;\n\n        if(full_frame):\n            highest_point = (highest_point[0], highest_point[1])\n            cv2.circle(frame, highest_point, 10, [255,0,255], -1)\n        else:\n            cv2.circle(img_left, highest_point, 10, [255,0,255], -1)\n            highest_point = (highest_point[0] + x1_crop, highest_point[1] + y1_crop)\n            cv2.circle(frame, highest_point, 10, [255,0,255], -1)\n\n        # Print number of fingers\n        textDefects(frame, count_defects,debug_var = False)\n\n        # Show required images\n        if(drawing_box):\n            cv2.rectangle(frame, (x1_crop, y1_crop), (int(width/3), y2_crop),(0,0,255), 1)\n            cv2.rectangle(frame, (int(width/3 * 2), y1_crop), (int(width), y2_crop),(0,0,255), 1)\n        cv2.imshow(\"Full Frame\", frame)\n\n        all_image_left = np.hstack((drawing_left, img_left))\n        cv2.imshow('Recognition Left', all_image_left)\n\n        all_image_right = np.hstack((drawing_right, img_right))\n        cv2.imshow('Recognition Right', all_image_right)\n\n        last.append(count_defects)\n        if(len(last) \u003e 5):\n            last = last[-5:]\n            # last = []\n\n\n        # Check if previously hand was wide open (3/4 fingers in previous frames), and is now a fist (0 fingers)\n        if(good_condition):\n            if(count_defects == 0 and 4 in last):\n                last = []\n                sendCommand(sock, UDP_IP, UDP_PORT, \"ACTION\")\n\n            elif(count_defects == 0 and 2 in last):\n                last = []\n                sendCommand(sock, UDP_IP, UDP_PORT, \"BACK\")\n\n        else:\n            if(count_defects == 0 and 4 in last):\n                last = []\n                sendCommand(sock, UDP_IP, UDP_PORT, \"ACTION\")\n\n        command = \"l \" + str(highest_point[0]) + \" \" + str(highest_point[1])\n\n\n        sendCommand(sock, UDP_IP, UDP_PORT, command, debug_var = False)\n\n    except:\n        pass\n\n    # Close the camera if 'q' is pressed\n    if cv2.waitKey(1) == ord('q'):\n        break\n\ncapture.release()\ncv2.destroyAllWindows()\n```\n\n## 屏幕\n\nhttps://github.com/dorisoy/Dorisoy.HandTracking/blob/main/Video/826bc7f6c296dfae62b87318218fe8eb.mp4\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdorisoy%2Fdorisoy.handtracking","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdorisoy%2Fdorisoy.handtracking","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdorisoy%2Fdorisoy.handtracking/lists"}