{"id":21867291,"url":"https://github.com/motapinto/computer-vision-structured-light","last_synced_at":"2025-10-08T15:31:52.472Z","repository":{"id":68250138,"uuid":"348767803","full_name":"motapinto/computer-vision-structured-light","owner":"motapinto","description":"Project developed for the Computer Vision course unit in FEUP","archived":false,"fork":false,"pushed_at":"2021-04-27T14:05:31.000Z","size":3603,"stargazers_count":10,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-04T04:13:10.742Z","etag":null,"topics":["computer-vision","edge-detection","jupyter-notebook","opencv","python","structured-light"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/motapinto.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-03-17T15:54:47.000Z","updated_at":"2024-10-22T05:35:13.000Z","dependencies_parsed_at":"2023-09-12T19:00:37.650Z","dependency_job_id":null,"html_url":"https://github.com/motapinto/computer-vision-structured-light","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/motapinto/computer-vision-structured-light","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/motapinto%2Fcomputer-vision-structured-light","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/motapinto%2Fcomputer-vision-structured-light/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/motapinto%2Fcomputer-vision-structured-light/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/motapinto%2Fcomputer-vision-structured-light/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/motapinto","download_url":"https://codeload.github.com/motapinto/computer-vision-structured-light/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/motapinto%2Fcomputer-vision-structured-light/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278969287,"owners_count":26077641,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-08T02:00:06.501Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","edge-detection","jupyter-notebook","opencv","python","structured-light"],"created_at":"2024-11-28T05:09:07.124Z","updated_at":"2025-10-08T15:31:52.466Z","avatar_url":"https://github.com/motapinto.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv class=\"cell markdown\"\u003e\n\n# Calculate X, Y, Z Real World Coordinates from Image Coordinates using OpenCV and Structured Light\n\n### Notebook by [André Madureira](https://github.com/Andremad-03), [José Guerra](https://github.com/LockDownPT), [Luis Ramos](https://github.com/luispramos), [Martim Pinto da Silva](https://github.com/motapinto)\n\n#### [Faculdade de Engenharia da Universidade do Porto](https://sigarra.up.pt/feup/en/web_page.inicial)\n\n#### It is recommended to [view this notebook in nbviewer]() for the best overall experience\n\n#### You can also execute the code on this notebook using [Jupyter Notebook](https://jupyter.org/), [Binder](https://mybinder.org/) or [Google Colab](https://colab.research.google.com/) (no local installation required)\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell markdown\"\u003e\n\n## Table of contents\n\n1.    - [Introduction](#Introduction)\n\n2.    - [Required libraries](#Required-libraries)\n\n3.    - [Camera calibration](#Camera-calibration)\n          - [Intrinsic parameters](#Intrinsic-parameters)\n          - [Extrinsic parameters](#Extrinsic-parameters)\n\n4.    - [Re-projection Error](#Re-projection-Error)\n\n5.    - [Undistortion](#Undistortion)\n\n6.    - [Perspective Projection Matrix](#Perspective-Projection-Matrix)\n\n7.    - [Line Detection](#Line-Detection)\n\n8.    - [Resources](#Resources)\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell markdown\"\u003e\n\n## Introduction\n\n[go back to the top](#Table-of-contents)\n\nStructured light techniques for 3D data acquisition play a central role\nin many 3D data acquisition applications, namelly when the surfaces to\nbe measured do not have feature points or when it is necessary to obtain\ndense 3D data. They are used in numerous applications: industrial (ex:\ndimensional control or quality inspection), reverse engineering, urban\n(ex: road inspection) and medical, are just a few examples.\n\nThese techniques are based on the acquisition of an image of a scene\nover which a light pattern is projected; this pattern ranges from a\nsingle light ray or a single light sheet to a set of parallel sheets or\na pseudo-random pattern. Frequently, laser light is used to simplify the\ndetection of the projected patterns.\n\nIn this work we will have the opportunity of implementing a 3D data\nacquisition system based on structured light, using a single sheet of\nlight/shadow\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell markdown\"\u003e\n\n## Required libraries\n\n[go back to the top](#Table-of-contents)\n\nThe primary libraries that we'll be using are:\n\n  - numpy: Provides a fast numerical array structure and helper\n    functions.\n  - cv2: OpenCV provides a real-time optimized Computer Vision library,\n    tools, and hardware.\n  - glob: Is used to retrieve files/pathnames matching a specified\n    pattern.\n  - matplotlib: Basic plotting library in Python, with capabilities of\n    showing images.\n  - sympy: Library for symbolic mathematics and for solving equations.\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell code\" data-execution_count=\"63\"\u003e\n\n``` python\nimport os\nimport numpy as np\nimport cv2\nimport glob\nimport math\nimport matplotlib.pyplot as plt\nfrom sympy import symbols, Eq, solve, poly\n```\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell code\" data-execution_count=\"64\"\u003e\n\n``` python\n# number of interior squares of chess Board\nn_grid = 7\n\n# chessboard square size in cm\nsquare_size = 2.5\n\n# termination criteria\ncriteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)\n\n# prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0)\nobjp = np.zeros((n_grid*n_grid,3), np.float32)\nobjp[:,:2] = np.mgrid[0:n_grid,0:n_grid].T.reshape(-1,2)\n\n# Arrays to store object points and image points from all the images.\nobj_points = [] # 3d point in real world space\nimg_points = [] # 2d points in image plane.\n\nline_detection_path = 'images/lineDetection'\ncalibration_path = 'images/calibration'\ncalibration_images = glob.glob(os.path.join(calibration_path,'*.JPG'))\n```\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell markdown\"\u003e\n\n## Camera calibration\n\n### Intrinsic parameters\n\nIntrinsic parameters are specific to a camera. They include information\nlike focal length (fx,fy) and optical centers (cx,cy). The focal length\nand optical centers can be used to create a camera matrix, which can be\nused to remove distortion due to the lenses of a specific camera. The\ncamera matrix is unique to a specific camera, so once calculated, it can\nbe reused on other images taken by the same camera.\n\n### Extrinsic parameters\n\nExtrinsic parameters corresponds to rotation and translation vectors\nwhich translates a coordinates of a 3D point to a coordinate system.\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell code\" data-execution_count=\"65\"\u003e\n\n``` python\nfor fname in calibration_images:\n    img = cv2.imread(fname)\n\n    # TODO: Why do we put thin in gray? Does it affect results?\n    gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)\n\n    # Find the chess board corners\n    ret, corners = cv2.findChessboardCorners(img, (n_grid,n_grid), None)\n\n    # If found, add object points, image points (after refining them)\n    if ret:\n        obj_points.append(objp)\n        img_point = cv2.cornerSubPix(gray, corners, (11, 11), (-1,-1), criteria)\n        img_points.append(img_point)\n\n# After acquiring the object and image points we need to calibrate the camera. For that we use the function, cv2.calibrateCamera() that returns the camera matrix, distortion coefficients, rotation and translation vectors.\n_, camera_matrix, dist, rvecs, tvecs = cv2.calibrateCamera(obj_points, img_points, gray.shape[::-1], None, None)\n```\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell markdown\"\u003e\n\n## Re-projection Error\n\nRe-projection error gives a good estimation of just how exact the found\nparameters are. The closer the re-projection error is to zero, the more\naccurate the parameters we found are.\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell code\" data-execution_count=\"66\" data-tags=\"[]\"\u003e\n\n``` python\ntotal_error = 0\nfor i in range(len(obj_points)):\n    img_points_tes, _  = cv2.projectPoints(obj_points[i], rvecs[i], tvecs[i], camera_matrix, dist)\n    error = cv2.norm(src1=img_points[i], src2=img_points_tes, normType=cv2.NORM_L2) / len(img_points_tes)\n    total_error += error\n\nprint(\"mean error: {}\".format(total_error / len(obj_points)))\n```\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell markdown\"\u003e\n\n## Undistortion\n\nNow we can take an image and undistort it using the distortion\ncoeficients.\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell code\" data-execution_count=\"67\"\u003e\n\n``` python\nimgcal = cv2.imread(os.path.join('images', 'img_cal.jpg'))\nh,  w = imgcal.shape[:2]\nnew_camera_mtx, roi=cv2.getOptimalNewCameraMatrix(camera_matrix, dist, (w,h), 1, (w,h))\n\n# undistort\nundst = cv2.undistort(imgcal, camera_matrix, dist, None, new_camera_mtx)\n\n# crop the image\nx,y,w,h = roi\nundst = undst[y:y+h, x:x+w]\ncv2.imwrite('images/calibresult.png', undst)\n```\n\u003c/div\u003e\n\n\u003cdiv class=\"cell markdown\" data-pycharm=\"{\u0026quot;name\u0026quot;:\u0026quot;#%% md\\n\u0026quot;}\"\u003e\n\n## Perspective Projection Matrix\n\nWe create the Perspective Projection Matrix using the Camera Matrix\n(obtained in the function calibrateCamera), the Rotation Matrix (which\nwe are going to calculate using Rodrigues function), and the Translation\nVector (later obtained in the SolvePnP function), where all these\nfunctions are a part of the cv2 library\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell code\" data-execution_count=\"68\" data-collapsed=\"false\" data-pycharm=\"{\u0026quot;name\u0026quot;:\u0026quot;#%%\\n\u0026quot;}\"\u003e\n\n``` python\n#The draw function implemented next draws the reference axis in our selected image\ndef draw(img, corners, imgpts):\n    corner = tuple(corners[0].ravel())\n    img = cv2.line(img, corner, tuple(imgpts[0].ravel()), (255,0,0), 5)\n    img = cv2.line(img, corner, tuple(imgpts[1].ravel()), (0,255,0), 5)\n    img = cv2.line(img, corner, tuple(imgpts[2].ravel()), (0,0,255), 5)\n    return img\n```\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell code\" data-execution_count=\"69\" data-collapsed=\"false\" data-pycharm=\"{\u0026quot;name\u0026quot;:\u0026quot;#%%\\n\u0026quot;}\"\u003e\n\n``` python\naxis = np.float32([[3,0,0], [0,3,0], [0,0,-3]]).reshape(-1,3)\n\n# prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0)\n#The np.zeros matrix was changed from (n_grid, n_grid) = (7,7) to (9,6)\n#because the new chessboard image has 9*6 inners corners\nobjp2 = np.zeros((9*6,3), np.float32)\nobjp2[:,:2] = np.mgrid[0:9,0:6].T.reshape(-1,2)\n\n# Arrays to store object points and image points from all the images.\nobj_points2 = [] # 3d point in real world space\nimg_points2 = [] # 2d points in image plane.\n\ngray = cv2.cvtColor(undst, cv2.COLOR_BGR2GRAY)\nret, corners = cv2.findChessboardCorners(gray, (9,6),None)\nif ret:\n    corners2 = cv2.cornerSubPix(gray,corners,(11,11),(-1,-1),criteria)\n    # Find the rotation and translation vectors.\n    ret, rvecs1, tvecs1, inlier = cv2.solvePnPRansac(objp2, corners2, camera_matrix, dist)\n\n    # project 3D reference into image plane\n    imgpts, jac = cv2.projectPoints(axis, rvecs1, tvecs1, camera_matrix, dist)\n    ref_img = draw(undst, corners, imgpts)\n    plt.imshow(ref_img)\n    plt.show()\n    rotM, _= cv2.Rodrigues(rvecs1)\n```\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell code\" data-execution_count=\"70\" data-collapsed=\"false\" data-pycharm=\"{\u0026quot;name\u0026quot;:\u0026quot;#%%\\n\u0026quot;}\"\u003e\n\n``` python\n#Prespective Projection Matrix -\nK = new_camera_mtx\n\n#In this block, we homogenize the matrices that are going to be a part of the Prespective Projection Matrix (PPMatrix) because\n# Using homogeneous coordinates, Rotation and Translation can be expressed by a single matrix\nHomog_K = np.array([[K[0][0], K[0][1], K[0][2], 0], [K[1][0], K[1][1], K[1][2], 0], [K[2][0], K[2][1], K[2][2], 0]])\nHomog_R = np.array([[rotM[0][0], rotM[0][1], rotM[0][2], 0],[rotM[1][0], rotM[1][1], rotM[1][2], 0], [rotM[2][0], rotM[2][1], rotM[2][2], 0],[0, 0, 0, 1]])\nHomog_T = np.array([[1, 0, 0, tvecs1[0][0]],[0, 1, 0, tvecs1[1][0]], [0, 0, 1, tvecs1[2][0]], [0, 0, 0, 1]])\nHomog_Ext = np.matmul(Homog_T, Homog_R)\nPPMatrix = np.matmul(Homog_K, Homog_Ext)\n```\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell markdown\"\u003e\n\n## Line Detection\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell code\" data-execution_count=\"80\" data-collapsed=\"false\" data-pycharm=\"{\u0026quot;name\u0026quot;:\u0026quot;#%%\\n\u0026quot;}\"\u003e\n\n``` python\n# Gray image\nimg_shadow_plane = cv2.imread(os.path.join(line_detection_path, 'img4.JPG'))\ngray = cv2.cvtColor(img_shadow_plane, cv2.COLOR_BGR2GRAY)\ngray = cv2.GaussianBlur(gray, (5,5), 0)\n\n# Threshold\nlow_threshold = 80\nhigh_threshold = 120\n_,thresh = cv2.threshold(gray, low_threshold, high_threshold, cv2.THRESH_BINARY)\nthresh = 255 - thresh\nthresh = cv2.erode(thresh, kernel=(1, 1), iterations=2)\n\n# Laplacian\nlaplacian = cv2.Laplacian(thresh, cv2.CV_8U)\n\nplt.rcParams[\"figure.figsize\"] = (20, 10)\nplt.rcParams.update({'font.size': 20})\nfig, ((ax1, ax2, ax3)) = plt.subplots(1, 3)\n\nax1.imshow(gray, cmap='gray')\nax1.set_xlabel('Gray Image')\n\nax2.imshow(thresh, cmap='gray')\nax2.set_xlabel('Threshold')\n\nax3.imshow(laplacian, cmap='gray')\nax3.set_xlabel('Laplacian')\n\nplt.show()\n```\n\n\u003c/div\u003e\n\u003cdiv class=\"cell code\" data-execution_count=\"72\"\u003e\n\n``` python\n# Then, use findContours to get the contours. You can adjust the parameters for better performance.\ncontours2, _ = cv2.findContours(laplacian, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)\ncont2 = cv2.drawContours(img_shadow_plane, contours2, -1, (0,255,0), 2)\n\nplt.imshow(cont2, cmap='gray')\nplt.show()\n```\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell markdown\"\u003e\nHere, we defined some variables in order to filtrate some of the\noverlapping lines obtained with findContours. Basically, what happens is\nthat we select 3 points on top of the shadow line: the highest left\npoint, the highest right point and the highest middle point.\n\nAlso, xl, yl, xr, yr, xm, ym are defined with the value -1 and width+1\nbecause they are supposed to represent pixels, and by assigning the\nvalue -1 and width + 1, there is no risk of actually missing a point\nvalue because these values do not belong in the image.\n\nIn order to regularize the obtained pixels we assure that the (xm, ym)\npixel refers to the line in the upper plane of our image, (xl, yl) to\nthe left line on the lower plane and (xr, yr) to the right line on the\nlower plane\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell code\" data-execution_count=\"73\" data-tags=\"[]\"\u003e\n\n``` python\nwidth = img_shadow_plane.shape[1]\nmid_point = math.ceil(width/2)\n\nxl=width+1\nyl=width+1\nxr=-1\nyr=width+1\nxm=-1\nym=width+1\n\n\nfor contour in contours2:\n    for point in contour:\n        if point[0][0] \u003c= xl:\n            if point[0][0] == xl:\n                if point[0][1] \u003c= yl:\n                    xl = point[0][0]\n                    yl = point[0][1]\n            else:\n                xl = point[0][0]\n                yl = point[0][1]\n        if point[0][0] \u003e= xr:\n            if point[0][0] == xr:\n                if point[0][1] \u003c= yr:\n                    xr = point[0][0]\n                    yr = point[0][1]\n            else:\n                xr = point[0][0]\n                yr = point[0][1]\n        if (mid_point-5 \u003c point[0][0] \u003c mid_point+5) and point[0][1] \u003c ym:\n            xm = point[0][0]\n            ym = point[0][1]\n```\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell markdown\" data-pycharm=\"{\u0026quot;name\u0026quot;:\u0026quot;#%% md\\n\u0026quot;}\"\u003e\n\nNow, it's only necessary to obtain the A, B, C, D and a, b, c, d\nvariables of each pixel plane in order to obtain the pixel values in our\nworld coordinate. For this, we only need to add the value of a selected\nplane which, in our case, was z=10 for (x10, y10) and z=0 for the other\n2 points.\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell code\" data-execution_count=\"74\" data-collapsed=\"false\" data-pycharm=\"{\u0026quot;name\u0026quot;:\u0026quot;#%%\\n\u0026quot;}\"\u003e\n\n``` python\n#1º Pixel\n#(xm,ym) it's in the plane z1=10\n\ncv2.circle(img_shadow_plane, (xm,ym), 5, (0,0,255), -1)\n\nA1 = PPMatrix[2][0]*xm - PPMatrix[0][0]\nB1 = PPMatrix[2][1]*xm - PPMatrix[0][1]\nC1 = PPMatrix[2][2]*xm - PPMatrix[0][2]\nD1 = PPMatrix[0][3] - PPMatrix[2][3]*xm\n\na1 = PPMatrix[2][0]*ym - PPMatrix[1][0]\nb1 = PPMatrix[2][1]*ym - PPMatrix[1][1]\nc1 = PPMatrix[2][2]*ym - PPMatrix[1][2]\nd1 = PPMatrix[1][3] - PPMatrix[2][3]*ym\n\nz1=-10/square_size\nx1,y1 = symbols('x1 y1')\neq1 = Eq(A1*x1 + B1*y1 + C1*z1 - D1, 0)\neq2 = Eq(a1*x1 + b1*y1 + c1*z1 - d1, 0)\n\nsol1 = solve((eq1,eq2), (x1,y1))\n\n#2º Pixel\n#(xl,yl) it's in the plane z2=0\n\ncv2.circle(img_shadow_plane, (xl,yl), 5, (0,0,255), -1)\n\nA2 = PPMatrix[2][0]*xl - PPMatrix[0][0]\nB2 = PPMatrix[2][1]*xl - PPMatrix[0][1]\nC2 = PPMatrix[2][2]*xl - PPMatrix[0][2]\nD2 = PPMatrix[0][3] - PPMatrix[2][3]*xl\n\na2 = PPMatrix[2][0]*yl - PPMatrix[1][0]\nb2 = PPMatrix[2][1]*yl - PPMatrix[1][1]\nc2 = PPMatrix[2][2]*yl - PPMatrix[1][2]\nd2 = PPMatrix[1][3] - PPMatrix[2][3]*yl\n\nz2=0\nx2,y2 = symbols('x2 y2')\neq3 = Eq(A2*x2 + B2*y2 + C2*z2 - D2, 0)\neq4 = Eq(a2*x2 + b2*y2 + c2*z2 - d2, 0)\n\nsol2 = solve((eq3,eq4), (x2,y2))\n\n#3º Pixel\n#(xr,yr) it's in the plane z2=0\n\ncv2.circle(img_shadow_plane, (xr,yr), 5, (0,0,255), -1)\n\n\nA3 = PPMatrix[2][0]*xr - PPMatrix[0][0]\nB3 = PPMatrix[2][1]*xr - PPMatrix[0][1]\nC3 = PPMatrix[2][2]*xr - PPMatrix[0][2]\nD3 = PPMatrix[0][3] - PPMatrix[2][3]*xr\n\na3 = PPMatrix[2][0]*yr - PPMatrix[1][0]\nb3 = PPMatrix[2][1]*yr - PPMatrix[1][1]\nc3 = PPMatrix[2][2]*yr - PPMatrix[1][2]\nd3 = PPMatrix[1][3] - PPMatrix[2][3]*yr\n\nz3=0\nx3,y3 = symbols('x3 y3')\neq5 = Eq(A3*x3 + B3*y3 + C3*z3 - D3, 0)\neq6 = Eq(a3*x3 + b3*y3 + c3*z3 - d3, 0)\n\nsol3 = solve((eq5,eq6), (x3,y3)) \n\nplt.imshow(img_shadow_plane)\nplt.show()\n```\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell markdown\" data-pycharm=\"{\u0026quot;name\u0026quot;:\u0026quot;#%% md\\n\u0026quot;}\"\u003e\n\nNow, we have 3 points: (x1, y1, z1); (x2, y2, z2); (x3, y3, z3); which\nis the number of points we need to calculate the shadow plane\\! This\nmeans that we only need to do some equation solving to obtain the A, B\nand C plane variables (to the variable D it is going to be attributed a\nconstant value of 1).\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell code\" data-execution_count=\"75\" data-collapsed=\"false\" data-pycharm=\"{\u0026quot;name\u0026quot;:\u0026quot;#%%\\n\u0026quot;}\"\u003e\n\n``` python\nvarA, varB, varC= symbols('A B C')\n\n#D is considered = 1 to calculate the shadow plane \nD = 1\n\neqABC1 = Eq(varA*sol1[x1] + varB*sol1[y1] + varC*z1, D)\neqABC2 = Eq(varA*sol2[x2] + varB*sol2[y2] + varC*z3, D)\neqABC3 = Eq(varA*sol3[x3] + varB*sol3[y3] + varC*z3, D)\nsolABC = solve((eqABC1, eqABC2, eqABC3), (varA, varB, varC))\nA, B, C = solABC[varA], solABC[varB], solABC[varC]\n\nprint('(A, B, C, D)')\nprint('{}, {}, {}, {})'.format(A, B, C, D))\n```\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell code\" data-execution_count=\"76\"\u003e\n\n``` python\n#This function calculates the 3D coordinates given the Perspective Projection Matrix (PPM), the plane coefficients (A, B, C, D) and the\n#pixel point we want to convert (i, j)\ndef calculate3DPointCoords(PPM, i, j, A, B, C, D):\n    x_var, y_var, z_var = symbols('X Y Z')\n    calcX = Eq((PPM[2][0] * i - PPM[0][0]) * x_var + (PPM[2][1]*i - PPM[0][1]) * y_var + (PPM[2][2]*i - PPM[0][2]) * z_var - PPM[0][3] + PPM[2][3]*i, 0)\n    calcY = Eq((PPM[2][0] * j - PPM[1][0]) * x_var + (PPM[2][1]*j - PPM[1][1]) * y_var + (PPM[2][2]*j - PPM[1][2]) * z_var - PPM[1][3] + PPM[2][3]*j, 0)\n    calcZ = Eq(A * x_var + B * y_var + C * z_var - D, 0)\n    sol = solve((calcX, calcY, calcZ), (x_var, y_var, z_var))\n\n    return sol[x_var], sol[y_var], -sol[z_var]\n```\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell markdown\" data-pycharm=\"{\u0026quot;name\u0026quot;:\u0026quot;#%% md\\n\u0026quot;}\" data-tags=\"[]\"\u003e\n\nHere, we get a new image with the same shadow plane and apply\nfindContours to obtain the line representation of that plane.\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell code\" data-execution_count=\"77\" data-collapsed=\"false\" data-pycharm=\"{\u0026quot;name\u0026quot;:\u0026quot;#%%\\n\u0026quot;}\"\u003e\n\n``` python\n# Gray image\nimg_to_represent = cv2.imread(os.path.join(line_detection_path, 'img1.JPG'))\ngray_rep = cv2.cvtColor(img_to_represent, cv2.COLOR_BGR2GRAY)\n\ngray_rep = cv2.GaussianBlur(gray_rep, (5,5), 0)\n\n# Threshold\nlow_threshold = 80\nhigh_threshold = 120 #low_threshold * 3\n_,thresh_rep = cv2.threshold(gray_rep, low_threshold, high_threshold, cv2.THRESH_BINARY)\nthresh_rep = 255 - thresh_rep\n\nkernel = cv2.getStructuringElement(cv2.MORPH_RECT, ksize=(2, 2))\n\n# Remove noise\nthresh_rep = cv2.erode(thresh_rep, kernel=(1,1), iterations=2)\n\n# Laplacian\nlaplacian_rep = cv2.Laplacian(thresh_rep, cv2.CV_8U)\n\n\ncontours, _ = cv2.findContours(laplacian_rep, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)\ncont = cv2.drawContours(img_to_represent, contours, -1, (0,255,0), 5)\nplt.imshow(cont, cmap='gray')\nplt.show()\n```\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell markdown\" data-collapsed=\"false\" data-pycharm=\"{\u0026quot;name\u0026quot;:\u0026quot;#%% md\\n\u0026quot;}\"\u003e\n\nAfter having found the contours of the new image, it is now possible to\ncalculate the height map of the object using the shadow plane\ncoefficients (A, B, C, D) found earlier.\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell code\" data-execution_count=\"78\" data-collapsed=\"false\" data-pycharm=\"{\u0026quot;name\u0026quot;:\u0026quot;#%%\\n\u0026quot;}\"\u003e\n\n``` python\n# Find highest pixel in contours\nheight = laplacian_rep.shape[0]\nmaximum = np.ones(width, dtype=int)*height\n\nfor contour in contours:\n    for point in contour:\n        i = point[0][0]\n        j = point[0][1]\n        if maximum[i] \u003e j:\n           maximum[i] = j\n\n# Compress maximum array and calculate 3D points into heightMap\nheightMap = []\ncurr_max = -1\n\nfor i, max_val in enumerate(maximum):\n    if max_val != curr_max and max_val != height:\n        heightMap.append(calculate3DPointCoords(PPMatrix, i, max_val, A, B, C, D))\n        curr_max = max_val\n\n# Create a height histogram with calculated height map\nhistHeight = [0.0, 1.0] * len(heightMap)\nhistHeight = np.array(histHeight).reshape((len(heightMap), 2))\n\nfor i, point in enumerate(heightMap):\n    histHeight[i][0] = point[2] * square_size\n    histHeight[i][1] = point[1] * square_size\n\nplt.plot(histHeight[:,1], histHeight[:,0])\nplt.xlabel('width(y)')\nplt.ylabel('height(z)')\nplt.show()\n```\n\n\u003c/div\u003e\n\n\u003cdiv class=\"cell markdown\" data-collapsed=\"false\" data-pycharm=\"{\u0026quot;name\u0026quot;:\u0026quot;#%% md\\n\u0026quot;}\"\u003e\n\nReferences\n\n  - \u003chttps://docs.opencv.org/3.3.0/dc/dbb/tutorial_py_calibration.html\u003e\n  - \u003chttps://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html\u003e\n  - \u003chttps://docs.opencv.org/3.3.0/d9/d0c/group__calib3d.html\u003e\n\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmotapinto%2Fcomputer-vision-structured-light","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmotapinto%2Fcomputer-vision-structured-light","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmotapinto%2Fcomputer-vision-structured-light/lists"}