{"id":18039677,"url":"https://github.com/timbokz/srv-dataset-tools","last_synced_at":"2025-04-05T01:14:18.889Z","repository":{"id":37592438,"uuid":"166048407","full_name":"TimboKZ/srv-dataset-tools","owner":"TimboKZ","description":null,"archived":false,"fork":false,"pushed_at":"2022-06-21T21:54:43.000Z","size":311002,"stargazers_count":0,"open_issues_count":3,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-10T09:29:58.100Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TimboKZ.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-01-16T13:52:29.000Z","updated_at":"2019-12-03T20:03:05.000Z","dependencies_parsed_at":"2022-08-25T22:12:10.697Z","dependency_job_id":null,"html_url":"https://github.com/TimboKZ/srv-dataset-tools","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TimboKZ%2Fsrv-dataset-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TimboKZ%2Fsrv-dataset-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TimboKZ%2Fsrv-dataset-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TimboKZ%2Fsrv-dataset-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TimboKZ","download_url":"https://codeload.github.com/TimboKZ/srv-dataset-tools/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247271527,"owners_count":20911587,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-30T14:10:01.283Z","updated_at":"2025-04-05T01:14:18.869Z","avatar_url":"https://github.com/TimboKZ.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Texture reconstruction data set tools\n\nThis repo contains the tools for texture reconstruction based on laparoscopic surgery data sets, made for Surgical \nVision and Robotics group at UCL. This repository goes together with Timur Kuzhagaliyev's final year project, \ndescribed in [this report](https://drive.google.com/open?id=1QT4nYPPCjCrJqFcet6b-yGSSvJUqmy7Y). If you're planning to\n use the code from this repository, you are strongly advised to look at sections 3 and 4 of the project report.\n\n\n# Project structure\n\nThe purpose of each directory is described below:\n\n* `ds_tools/`: This folder contains all of the Python source code for the project. Note that it's meant to act a \nself-contained Python package, meaning that code inside it uses package-absolute imports, e.g. \n`from ds_tools.shared import util`.\n* `matlab_issi/`: Matlab code used to process data from the da Vinci surgical robot (acquired using a dvLogger). It's\n mostly based on the ISSI file processing functions provided by Intuitive Surgical Inc. **This code is not available\n in the public version of the thesis (see notice inside the folder).**\n* `notebooks/`: This folder contains various Jupyter notebooks, mostly used for calibration purposes. The reason the \ncalibration code is presented in the form of Jupyter notebooks is to make it easier to select the correct filters and\n debug the calibration process. Notes on how to use the notebooks can be found below.\n* `data/`: The folder meant to hold data used for texture reconstruction. Since this usually includes large video and\n CSV files, the contents of this folder are ignored via `.gitignore`. \n* `resources/`: This folder contains various 3D assets and example output of projective texture mapping/texture \nreconstruction algorithms.\n\n# Python packages\n\nThis project uses Python 3.5.2. You will need the following Python Pip packages to run the code in this project:\n```\nopencv-contrib-python==3.4.0.12\nopencv-python==4.0.0.21\nmatplotlib==3.0.2\ntrimesh==2.36.16\npanda3d==1.10.0\npandas==0.23.4\nnumpy==1.15.4\npycpd==1.0.5\nscipy==1.2.0\n```\n\nThese packages can be installed by running the following command at the root folder of this repository:\n```bash\npip3 install -r requirements.txt\n```\n\n# Using custom data\n\nAs you might have noticed, this repository includes a `data/` folder. This folder is empty (except for a `.gitignore`\nfile) and is ignored by Git, which means that you can put anything in there and it won't affect your commits.\n\nA lot of scripts and Jupyter notebooks in this repository assume that relevant files are located in the `data/` folder.\nFor example, the code that assumes that files `data/EndoscopeImageMemory_0.avi` and `data/pose_ecm.csv` exist will look\nas follows:\n\n```python\nfrom os import path\nfrom ds_tools.shared import util\n\ndata_dir = util.get_data_dir()\nvideo_path = path.join(data_dir, 'EndoscopeImageMemory_0.avi')\npose_ecm_path = path.join(data_dir, 'pose_ecm.csv')\n```\n\nYou are free to adjust the paths and filenames for your convenience. Note that you don't *have* to put your data into \nthe `data/` folder - you can use your own absolute paths, e.g. `/home/my_user/my_data/pose_ecm.csv`. Keep in mind \nthat the **output** some scripts generate might go into the `data/` folder by default.\n\n**Important:** Example data for placenta phantom texture reconstruction can be downloaded [here](https://drive.google.com/open?id=1t78ZC9vEUOBj5gDyg9efxpATznlVgkvD). This archive has to be placed inside the `data/` folder and unzipped. The code in this repository has been \nadjusted to use this data set. You can use it as a reference when preparing your own data set for texture \nreconstruction.\n\n# Using Jupyter notebooks\n\nThe `notebooks/` folder and its sub-directories contain Jupyter notebooks that showcase various tools from this \nrepository. The Python import statements in these notebooks are relative to the `.ipynb` files, which means that the \nnotebooks should work regardless of where you run `jupyter notebook`. That said, it's recommended to start the \nJupyter server as follows:\n```bash\ncd notebooks/\njupyter notebook\n```\n\n# User manual\n\nThis section provides instructions on how to use each individual part of the project.\n\n## Converting dvLogger output CSV files\n\ndvLogger recorder is the tool used to record da Vinci kinematics and stereo endoscope footage. It starts logging data\n the moment it boots up, given it's connected to the da Vinci surgical system. The data for each recording session \n contains multiple `.issi` files and 2 `.avi`. The two videos are the video footage from the left and right stereo \n endoscope cameras respectively, and 3 of the `.issi` contain da Vinci kinematics and timestamps for video frames for\n each camera. Typical output of the dvLogger looks like this:\n ```\n.\n├── ...\n├── DaVinciSiMemory.issi\n├── EndoscopeImageMemory_0.avi\n├── EndoscopeImageMemory_0.issi\n├── EndoscopeImageMemory_1.avi\n└── EndoscopeImageMemory_1.issi\n```\n  \nTo make these `.issi` files easier to use with Python, they must first be converted into the `.csv` format. This can \nbe done using the `ConvertAll.m` script from the `matlab_issi/` folder. Place the `.issi` files listed above into \n`matlab_issi/`, then either run `ConvertAll.m` or run the following Matlab code directly:\n```matlab\ndvsm = ISSILoad('DaVinciSiMemory.issi');\neim0 = ISSILoad('EndoscopeImageMemory_0.issi');\neim1 = ISSILoad('EndoscopeImageMemory_1.issi');\n\ndvsmData2csv(dvsm, 'DaVinciSiMemory.csv');\neimData2csv(eim0, 'EndoscopeImageMemory_0.csv');\neimData2csv(eim1, 'EndoscopeImageMemory_1.csv');\n```\n\nThis code will convert all 3 files into CSV format, and save them into `matlab_issi/`, but it might be a good idea to\n move them to the `data/` folder. Note that this Matlab code can only be ran on a Windows edition of Matlab, since \n some of the proprietary ISSI loading code from `matlab_issi/` folders uses encrypted binaries which are only \n compatible with Windows.\n \n## Syncing timestamps in dvLogger data\n \nThe timestamps in the CSV files generated above are not synced by default - some of the might skip several \nframes or have slightly different rates. The 3 CSV files and 2 AVI files need to be processed further to make sure \nthe data is in sync. This can be done using the `sync_timestamps.py` script from the `ds_tools/scripts/` folder. Open\n`sync_timestamps.py` and change the `capture_dir` variable to reflect the directory where your files are located.\n For example, if you were to place the files in `data/my_capture/`, you will have to update `capture_dir` to look as \n follows:\n```python\n# ...\ndef main():\n    data_dir = util.get_data_dir()\n    capture_dir = path.join(data_dir, 'capture_dir')\n    # ...\n```\n\nYou can then run the script using:\n```bash\npython3 ds_tools/scripts/sync_timestamps.py\n```\n\nOnce the script terminates, a folder called `synced/` will be created inside the directory you specified. This folder\n will contain the processed CSV files and AVI videos, all synced to 30 FPS.\n \n## Segmenting 3D models from CT scans, UV unwrapping\n\nCT scans almost always come as a collection of DICOM images, where each image represents a single slice through the \nvolume of the scanned scene. `.DICOM` files can be viewed using [itk-SNAP](http://www.itksnap.org/pmwiki/pmwiki.php).\n Relevant objects, such as markers on the calibration object or markers on the endoscope collar, can be easily \n segmented from the CT scan using one of [itk-SNAP's automatic segmentation tools](http://www.itksnap.org/docs/viewtutorial.php?chapter=TutorialSectionIntroductionToAutomatic) (snakes evolution tool is recommended). \n\nNote that groups of markers, e.g. all endoscope collar markers, must all be segmented into the same label, and each \n distinct group should have its own label. The recorded surface, e.g. the walls of the recorded surgical phantom, can\n be segmented using the same approach.\n  \nOnce segmentation is done, each label should be exported as a separate `.stl` mesh. This can be done using a built-in\n itk-SNAP tool for mesh export.\n\n---\n\nThe object of the mesh whose texture will be reconstructed (e.g. surgical phantom) has to go through extra processing\n steps, such as smoothing and UV unwrapping. These steps can be carried out using [Blender](https://www.blender.org/)\n software. The relevant `.stl` can be imported into Blender using the built-in STL plugin (with orientation Z-up \n Y-forward and same scaling as original model).\n \nThe number of faces can be reduced using Blender's [decimate modifier](https://docs.blender.org/manual/en/latest/modeling/modifiers/generate/decimate.html), and smoothing can be applied using [Laplacian \nsmooth modifier](https://docs.blender.org/manual/en/latest/modeling/modifiers/generate/decimate.html). After smooth \nis complete, the mesh needs to be UV unwrapped. This can be done using angle based flattening (ABF) in Blender's smart\n UV projects (see [unwrapping guide](https://docs.blender.org/manual/en/latest/editors/uv_image/uv/editing/unwrapping/index.html)).\nSome arbitrary texture must then be applied to the object (e.g. a chessboard pattern of size `2048 x 2048` pixels) to\n give texture reconstruction code some reference for the desired size of the figure.\n \nOnce the model is UV unwrapped and a texture is applied to it, it needs to be exported as an `.obj` file (since `.stl` \ndoesn't encode UV mappings). This `.obj` file can then be used in texture reconstruction.\n\n\n## Camera calibration\n\nThe code for camera calibration is expressed as Jupyter Notebooks found in `notebooks/calibration/`. The main goal of\n the calibration process is to produce `data/intrinsics.json` and `data/extrinsics.json` files which encode the \n intrinsics and extrinsics parameters of the camera respectively. These can later be used to estimate the camera pose\n in various shot of the object whose texture is reconstructed.\n \nAll of the notebooks contain some default values which are built around the placenta phantom data set mentioned \nearlier, so it's a good idea to download it and use as reference.\n\n**Intrinsics calibration**: Intrinsics are calibrated in the `intrinsics_calibration.ipynb` notebook. Calibration is \ndone using footage of a planar chessboard pattern moving in front of the camera. Calibration can be done using a \nshort video or by specifying a path to a folder that contains frames with calibration object visible (such frames can\n be extracted using video players, e.g. [VLC](https://www.videolan.org/vlc/index.en-GB.htm)). The output is \n `data/intrinsics.json`, which records the intrinsic matrix, distortion coefficients and the image size of the camera.\n \nIf necessary, radial distortion coefficients can be refined by marking straight lines one some video frames \nusing `ds_tools/scripts/mark_points.py` and then using `intrinsics_refining.ipynb`. Once intrinsics of the camera are\n known, `rectify_images.ipynb` can be used to correct distortion on arbitrary images taken by the camera.\n \n**Extrinsics calibration**: Extrinsics calibration is done in two steps. First, we extract the shape of rigid body \nassociated with the endoscope collar using `endoscope_rigidbody_calibration.ipynb`. To do that, we need the segmented\n endoscope markers from the calibration scene and the segmented endoscope markers from the actual data capture scene \n (both extracted from different CT scans). Only the first set of markers is used to extract the shape of the rigid \n body, the second set is just used to verify how accurate the segmentation is. This notebook creates the file \n `data/extrinsics.json`.\n \nThe second step involves working out the extrinsics of the camera with respect to the rigid body defined in the \nprevious step. This is done in `endoscope_camera_calibration.ipynb`. It requires meshes of endoscope markers (mesh 1)\nand the mesh of markers of the calibration object (mesh 2), segmented from the CT scan of the calibration scene. It \nalso requires an image from the endoscope cameras, taken at the time when CT scan was taken. All calibration markers\nmust be visible in the image. This process works out the transformation between the camera and the reference rigid \nbody, and saves the result into `data/extrinsics.json`.\n\n**3D render test**: To make sure that extrinsics and intrinsics are calibrated correctly, `3d_render_test.ipynb` is used\nto generate a synthetic render of the scene with the phantom. It loads the JSON files generated in the previous \nsteps, as well as the 3D mesh of the recorded object. The results for the placenta data set can be seen below.\n\nThis notebook also outputs the `data/phantom_scene.json` file, which contains information about the final camera \nparameters calculated explicitly for this scene. These parameters are automatically imported when you run \nthe `ds_tools/calibration/render.py` script. The script lets you move the camera around to estimate camera poses for \nother frames from the endoscope video footage.\n\n| ![](figures/3d_render_test_synthetic.png) | ![](figures/3d_render_test_real.png) |\n|-------------------------------------------|--------------------------------------|\n\n## Projective texture mapping\n\nTo prepare the data for texture reconstruction, one should extract several frames that best cover the surface of \ninterest from the endoscope footage, and estimate the camera pose using the method described above. Script \n`ds_tools/scripts/visualise_kinematics.py` provides a convenient way to preview videos from stereo cameras, allowing \nthe user to take snapshots of the current frame from both cameras. Before it can be used, the file \ncontaining da Vinci kinematics, `DaVinciSiMemory.csv`, must be broken doing into individual CSV files (covering each \nmanipulator), which can be done by running `ds_tools/scripts/split_join_csv.py`.\n \nOnce this data is available, images need to be renamed into the format `\u003cindex\u003e_screenshot.png`, and camera \nparameters need to be stored in `capture_data.json` in the same order as the images. See \n`resources/placenta_images/` for reference. The alignment of images can be improved using \n`ds_tools/texture_mapping/camera_pose_refinement.py`, given that overlapping image projections have enough distinct \nedges.\n\n**Converting data to UV space**: Once all of the images have correct names and `capture_data.json` is populated with \ncamera poses, all of te data needs to be converted into UV space. This is done by running the \n`ds_tools/texture_mapping/texture_mapping_renderer.py` script. It initialises the 3D renderer, loads relevant meshes \nand images, and generates projections of every image using projective texture mapping. Example output of this script \ncan be seen in the `resources/placenta_texture/` folder.\n\n## Texture reconstruction\n\nAfter all of the data is converted into UV space, the texture can be reconstructed by running the \n`ds_tools/texture_mapping/texture_merger.py` script. It generates confidence maps for every image projection, and \nthen merges projection based on normalised confidence scores. Example output of this process can be seen on \n`resources/placenta_texture/base_final.png`.\n\nFor the example data, the texture was reconstructed using the mesh `resources/3d_assets/placenta.obj`. To view the \nfinal texture in 3D, this mesh can be imported into Blender, with its texture can be set to  \n`resources/placenta_texture/base_final.png`, obtaining a result similar to:\n![](figures/texture_reconstruction.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimbokz%2Fsrv-dataset-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftimbokz%2Fsrv-dataset-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimbokz%2Fsrv-dataset-tools/lists"}