{"id":13445503,"url":"https://github.com/ArashJavan/DeepLIO","last_synced_at":"2025-03-20T21:30:43.756Z","repository":{"id":38307270,"uuid":"241644304","full_name":"ArashJavan/DeepLIO","owner":"ArashJavan","description":"Deep Lidar Inertial Odometry","archived":false,"fork":false,"pushed_at":"2021-07-29T09:38:12.000Z","size":21127,"stargazers_count":53,"open_issues_count":7,"forks_count":10,"subscribers_count":9,"default_branch":"master","last_synced_at":"2024-08-01T05:14:58.356Z","etag":null,"topics":["deep-learning","imu","imu-sensor","lidar-point-cloud","odometry"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ArashJavan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-02-19T14:36:06.000Z","updated_at":"2024-07-26T14:53:37.000Z","dependencies_parsed_at":"2022-08-25T21:23:20.555Z","dependency_job_id":null,"html_url":"https://github.com/ArashJavan/DeepLIO","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArashJavan%2FDeepLIO","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArashJavan%2FDeepLIO/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArashJavan%2FDeepLIO/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArashJavan%2FDeepLIO/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ArashJavan","download_url":"https://codeload.github.com/ArashJavan/DeepLIO/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221807711,"owners_count":16883638,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","imu","imu-sensor","lidar-point-cloud","odometry"],"created_at":"2024-07-31T05:00:34.985Z","updated_at":"2025-03-20T21:30:43.750Z","avatar_url":"https://github.com/ArashJavan.png","language":"Jupyter Notebook","funding_links":[],"categories":["5 Localization and SLAM"],"sub_categories":["3.4 High Performance Inference"],"readme":"# DeepLIO\n__Deep Lidar Inertial Odometry__\n\n\n1. Intorduction\n\nDeepLIO is an deep learning based odometry estimation by fusing lidar and IMU. \n\nCloning deeplio\n```\ngit clone https://github.com/ArashJavan/DeepLIO.git\ncd DeepLIO\n```\n\n\n### __1.1 Dependencies__\n\nFollowing packages should be already installed, before you can start using deeplio.\n- pytorch (\u003e v1.4)\n- tensorboard \n- [pytorch-model-summary](https://pypi.org/project/pytorch-model-summary/)\n- tqdm (optional)\n- open3d (optinal)\n\n### __1.2 Preparing the KITTI Dataset__\n\n__Downloading KITTI__\n\nIn this project the synced and unsyced (extract) version of the KITTI dataset is used.\nFrom unsynced (extract) data we only need the IMu measurments and from the synced data the lidar frames and the ground truth. \nSo if you want to reduce the storage space, you can safely remove the images folder and also the lidar frames from the unsyced versoin.\nEithter you download the datasets by yourself or you can also  run the _download_kitti_raw.sh_ script to download the KITTI raw sequences.\n\n__Note__: You will need at least 300 GB free sapce on your hard drive.\n\n```\n$ mkdir -p datasets/KITTI/sync\n$ download_kitti_raw.sh datasets/KITTI  sync\n\n$ mkdir -p datasets/KITTI/extract\n$ download_kitti_raw.sh datasets/KITTI  extract\n```\nNow wait till download is completed. Well it will take a long long time so go get some coffee :)\n\nAt the end you will find all sequences extracted under KITTI folder\n```\nKITTI\n|\n extract|\n        |\u003e 2011_09_30\n        |-\u003e 2011_09_30_drive_0016_extract\n            |-\u003e image0 .. imahe3\n            |-\u003eoxts\n            |-\u003evelodyne_points\n        .\n        .\n        .\n    |-\u003e 2011_10_03\n        |-\u003e 2011_09_03_drive_0027_extract\n    .\n    .\n    .\n sync|\n    .\n    .\n    .\n```\n\n### __1.3 DeepLIO Architecture__\nDeepLIO is made completely modular and configurable, e.g. every part and module can be combined with other modules to build\nthe whole network architecture. As you can see from the following figure, there are four main modules.\n- __LiDAR-Feature-Nets__:\nThis module is responsible for extracting (learn) and encoding the LiDAR frames which are first transformed by spherical projection.\n- __IMU-Feature-Nets__:\nThis module is responsible for extracting (learn) and encoding the IMU measurements, which consists of linear acceleration and \n angular velocity (dim=6).\n- __Fusion-Net__:\nThis module is responsible for fusing the features extract fom LiDAR and IMU.\n- __Odometry-Net__:\nAt least in this module the fused features are used to learn the hidden state, which shall explain the odometry information \nencoded in these features.\n\n\u003cimg src=\"resources/images/deepliov2.png\" alt=\"deeplio\" width=\"500\"/\u003e\n\n#### __1.3.1 LiDAR-Feature-Nets__\nThe inputs to this module are the projected LiDAR frames and their normal images, which in the current configuration is built by stacking two remission (intensity) and\ndepth channels together. The module itself is made of an arbitrary Convolutional Neural Network (CNN). These networks are \nimplemented as a [siamese netowrk](https://github.com/ArashJavan/DeepLIO/blob/master/deeplio/models/nets/lidar_feat_nets.py#L135).\nAt this point in time there are four CNN-Architectures implemented:\n\n1.  [FlowNet-Feature-Net]\n\n__TODO__ adding link to flownet here.\n\n2.  [ResNet-Feature-Net]\n\n__TODO__ adding link to resnet here.\n\n3.  [Simple-Feature-Net]\n\n4. [PointSeg-Feature-Net](https://github.com/ArashJavan/DeepLIO/blob/master/deeplio/models/nets/lidar_feat_nets.py#L43)\nThis network is the implementation of the more demanding [PointSeg Netowrk](https://github.com/ArashJavan/PointSeg). \nthe network it self is made of three functional layers, the so called Fire layers,  squeeze reweighting\nlayer and enlargement layer. An interested reader may find more information about each of those layers in the original\n[PointSeg paper](https://arxiv.org/pdf/1807.06288.pdf).\n\nAll the above listed networks can be found and configured in the configuration file:\n\n```yaml\n### Lidar Feature Netowrks ###########################\n# feature network simple0 with conv layers\n\n# feature network pointseg\nlidar-feat-pointseg:  # pointseg feature\n  dropout: 0.1\n  classes: ['unknown', 'object']\n  bypass: \"simple\"\n  fusion: add # [cat, sub, add]\n  part: \"encoder\" # [encoder]\n\nlidar-feat-flownet:\n  dropout: 0.\n  fusion: add # [cat, sub, add]\n\nlidar-feat-resnet:\n  dropout: 0.25\n  fusion: add # [cat, sub, add]\n\n# feature network simple0 with bypassed conv layers\nlidar-feat-simple-1:\n  dropout: 0.25\n  fusion: add # [cat, sub, add]\n  bypass: false\n```\n\nWith the keyword _fusion_ you can define, if the both threads of the siamese network should be concatenated together or subtracted.\nPlease note that by subtracting you will have the half number of feature as by concatenating. \n\n#### __1.3.3 IMU-Feature-Nets__\nThe inputs to this modules are sequences of the IMU measurements. Basically any kind of Fully Connected (FC) Network\ncan be configured here. Since the IMU measurements are also time series it is quit appealing to handle them \nwith an RNN based network. For this reason both of these networks - FC and RNN (LSTM, GRU), are also implemented \nin the  [IMU-Module](https://github.com/ArashJavan/DeepLIO/blob/master/deeplio/models/nets/imu_feat_nets.py).\nAgain you can freely configure each network in the config-file section:\n\n```yaml\n### IMU Feature Netowrks ###########################\n# IMU Feature Netowrks\nimu-feat-fc:  # FC\n  input-size: 6  # !fixed! do not chanage\n  hidden-size: [1024, 512, 256]\n  dropout: 0.\n\nimu-feat-rnn: # RNN\n  type: \"lstm\"\n  input-size: 6  # !fixed! do not chanage\n  hidden-size: 128\n  num-layers: 2\n  bidirectional: true\n  dropout: 0.\n```\n\nPlease note that the networks are build dynamically, e.g. you can change number of hidden layers an other stuff as required.\n\n#### __1.3.2 Fusion-Layer__\nAfter we have encoded the features in each sensory information by passing them through each IMU and LiDAR Network, \nwe should some how fusion them together for the task of odometry inference. One classical approach may be using some kind of \nfilter like for example Kalman-Filter, but since we want a fully differentiable system, we should integrate this part also \nin the network. One simple procedure would be just concatenating them together, or use another network to learn the \ncorrelation between those features, as in attention based networks. For now only concatenating procedure is implemented.\n\n```yaml\n### Fusion-Layer for DeepLIO Net ###########################\nfusion-layer:\n  type: \"cat\" # [cat, soft-fusion]\n```\n\n#### __1.3.4 Odometry-Feature-Nets__\nThis layer should learn how to extract information needed for the task fo odometery inference from the fused features.\nSince odometry inference can be  typically seen as an inference in a time series, both RNN-based an FC-based networks \ncan be used here. Like in IMU-Features you can configure the odom-feat-network as you need.\n\n```yaml\n### Odometry Feature Netowrks ###########################\n# odometry feature network with fully connected layers\nodom-feat-fc:\n  size: [256]\n  dropout: 0.\n\n# odometry feature network with rnn-layers\nodom-feat-rnn:\n  type: \"lstm\"\n  hidden-size: 512\n  num-layers: 2\n  bidirectional: true\n  dropout: 0.\n```\n \n### __1.4 Configuring DeepLIO__\nAs mentioned before you can configure each module of the network as you need, e.g. you can also forego some modules if you\nwant.  \n\n```yaml\n### DeepLIO Network ##############################\ndeeplio:\n  dropout: 0.25\n  pretrained: false\n  model-path: \"\"\n  lidar-feat-net:\n    name: \"lidar-feat-simple-1\"\n    pretrained: false\n    model-path: \"\"\n  imu-feat-net:\n    name: \"imu-feat-rnn\"\n    pretrained: false\n    model-path: \"\"\n  odom-feat-net:\n    name: \"odom-feat-fc\"\n    pretrained: false\n    model-path: \"\"\n  fusion-net: \"fusion-layer\"\n```\n\nYou can choose between different network implementations by just replacing the name of the desired network.  \n\n\n#### __1.4.1 DeepIO__\n__Deep Inertial Odometry__\nYou can configure the network in a way that it only uses IMU measurements for odometry prediction. You do it \njust by commenting out '#' the name of all other modules, as below:\n\n```yaml\n ### DeepLIO Network ##############################\ndeeplio:\n  dropout: 0.25\n  pretrained: false\n  model-path: \"\"\n  lidar-feat-net:\n    name: # \"lidar-feat-simple-1\" # See the name is commented out\n    pretrained: false\n    model-path: \"\"\n  imu-feat-net:\n    name: \"imu-feat-rnn\"\n    pretrained: false\n    model-path: \"\"\n  odom-feat-net:\n    name: # \"odom-feat-fc\" # See the name is commented out\n    pretrained: false\n    model-path: \"\"\n  fusion-net: \"fusion-layer\"\n```\n\nPlease also note that you can for example have IMU and Odometry module by just commenting Lidar-Module out!\n\n#### __1.4.2 DeepLO__ \n__Deep LiDAR Odometry__\nIn the same way you can configure the network in a way that it only uses Lidar measurements for odometry prediction. You do it \njust by commenting out '#' the name of all other modules, as below:\n\n```yaml\n ### DeepLIO Network ##############################\ndeeplio:\n  dropout: 0.25\n  pretrained: false\n  model-path: \"\"\n  lidar-feat-net:\n    name: \"lidar-feat-simple-1\"\n    pretrained: false\n    model-path: \"\"\n  imu-feat-net:\n    name: \"imu-feat-rnn\"  # See the name is commented out\n    pretrained: false\n    model-path: \"\"\n  odom-feat-net:\n    name: # \"odom-feat-fc\" # See the name is commented out\n    pretrained: false\n    model-path: \"\"\n  fusion-net: \"fusion-layer\"\n```\n\nPlease also note that you can for example have LiDAR and Odometry module by just commenting IMU-Module out!\nAs you can easily guess you will get the whole __DeepLIO__ netowrk by naming all the modules you want in the network.  \n\n\n### __1.5 Data Normalization__\nThe inputs to the LiDAR-Feat-Nets are as mentioned before the projected frame, consisting of two remission and intensity channels.\nAs you can imagine they do not have the same scale, for this reason they should be normalized before we pass them to the network.\nThe same is true for IMU-measurments. Mean and standard deviation of each inputs to the netowrk \nis  calculated by [Mean-Std-Script](https://github.com/ArashJavan/DeepLIO/blob/master/scripts/mean_std_dataset.py).\nFor the KITTI dataset the values are already incorporated in the config-file.\n```yaml\n    # channesl: x, y, z, range(x,y,z), remissions)\n    mean-image: [-0.05815, 0.4165, -0.8585, 0.1029, 0.2164]\n    std-image: [10.0366, 7.4936, 0.8761, 0.0857, 0.1740]\n    mean-imu: [-0.0635, 0.186, 9.7966, 0., 0.0005, 0.0082]\n    std-imu: [0.8525, 0.9516, 0.3401, 0.0198, 0.02204, 0.1383]\n```\n\n__Note__: In this work we use in the computer vision community more popular inverse depth representation\ninsted of depth values directly.\n\n\u003cimg src=\"resources/images/histogram_mean_and_std_inv_depth.png\" alt=\"deeplio\" width=\"500\"/\u003e\n\n### __1.6 Training DeepLIO___\nYou can train the network by first preparing the KITTI datset as explained before, then performing the required configurations\nin the config-file depending on your needs and at least starting the training by executing:\n\n```shell script\npython deeplio/train.py\n``` \nor \n```shell script\npython deeplio/train.py --lr 0.01 --weight-decay 0.0001 -b 32 --epochs 100 -j 6 --device cuda --print-freq 10 --lr-decay 20\n``` \n\nFor each traning instance one folder with the current timestampm will be created and the outputs of the netowork are all saved in there.\nThe Ouputs are:\n- Checkpoints. \n- Logfile.\n- Tensorboard logfile.\n\n\n### Citation\n\nIf you use this package for your research, a footnote with the link to this repository is appreciated: https://github.com/ArashJavan/DeepLIO.\nFor citation with BibTeX:\n\n```\n@article{isprs-annals-V-1-2021-47-2021,\nauthor = {Javanmard-Gh., A and Iwaszczuk, D and Roth, S},\ndoi = {10.5194/isprs-annals-V-1-2021-47-2021},\njournal = {ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences},\npages = {47--54},\ntitle = {{DEEPLIO: DEEP LIDAR INERTIAL SENSOR FUSION FOR ODOMETRY ESTIMATION}},\nurl = {https://www.isprs-ann-photogramm-remote-sens-spatial-inf-sci.net/V-1-2021/47/2021/},\nvolume = {V-1-2021},\nyear = {2021}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FArashJavan%2FDeepLIO","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FArashJavan%2FDeepLIO","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FArashJavan%2FDeepLIO/lists"}