{"id":13437466,"url":"https://github.com/Eagleflag88/MultiModalEvnPercption","last_synced_at":"2025-03-19T06:31:20.089Z","repository":{"id":121169472,"uuid":"348214266","full_name":"Eagleflag88/MultiModalEvnPercption","owner":"Eagleflag88","description":"Experimental platform to achieve fused environment perception using different modalities of sensors","archived":false,"fork":false,"pushed_at":"2021-12-14T09:31:48.000Z","size":16605,"stargazers_count":5,"open_issues_count":0,"forks_count":6,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-10-27T21:51:51.816Z","etag":null,"topics":["computer-vision","ekf-slam","factor-graph","imu","lidar-point-cloud","ros","slam"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Eagleflag88.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2021-03-16T04:33:35.000Z","updated_at":"2024-04-08T09:02:18.000Z","dependencies_parsed_at":"2024-03-11T16:46:17.945Z","dependency_job_id":null,"html_url":"https://github.com/Eagleflag88/MultiModalEvnPercption","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Eagleflag88%2FMultiModalEvnPercption","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Eagleflag88%2FMultiModalEvnPercption/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Eagleflag88%2FMultiModalEvnPercption/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Eagleflag88%2FMultiModalEvnPercption/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Eagleflag88","download_url":"https://codeload.github.com/Eagleflag88/MultiModalEvnPercption/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244371105,"owners_count":20442335,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","ekf-slam","factor-graph","imu","lidar-point-cloud","ros","slam"],"created_at":"2024-07-31T03:00:57.382Z","updated_at":"2025-03-19T06:31:15.077Z","avatar_url":"https://github.com/Eagleflag88.png","language":"C++","funding_links":[],"categories":["C++"],"sub_categories":[],"readme":"# MultiModalEvnPercption\n\nA ROS-based experimental software platform for Multi-Modal Simultaneous Localization And Mapping.\n\nFor the purpose of studying and research in the field of SLAM, based on the infrastructure of ROS, this repo provide a software platform which is highly \nmodularized and is able to fuse the information from a number of different modalities to achieve to an accurate and consistent environmental understanding. \n\n# Software Architecture\n\nThe whole package of software consists of three layers, as depicted in following picture:\n\n![Software Architecture](https://github.com/Eagleflag88/MultiModalEvnPercption/blob/main/SoftwareArch.jpg)\n\n## Sensors\nModeling of the sensor driver, which acts as the source of sensor measurements. In the case of simulation, \nthese nodes take the playback of rosbag or other compliant data format.\n\n## Frontend\nResponsible for preprocessing of the measurements, such as filtering and undistortion. \nAs next steps, feature extraction and motion estimation are implemented. \nMeanwhile, objection detection based on deep learning is implemented for object-level environmental understanding. \nIn addition, a tracker node is exploited to take care of data association, in feature and object-level.\n\n## Backend \nWhile the frontend focuses on the real-time performance, the nodes of the backend try to fuse the initial guesses \nfrom frontend to obtain more accurate estimate of the ego poses and landmarks. \nTo this end, a sliding window is maintained to constrain the computational burden and complexity. \nWithin the sliding window, maximum-a-posteriori estimates are calculated based on the inference of a factor graph. \nAlternatively, an EKF can be applied.\n\n\n# Dependencies\n\n## Basic Software Environment\nOS: Ubuntu 20.04 \\\nROS: Noetic\\\nOpenCV \u0026 Opencv_Contrib: 4.4.0 (For Compatibility with Cuda 11), Consult the Installation Instruction\\\nOpenMP:\n```\nsudo apt install libomp-dev    \n```\n\n## Software for DL Based Environment Perception\nCuda: 11: Consult the Installation Instruction\\\nCudnn: 8: Consult the Installation Instruction\\\nTensorRT: Consult the Installation Instruction\\\nDarknet For Yolo: Consult the Installation Instruction\n\n## SLAM Related\nCeres: Consult the Installation Instruction; SuiteSparse and CXSparse are included \\\nEigen 3:\n```\nsudo apt install libeigen3-dev\n```\ng2o: Comes with the repo, github commit a48ff8c\\\nGeographic_Lib: A library dealing with geographic coordinate conversion, comes with the repo\n\n# Building\n\n## Clone the Repo\n```\ngit clone https://github.com/Eagleflag88/MultiModalEvnPercption.git\n```\n## Build the Ros Package\n```\n./build.sh\n```\n\n# Todo\n\n0. Refactor the codebase.\n1. Implement the keyframe_node, factor_graph_node and ekf_node.\n2. Create launch files for different functional modes, for example, camera_only, lidar_only, visual inertial, etc.\n3. Add the node for place recognition into the software architecture.\n4. Add the HD-Map as a further source of information.\n5. Add semantic pipeline\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FEagleflag88%2FMultiModalEvnPercption","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FEagleflag88%2FMultiModalEvnPercption","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FEagleflag88%2FMultiModalEvnPercption/lists"}