https://github.com/buvnswrn/droneinspection
Repository for Drone Inspection Digital Twin
https://github.com/buvnswrn/droneinspection
avro-schema basys digital-twins dji-sdk drone java kafka maven numpy object-detection opencv-python pose-estimation python pytorch restx-api roboflow robots unity3d
Last synced: 15 days ago
JSON representation
Repository for Drone Inspection Digital Twin
- Host: GitHub
- URL: https://github.com/buvnswrn/droneinspection
- Owner: buvnswrn
- Created: 2022-12-23T09:51:21.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2024-04-07T17:46:35.000Z (about 2 years ago)
- Last Synced: 2025-09-10T01:31:09.171Z (9 months ago)
- Topics: avro-schema, basys, digital-twins, dji-sdk, drone, java, kafka, maven, numpy, object-detection, opencv-python, pose-estimation, python, pytorch, restx-api, roboflow, robots, unity3d
- Homepage:
- Size: 2.62 MB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# droneinspection
[](LICENSE.md)
This project was created as part of a Seminar - SupRTwin: Sensing, Understanding, and Provisioning of Robotic Digital Twins from Universität des Saarlandes. The objective of the project is to create a Digital Twin of objects that are detected during an inspection flight conducted by a drone. The Digital twins of the objects and the drone are represented in a realtime simulation environment in Unity Game Engine.
Original Repository: https://mrk40.dfki.de/drone-inspection-seminar/droneinspection
## Introduction
This repository contains the presentation files used for the final seminar presentation and some set of Experiment data.
The end to end flow looks like:
Have a look at the working Demo here:
[](https://youtu.be/lHDcypc_lmY)
## Component and Explanation
### MQTT Drone API
The drone uses [MQTT Protocol](https://mqtt.org/) to communicate. In short, the following topics are of our interest
- Mavic2/state/pose - Pose information obtained from markers of drone using a Motion Capture system (OptiTrack system) and published to MQTT through [Motive App](https://optitrack.com/software/motive/)
- Mavic2/state/physical - Current Physical State obtained using [DJI SDK](https://developer.dji.com/mobile-sdk-v4/) and [Android dronecontrol app](https://mrk40.dfki.de/mrk-4.0/androiddronecontrol)
### Drone Live Monitoring
The drone can be monitored live and inputs such as waypoints and certain commands such as starting live image can be provided to the drone through the Live Monitoring Web App.
### Drone Inspection Service API
- The project involves creation of two main services - Inspection Flight and Object Detection. They are created using [Flask-RestX](https://flask-restx.readthedocs.io/en/latest/).
### Drone Object Detection
- We use custom trained [YoloV5](https://github.com/ultralytics/yolov5) model to detect the objects
### Pose Estimation
- We use inverse Homography technique to estimate the pose of the objects detected.
- The flow of the pose estimation is given below
### Process Control and States
- The process control is done with the help of Camunda Process Model and [PackML](https://www.omac.org/packml) standard is used to maintain robotic states across the PPR Dashboard along with other robots.
- The states can be monitored in PPR Dashboard or using [OPC UA Client](https://opcfoundation.org/about/opc-technologies/opc-ua/).
- The [PPR Dashboard](https://github.com/dfkibasys/ppr-dashboard) is a part of the BaSys project which looks like the one below
### Simulation
- The digital twin of the detected object is then visualized using simulation framework - SiMRK
### Tech Stack :
[//]: # ()
