https://github.com/pulp-platform/pulp-detector
https://github.com/pulp-platform/pulp-detector
Last synced: 5 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/pulp-platform/pulp-detector
- Owner: pulp-platform
- Created: 2023-01-20T13:47:10.000Z (over 3 years ago)
- Default Branch: master
- Last Pushed: 2025-04-28T14:03:34.000Z (about 1 year ago)
- Last Synced: 2025-04-28T15:24:27.574Z (about 1 year ago)
- Language: C
- Size: 17.6 MB
- Stars: 13
- Watchers: 3
- Forks: 3
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE.Apache.md
Awesome Lists containing this project
README
PULP-Detector
Lorenzo Lamberti, Luca Bompani, Victor Javier Kartsch, Manuele Rusci, Daniele Palossi, Luca Benini.
Copyright (C) 2023 University of Bologna, ETH Zürich. All rights reserved.
**Video:** [YouTube](https://youtu.be/BTin8g0nyko)
**Citing:** *"Bio-inspired Autonomous Exploration Policies with CNN-based Object Detection on Nano-drones"* ([IEEExplore](https://ieeexplore.ieee.org/abstract/document/10137154), [arXiv](https://arxiv.org/abs/2301.12175))
~~~~
@INPROCEEDINGS{pulp_detector,
author={Lamberti, Lorenzo and Bompani, Luca and Kartsch, Victor Javier and Rusci, Manuele and Palossi, Daniele and Benini, Luca},
booktitle={2023 Design, Automation \& Test in Europe Conference \& Exhibition (DATE)},
title={{{Bio-inspired Autonomous Exploration Policies with CNN-based Object Detection on Nano-drones}}},
year={2023},
volume={},
number={},
pages={1-6},
doi={10.23919/DATE56975.2023.10137154}}
~~~~
## 1. Introduction
### What is PULP-Detector ?
**PULP-Detector** is a nano-drone system that strives for both maximizing the exploration of a room while performing visual object detection.
The Exploration policies as implemented as lightweight and bio-inpired state machines.
The object detection CNN is based on the MobilenetV2-SSD network.
The drone performs obstacle avoidance thanks to Time-of-flight sensors.
The drone is completely autonomous -- **no human operator, no ad-hoc external signals, and no remote laptop!**
- **Software component:**
Object detection CNN: is a shallow convolutional neural network (CNN) composed of Mobilenet-v2 backbone plus the SSD (single-shot detector) heads.
It runs at 1.6-4.3 FPS onboard.
- **Hardware components:**
The hardware soul of PULP-Detector is an ultra-low power visual navigation module embodied by a pluggable PCB (called *shield* or *deck*) for the [Crazyflie 2.0](https://www.bitcraze.io/crazyflie-2/)/[2.1](https://www.bitcraze.io/crazyflie-2-1/) nano-drone. The shield features a Parallel Ultra-Low-Power (PULP) GAP8 System-on-Chip (SoC) from GreenWaves Technologies (GWT), an ultra-low power HiMax HBM01 camera, and off-chip Flash/DRAM memory; This pluggable PCB has evolved over time, from the [*PULP-Shield*](https://ieeexplore.ieee.org/document/8715489) , the first custom-made prototype version developed at ETH Zürich, and its commercial off-the-shelf evolution, the [*AI-deck*](https://store.bitcraze.io/products/ai-deck).
Summary of characteristics:
- **Hardware:** [*AI-deck*](https://store.bitcraze.io/products/ai-deck)
- **Deep learning framework:** Tensorflow 1.15 ([Tensorflow Object detection API](??))
- **Quantization**: fixed-point 8 bits, fully automated with [NNTool](https://greenwaves-technologies.com/sdk-manuals/nn_quick_start_guide/)
- **Deployment**: fully automated with [AutoTiler](https://greenwaves-technologies.com/sdk-manuals/nn_quick_start_guide/)
We release here, as open source, all our code, hardware designs, datasets, and trained networks.
## Setup
Clone recursively to download all submodules
```
git clone git@github.com:pulp-platform/pulp-detector.git --recursive
```
## PULP-Platforms refs
[PULP Platform Youtube](https://www.youtube.com/c/PULPPlatform) channel (subscribe it!)
[PULP Platform Website](https://pulp-platform.org/).
## Licenses
All files under:
* `./crazyflie_app/random-following-spiral`
* `./crazyflie_app/rotate`
* `./gap8_app/SSD_tin_can_bottle.c`
are original and licensed under Apache-2.0, see [LICENSE.Apache.md](LICENSE.Apache.md).
The images used for the training and testing need to be downloaded and copied into the following folder:
* `dataset/`
all the files can be downloaded from this [link](https://zenodo.org/records/10992237) and are under the Creative Commons Attribution Non Commercial No Derivatives 4.0 International see [LICENSE.CC.md](LICENSE.CC.md)
All files under:
* `./training/`
Are from [Tensorflow](https://github.com/tensorflow/models/), released under Apache-2.0 License, see [LICENSE.Apache.md](LICENSE.Apache.md).
All files under:
* `./gap8_app/` (except for `./gap8_app/SSD_tin_can_bottle.c`)
Are from [GreenWaves Technologies](https://github.com/GreenWaves-Technologies), released under a BSD License, see [LICENSE.BSD.md](LICENSE.BSD.md)
The external modules under:
* `./viewer-pulp-detector/`
* `./crazyflie_app/crazyflie-firmware`
* `./crazyflie_app/crazyflie-firmware-modified`
Are from [Bitcraze](https://github.com/bitcraze), released under a GPL-3.0 license.