https://github.com/loustoc/sentiers-aveugles
2nd Year project at Gobelins Paris - Sound mapping using Kinect v1, Three.JS and M5StickC-Plus
https://github.com/loustoc/sentiers-aveugles
art bluetooth computer-vision electron hardware installation interactive kinect libfreenect osc threejs
Last synced: about 19 hours ago
JSON representation
2nd Year project at Gobelins Paris - Sound mapping using Kinect v1, Three.JS and M5StickC-Plus
- Host: GitHub
- URL: https://github.com/loustoc/sentiers-aveugles
- Owner: Loustoc
- Created: 2024-06-27T00:53:43.000Z (12 months ago)
- Default Branch: main
- Last Pushed: 2024-06-27T05:06:19.000Z (12 months ago)
- Last Synced: 2025-02-23T14:43:04.979Z (4 months ago)
- Topics: art, bluetooth, computer-vision, electron, hardware, installation, interactive, kinect, libfreenect, osc, threejs
- Language: C
- Homepage:
- Size: 27.5 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Sentiers Aveugles (Blind Paths)
Final Year Project / POC of my 2nd Year at Gobelins - Paris.
**Theme :** `The 5 Senses and the 4 Elements`.
---
A defined area is subdivided in a (by default 3x3) grid, each subdivision then having a random sound mapped onto it.
The user's position in that space is calculated by weighting out the number of points in each subdivision, by depth and position on the X axis.
The gyroscope allows for the app to take full advantage of the Web Audio Spatialization API.
**1. Building the Kinect App:**
```
cd ./kinect
cmake .
cmake --build .
```
**2. Building and running the Electron app:**
```
cd ../app
npm i
npm run start
```
You should then see (by default) 9 spheres in a grid, along with a Sphere Mesh. The red section of the latter represents the eyes of the user.
**3. Setting up and Connecting the M5StickC-Plus:**
You'll need the following libs :
- [EasyButton](https:/github.com/evert-arias/EasyButton)
- [OSC](https:/github.com/CNMAT/OSC)
- [M5StickCPlus](https:/www.github.com/m5stack/M5StickC-Plus.git)
Keep in mind you'll need to add the M5Stack boards to your board manager, by using the following link :
[Boards JSON](https://static-cdn.m5stack.com/resource/arduino/package_m5stack_index.json)
After uploading the firmware to the board, you'll need to connect it to **the same WiFi network as the computer the apps run on**.
To do this, install the BlueFruit Connect app: [Android](https:/play.google.com/store/apps/details?id=com.adafruit.bluefruit.le.connect) / [iOS](https:/apps.apple.com/us/app/bluefruit-connect/id830125974).
Then, connect to the M5Stick. It should be called "SentiersAveugles-Gyro-XX".
You can now send your network info in that order (Check the M5Stick's screen for any error) :
- SSID
- PWD
- Local IP Address of the computer (you can scan the QRCode on the electron app to get it)
- OSC Port (default is 9419)
**4. Calibration:**
- Press `E`
on the Kinect App to calibrate the depth. This will allow the program to detect pixels that have changed depth (thus if something moved at that specific position), compared to a reference frame.
- After ensuring the red section on the Sphere Mesh moves when rotating the gyroscope along the Z axis, click on the `A` button when the red section faces the left side. Then, strap the gyroscope onto the headphones, face the kinect, and press the `A` button again. You're now all set !
## Additionnal Information
I'm using the M5StickC-Plus solely for the fact that it packs a gyroscope/accelerometer, WiFi, Bluetooth and a battery, without the need to plug anything else in. Since it's based off a widely used chip, the ESP32, and that I'm not using any specific chip reliant functionnality, it should be fairly easy to adapt.
## Shortcuts
- Kinect App
- `E` : Calibration
- `O` and `P`: Decrease / Increase Depth Threshold
- `W` and `S`: Zoom in/out
- `V`: Reset Calibration and deactivate OSC connection
- M5StickC-Plus
- `A`(Short-Press): Toggle OSC connection
- `A`(Long-Press): Reset Conections (Both OSC and WiFi, Bluetooth will remain)
- `B`(Long-Press): Turn ON/OFF