https://github.com/charliegerard/gestures-ml-js

[WIP] - Gesture recognition using hardware and Tensorflow.js
https://github.com/charliegerard/gestures-ml-js

arduino creative-coding front-end-development javascript machine-learning tensorflow-js

Last synced: 2 months ago
JSON representation

[WIP] - Gesture recognition using hardware and Tensorflow.js

• Host: GitHub
• URL: https://github.com/charliegerard/gestures-ml-js
• Owner: charliegerard
• License: gpl-3.0
• Created: 2019-05-14T08:38:13.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2019-09-01T12:29:15.000Z (over 5 years ago)
• Last Synced: 2024-08-05T09:14:47.847Z (6 months ago)
• Topics: arduino, creative-coding, front-end-development, javascript, machine-learning, tensorflow-js
• Language: JavaScript
• Homepage: https://bit.ly/sf-ml
• Size: 19.4 MB
• Stars: 77
• Watchers: 6
• Forks: 11
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • License: LICENSE.md

Awesome Lists containing this project

README

        
# Gesture recognition using hardware and Tensorflow.js

**This project is very experimental and in active development.**

Experiments building a gesture recognition system using an Arduino, a Daydream controller and a mobile phone, with Tensorflow.js

*Inspired by a [similar project](https://blog.mgechev.com/2018/10/20/transfer-learning-tensorflow-js-data-augmentation-mobile-net/) by [Minko Gechev](https://twitter.com/mgechev) using the webcam*.

## Demo:

![Demo](demo.gif)

Each project has 2 demos: one to play a game of street fighter, and one to predict magic wand movements.

Sprites used in the Street Fighter demo come from this [Codepen](https://codepen.io/jkneb/pen/smtHA) and [this repo](https://github.com/jkneb/street-fighter-css)

## How it works:

Using an accelerometer/gyroscope (MPU6050 for the Arduino, and built-in sensors for the Daydream and phone), we can record data streamed while performing a gesture. By repeating and recording gestures multiple times, we can feed all this data to a machine learning algorithm to find patterns in the data. Once a model is created, we can use it to predict new live data and classify it to use as input for an interface or device.

## Arduino project:

See [arduino-mkr1000](arduino-mkr1000/) folder.

## Daydream project:

See [daydream](daydream/) folder.

## Phone project:

See [phone](phone/) folder.

---

## Blog post

More details on in [this blog post](https://dev.to/devdevcharlie/play-street-fighter-with-body-movements-using-arduino-and-tensorflow-js-4kbi).

---