https://github.com/emilamaj/syntomid
Extract notes from Synthesia piano videos on youtube, and export songs to MIDI files.
https://github.com/emilamaj/syntomid
converter midi opencv piano synthesia youtube
Last synced: 26 days ago
JSON representation
Extract notes from Synthesia piano videos on youtube, and export songs to MIDI files.
- Host: GitHub
- URL: https://github.com/emilamaj/syntomid
- Owner: emilamaj
- Created: 2023-03-29T10:59:55.000Z (about 2 years ago)
- Default Branch: master
- Last Pushed: 2023-05-16T13:13:27.000Z (almost 2 years ago)
- Last Synced: 2025-03-29T09:41:47.047Z (about 2 months ago)
- Topics: converter, midi, opencv, piano, synthesia, youtube
- Language: Python
- Homepage:
- Size: 565 KB
- Stars: 4
- Watchers: 2
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# SynToMid
This tool helps you convert synthesia-like piano videos of Youtube to a MIDI (.mid) file that can be read by most music tools. It makes extensive use of OpenCV to process the video frames, and to extract the notes played.
Currently implemented:
- YoutubeStitch.py: Stitch together the frames of the Youtube video into a tall png image of the keys pressed.
- WaterfallProcess.py: Convert the stitched image to a list of list of rectangles representing the keys pressed. **Very buggy, not yet usable for note extraction.**
Todo:
- [ ] Better processing of the notes, especially close ones. (Try again pseudo-gradient descent but with cost function in absolute pixels, like L=[10*black_pixels - 1*white_pixels], where the pixels proposed are the ones enclosed by a rectangle. Beware of the fact that close rectangles will mask each otehr, L also needs an overlap term.)
- [ ] Implement ReadNotes.py, which converts the processed rectangles to a list of notes.
- [ ] Fix YoutubeStitch.py so that the note timings are respected. (Find average scroll rate, and blindly stitch ? Scroll rate might not be constant...)
## YoutubeStitch.py
This is the tool that stitches together the frames of the Youtube video into a tall png image of the keys pressed.
This is the result of the script for the first 30 seconds of Chopin - Ballade No. 1 played by Rousseau (https://www.youtube.com/watch?v=Zj_psrTUW_w)
This project will consist in 3 parts:
- YoutubeStitch.py
- WaterfallProcess.py
- ReadNotes.py
# YoutubeStitch.py
Here is the usage for YoutubeStitch.py:
```bash
python YoutubeStitch.py
```
- url (string): Url of the Youtube video
- height (string): Percent of the height of the video to process (starting from the top, to ignore the hands of the player)
- interval (float): Interval in seconds between the frames, to allow the script to run faster
- start (float): Start position of the video, in seconds (to ignore intros)
- stop (float): Stop position of the video, in seconds
The process can run for several minutes.
You can speedup the script by lowering the height processed. It also removes a lot of the visual artefacts that might remain at the end.
Reducing the height however completely messes up the duration of the long notes and the long silences.
It produces a file called `output.png`, which is the stitched image of the video.
# WaterfallProcess.py
This is the tool that converts the stitched image to a list of list of rectangles representing the keys pressed.
Usage:
```bash
python WaterfallProcess.py
```
Here is the current state of the image processing done with OpenCV:
