Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/lnstadrum/beatmup

Beatmup: image and signal processing library
https://github.com/lnstadrum/beatmup

android gpu gpu-computing image-processing linux opengl opengl-es raspberry-pi signal-processing windows

Last synced: 3 days ago
JSON representation

Beatmup: image and signal processing library

Awesome Lists containing this project

README 

        
![android](https://github.com/lnstadrum/beatmup/workflows/android/badge.svg) ![linux](https://github.com/lnstadrum/beatmup/workflows/linux/badge.svg) ![windows](https://github.com/lnstadrum/beatmup/workflows/windows/badge.svg) ![Python packages](https://github.com/lnstadrum/beatmup/workflows/Build%20Python%20packages/badge.svg)



# Overview



Beatmup is an extensible asynchronous image and signal processing framework.



### Beatmup is for image processing, mainly.

  - It enables low-level user-defined processing tasks execution

    - in parallel and using hardware acceleration for the maximum speed,

    - asynchronously, in an application-friendly fashion.



  - It implements basic building blocks and predefined processing operations:

    - Image resampling and geometric operations

    - Basic color filtering

    - Flood fill and contour extraction

    - Realtime rendering



### Beatmup runs quite everywhere.

  - Used in **linux**, **Windows** and **Android**

  - Written in **C++11**, it has **Java** and **Python** bindings

  - **Prebuilt Python packages** are available for x64 desktop Windows and Ubuntu-based linux distributions

  - Few to no dependencies: Beatmup is extremely easy to build for a different platform

    - git, CMake and a C++ compiler is all you need

    - Beatmup compiles with gcc, clang and msvc



### Beatmup speeds up processing by using GPU regardless its vendor, price and release year.

  - It runs on

    - desktop GPUs from mainstream vendors,

    - integrated desktop GPUs,

    - Android GPUs,

    - Raspberry Pi GPU (all models including Pi Zero W),

    - NVIDIA Jetson Nano,

    - ...

  - It uses the GPU for

    - *x2*, neural network-based image upscaling technique,

    - running inference of small user-defined neural nets,

    - applying user-defined GLSL shaders to process images,

    - realtime camera image processing in Android,

    - scene rendering and predefined image processing tasks.



# Highlights



## *x2*: a neural net-based image upscaler



Beatmup is arguably the easiest way to get a fast hardware-accelerated neural network-based image superresolution running on any decent GPU. It does not provide a state-of-the-art PSNR score, but can run at a hundred of images per second on a desktop machine.



More details on the neural network design and the inference implementation: [Inferring a super-resolution neural network on Raspberry Pi GPU](https://medium.com/analytics-vidhya/inferring-a-super-resolution-neural-network-on-raspberry-pi-gpu-89b5456d21ef)



Update in 2.1 release: the fine-tuned version of the model (trained with multiple degradation kernels) achieves now 32.64 dB on DIV2K validation set compared to the initial score of 32.57 dB.



# Available resources



## Docs

 * C++ documentation is available [here](https://lnstadrum.github.io/beatmup/cpp/html). This is the most extensive documentation, it also contains additional explanations that do not relate to a specific programming language:

   - [Programming model](https://lnstadrum.github.io/beatmup/cpp/html/ProgrammingModel.html)

   - [NNets module overview](https://lnstadrum.github.io/beatmup/cpp/html/NNetsModuleOverview.html)

 * Java package API documentation is [here](https://lnstadrum.github.io/beatmup/java/html).

 * Python documentation is [here](https://lnstadrum.github.io/beatmup/python/python.html).



## C++

There is a number of test apps showcasing the use of Beatmup with additional explanations in [apps](apps) folder. The code is in C++, but can be universally helpful. Few examples:

 * [apps/shaderer](apps/shaderer/app.cpp) processes images with a custom GLSL fragment shader read from the standard input.

 * [apps/x2](apps/x2/app.cpp) is a test app for the *x2* neural net.

 * [apps/classify](apps/classify/app.cpp) is a dog image classifier, a variant of ResNeXt trained on a subset of ImageNet containing 120 classes of dogs and cats images. The inference is implemented with OpenGL shaders. Top-1 validation accuracy achieved on Raspberry Pi Zero W is 72.08%. Classifying a 385*385 image takes ~595 ms there.



## Python

[python/examples](python/examples) folder contains detailed examples of scripts using Beatmup in Python.

 * [cifar100.py](python/examples/cifar100.py) explains how to train a neural net with TensorFlow/Keras, convert it into a Beatmup-compliant model and run its inference on CIFAR100 test set.

 * [x2_superresolution.py](python/examples/x2_superresolution.py) and [x2_superresolution_video.py](python/examples/x2_superresolution_video.py) showcase applying x2 upsampling neural net to an image and a video.



Unit tests in Python are available in [python/tests/test.py](python/tests/test.py) and [python/tests/test_nnets.py](python/tests/test_nnets.py).



## Java (Android)

An Android test app is available in [android/app](android/app) folder. It consists of independent examples showing how to use Beatmup in Android.

 * [BasicCameraUse.java](android/app/src/main/java/xyz/beatmup/androidapp/samples/BasicCameraUse.java) shows how to get the camera preview, apply a simple shader and display the result.

 * [UpsamplingConvnetOnCamera.java](android/app/src/main/java/xyz/beatmup/androidapp/samples/UpsamplingConvnetOnCamera.java) implements the application of x2 upsampling neural net to the camera preview.

 * [DogClassifier.java](android/app/src/main/java/xyz/beatmup/androidapp/samples/DogClassifier.java) implements the dog classifier taking the camera preview image as input.



# Quick start: compiling Beatmup



To get things running, make sure you have *git* and *CMake* installed, then pick one of the following copy-paste recipes.



## Linux



Building **X2** app upscaling an image using a neural net inferred with OpenGL:



    git clone https://github.com/lnstadrum/beatmup.git

    cd beatmup

    git submodule update --init --recursive

    mkdir -p build && cd build

    cmake -DUSE_GLX=ON ..

    make X2



 * Try to use `-DUSE_OPENGL=ON` instead `-DUSE_GLX=ON` starting from a clean build folder if you run into trouble.

 * On **Raspberry Pi prior to series 4** use the EGL backend with following CMake command:



    `cmake -DUSE_EGL=ON -DUSE_BRCM_LIBS=ON -DGLES_VERSION=20 ..`



More details on Raspberry Pi setup [here](https://github.com/lnstadrum/beatmup/wiki/Compiling-Beatmup-on-Raspberry-Pi-prior-to-4).



 * On **Raspberry Pi 4** rather use the following CMake command:



    `cmake -DUSE_EGL_DRM=ON -DGLES_VERSION=20 ..`



More details on Raspberry Pi 4 setup [here](https://github.com/lnstadrum/beatmup/wiki/Compiling-Beatmup-on-Raspberry-Pi-4).



Once the app is built, you can feed it with an image of your choice and get the upscaled result as follows:



    ./X2 .bmp x2-result.bmp



## Windows



To build the **X2** app in Windows you may install *Visual Studio* and generate the VS solution as follows:



    git clone https://github.com/lnstadrum/beatmup.git

    cd beatmup

    git submodule update --init --recursive

    mkdir build && cd build

    cmake -DUSE_OPENGL=ON ..



Then open the *.sln* solution file in Visual Studio and build the executable.



## Android



There is Android project containing a library and a test app modules (*lib* and *app*). To build them install Android Studio and Android NDK bundle, clone the repository and open the project situated in *android* folder in Android studio.



You can build an apk in a docker container as well. Having docker installed, you can run the following script to build *beatmupApp.apk* installation package which you may then copy to your Android phone and open it there to install the test app:



    git clone https://github.com/lnstadrum/beatmup.git

    cd beatmup

    git submodule update --init --recursive

    docker build -f dockerfiles/android.Dockerfile -t beatmup-android .

    docker run -v $(pwd):/opt/android-build beatmup-android



# Quicker start with Python



## Windows



Prebuilt python packages are available in 64-bit Windows for Python versions from **3.6** to **3.11**:



    python -m pip install --upgrade pip

    python -m pip install beatmup



## Ubuntu-based linux



Prebuilt packages for x64 desktop Ubuntu-based linux distributions are available for downloading on the [releases page](https://github.com/lnstadrum/beatmup/releases/).



## Compiling Python package



If there is no prebuilt package available for your platform/OS, you can easily build a Python wheel installable with `pip` on your own.



First, get the code and try to compile an app as explained above. Then in the repository root folder run



    cd build

    make beatmup

    cd ../python

    python3 setup.py bdist_wheel clean



The installation package is now available in `python/dist` folder. You can install it and make sure it works:



    python3 -m pip install --no-index --find-links=dist beatmup

    python3 -c "import beatmup; beatmup.say_hi()"



# Licence and contributions



The project is licensed under the GNU GPL v3 licence.



The contributions are governed by the [Contribution Licence Agreement (CLA)](CLA.md). In short, under this Agreement you remain the copyright owner of your contribution, but you allow us to use and distribute your contribution under the terms of a possibly different licence.



If you want to contribute:

 * make sure you own the copyright of your contribution,

 * raise a pull request,

 * send a signed CLA on [email protected].



Otherwise:

 * raise an issue if you have a question or a feature request,

 * contact us by email.