{"id":25067026,"url":"https://github.com/shoyamanishi/androidmfcc","last_synced_at":"2025-04-14T19:33:26.443Z","repository":{"id":187800776,"uuid":"228101687","full_name":"ShoYamanishi/AndroidMFCC","owner":"ShoYamanishi","description":"26-Point MFCC \u0026 512-Point FFT Generator \u0026 Visualizer in Java, C++, and NEON intrinsics","archived":false,"fork":false,"pushed_at":"2019-12-26T16:57:19.000Z","size":6312,"stargazers_count":15,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-28T07:51:12.234Z","etag":null,"topics":["android","asr","audio","cpp","fft","java","mfcc","neon-simd-intrinsics","spectrum","visualizer"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ShoYamanishi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-12-14T23:10:29.000Z","updated_at":"2025-01-23T00:09:51.000Z","dependencies_parsed_at":null,"dependency_job_id":"90f49939-bf38-4e22-95d4-25b813693df6","html_url":"https://github.com/ShoYamanishi/AndroidMFCC","commit_stats":null,"previous_names":["shoyamanishi/androidmfcc"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShoYamanishi%2FAndroidMFCC","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShoYamanishi%2FAndroidMFCC/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShoYamanishi%2FAndroidMFCC/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShoYamanishi%2FAndroidMFCC/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ShoYamanishi","download_url":"https://codeload.github.com/ShoYamanishi/AndroidMFCC/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248945916,"owners_count":21187409,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["android","asr","audio","cpp","fft","java","mfcc","neon-simd-intrinsics","spectrum","visualizer"],"created_at":"2025-02-06T20:29:05.788Z","updated_at":"2025-04-14T19:33:26.418Z","avatar_url":"https://github.com/ShoYamanishi.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Real-time 26-Point MFCC \u0026 512-Point Radix-2 FFT Generator \u0026 Visualizer on Android in Java, C++ and NEON Intrinsics\n\n\u003ca href=\"doc/440Hz.png\"\u003e \u003cimg src=\"doc/440Hz_thumb.png\" height=\"100\"\u003e\u003c/a\u003e\n\u003ca href=\"doc/AndroidAudio.png\"\u003e \u003cimg src=\"doc/AndroidAudio_thumb.png\" height=\"100\"\u003e\u003c/a\u003e\n\u003ca href=\"doc/BlahBlahBlah.png\"\u003e \u003cimg src=\"doc/BlahBlahBlah_thumb.png\" height=\"100\"\u003e\u003c/a\u003e\n\u003ca href=\"doc/WirSindAlleProgrammiertTimBendzko.png\"\u003e \u003cimg src=\"doc/WirSindAlleProgrammiertTimBendzko_thumb.png\" height=\"100\"\u003e\u003c/a\u003e\n\u003ca href=\"doc/GrandPianoSingleTonesC3andC4.png\"\u003e \u003cimg src=\"doc/GrandPianoSingleTonesC3andC4_thumb.png\" height=\"100\"\u003e\u003c/a\u003e\n\u003ca href=\"doc/GrandPianoCMaj9.png\"\u003e \u003cimg src=\"doc/GrandPianoCMaj9_thumb.png\" height=\"100\"\u003e\u003c/a\u003e\n\u003ca href=\"doc/SynthLeadSpainBEGGbDBE.png\"\u003e \u003cimg src=\"doc/SynthLeadSpainBEGGbDBE_thumb.png\" height=\"100\"\u003e\u003c/a\u003e\n\u003ca href=\"doc/CoryWongSlowlyJamming.png\"\u003e \u003cimg src=\"doc/CoryWongSlowlyJamming_thumb.png\" height=\"100\"\u003e\u003c/a\u003e\n\u003ca href=\"doc/CoryHenryNickSemrad.png\"\u003e \u003cimg src=\"doc/CoryHenryNickSemrad_thumb.png\" height=\"100\"\u003e\u003c/a\u003e\n\u003ca href=\"doc/JacoPortraitOfTracy.png\"\u003e \u003cimg src=\"doc/JacoPortraitOfTracy_thumb.png\" height=\"100\"\u003e\u003c/a\u003e\n\u003ca href=\"doc/Naturally7.png\"\u003e \u003cimg src=\"doc/Naturally7_thumb.png\" height=\"100\"\u003e\u003c/a\u003e\n\u003ca href=\"doc/TheoKatzmanHalfTheWay.png\"\u003e \u003cimg src=\"doc/TheoKatzmanHalfTheWay_thumb.png\" height=\"100\"\u003e\u003c/a\u003e\n\u003ca href=\"doc/ToriKelly.png\"\u003e \u003cimg src=\"doc/ToriKelly_thumb.png\" height=\"100\"\u003e\u003c/a\u003e\n\n# Performance Comparison of 3 Implementations\nThe following table and figure shows the average time in seconds observed for MFCC generation per 400-sample frame.\n\n\u003ca href=\"doc/performance.png\"\u003e \u003cimg src=\"doc/performance.png\"\u003e\u003c/a\u003e\n\n* Java : All written in Java code\n \n* C++ : All written in Native C++ with JNI interface.\n\n* C++ \u0026 NEON/SSE : Written in C++ with 4-lane ArmV7 NEON/SSE SIMD intrinsics for Hamming, FFT, and DCT.\n\n## Conditions\n\n* Galaxy S9 -O0 : Galaxy S9 with 8-Core Snapdragon 845 with C++ compiler optimization level 0\n\n* Galaxy S9 -O3 : Galaxy S9 with 8-Core Snapdragon 845 with C++ compiler optimization level 3\n\n* Emulator (X86) -O0 : Android Emulator on a Host PC with 4-Core emulation with C++ compiler optimization level 0 \n\n* Emulator (X86) -O3 : Android Emulator on a Host PC with 4-Core emulation with C++ compiler optimization level 3\n\nThe numbers are all in seconds.\n\n| Tables            | Java          |      C++      | C++ NEON SSE  |\n| ----------------- |---------------|---------------|---------------|\n| Galaxy S9 -O0     |   0.0016      |    0.0015     |    0.0011     |\n| Galaxy S9 -O3     |   0.0016      |    0.00050    |    0.00034    |\n| Emulator(X86) -O0 |   0.00020     |    0.00012    |    0.00012    |\n| Emulator(X86) -O3 |   0.00020     |    0.000049   |    0.000047   |\n\n## Remarks\nThe Java implementation works surprisingly well. On the test target, it takes 2[ms] to process one\nframe. Assuming one of the core is availalble all the time, the realtime factor is greater than 5.\n\nThe author is not capable of further tuning with assembler beyond the intrinsics, \nbut further performance improvement by CPU-specific assembler-level optimization may be possible.\n\n# Install\n\n1. Download the contents and open with AndroidStudio.\n\n2. Copy [`NEON_2_SSE.h`](https://github.com/intel/ARM_NEON_2_x86_SSE/blob/master/NEON_2_SSE.h) \nfrom `https://github.com/intel/ARM_NEON_2_x86_SSE` into [app/src/main/cpp/](app/src/main/cpp/).\n\n3. Build.\n\nIt was tested with the following environment.\n\n* Android Studio 3.5.1\n\n* Min SDK Version 21\n\n* Virtual device API29, Android 10.0, x86\n\nIf it does not work, check the App permission for Mic on the device.\nAlso, try chanding RECORDING_RATE in AudioReceiver.\n\n# Description\nOriginally motivated to measure the real-time performance of audio signal processing on Android devices.\nThis is a study implementation as a bench-mark for a Native C++ implematation with 4-lane ARM NEON SIMD intrinsics.\n\n* Audio input 16KHz monaural linear PCM taken from AudioRecorder\n\n* Frame size 400 samples (25[ms]), Frame shift 160 samples (10[ms])\n\n* Pre-emphasis (tap 0.96)\n\n* Hamming window per frame\n\n* 512-Point Radix-2 Cooley-Tukey recursive FFT\n\n* Mel Filterbank, 26 banks, top 8KHz, bottom 300Hz, with flooring at 1.0\n\n* DCT into 26-point MFCC [quefrency] with DC.\n\n\n# Spectrum Visualization for Fun\nThe upper part is the 26-point MFCC. The lower part is the 256-point spectrum taken from 512-point FFT.\nPlese click the thumbnails to enlarge.\n\n* \u003ca href=\"doc/440Hz.png\"\u003e \u003cimg src=\"doc/440Hz_thumb.png\" height=\"100\"\u003e\u003c/a\u003e\n[ 440Hz Sine ](doc/440Hz.png) : 440 Hz Sine wave with some background noise.\n\n\n* \u003ca href=\"doc/AndroidAudio.png\"\u003e \u003cimg src=\"doc/AndroidAudio_thumb.png\" height=\"100\"\u003e\u003c/a\u003e\n[ 'Android audio' ](doc/AndroidAudio.png) : Me saying \"Android audio\".\n\n\n* \u003ca href=\"doc/BlahBlahBlah.png\"\u003e \u003cimg src=\"doc/BlahBlahBlah_thumb.png\" height=\"100\"\u003e\u003c/a\u003e\n[ 'Blah Blah Blah' ](doc/BlahBlahBlah.png ) : Me saying \"Blah Blah Blah...\".\n\n\n* \u003ca href=\"doc/WirSindAlleProgrammiertTimBendzko.png\"\u003e \u003cimg src=\"doc/WirSindAlleProgrammiertTimBendzko_thumb.png\" height=\"100\"\u003e\u003c/a\u003e\n[ 'Wir sind alle programmiert.' ](doc/WirSindAlleProgrammiertTimBendzko.png) : Me saying \"Wir sind alle programmiert.\"\n\n\n* \u003ca href=\"doc/GrandPianoSingleTonesC3andC4.png\"\u003e \u003cimg src=\"doc/GrandPianoSingleTonesC3andC4_thumb.png\" height=\"100\"\u003e\u003c/a\u003e\n[ Piano Single Tones ](doc/GrandPianoSingleTonesC3andC4.png) : Grand Piano Single Tones C3 and then C4.\n\n\n* \u003ca href=\"doc/GrandPianoCMaj9.png\"\u003e \u003cimg src=\"doc/GrandPianoCMaj9_thumb.png\" height=\"100\"\u003e\u003c/a\u003e\n[ Piano Chord ](doc/GrandPianoCMaj9.png) : Grand Piano Chord C maj9\n\n\n* \u003ca href=\"doc/SynthLeadSpainBEGGbDBE.png\"\u003e \u003cimg src=\"doc/SynthLeadSpainBEGGbDBE_thumb.png\" height=\"100\"\u003e\u003c/a\u003e\n[ Synth Lead Melody ](doc/SynthLeadSpainBEGGbDBE.png) : Sythe lead tracing a line of Spain by Chick Corea (B-\u003eE-\u003eG-\u003eGb-\u003eD-\u003eB-\u003eE).\n\n\n* \u003ca href=\"doc/CoryWongSlowlyJamming.png\"\u003e \u003cimg src=\"doc/CoryWongSlowlyJamming_thumb.png\" height=\"100\"\u003e\u003c/a\u003e\n[ Cory Wong Jamming ](doc/CoryWongSlowlyJamming.png) : Cory Wong slowly jamming with his guitar. (https://youtu.be/i14pnaRzflU?t=208)\n\n\n* \u003ca href=\"doc/CoryHenryNickSemrad.png\"\u003e \u003cimg src=\"doc/CoryHenryNickSemrad_thumb.png\" height=\"100\"\u003e\u003c/a\u003e\n[ Cory Henry \u0026 Nick Semrad @NAMM ](doc/CoryHenryNickSemrad.png) : Cory Henry and Nick Semrad playing Gospel at NAMM show. (https://www.youtube.com/watch?v=8TwhLplrFNo)\n\n\n* \u003ca href=\"doc/JacoPortraitOfTracy.png\"\u003e \u003cimg src=\"doc/JacoPortraitOfTracy_thumb.png\" height=\"100\"\u003e\u003c/a\u003e\n[ Jaco Portrait of Tracy ](doc/JacoPortraitOfTracy.png) : Tried to capture the incredible harmonics of Portrait of Tracy, but not much seen. (https://www.youtube.com/watch?v=nsZ_1mPOuyk)\n\n\n* \u003ca href=\"doc/Naturally7.png\"\u003e \u003cimg src=\"doc/Naturally7_thumb.png\" height=\"100\"\u003e\u003c/a\u003e\n[ Naturally 7 Human Voices ](doc/Naturally7.png) : Successfully captured the incredible grooves/vibrato of their voices. (https://www.youtube.com/watch?v=AF-KagTq7qY)\n\n\n* \u003ca href=\"doc/TheoKatzmanHalfTheWay.png\"\u003e \u003cimg src=\"doc/TheoKatzmanHalfTheWay_thumb.png\" height=\"100\"\u003e\u003c/a\u003e\n[ Vulfpeck ](doc/TheoKatzmanHalfTheWay.png) : Theo Katzman's singing voice. (https://www.youtube.com/watch?v=6HUkbf44iAA)\n\n\n* \u003ca href=\"doc/ToriKelly.png\"\u003e \u003cimg src=\"doc/ToriKelly_thumb.png\" height=\"100\"\u003e\u003c/a\u003e\n[ Tori Kelly ](doc/ToriKelly.png) : Tori Kelly's incredible voice visualized. (https://www.youtube.com/watch?v=Jv8IqJm6q7w)\n\n\n\n# Code\n\n## Signal Processing in Java\n\n* [HammingWindowJava](app/src/main/java/com/example/android_mfcc/HammingWindowJava.java): Pre-emphasis \u0026 Hamming for a 400-sample frame\n* [FFT512Java](app/src/main/java/com/example/android_mfcc/FFT512Java.java): 512-point Radix-2 Cooley-Tukey recursive FFT with pre-calculated Twiddle table\n* [MelFilterBanksJava](app/src/main/java/com/example/android_mfcc/MelFilterBanksJava.java): Generates MelFilterBanks log energy coefficients with Bins and precalculated table.\n* [DCTJava](app/src/main/java/com/example/android_mfcc/DCTJava.java): 26-point DCT with a pre-calculated table.\n\n## Signal Processing in C++\nAll the parts related to NEON intrinsics are enclosed by `#ifdef HAVE_NEON ... #endif`.\n\n* [mfcc_impl01.cpp](app/src/main/cpp/mfcc_impl01.cpp): This file contains the following classes and some JNI glue code.\n \n  * `class HamminwWindow` : Pre-emphasis \u0026 Hamming for a 400-sample frame. It utiizes NEON for the float mult loop.\n\n  * `class FFT512` : 512-point Radix-2 Cooley-Tukey recursive FFT with pre-calculated Twiddle table. It utlizes NEON for the even-odd splitting and the butterfly calculations.\n\n  * `class MelFilterBanks` : Generates MelFilterBanks log energy coefficients with Bins and precalculated table. It does not utilize NEON.\n\n  * `class DCT` : 26-point DCT with a pre-calculated table. It utilizes NEON in the inner-loop of mult-add.\n\n\nVisualization\n\n* [ScrollingHeatMapView](app/src/main/java/com/example/android_mfcc/ScrollingHeatMapView.java): ImageView for real-time scrolling spectrum visuzliation.\n\n\nOthers\n\n* [AudioReceiver](app/src/main/java/com/example/android_mfcc/AudioReceiver.java): receives audio with android.media.AudioRecorder in chunks in realtime.\n\n* [AudioChunkAggregator](app/src/main/java/com/example/android_mfcc/AudioChunkAggregator.java): arranges the audio data into 400[ms] frames with 10[ms] frame shift.\n\n\n# Dependencies\n\n* [cpu_features](https://github.com/google/cpu_features) : linked to the binary to obtain processor info for convenience. Apache License.\n\n* [NEON_2_SSE](https://github.com/intel/ARM_NEON_2_x86_SSE) : used to to compile NEON intrinsics for X86 Android Emulator. It converts ARM NEON intrinsics to equivalents in Intel SSE. Intel's own license but basically distributable retaining the original copyright notice.\n\n\n# References\n\n* J. S. Bridle and M. D. Brown (1974), \"An Experimental Automatic Word-Recognition System\", JSRU Report No. 1003, Joint Speech Research Unit, Ruislip, England.\n\n* \"Digital signal processing\" by Proakis, Manolakis 4th edition Chap 8: Efficient Computation of the DFT: Fast Fourier Transform\n\n* [Mel Frequency Cepstral Coefficient (MFCC) tutorial](http://practicalcryptography.com/miscellaneous/machine-learning/guide-mel-frequency-cepstral-coefficients-mfccs) : Nice tutorial.\n\n* [libmfcc](https://github.com/rohithkd/libmfcc) : C-implementation.\n\n* [MFCC.cpp](https://github.com/MTG/miredu/blob/master/src/MFCC.cpp) : another nice C-implemetation\n\n\n# Contact\n\nFor technical and commercial inquiries, please contact: Shoichiro Yamanishi\n\nyamanishi72@gmail.com","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshoyamanishi%2Fandroidmfcc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshoyamanishi%2Fandroidmfcc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshoyamanishi%2Fandroidmfcc/lists"}