{"id":25401948,"url":"https://github.com/helloooideeeeea/realtimecutvadlibrary","last_synced_at":"2025-04-12T08:59:24.797Z","repository":{"id":275588714,"uuid":"926544523","full_name":"helloooideeeeea/RealTimeCutVADLibrary","owner":"helloooideeeeea","description":"A real-time Voice Activity Detection (VAD) library for iOS and macOS using Silero models powered by ONNX Runtime. Includes advanced noise suppression and audio preprocessing with WebRTC APM, supporting seamless WAV data output with header metadata.","archived":false,"fork":false,"pushed_at":"2025-04-06T08:34:39.000Z","size":4219,"stargazers_count":7,"open_issues_count":1,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-12T08:59:16.778Z","etag":null,"topics":["ios","macos","onnxruntime","silero-vad","vad","webrtc-audio-processing"],"latest_commit_sha":null,"homepage":"","language":"Swift","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/helloooideeeeea.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-03T12:55:14.000Z","updated_at":"2025-04-06T08:34:42.000Z","dependencies_parsed_at":null,"dependency_job_id":"c8043565-72ae-4dc8-bcb4-22e0fe311b8d","html_url":"https://github.com/helloooideeeeea/RealTimeCutVADLibrary","commit_stats":null,"previous_names":["helloooideeeeea/realtimecutvadlibrary"],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/helloooideeeeea%2FRealTimeCutVADLibrary","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/helloooideeeeea%2FRealTimeCutVADLibrary/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/helloooideeeeea%2FRealTimeCutVADLibrary/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/helloooideeeeea%2FRealTimeCutVADLibrary/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/helloooideeeeea","download_url":"https://codeload.github.com/helloooideeeeea/RealTimeCutVADLibrary/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248543861,"owners_count":21121838,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ios","macos","onnxruntime","silero-vad","vad","webrtc-audio-processing"],"created_at":"2025-02-16T01:20:50.903Z","updated_at":"2025-04-12T08:59:24.785Z","avatar_url":"https://github.com/helloooideeeeea.png","language":"Swift","funding_links":[],"categories":[],"sub_categories":[],"readme":"# RealTime Silero VAD iOS/macOS Library\n\nA real-time Voice Activity Detection (VAD) library for iOS and macOS using Silero models. This library helps detect human voice in real-time, allowing developers to implement efficient voice-based features in their applications.\n\n---\n\n## Features\n\n- **Real-time Voice Activity Detection (VAD)**\n- **Supports Silero Model Versions v4 and v5**\n- **Customizable audio sample rates**\n- **Outputs WAV data with automatic sample rate conversion to 16 kHz**\n- **iOS and macOS support**\n- **Supports CocoaPods and Swift Package Manager (SPM)**\n- **🆕 Real-time PCM stream callback (`voiceDidContinueWithPCMFloatData`)**\n\n---\n\n## Sample iOS App Demo\n\nCheck out the sample iOS app demonstrating real-time VAD:\n\n[Sample iOS App Demo](https://github.com/user-attachments/assets/6e4d6ae5-4d34-4114-930b-f399bcf123ba)\n\n---\n\n## Installation\n\n### Using CocoaPods\n\nAdd the following to your `Podfile` to integrate the library:\n\n```ruby\npod 'RealTimeCutVADLibrary', '~\u003e 1.0.9'\n```\n\nThen, run:\n\n```bash\npod install\n```\n\n### Using Swift Package Manager (SPM)\n\nYou can also integrate the library using Swift Package Manager. Add the following to your `Package.swift` file:\n\n```swift\n.dependencies: [\n    .package(url: \"https://github.com/helloooideeeeea/RealTimeCutVADLibrary.git\", from: \"1.0.9\")\n]\n```\n\nOr, add the URL directly through Xcode's **File \u003e Swift Packages \u003e Add Package Dependency**.\n\n---\n\n## Usage\n\nImport the library and set up VAD in your `ViewController`:\n\n```swift\nimport RealTimeCutVADLibrary\n\nclass ViewController: UIViewController {\n    var vadManager: VADWrapper?\n\n    override func viewDidLoad() {\n        super.viewDidLoad()\n\n        // Initialize VAD Manager\n        vadManager = VADWrapper()\n\n        // Set VAD delegate to receive callbacks\n        vadManager?.delegate = self\n\n        // Set Silero model version (v4 or v5). Version v5 is recommended.\n        vadManager?.setSileroModel(.v5)\n\n        // Calling setVADThreshold is optional. If not called, the recommended default values will be used.\n        // vadManager?.setThresholdWithVadStartDetectionProbability(0.7,0.7,0.5,0.95,10,57)\n\n        // Set audio sample rate (8, 16, 24, or 48 kHz)\n        vadManager?.setSamplerate(.SAMPLERATE_48)\n\n        // Retrieve audio channel data from Microphone\n        guard let channelData = buffer.floatChannelData else {\n            return\n        }\n\n        // Extract frame length from the audio buffer\n        let frameLength = UInt(buffer.frameLength)\n\n        // Select the first channel as mono audio data\n        let monoralData = channelData[0] // This is UnsafeMutablePointer\u003cFloat\u003e\n\n        // Send the audio data directly to VAD processing\n        vadManager?.processAudioData(withBuffer: monoralData, count: frameLength)\n\n        // ❌ Deprecated Usage. Do NOT use this method: Slow due to NSNumber conversion\n        var monoralDataArray: [NSNumber] = []\n        for i in 0..\u003cframeLength {\n            monoralDataArray.append(NSNumber(value: monoralData[i]))\n        }\n        // ⚠️ Deprecated method call\n        vadManager?.processAudioData(monoralDataArray)\n}\n\nextension ViewController: VADDelegate {\n    // Called when voice activity starts\n    func voiceStarted() {\n        print(\"Voice detected!\")\n    }\n\n    // Called when voice activity ends, providing WAV data at 16 kHz\n    // The Data object includes the WAV file header metadata, making it ready for playback or saving directly.\n    func voiceEnded(withWavData wavData: Data!) {\n        print(\"Voice ended. WAV data length: \\(wavData.count) bytes\")\n    }\n\n    // 🆕 Called in real-time with each chunk of PCM float audio data during voice activity\n    // This data is 32-bit float, mono, 16 kHz, and can be concatenated for file output or streamed live.\n    // ⚠️ Make sure to handle this on a background thread, as processing in real-time on the main thread may lead to dropped audio or UI freeze.\n    // For testing, save the concatenated PCM data and convert to WAV using sox:\n    // $ sox -t raw -r 16000 -c 1 -e float -b 32 test_16khz_32bit.pcm test_16khz_32bit.wav\n    func voiceDidContinue(withPCMFloat pcmFloatData: Data!) {\n        print(\"voiceDidContinue\")\n        // ex. private let pcmDataQueue = DispatchQueue(label: \"com.example.pcmDataQueue\")\n        pcmDataQueue.async { [weak self] in\n            guard let self = self, let data = pcmFloatData else { return }\n            self.collectedPCMData.append(data)\n        }\n    }\n}\n```\n\n---\n\n## Configuration Options\n\n### Sample Rates\nYou can set the audio sample rate using `setSamplerate`:\n\n- `.SAMPLERATE_8`  (8 kHz)\n- `.SAMPLERATE_16` (16 kHz)\n- `.SAMPLERATE_24` (24 kHz)\n- `.SAMPLERATE_48` (48 kHz)\n\n### Silero Model Versions\nChoose between Silero model versions:\n\n- `.v4` - Silero Model Version 4\n- `.v5` - Silero Model Version 5 (recommend)\n\n### VAD Threshold Configuration\nCustomize VAD detection sensitivity with `setThresholdWithVadStartDetectionProbability`:\n\n```swift\nvadManager?.setThresholdWithVadStartDetectionProbability(\n    0.7,  // Start detection probability threshold\n    0.7,  // End detection probability threshold\n    0.5,  // True positive ratio for voice start\n    0.95, // False positive ratio for voice end\n    10,   // Frames to confirm voice start (0.32s)\n    57    // Frames to confirm voice end (1.824s)\n)\n```\n\n### **Threshold Explanation**\n- **Start detection probability threshold (0.7)**: The VAD model must predict speech probability above this threshold to trigger voice start.\n- **End detection probability threshold (0.7)**: The VAD model must predict speech probability below this threshold to trigger voice end.\n- **True positive ratio for voice start (0.5)**: 50% of frames in a given window must be speech for voice activity to begin.\n- **False positive ratio for voice end (0.95)**: 95% of frames in a given window must be silence for voice activity to end.\n- **Start frame count (10 frames ≈ 0.32s)**: Number of frames required to confirm voice activity.\n- **End frame count (57 frames ≈ 1.824s)**: Number of frames required to confirm silence before stopping voice detection.\n\n#### **Important Notes:**\n- **Stricter VAD Detection in Silero v5**:\nBased on observations, Silero v5 appears to apply a stricter VAD detection mechanism compared to v4. \n\n- **Differences in Speech Start Detection**:\nIn Silero v4, speech is considered to have started if, within 10 frames (0.32s), **80%** of the frames exceed a VAD probability of 70%.\nIn Silero v5, this condition is relaxed, and speech is considered started if **50%** of the frames within 10 frames (0.32s) exceed a VAD probability of 70%.\nAdjusting Sensitivity for Voice Activity Detection\nIf you need to fine-tune the sensitivity of voice segmentation, use the following function to customize the thresholds:\n\n```swift\nvadManager?.setThresholdWithVadStartDetectionProbability(\u003c#T##Float#\u003e, \n    vadEndDetectionProbability: \u003c#T##Float#\u003e, \n    voiceStartVadTrueRatio: \u003c#T##Float#\u003e, \n    voiceEndVadFalseRatio: \u003c#T##Float#\u003e, \n    voiceStartFrameCount: \u003c#T##Int32#\u003e, \n    voiceEndFrameCount: \u003c#T##Int32#\u003e)\n```\nBy adjusting these parameters, you can fine-tune the strictness of voice segmentation to better suit your application needs.\n- **Silero v5 Performance**:\nThe performance of Silero model v5 may vary, and adjusting the thresholds might be necessary to achieve optimal results. There are also discussions on this topic, such as [this one](https://github.com/SYSTRAN/faster-whisper/issues/934#issuecomment-2439340290).\n\n## Algorithm Explanation\n\n### ONNX Runtime for Silero VAD\nThis library leverages **ONNX Runtime (C++)** to run the Silero VAD models efficiently. By utilizing ONNX Runtime, the library achieves high-performance inference across different platforms (iOS/macOS), ensuring fast and accurate voice activity detection.\n\n### Why Use WebRTC's Audio Processing Module (APM)?\nThis library utilizes WebRTC's APM for several key reasons:\n\n- **High-pass Filtering**: Removes low-frequency noise.\n- **Noise Suppression**: Reduces background noise for clearer voice detection.\n- **Gain Control**: Adaptive digital gain control enhances audio levels.\n- **Sample Rate Conversion**: Silero VAD requires a sample rate of 16 kHz, and APM ensures conversion from other sample rates (8, 24, or 48 kHz).\n\n### Audio Processing Workflow\n\n1. **Input Audio Configuration**: The library supports sample rates of 8 kHz, 16 kHz, 24 kHz, and 48 kHz.\n2. **Audio Preprocessing**:\n   - The audio is split into chunks based on the sample rate.\n   - APM processes these chunks with filters and gain adjustments.\n   - Audio is converted to 16 kHz for Silero VAD compatibility.\n\n3. **Voice Activity Detection**:\n   - The processed audio chunks are passed to Silero VAD.\n   - VAD outputs a probability score indicating voice activity.\n\n4. **Algorithm for Voice Detection**:\n   - **Voice Start Detection**: When the VAD probability exceeds the threshold, a pre-buffer stores audio frames to capture speech onset.\n   - **Voice End Detection**: Once silence is detected over a set number of frames, recording stops, and the audio is output as WAV data.\n\n5. **Output**:\n   - The resulting audio data is provided as WAV with a sample rate of 16 kHz.\n\n### WebRTC APM Configuration\n\nThe following configurations are applied to optimize voice detection:\n\n```cpp\nconfig.gain_controller1.enabled = true;\nconfig.gain_controller1.mode = webrtc::AudioProcessing::Config::GainController1::kAdaptiveDigital;\nconfig.gain_controller2.enabled = true;\nconfig.high_pass_filter.enabled = true;\nconfig.noise_suppression.enabled = true;\nconfig.transient_suppression.enabled = true;\nconfig.voice_detection.enabled = false;\n```\n\n---\n\n## Additional Resources\n\n- [RealTimeCutVADCXXLibrary](https://github.com/helloooideeeeea/RealTimeCutVADCXXLibrary)\n\n---\n\n## License\n\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhelloooideeeeea%2Frealtimecutvadlibrary","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhelloooideeeeea%2Frealtimecutvadlibrary","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhelloooideeeeea%2Frealtimecutvadlibrary/lists"}