https://github.com/arthurfdlr/speech-transcription-and-diarization
💬 Combines Whisper and PyAnnote for Automatic Speaker Diarized Speech Recognition
https://github.com/arthurfdlr/speech-transcription-and-diarization
Last synced: 16 days ago
JSON representation
💬 Combines Whisper and PyAnnote for Automatic Speaker Diarized Speech Recognition
- Host: GitHub
- URL: https://github.com/arthurfdlr/speech-transcription-and-diarization
- Owner: ArthurFDLR
- Created: 2024-05-09T03:54:58.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2024-06-19T17:49:49.000Z (about 2 years ago)
- Last Synced: 2025-10-26T02:41:54.767Z (9 months ago)
- Language: Python
- Size: 29.3 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.ipynb
Awesome Lists containing this project
README
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Speech Transcription and Diarization\n",
"\n",
"💬 YoutubeCC - Parse JSON3 Youtube Closed Captions"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/arthur/Documents/02.workspace/02.active/speech-transcription-and-diarization/venv/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
" from .autonotebook import tqdm as notebook_tqdm\n"
]
}
],
"source": [
"import dotenv, json\n",
"from pathlib import Path\n",
"from stad import WhisperPipeline, DiarizationPipeline, assign_speaker_to_transcript\n",
"\n",
"# Load HF_TOKEN from .env file\n",
"dotenv.load_dotenv()\n",
"\n",
"AUDIO_PATH = Path.cwd() / \".data\" / \"audio.wav\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Automatic Speech Recognition with Hugging Face Transformers implementation of Whisper"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n",
"WhisperModel is using WhisperSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True` or `layer_head_mask` not None. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation=\"eager\"` when loading the model.\n",
"Whisper did not predict an ending timestamp, which can happen if audio is cut off in the middle of a word. Also make sure WhisperTimeStampLogitsProcessor was used during generation.\n"
]
},
{
"data": {
"text/html": [
"
\n",
"\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"\n",
"\n",
" \n",
" \n",
" \n",
" text\n",
" start\n",
" end\n",
" \n",
" \n",
" \n",
" \n",
" 0\n",
" What's\n",
" 0 days 00:00:00.040000\n",
" 0 days 00:00:00.300000\n",
" \n",
" \n",
" 1\n",
" up\n",
" 0 days 00:00:00.300000\n",
" 0 days 00:00:00.440000\n",
" \n",
" \n",
" 2\n",
" everybody\n",
" 0 days 00:00:00.440000\n",
" 0 days 00:00:00.700000\n",
" \n",
" \n",
" 3\n",
" welcome\n",
" 0 days 00:00:00.700000\n",
" 0 days 00:00:01\n",
" \n",
" \n",
" 4\n",
" to\n",
" 0 days 00:00:01\n",
" 0 days 00:00:01.220000\n",
" \n",
" \n",
" ...\n",
" ...\n",
" ...\n",
" ...\n",
" \n",
" \n",
" 25428\n",
" We'll\n",
" 0 days 02:03:16.760000\n",
" 0 days 02:03:17.040000\n",
" \n",
" \n",
" 25429\n",
" see\n",
" 0 days 02:03:17.040000\n",
" 0 days 02:03:17.160000\n",
" \n",
" \n",
" 25430\n",
" you\n",
" 0 days 02:03:17.160000\n",
" 0 days 02:03:17.300000\n",
" \n",
" \n",
" 25431\n",
" there.\n",
" 0 days 02:03:17.300000\n",
" 0 days 02:03:17.500000\n",
" \n",
" \n",
" 25432\n",
" Peace.\n",
" 0 days 02:03:17.500000\n",
" 0 days 02:03:18.320000\n",
" \n",
" \n",
"\n",
"25433 rows × 3 columns
\n",
""
],
"text/plain": [
" text start end\n",
"0 What's 0 days 00:00:00.040000 0 days 00:00:00.300000\n",
"1 up 0 days 00:00:00.300000 0 days 00:00:00.440000\n",
"2 everybody 0 days 00:00:00.440000 0 days 00:00:00.700000\n",
"3 welcome 0 days 00:00:00.700000 0 days 00:00:01\n",
"4 to 0 days 00:00:01 0 days 00:00:01.220000\n",
"... ... ... ...\n",
"25428 We'll 0 days 02:03:16.760000 0 days 02:03:17.040000\n",
"25429 see 0 days 02:03:17.040000 0 days 02:03:17.160000\n",
"25430 you 0 days 02:03:17.160000 0 days 02:03:17.300000\n",
"25431 there. 0 days 02:03:17.300000 0 days 02:03:17.500000\n",
"25432 Peace. 0 days 02:03:17.500000 0 days 02:03:18.320000\n",
"\n",
"[25433 rows x 3 columns]"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"with WhisperPipeline.create(\n",
" return_timestamps=\"word\",\n",
" attn_implementation=\"sdpa\",\n",
" batch_size=16,\n",
") as whisper:\n",
" transcript_df = whisper(\n",
" audio_path=AUDIO_PATH,\n",
" language=\"english\",\n",
" )\n",
"\n",
"transcript_df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Speaker Diarization with PyAnnote [`pyannote/speaker-diarization-3.1`](https://huggingface.co/pyannote/speaker-diarization-3.1)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"torchvision is not available - cannot save figures\n"
]
},
{
"data": {
"text/html": [
"\n",
"\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"\n",
"\n",
" \n",
" \n",
" \n",
" label\n",
" speaker\n",
" start\n",
" end\n",
" \n",
" \n",
" \n",
" \n",
" 0\n",
" A\n",
" SPEAKER_03\n",
" 0 days 00:00:00.132218750\n",
" 0 days 00:00:46.319093750\n",
" \n",
" \n",
" 1\n",
" B\n",
" SPEAKER_02\n",
" 0 days 00:00:24.870968750\n",
" 0 days 00:00:25.124093750\n",
" \n",
" \n",
" 2\n",
" C\n",
" SPEAKER_03\n",
" 0 days 00:00:47.567843750\n",
" 0 days 00:00:58.705343750\n",
" \n",
" \n",
" 3\n",
" D\n",
" SPEAKER_04\n",
" 0 days 00:00:56.376593750\n",
" 0 days 00:00:57.540968750\n",
" \n",
" \n",
" 4\n",
" E\n",
" SPEAKER_03\n",
" 0 days 00:00:59.430968750\n",
" 0 days 00:01:04.206593750\n",
" \n",
" \n",
" ...\n",
" ...\n",
" ...\n",
" ...\n",
" ...\n",
" \n",
" \n",
" 3208\n",
" DSK\n",
" SPEAKER_03\n",
" 0 days 02:03:08.597843750\n",
" 0 days 02:03:09.272843750\n",
" \n",
" \n",
" 3209\n",
" DSL\n",
" SPEAKER_03\n",
" 0 days 02:03:10.049093750\n",
" 0 days 02:03:11.669093750\n",
" \n",
" \n",
" 3210\n",
" DSM\n",
" SPEAKER_00\n",
" 0 days 02:03:11.669093750\n",
" 0 days 02:03:11.702843750\n",
" \n",
" \n",
" 3211\n",
" DSN\n",
" SPEAKER_00\n",
" 0 days 02:03:15.145343750\n",
" 0 days 02:03:17.541593750\n",
" \n",
" \n",
" 3212\n",
" DSO\n",
" SPEAKER_00\n",
" 0 days 02:03:17.963468750\n",
" 0 days 02:03:18.300968750\n",
" \n",
" \n",
"\n",
"3213 rows × 4 columns
\n",
""
],
"text/plain": [
" label speaker start end\n",
"0 A SPEAKER_03 0 days 00:00:00.132218750 0 days 00:00:46.319093750\n",
"1 B SPEAKER_02 0 days 00:00:24.870968750 0 days 00:00:25.124093750\n",
"2 C SPEAKER_03 0 days 00:00:47.567843750 0 days 00:00:58.705343750\n",
"3 D SPEAKER_04 0 days 00:00:56.376593750 0 days 00:00:57.540968750\n",
"4 E SPEAKER_03 0 days 00:00:59.430968750 0 days 00:01:04.206593750\n",
"... ... ... ... ...\n",
"3208 DSK SPEAKER_03 0 days 02:03:08.597843750 0 days 02:03:09.272843750\n",
"3209 DSL SPEAKER_03 0 days 02:03:10.049093750 0 days 02:03:11.669093750\n",
"3210 DSM SPEAKER_00 0 days 02:03:11.669093750 0 days 02:03:11.702843750\n",
"3211 DSN SPEAKER_00 0 days 02:03:15.145343750 0 days 02:03:17.541593750\n",
"3212 DSO SPEAKER_00 0 days 02:03:17.963468750 0 days 02:03:18.300968750\n",
"\n",
"[3213 rows x 4 columns]"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"with DiarizationPipeline() as diarization:\n",
" speakers_df = diarization(audio_path=AUDIO_PATH)\n",
"\n",
"speakers_df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Assign speaker labels to each chunk in the transcript"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"\n",
"\n",
" \n",
" \n",
" \n",
" text\n",
" start\n",
" end\n",
" speaker\n",
" \n",
" \n",
" \n",
" \n",
" 0\n",
" What's\n",
" 0 days 00:00:00.040000\n",
" 0 days 00:00:00.300000\n",
" SPEAKER_03\n",
" \n",
" \n",
" 1\n",
" up\n",
" 0 days 00:00:00.300000\n",
" 0 days 00:00:00.440000\n",
" SPEAKER_03\n",
" \n",
" \n",
" 2\n",
" everybody\n",
" 0 days 00:00:00.440000\n",
" 0 days 00:00:00.700000\n",
" SPEAKER_03\n",
" \n",
" \n",
" 3\n",
" welcome\n",
" 0 days 00:00:00.700000\n",
" 0 days 00:00:01\n",
" SPEAKER_03\n",
" \n",
" \n",
" 4\n",
" to\n",
" 0 days 00:00:01\n",
" 0 days 00:00:01.220000\n",
" SPEAKER_03\n",
" \n",
" \n",
" ...\n",
" ...\n",
" ...\n",
" ...\n",
" ...\n",
" \n",
" \n",
" 25428\n",
" We'll\n",
" 0 days 02:03:16.760000\n",
" 0 days 02:03:17.040000\n",
" SPEAKER_03\n",
" \n",
" \n",
" 25429\n",
" see\n",
" 0 days 02:03:17.040000\n",
" 0 days 02:03:17.160000\n",
" SPEAKER_00\n",
" \n",
" \n",
" 25430\n",
" you\n",
" 0 days 02:03:17.160000\n",
" 0 days 02:03:17.300000\n",
" SPEAKER_03\n",
" \n",
" \n",
" 25431\n",
" there.\n",
" 0 days 02:03:17.300000\n",
" 0 days 02:03:17.500000\n",
" SPEAKER_00\n",
" \n",
" \n",
" 25432\n",
" Peace.\n",
" 0 days 02:03:17.500000\n",
" 0 days 02:03:18.320000\n",
" SPEAKER_03\n",
" \n",
" \n",
"\n",
"25433 rows × 4 columns
\n",
""
],
"text/plain": [
" text start end speaker\n",
"0 What's 0 days 00:00:00.040000 0 days 00:00:00.300000 SPEAKER_03\n",
"1 up 0 days 00:00:00.300000 0 days 00:00:00.440000 SPEAKER_03\n",
"2 everybody 0 days 00:00:00.440000 0 days 00:00:00.700000 SPEAKER_03\n",
"3 welcome 0 days 00:00:00.700000 0 days 00:00:01 SPEAKER_03\n",
"4 to 0 days 00:00:01 0 days 00:00:01.220000 SPEAKER_03\n",
"... ... ... ... ...\n",
"25428 We'll 0 days 02:03:16.760000 0 days 02:03:17.040000 SPEAKER_03\n",
"25429 see 0 days 02:03:17.040000 0 days 02:03:17.160000 SPEAKER_00\n",
"25430 you 0 days 02:03:17.160000 0 days 02:03:17.300000 SPEAKER_03\n",
"25431 there. 0 days 02:03:17.300000 0 days 02:03:17.500000 SPEAKER_00\n",
"25432 Peace. 0 days 02:03:17.500000 0 days 02:03:18.320000 SPEAKER_03\n",
"\n",
"[25433 rows x 4 columns]"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"assign_speaker_to_transcript(\n",
" speakers_df=speakers_df,\n",
" transcript_df=transcript_df,\n",
" inplace=True,\n",
")\n",
"\n",
"transcript_df.to_json(\n",
" AUDIO_PATH.with_suffix(\".json\"), orient=\"records\", indent=4\n",
")\n",
"\n",
"transcript_df"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.9"
}
},
"nbformat": 4,
"nbformat_minor": 2
}