{"id":14959134,"url":"https://github.com/philipperemy/tensorflow-ctc-speech-recognition","last_synced_at":"2025-05-02T12:31:18.158Z","repository":{"id":54191487,"uuid":"90033372","full_name":"philipperemy/tensorflow-ctc-speech-recognition","owner":"philipperemy","description":"Application of Connectionist Temporal Classification (CTC) for Speech Recognition (Tensorflow 1.0 but compatible with 2.0).","archived":false,"fork":false,"pushed_at":"2021-03-04T06:29:48.000Z","size":649,"stargazers_count":130,"open_issues_count":4,"forks_count":46,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-04-07T01:52:23.393Z","etag":null,"topics":["ctc","ctc-loss","deep-learning","machine-learning","speech-analysis","speech-recognition","speech-to-text","tensorflow","tensorflow-1-0","tutorial"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/philipperemy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"github":["philipperemy"]}},"created_at":"2017-05-02T12:55:10.000Z","updated_at":"2024-08-12T19:29:33.000Z","dependencies_parsed_at":"2022-08-13T08:50:27.660Z","dependency_job_id":null,"html_url":"https://github.com/philipperemy/tensorflow-ctc-speech-recognition","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipperemy%2Ftensorflow-ctc-speech-recognition","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipperemy%2Ftensorflow-ctc-speech-recognition/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipperemy%2Ftensorflow-ctc-speech-recognition/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipperemy%2Ftensorflow-ctc-speech-recognition/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/philipperemy","download_url":"https://codeload.github.com/philipperemy/tensorflow-ctc-speech-recognition/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252038131,"owners_count":21684629,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ctc","ctc-loss","deep-learning","machine-learning","speech-analysis","speech-recognition","speech-to-text","tensorflow","tensorflow-1-0","tutorial"],"created_at":"2024-09-24T13:18:54.364Z","updated_at":"2025-05-02T12:31:17.481Z","avatar_url":"https://github.com/philipperemy.png","language":"Python","funding_links":["https://github.com/sponsors/philipperemy"],"categories":[],"sub_categories":[],"readme":"# Tensorflow CTC Speech Recognition\n- Compatible with Tensorflow through v1 compat.\n- Application of Connectionist Temporal Classification (CTC) for Speech Recognition (Tensorflow 1.0) \n- On the VCTK Corpus (same corpus as the one used by WaveNet).\n\n## How to get started?\n```bash\ngit clone https://github.com/philipperemy/tensorflow-ctc-speech-recognition.git ctc-speech\ncd ctc-speech\npip3 install -r requirements.txt # inside a virtualenv\n\n# Download the VCTK Corpus here: http://homepages.inf.ed.ac.uk/jyamagis/page3/page58/page58.html\n# OR use this file (~65MB) that contains all the utterances for speaker p225.\nwget https://www.dropbox.com/s/xecprghgwbbuk3m/vctk-pc225.tar.gz\ntar xvzf vctk-pc225.tar.gz \u0026\u0026 rm -rf vctk-pc225.tar.gz\npython generate_audio_cache.py --audio_dir vctk-p225\n\n\n\nwget http://homepages.inf.ed.ac.uk/jyamagis/release/VCTK-Corpus.tar.gz # 10GB!l\npython3 ctc_tensorflow_example.py # to run the experiment defined in the section First Experiment.\n```\n\nYou can also download only the relevant files here [https://www.dropbox.com/s/xecprghgwbbuk3m/vctk-pc225.tar.gz?dl=1](https://www.dropbox.com/s/xecprghgwbbuk3m/vctk-pc225.tar.gz?dl=1) (~69MB). Thanks to @Burak Bayramli.\n\n## Requirements\n- **dill**: improved version of pickle\n- **librosa**: library to interact with audio wav files\n- **namedtupled**: dictionary to named tuples\n- **numpy**: scientific library\n- **python_speech_features**: extracting relevant features from raw audio data\n- **tensorflow**: machine learning library\n- **progressbar2**: progression bar\n\n## First experiment\n\nThe code to reproduce this experiment is no longer in the latest commit.\n\n```\ngit checkout ba6c10fba2383cd4933d47896f95d30248458161\n```\n\n### Set up\nSpeech Recognition is a very difficult topic. In this first experiment, we consider:\n- A very small subset of the VCTK Corpus composed of only one speaker: p225.\n- Only 5 sentences of this speaker, denoted as: 001, 002, 003, 004 and 005.\n\nThe network is defined as:\n- One LSTM layer `rnn.LSTMCell` with 100 units, completed by a softmax.\n- Batch size of 1.\n- Momentum Optimizer with learning rate of 0.005 and momentum of 0.9.\n\nThe validation set is obtained by constantly truncating the audio files randomly at the beginning (between 0 and 125ms max). We make sure that we do not cut when the speaker is speaking. Using 5 unseen sentences would be more realistic, however, it's almost impossible for the network to pick it up since a training set of only 5 sentences is way too small to cover all the possible phonemes of the english language. By truncating randomly the silences at the beginning, we make sure that the network does not learn the mapping audio from sentence -\u003e text in a dumb way.\n\n\n### Results\n\nMost of the time, the network can guess the correct sentence. Sometimes, it misses a bit but still encouraging.\n\nExample 1\n```\nOriginal training: diving is no part of football\nDecoded training: diving is no part of football\nOriginal validation: theres still a bit to go\nDecoded validation: thers still a bl to go\nEpoch 3074/10000, train_cost = 0.032, train_ler = 0.000, val_cost = 9.131, val_ler = 0.125, time = 1.648\n```\n\nExample 2\n```\nOriginal training: three hours later the man was free\nDecoded training: three hours later the man was free\nOriginal val: and they were being paid \nDecoded val: nand they ere being paid  \nEpoch 3104/10000, train_cost = 0.075, train_ler = 0.000, val_cost = 2.945, val_ler = 0.077, time = 1.042\n```\n\nExample 3\n```\nOriginal training: theres still a bit to go\nDecoded training: theres still a bit to go\nOriginal val: three hours later the man was free\nDecoded val: three hors late th man wasfree\nEpoch 3108/10000, train_cost = 0.032, train_ler = 0.000, val_cost = 12.532, val_ler = 0.118, time = 0.859\n```\n\n\u003cp align=\"center\"\u003e\n  \u003cb\u003eCTC Loss\u003c/b\u003e\u003cbr\u003e\n  \u003cimg src=\"veusz/ctc_loss_1.png\" width=\"600\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cb\u003eCTC Loss (Log scale)\u003c/b\u003e\u003cbr\u003e\n  \u003cimg src=\"veusz/ctc_loss_2.png\" width=\"600\"\u003e\n\u003c/p\u003e\nCTC Loss is the raw loss defined in the paper by Alex Graves.\n\n\u003cp align=\"center\"\u003e\n  \u003cb\u003eLER Loss\u003c/b\u003e\u003cbr\u003e\n  \u003cimg src=\"veusz/ler_loss_1.png\" width=\"600\"\u003e\n\u003c/p\u003e\nLER (Label Error Rate) measures the inaccuracy between the predicted and the ground truth texts.\n\nClearly we can see that the network learns very well on just 5 sentences! It's far from being perfect but quite appealing for a first try.\n\n## Second experiment\n\n- LSTM with 256 cells.\n- Only one speaker: p225.\n- 15 shortest utterances used as testing set.\n- Rest used as training set.\n- Can now define a batch size different than 1.\n\n```\nEpoch 2723/3000, train_cost = 1.108, train_ler = 0.000, val_cost = 59.116, val_ler = 0.467, time = 2.559\n- Original (training) : but the commission is on a collision course with the government \n- Decoded  (training) : but the commission is on a collision course with the government \n- Original (training) : this action reflects a slump in bookings \n- Decoded  (training) : this action reflects a slump in bookings \n- Original (training) : they had to learn to work from the consumer back \n- Decoded  (training) : they had to learn to work from the consumer back \n- Original (training) : it depends on the internal discussions in the ministry of defence \n- Decoded  (training) : it depends on the internal discussions in the ministry of defence \n- Original (training) : irvine said his company was intent on supporting the scottish dairy industry \n- Decoded  (training) : irvine said his company was intent on supporting the scottish dairy industry \n- Original (training) : the pain was almost too much to bear \n- Decoded  (training) : the pain was almost too much to bear \n- Original (training) : this is a very common type of bow one showing mainly red and yellow with little or no green or blue \n- Decoded  (training) : this is a very common type of bow one showing mainly red and yellow with little or no green or blue \n- Original (training) : in fact he should never have been in the field \n- Decoded  (training) : in fact he should never have been in the field \n- Original (training) : saddam is not the only example of evil in our world \n- Decoded  (training) : saddam is nat the only example of evil in our world \n- Original (training) : so did she meet him  \n- Decoded  (training) : so did she meet him  \n- Original (validation) : it is a court case \n- Decoded  (validation) : it is a cot ase    \n```\n\n\u003cp align=\"center\"\u003e\n  \u003cb\u003eLER Loss\u003c/b\u003e\u003cbr\u003e\n  \u003cimg src=\"veusz/exp2.png\" width=\"600\"\u003e\n\u003c/p\u003e\n\nThis experiment is interesting. The network seems to generalize a bit on unseen audio files (of the same speaker). I didn't expect the network to perform well on such a small dataset. However, let's keep in mind that the generalization power is quite poor here. It's overfitting as well.\n\n## Special Thanks\n- Igor Maceda for providing the basic scripts here [ctc_tensorflow_example](https://github.com/igormq/ctc_tensorflow_example)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphilipperemy%2Ftensorflow-ctc-speech-recognition","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fphilipperemy%2Ftensorflow-ctc-speech-recognition","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphilipperemy%2Ftensorflow-ctc-speech-recognition/lists"}