{"id":13598369,"url":"https://github.com/sushant097/Handwritten-Line-Text-Recognition-using-Deep-Learning-with-Tensorflow","last_synced_at":"2025-04-10T09:31:04.620Z","repository":{"id":47078200,"uuid":"201054077","full_name":"sushant097/Handwritten-Line-Text-Recognition-using-Deep-Learning-with-Tensorflow","owner":"sushant097","description":"Use Convolutional Recurrent Neural Network to recognize the Handwritten line text image without pre segmentation into words or characters. Use CTC loss Function to train.","archived":false,"fork":false,"pushed_at":"2022-10-28T09:48:45.000Z","size":2295,"stargazers_count":248,"open_issues_count":4,"forks_count":119,"subscribers_count":15,"default_branch":"master","last_synced_at":"2023-11-07T14:17:43.753Z","etag":null,"topics":["blstm","cnn","crnn-tensorflow","ctc-loss","deep-learning","deep-neural-networks","handwritten-text-recognition","iam-dataset","python3","recurrent-neural-network","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sushant097.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-08-07T13:16:17.000Z","updated_at":"2023-10-30T13:01:36.000Z","dependencies_parsed_at":"2023-01-20T10:35:18.135Z","dependency_job_id":null,"html_url":"https://github.com/sushant097/Handwritten-Line-Text-Recognition-using-Deep-Learning-with-Tensorflow","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sushant097%2FHandwritten-Line-Text-Recognition-using-Deep-Learning-with-Tensorflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sushant097%2FHandwritten-Line-Text-Recognition-using-Deep-Learning-with-Tensorflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sushant097%2FHandwritten-Line-Text-Recognition-using-Deep-Learning-with-Tensorflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sushant097%2FHandwritten-Line-Text-Recognition-using-Deep-Learning-with-Tensorflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sushant097","download_url":"https://codeload.github.com/sushant097/Handwritten-Line-Text-Recognition-using-Deep-Learning-with-Tensorflow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248191655,"owners_count":21062547,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blstm","cnn","crnn-tensorflow","ctc-loss","deep-learning","deep-neural-networks","handwritten-text-recognition","iam-dataset","python3","recurrent-neural-network","tensorflow"],"created_at":"2024-08-01T17:00:52.070Z","updated_at":"2025-04-10T09:31:04.605Z","avatar_url":"https://github.com/sushant097.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Handwritten Line Text Recognition using Deep Learning with Tensorflow\r\n![GitHub stars](https://img.shields.io/github/stars/sushant097/Handwritten-Line-Text-Recognition-using-Deep-Learning-with-Tensorflow) \r\n![GitHub forks](https://img.shields.io/github/forks/sushant097/Handwritten-Line-Text-Recognition-using-Deep-Learning-with-Tensorflow)\r\n[![Maintenance](https://img.shields.io/badge/maintained-yes-green.svg)](https://github.com/sushant097/Handwritten-Line-Text-Recognition-using-Deep-Learning-with-Tensorflow///commits/master)\r\n[![Website shields.io](https://img.shields.io/badge/website-up-yellow)](https://github.com/sushant097/Handwritten-Line-Text-Recognition-using-Deep-Learning-with-Tensorflow)\r\n[![Ask Me Anything !](https://img.shields.io/badge/ask%20me-linkedin-1abc9c.svg)](https://www.linkedin.com/in/susan-gautam/)\r\n[![License](http://img.shields.io/:license-mit-blue.svg?style=flat-square)](http://badges.mit-license.org)\r\n\r\n\r\n### Description\r\nUse Convolutional Recurrent Neural Network to recognize the Handwritten line text image without pre segmentation into words or characters. Use CTC loss Function to train.\r\nMore read this [Medium Post](https://susant.medium.com/learn-and-use-handwritten-line-text-recognition-using-deep-learning-with-tensorflow-b661434b5e3b?source=friends_link\u0026sk=f22713a4c39f144ee26acf9a31d757af)\r\n\r\n### Why Deep Learning?\r\n![Why Deep Learning](images/WhyDeepLearning.png?raw=true \"Why Deep Learning\")\r\n\u003e Deep Learning self extracts features with a deep neural networks and classify itself. Compare to traditional Algorithms it performance increase with Amount of Data.\r\n\r\n## \u003ci\u003e Basic Intuition on How it Works.\r\n![Step_wise_detail](images/Step_wise_detail_of_workflow.png?raw=true \"Step_Wise Detail\")\r\n* First Use Convolutional Recurrent Neural Network to extract the important features from the handwritten line text Image.\r\n* The output before CNN FC layer (512x1x100) is passed to the BLSTM which is for sequence dependency and time-sequence operations.\r\n* Then CTC LOSS [Alex Graves](https://www.cs.toronto.edu/~graves/icml_2006.pdf) is used to train the RNN which eliminate the Alignment problem in Handwritten, since handwritten have different alignment of every writers. We just gave the what is written in the image (Ground Truth Text) and BLSTM output, then it calculates loss simply as `-log(\"gtText\")`; aim to minimize negative maximum likelihood path.\r\n* Finally CTC finds out the possible paths from the given labels. Loss is given by for (X,Y) pair is: ![Ctc_Loss](images/CtcLossFormula.png?raw=true \"CTC loss for the (X,Y) pair\")\r\n* Finally CTC Decode is used to decode the output during Prediction.\r\n\u003c/i\u003e\r\n\r\n\r\n#### Detail Project Workflow\r\n![Architecture of Model](images/ArchitectureDetails.png?raw=true \"Model Architecture\")\r\n\r\n* Project consists of Three steps:\r\n  1. Multi-scale feature Extraction --\u003e Convolutional Neural Network 7 Layers\r\n  2. Sequence Labeling (BLSTM-CTC)  --\u003e Recurrent Neural Network (2 layers of LSTM) with CTC \r\n  3. Transcription --\u003e Decoding the output of the RNN (CTC decode)\r\n![DetailModelArchitecture](images/DetailModelArchitecture.png?raw=true \"DetailModelArchitecture\")\r\n\r\n# Requirements\r\n1. Tensorflow 1.8.0 ; You can upgrade to Tensorflow v2 with this [link](https://www.tensorflow.org/guide/upgrade)\r\n2. Flask\r\n3. Numpy\r\n4. OpenCv 3\r\n5. Spell Checker `autocorrect` \u003e=0.3.0 ``pip install autocorrect``\r\n\r\n#### Dataset Used\r\n* IAM dataset download from [here](http://www.fki.inf.unibe.ch/databases/iam-handwriting-database)\r\n* Only needed the lines images and lines.txt (ASCII).\r\n* Place the downloaded files inside data directory  \r\n\r\n###### The Trained model is available and download from this [link](https://drive.google.com/file/d/10HHNZPqPQZCQCLrKGQOq5E7zFW5wGcA4/view?usp=sharing). The trained model available have CER=8.32% and trained on IAM dataset with some additional created dataset. The final model have 3.42% CER which is not available publicly.\r\n\r\n\r\nTo Train the model from scratch\r\n```markdown\r\n$ python main.py --train\r\n\r\n```\r\nTo validate the model\r\n```markdown\r\n$ python main.py --validate\r\n```\r\nTo Prediction\r\n```markdown\r\n$ python main.py\r\n```\r\n\r\nRun in Web with Flask\r\n```markdown\r\n$ python upload.py\r\nValidation character error rate of saved model: 8.654728%\r\nPython: 3.6.4 \r\nTensorflow: 1.8.0\r\nInit with stored values from ../model/snapshot-24\r\nWithout Correction clothed leaf by leaf with the dioappoistmest\r\nWith Correction clothed leaf by leaf with the dioappoistmest\r\n```\r\n**Prediction output on IAM Test Data**\r\n![PredictionOutput](images/IAM_dataset_Prediction_Output.png?raw=true \"Prediction Output On Iam Dataset\")\r\n\r\n**Prediction output on Self Test Data**\r\n![PredictionOutput](images/PredictionOutput.png?raw=true \"Prediction Output on Self Data\")\r\n\r\n\u003ci\u003eSee the project [Devnagari Handwritten Word Recognition with Deep Learning](https://github.com/sushant097/Devnagari-Handwritten-Word-Recongition-with-Deep-Learning) for more insights.\u003c/i\u003e\r\n# Further Improvement\r\n* Using MDLSTM to recognize whole paragraph at once [Scan, Attend and Read: End-to-End Handwritten Paragraph Recognition with MDLSTM Attention](https://arxiv.org/abs/1604.03286)\r\n* Line segmentation can be added for full paragraph text recognition. To segment a paragraph into lines, you can use an A* path planning algorithm or CNN model.\r\n* Better Image preprocessing, such as: reducing background noise to handle real-time images more accurately.\r\n* Better Decoding approach to improve accuracy. Some of the CTC Decoder found [here](https://github.com/githubharald/CTCDecoder)  \r\n\r\n\r\n# Citation\r\nIf you use any part of this project in your work, please cite:\r\n\r\n```bash\r\n@techreport{Handwritten-Line-text-recognition-using-deep-learning-2019,\r\n  title={Handwritten Line Text Recognition},\r\n  author={Gautam Sushant},\r\n  institution={Tribhuvan University},\r\n  year={2019}\r\n}\r\n```\r\nFeel Free to improve this project with a pull Request.\r\n\r\n\r\n\r\n* This is a work from my  undergraduate thesis in computer engineering at Tribhuvan University in 2019.\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsushant097%2FHandwritten-Line-Text-Recognition-using-Deep-Learning-with-Tensorflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsushant097%2FHandwritten-Line-Text-Recognition-using-Deep-Learning-with-Tensorflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsushant097%2FHandwritten-Line-Text-Recognition-using-Deep-Learning-with-Tensorflow/lists"}