{"id":21523183,"url":"https://github.com/harismuneer/Handwritten-Digits-Classification-Using-KNN-Multiclass_Perceptron-SVM","last_synced_at":"2025-07-16T07:31:20.659Z","repository":{"id":108166927,"uuid":"139596782","full_name":"OSSpk/Handwritten-Digits-Classification-Using-KNN-Multiclass_Perceptron-SVM","owner":"OSSpk","description":"🏆 A Comparative Study on Handwritten Digits Recognition using Classifiers like K-Nearest Neighbours (K-NN), Multiclass Perceptron/Artificial Neural Network (ANN) and Support Vector Machine (SVM) discussing the pros and cons of each algorithm and providing the comparison results in terms of accuracy and efficiecy of each algorithm.","archived":false,"fork":false,"pushed_at":"2024-11-22T12:01:08.000Z","size":13570,"stargazers_count":62,"open_issues_count":0,"forks_count":17,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-11-22T13:17:43.302Z","etag":null,"topics":["accuracy-analysis","artificial-neural-network","comparative-study","delta-rule","efficiency-analysis","handwritten-digit-recognition","k-nearest-neighbours","knn","learning-algorithms-comparison","logistic-regression","machine-learning-algorithms","mnist-classification-logistic","mnist-dataset","multiclass-perceptron","perceptron-learning-algorithm","sigmoid-function","supervised-machine-learning","support-vector-classifier","support-vector-machines","svm"],"latest_commit_sha":null,"homepage":"https://github.com/harismuneer","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OSSpk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":null,"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":["https://www.cyfylabs.com"]}},"created_at":"2018-07-03T14:37:27.000Z","updated_at":"2024-11-22T12:01:12.000Z","dependencies_parsed_at":null,"dependency_job_id":"8f4fc46a-3627-4f86-aaaf-71d261906080","html_url":"https://github.com/OSSpk/Handwritten-Digits-Classification-Using-KNN-Multiclass_Perceptron-SVM","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSSpk%2FHandwritten-Digits-Classification-Using-KNN-Multiclass_Perceptron-SVM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSSpk%2FHandwritten-Digits-Classification-Using-KNN-Multiclass_Perceptron-SVM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSSpk%2FHandwritten-Digits-Classification-Using-KNN-Multiclass_Perceptron-SVM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSSpk%2FHandwritten-Digits-Classification-Using-KNN-Multiclass_Perceptron-SVM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OSSpk","download_url":"https://codeload.github.com/OSSpk/Handwritten-Digits-Classification-Using-KNN-Multiclass_Perceptron-SVM/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226112968,"owners_count":17575438,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["accuracy-analysis","artificial-neural-network","comparative-study","delta-rule","efficiency-analysis","handwritten-digit-recognition","k-nearest-neighbours","knn","learning-algorithms-comparison","logistic-regression","machine-learning-algorithms","mnist-classification-logistic","mnist-dataset","multiclass-perceptron","perceptron-learning-algorithm","sigmoid-function","supervised-machine-learning","support-vector-classifier","support-vector-machines","svm"],"created_at":"2024-11-24T01:12:55.594Z","updated_at":"2025-07-16T07:31:20.616Z","avatar_url":"https://github.com/OSSpk.png","language":"Python","funding_links":["https://www.cyfylabs.com"],"categories":[],"sub_categories":[],"readme":"# 🏆 A Comparative Study on Handwritten Digits Recognition using Classifiers like K-NN, Multiclass Perceptron and SVM\n\n\u003ca href=\"https://github.com/harismuneer\"\u003e\u003cimg alt=\"views\" title=\"Github views\" src=\"https://komarev.com/ghpvc/?username=harismuneer\u0026style=flat-square\" width=\"125\"/\u003e\u003c/a\u003e\n[![Open Source Love svg1](https://badges.frapsoft.com/os/v1/open-source.svg?v=103)](#)\n[![GitHub Forks](https://img.shields.io/github/forks/harismuneer/Handwritten-Digits-Classification-Using-KNN-Multiclass_Perceptron-SVM.svg?style=social\u0026label=Fork\u0026maxAge=2592000)](https://www.github.com/harismuneer/Handwritten-Digits-Classification-Using-KNN-Multiclass_Perceptron-SVM/fork)\n[![GitHub Issues](https://img.shields.io/github/issues/harismuneer/Handwritten-Digits-Classification-Using-KNN-Multiclass_Perceptron-SVM.svg?style=flat\u0026label=Issues\u0026maxAge=2592000)](https://www.github.com/harismuneer/Handwritten-Digits-Classification-Using-KNN-Multiclass_Perceptron-SVM/issues)\n[![contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat\u0026label=Contributions\u0026colorA=red\u0026colorB=black\t)](#)\n\n\n\nFor the full report, refer to the file named [Detailed Report.pdf](../master/Detailed_Report.pdf).\n\n## Problem Statement\nThe task at hand is to classify handwritten digits using supervised machine learning methods. The digits belong to classes of **0 to 9**. \n\n*“Given a query instance (a digit) in the form of an image, our machine learning model must correctly classify its appropriate class.”*\n\n## Dataset\nMNIST Handwritten Digits dataset is used for this task. It contains images of digits taken from a variety of scanned documents, normalized in size and centered.\nEach image is a 28 by 28 pixel square (784 pixels total). The dataset contains 60,000 images for model training and 10,000 images for the evaluation of the model.\n\n## Methodology\nWe have used **supervised machine learning models** to predict the digits. Since this is a **comparative study** hence we will first describe the **K-Nearest Neighbors Classifier** as the baseline method which will then be compared to **Multiclass Perceptron Classifier** and **SVM Classifier**.\n\n### 1) K-Nearest Neighbors Classifier – Our Baseline Method\nk-Nearest Neighbors (k-NN) is an algorithm, which:\n*\tfinds a group of k objects in the training set that are closest to the test object, and\n*\tbases the assignment of a label on the predominance of a class in this neighborhood.\n\nWhen we used the K-NN method the following pros and cons were observed:\n\n#### Pros\n*\tK-NN executes quickly for small training data sets.\n*\tNo assumptions about data — useful, for example, for nonlinear data\n*\tSimple algorithm — to explain and understand/interpret\n*\tVersatile — useful for classification or regression\n*\tTraining phase is extremely quick because it doesn’t learn any data\n\n#### Cons\n*\tComputationally expensive — because the algorithm compares the test data with all examples in training data and then finalizes the label\n*\tThe value of K is unknown and can be predicted using cross validation techniques\n*\tHigh memory requirement – because all the training data is stored\n*\tPrediction stage might be slow if training data is large\n\n### 2) Multiclass Perceptron Classifier:\nA multiclass perceptron classifier can be made using multiple binary class classifiers trained with 1 vs all strategy. In this strategy, while training a perceptron the training labels are such that e.g. for the classifier 2 vs all, the labels with 2 will be labeled as 1 and rest will be labeled as 0 for Sigmoid Unit while for Rosenblatt’s perceptron the labels would be 1 and -1 respectively for positive and negative examples. \n\nNow all we have to do is to train (learn the weights for) 10 classifiers separately and then feed the query instance to all these classifiers (as shown in figure above). The label of classifier with highest confidence will then be assigned to the query instance. \n\n#### How Multiclass Perceptron mitigates the limitations of K-NN:\n\nAs we already discussed, K-NN stores all the training data and when a new query instance comes it compares its similarity with all the training data which makes it expensive both computationally and memory-wise. There is no learning involved as such. On the other hand, Multiclass perceptron takes some time in learning phase but after its training is done, it learns the new weights which can be saved and then used. Now, when a query instance comes, it only has to take to dot product of that instance with the weights learned and there comes the output (after applying activation function). \n\n*\tThe prediction phase is extremely fast as compared to that of K-NN. \n*\tAlso, it’s a lot more efficient in terms of computation (during prediction phase) and memory (because now it only has to store the weights instead of all the training data). \n\n### 3) SVM Classifier using Histogram of Oriented Gradients (HOG) Features:\n\nJust for comparison purposes, we have also used a third supervised machine learning technique named Support Vector Machine Classifier.\nThe model isn’t implemented. Its imported directly from scikit learn module of python and used.\n\nIn K-NN and Multiclass Perceptron Classifier we trained our models on raw images directly instead of computing some features from the input image and training the model on those computed measurements/features. \n\nA feature descriptor is a representation of an image that simplifies the image by extracting useful information and throwing away extraneous information. Now we are going to compute the Histogram of Oriented Gradients as features from the digit images and we will train the SVM Classifier on that. The HOG descriptor technique counts occurrences of gradient orientation in localized portions of an image - detection window.\n\n\n## Analysis\n\nNow the final phase. After running the experiment with different algorithms, the results are summarized. First comparing the techniques on basis of Accuracy:\n\n### Accuracy (Performance):\n\n\u003cp align=\"middle\"\u003e\n  \u003cimg src=\"../master/Images/r1.png\" width=\"600\"/\u003e\n\u003c/p\u003e\n\nWhen we compare the K-NN method with Multiclass Perceptron and SVM on basis of accuracy then its accuracy is similar to that of other two classifiers which means despite its simplicity K-NN is really a good classifier.\n\n### Prediction Time (Efficiency):\n\n#### Our Observations:\n\nOne of the main limitations of K-NN was that it was computationally expensive. Its prediction time was large because whenever a new query instance came it had to compare its similarity with all the training data and then sort the neighbors according to their confidence and then separating the top k neighbors and choosing the label of the most occurred neighbor in top k. In all this process, it takes a comparable amount of time.\n\nWhile for Multiclass Perceptron Classifier we observed it will mitigate this limitation in efficiency such that its prediction time will be short because now it will only compute the dot product in the prediction phase. The majority of time is spent only once in its learning phase. Then it’s ready to predict the test instances. \n\n#### Results:\n\n\u003cp align=\"middle\"\u003e\n  \u003cimg src=\"../master/Images/r2.png\" width=\"600\"/\u003e\n\u003c/p\u003e\n\n\n## Conclusion:\n\nWhen the times were calculated for the prediction phases of K-NN, Multiclass Perceptron and SVM, the Multiclass Perceptron clearly stands out with the shortest prediction time while on the other side, K-NN took a large time in predicting the test instances.\nHence Multiclass Perceptron clearly leaves K-NN behind in terms of efficiency in Prediction Time and also in terms of computation and memory load. Thus, it mitigates the limitations of our baseline method K-NN.\n\n----------------------------------------------------------------------------------------------------------------------------------------\n\n## How to Run Code\nThe code files are in running condition and are directly executable.\n\n(To install all the necessary packages at once, install [Anaconda](https://www.anaconda.com/download/#download))\n\n----------------------------------------------------------------------------------------------------------------------------------------\n\n\u003chr\u003e\n\n\n## Author\nYou can get in touch with me on my LinkedIn Profile: [![LinkedIn Link](https://img.shields.io/badge/Connect-harismuneer-blue.svg?logo=linkedin\u0026longCache=true\u0026style=social\u0026label=Follow)](https://www.linkedin.com/in/harismuneer)\n\nYou can also follow my GitHub Profile to stay updated about my latest projects: [![GitHub Follow](https://img.shields.io/badge/Connect-harismuneer-blue.svg?logo=Github\u0026longCache=true\u0026style=social\u0026label=Follow)](https://github.com/harismuneer)\n\nIf you liked the repo then kindly support it by giving it a star ⭐ and share in your circles so more people can benefit from the effort.\n\n\n## Contributions Welcome\n[![GitHub Issues](https://img.shields.io/github/issues/harismuneer/Handwritten-Digits-Classification-Using-KNN-Multiclass_Perceptron-SVM.svg?style=flat\u0026label=Issues\u0026maxAge=2592000)](https://www.github.com/harismuneer/Handwritten-Digits-Classification-Using-KNN-Multiclass_Perceptron-SVM/issues)\n\nIf you find any bugs, have suggestions, or face issues:\n\n- Open an Issue in the Issues Tab to discuss them.\n- Submit a Pull Request to propose fixes or improvements.\n- Review Pull Requests from other contributors to help maintain the project's quality and progress.\n\nThis project thrives on community collaboration! Members are encouraged to take the initiative, support one another, and actively engage in all aspects of the project. Whether it’s debugging, fixing issues, or brainstorming new ideas, your contributions are what keep this project moving forward.\n\nWith modern AI tools like ChatGPT, solving challenges and contributing effectively is easier than ever. Let’s work together to make this project the best it can be! 🚀\n\n\n## License\n[![MIT](https://img.shields.io/cocoapods/l/AFNetworking.svg?style=style\u0026label=License\u0026maxAge=2592000)](../master/LICENSE)\n\nCopyright (c) 2018-present, harismuneer                                                        \n\n\n\u003c!-- PROFILE_INTRO_START --\u003e\n\n\u003chr\u003e\n\n\u003ch1\u003e \u003ca href=\"#\"\u003e\u003cimg src=\"https://media.giphy.com/media/hvRJCLFzcasrR4ia7z/giphy.gif\" alt=\"Waving hand\" width=\"28\"\u003e\u003c/a\u003e\nHey there, I'm \u003ca href=\"https://www.linkedin.com/in/harismuneer/\"\u003eHaris Muneer\u003c/a\u003e 👨🏻‍💻\n\u003c/h1\u003e\n\n\n\u003ca href=\"https://github.com/harismuneer\"\u003e\u003cimg src=\"https://img.shields.io/github/stars/harismuneer\" alt=\"Total Github Stars\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/harismuneer?tab=followers\"\u003e\u003cimg src=\"https://img.shields.io/github/followers/harismuneer\" alt=\"Total Github Followers\"\u003e\u003c/a\u003e\n\n\u003chr\u003e\n\n- \u003cb\u003e🛠️ Product Builder:\u003c/b\u003e Agile Product Manager with 5+ years of hands-on experience delivering SaaS solutions across sales, recruiting, AI, social media, and public sector domains. Background in Computer Science, with a proven track record of scaling products from inception to $XXM+ ARR, launching 3 top-ranking tools on Product Hunt, and developing solutions adopted by 250+ B2B clients in 40+ countries.  \n \n- \u003cb\u003e🌟 Open Source Advocate:\u003c/b\u003e Passionate about making technology accessible, I’ve developed and open-sourced several software projects for web, mobile, desktop, and AI on my \u003ca href=\"https://github.com/harismuneer\"\u003eGitHub profile\u003c/a\u003e. These projects have been used by thousands of learners worldwide to enhance their skills and knowledge.\n\n- \u003cb\u003e📫 How to Reach Me:\u003c/b\u003e To learn more about my skills and work, visit my \u003ca href=\"https://www.linkedin.com/in/harismuneer\"\u003eLinkedIn profile\u003c/a\u003e. For collaboration or inquiries, feel free to reach out via \u003ca href=\"mailto:haris.muneer5@gmail.com\"\u003eemail\u003c/a\u003e.\n\n\u003chr\u003e\n\n\u003ch2 align=\"left\"\u003e🤝 Follow my journey\u003c/h2\u003e\n\u003cp align=\"left\"\u003e\n  \u003ca href=\"https://www.linkedin.com/in/harismuneer\"\u003e\u003cimg title=\"Follow Haris Muneer on LinkedIn\" src=\"https://img.shields.io/badge/LinkedIn-0077B5?style=for-the-badge\u0026logo=linkedin\u0026logoColor=white\"/\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/harismuneer\"\u003e\u003cimg title=\"Follow Haris Muneer on GitHub\" src=\"https://img.shields.io/badge/GitHub-100000?style=for-the-badge\u0026logo=github\u0026logoColor=white\"/\u003e\u003c/a\u003e\n  \u003ca href=\"https://www.youtube.com/@haris_muneer?sub_confirmation=1\"\u003e\u003cimg title=\"Subscribe on YouTube\" src=\"https://img.shields.io/badge/YouTube-FF0000?style=for-the-badge\u0026logo=youtube\u0026logoColor=white\"/\u003e\u003c/a\u003e \n  \u003ca href=\"mailto:haris.muneer5@gmail.com\"\u003e\u003cimg title=\"Email\" src=\"https://img.shields.io/badge/Gmail-D14836?style=for-the-badge\u0026logo=gmail\u0026logoColor=white\"/\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\n\n\u003c!-- PROFILE_INTRO_END --\u003e\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharismuneer%2FHandwritten-Digits-Classification-Using-KNN-Multiclass_Perceptron-SVM","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fharismuneer%2FHandwritten-Digits-Classification-Using-KNN-Multiclass_Perceptron-SVM","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharismuneer%2FHandwritten-Digits-Classification-Using-KNN-Multiclass_Perceptron-SVM/lists"}