{"id":26348932,"url":"https://github.com/raphaelsenn/lenet-1","last_synced_at":"2025-03-16T08:17:29.061Z","repository":{"id":278511508,"uuid":"935405429","full_name":"raphaelsenn/lenet-1","owner":"raphaelsenn","description":"Implementation of \"Backpropagation Applied to Handwritten Zip Code Recognition\" in PyTorch.","archived":false,"fork":false,"pushed_at":"2025-03-06T15:55:27.000Z","size":413,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-06T16:38:00.782Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/raphaelsenn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-19T11:48:11.000Z","updated_at":"2025-03-06T15:55:31.000Z","dependencies_parsed_at":"2025-03-06T16:38:12.294Z","dependency_job_id":"68ad1649-592c-4111-896b-8e950e1e09ce","html_url":"https://github.com/raphaelsenn/lenet-1","commit_stats":null,"previous_names":["raphaelsenn/lecun-backprop89","raphaelsenn/lenet-1989","raphaelsenn/lenet-1"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raphaelsenn%2Flenet-1","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raphaelsenn%2Flenet-1/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raphaelsenn%2Flenet-1/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raphaelsenn%2Flenet-1/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/raphaelsenn","download_url":"https://codeload.github.com/raphaelsenn/lenet-1/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243841152,"owners_count":20356446,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-16T08:17:28.559Z","updated_at":"2025-03-16T08:17:29.051Z","avatar_url":"https://github.com/raphaelsenn.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LeNet-1\nImplementation of the convolutional neural network (LeNet-1) described in the paper [Backpropagation Applied to Handwritten Zip Code Recognition](https://ieeexplore.ieee.org/document/6795724) in PyTorch.\n\n![image](res/architecture.png)\n\n## Usage\n\n#### Automatic training\n\nJust run:\n```bash\npython3 train.py\n```\n\n#### Manual training\n\nCreating the data:\n```python\nimport torch\nfrom create_data import create_data\n\n\n# set random seed\nseed = 42\ntorch.manual_seed(seed)\n\n# create training and testing data\ndataloader_train, dataloader_test = create_data(seed=seed)\n```\n\nCreating the model:\n```python\nfrom lenet1.lenet1 import LeNet1\n\n\n# creating the model\nlenet = LeNet1()\n\n# forward pass\ny_pred = lenet.forward(x)\n\n# printing model stats\nprint(lenet)\n```\n\n```bash\nStats LeNet-1\ntotal units:              1256\ntotal connections:        64660\nindependent parameters:   9760\n```\n\nStart training:\n```python\nfrom train import train\n\n\n# start training\ntrain(\n  lenet,              # model\n  dataloader_train,   # training data\n  dataloader_test,    # test data\n  0.15,               # learning rate\n  23,                 # training passes\n  'cpu',              # device\n  True                # printing mse, error rate, ... while training\n)\n```\n\n## Results\n\nResults of the paper after 23 passes:\n\n```text\npass: 23\ntrain report - loss: 0.00250     error: 0.0014   missclassifications: 10\ntest  report - loss: 0.01800     error: 0.0500   missclassifications: 102\n```\nMy results after 23 passes:\n\n```text\npass: 23\ntrain report - loss: 0.00101    error: 0.00521  missclassifications: 38\ntest  report - loss: 0.00811    error: 0.04933  missclassifications: 99\n```\n\nThese results match pretty much the results from the original paper. Maybie with some hyperparameter optimization (i.e. for the best learning rate) we could achive much better results.\n\n## Notes\n\n#### About the Convolutional Neural Network\n\n* \"LeNet-1\" consists of three hidden layers (H1 to H3), and one output layer\n\n* Input is a $(16 \\times 16)$ greyscale image (range between [-1, 1]), resulting in 16 * 16 = 256 input neurons\n\n* *\"For units in layer H1 that are one unit apart, their receptive fields (in the input layer) are two pixels apart.*\" this implies `stride=2` (same between layer H1 and H2)\n\n\n##### Layer H1\n\n* Layer H1 uses 12 $(5 \\times 5)$-kernels resulting in 12 feature maps H1.1, ..., H1.12 where each feature map has a $(8 \\times 8)$-shape\n\n* The 12 $(12 \\times 12)$-kernels result in 12 * 5 * 5 = 300 learnable parameters (weights)\n\n* Each Unit in H1.X with $X \\in \\{1, ..., 12\\}$ has its own bias (Conv2D in PyTorch uses 1-bias for each feature map instead), resulting in 12 * 8 * 8 = 768 biases\n\n* Therefore layer $H1$ consists of 300 + 768 = 1068 learnable parameters\n\n##### Layer H2\n\n* Layer H2 features 12 feature maps, each feature map consists of 8 $(5 \\times 5)$-kernels, resulting in 12 * 8 * 5 * 5 = 2400 learnable parameteres (weights) \n\n* Each unit in H2 combines local information coming from 8 of the 12 different feature maps in H1\n\n* There is **NO** clear explanation how to select 8 of the 12 feature maps between layer H1 and H2, i did it like [@karpathy](https://github.com/karpathy)\n\n* Each unit in H2.X with $X \\in \\{1, ..., 12\\}$ has 8 * 5 * 5 = 200 inputs coming from 8 $(8x8)$ feature maps (from H1) **AND** 1-bias\n\n* H2 consists of 12 * 4 * 4 = 192-biases\n\n* Therefore layer H2 consists of 2400 + 192 = 2592 learnable parameters\n\n##### Layer H3\n\n* Layer H3 is fully connected to H2 (the 12 feature maps H2.1, ..., H2.12 which are 12 * 4 * 4 = 192 units)\n\n* H3 consists of 30 units and biases, resulting in 192 * 30 + 30=5790 learnable parameters \n\n##### Output layer\n\n* The output layer is fully-connected to layer H3\n\n* The output layer consists of 10 units and biases, resulting in 30 * 10 + 10 = 310 learnable parameters\n\n##### More notes about the neural net\n\n* No information about padding\n\n* No information about used hyperparameters (i.e. learning rate)\n\n* No information how the biases were initialized (assumed to be zero)\n\n* They used the mean squared error as an objective, instead of cross-entropy\n\n* I one hot encoded the targets (during training), because of the MSE objective\n\n#### About the data\n\n* They used *\"9298 segmented numerals digitized from handwritten zip codes that appeared on U.S. mail passing through the Buffalo, NY post office. \"*\n\n* I couldn't find this dataset in the internet, so i simulated it using MNIST.\n\n## Insights about the Data\n\n#### Viewing some random numbers\n\n![image](res/random_numbers.png)\n\n#### Number distribution\n\n![image](res/number_distribution.png)\n\n## Citations\n\n```bibtex\n@article{LeCun1989,\n  title     = {Backpropagation Applied to Handwritten Zip Code Recognition},\n  author    = {Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackel},\n  journal   = {Neural Computation},\n  year      = {1989},\n  publisher = {MIT Press}\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraphaelsenn%2Flenet-1","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fraphaelsenn%2Flenet-1","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraphaelsenn%2Flenet-1/lists"}