{"id":21711855,"url":"https://github.com/gemarcano/fpga_resnet_compression","last_synced_at":"2025-03-20T18:22:03.587Z","repository":{"id":75914351,"uuid":"371255972","full_name":"gemarcano/fpga_resnet_compression","owner":"gemarcano","description":"CSE145/237D Project, evaluating the effectiveness of knowledge distillation for compressing ResNets","archived":false,"fork":false,"pushed_at":"2021-06-06T20:39:05.000Z","size":2046,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-25T16:44:44.789Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gemarcano.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-05-27T05:24:33.000Z","updated_at":"2023-06-25T14:18:15.000Z","dependencies_parsed_at":null,"dependency_job_id":"bd612730-1905-4515-8362-ae8a31e82283","html_url":"https://github.com/gemarcano/fpga_resnet_compression","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gemarcano%2Ffpga_resnet_compression","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gemarcano%2Ffpga_resnet_compression/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gemarcano%2Ffpga_resnet_compression/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gemarcano%2Ffpga_resnet_compression/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gemarcano","download_url":"https://codeload.github.com/gemarcano/fpga_resnet_compression/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244666788,"owners_count":20490314,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-25T23:30:09.940Z","updated_at":"2025-03-20T18:22:03.564Z","avatar_url":"https://github.com/gemarcano.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Compressing neural networks for use in FPGAs\n\n## Introduction\n\nArtificial neural networks (ANNs) are being deployed to solve difficult real-world problems, including object identification in images and video, voice recognition, and autonomous driving. The problem with typical ANNs is that they require a lot of computational power to run, limiting their deployment to systems that can host power-hungry graphic processing units (GPUs). Field programmable gate arrays (FPGAs) could be used as an alternative to GPUs for running ANNs with lower power requirements, but FPGAs tend to be resource constrained, making it difficult to port ANNs to them. One solution to this problem is improving neural network compression techniques to help downsize larger neural networks to fit into FPGAs.\n\nThis project explores the use of a specific algorithm to help compress residual neural networks (ResNets). Specifically, ResNets have skip connections between their layers which improve the network's capacity to converge on a solution, but adds complexity to FPGA implementations. Prior work done at UC San Diego began to explore the effects of removing skip connections and quantization of network weights on classification accuracy. We build off that work and evaluate the impact on classification accuracy of a special application of knowledge distillation to train ResNets with skip connections removed iteratively, and of quantizing the network with different parameters.\n\nOur implementation leverages Keras and QKeras to implement the neural networks and related training routines, and hls4ml and Vivado HLS to convert ResNets and their skip-less variants to FPGA bitstreams. We trained our networks using the CIFAR10 dataset. Our initial results indicate that there is little to no difference between the iterative approach for removing skips versus training the skip-less network normally. We hypothesize that this difference in results is due to a limitation in Keras, specifically of Tensorflow, that prevented us from implementing the iterative approach in a single training pass, as was the case for prior work. We suspect that prior work might have relied in the optimizer state adjusting to the skips being iteratively removed mid-training in order to achieve better results that we have. Nonetheless, we demonstrate that knowledge distillation approaches yield better Top 1 classification accuracy compared to regular training for larger ResNets on CIFAR10.\n\n## Team\n\n - Doheon Lee - Senior undergraduate student\n - Gabriel Marcano - PhD student\n\nThis project was performed for CSE 145/237D under the direction of Ryan Kastner, PhD., and in collaboration with Olivia Weng and Alireza Khodamoradi.\n\n## Layout\n\nThis repository contains artifacts related to the project and presentations. Contact Gabriel Marcano via gmarcano@ucsd.edu to gain access to the code repository. For those people with access, the code can be found on [GitLab here](https://gitlab.com/gemarcano/fpga_skip_hls) and it is mirrored on [Github here](https://github.com/gemarcano/fpga_resnet_skip).\n\nThe `Documents` folder contains all of the reports made for the project. The project specification outlines the initial idea of the project, which evolved as we gained a better understanding over machine learning and as we implemented the iterative skip removal algorithm for compressing ResNets. The second document, the milestone document, outlines the updated goals for our project. The final report will be placed here when completed, and will contain a summary of all of our work this quarter, our results, and discussions about why our data disagrees with prior work, as well as possible remedies and work to continue.\n\nThe `Presentations` folder contains all of the presentations made for the project, along with our speaker notes. The project overview presentation was prepared at the beginning of the quarter and was meant to be a pitch motivating our project. The mid quarter presentation shows the initial work and results as of the middle of the academic quarter. The final presentation and video will be posted here once they are completed.\n\n## Additional details\n\nThe following figure is a sketch showing a skip or residual connection skipping over some of the network's layers and recombining with the output of the layers.\n\n![ResNet skip example](./Pictures/ResNet_skip.png)\n\nResNets consist of many of these blocks connected in sequence. For example, ResNet20 contains 9 skip connections, while ResNet110 contains 54.\n\nPer the original paper describing ResNets [1], residual connections serve to help deep networks converge to a solution. As skip connections are removed, training the modified ResNets yields lower Top 1 classification accuracy. From our experiments, the deeper the ResNet is, the worse the drop in the Top 1 classification accuracy.\n\nKnowledge distillation is an approach to training neural networks that uses a teacher network to instruct a student model how to act [2]. The teacher model is static and does not change, while the student model updates its weights as the training progresses in an attempt to match the output of the teacher. From our data, smaller ResNets with all skip connections removed trained via knowledge distillation have similar Top 1 accuracy as the same network trained normally, but for much larger networks, like ResNet110, the knowledge distillation approach yields significantly better inaccuracies compared to training the network normally. Refer to our final report for more details.\n\nWe are using the Keras framework, on top of Tensorflow, to implement our ResNets, as it is possible to use HLS4ML to convert these models to FPGA bitstreams.\n\n## Future work\n\nAll of our work was done with the CIFAR10 dataset, so our results would have to be verified against other datasets to ensure that they are generalizable. As our work and prior work disagree about the efficacy of the iterative skip removal approach, we need to perform more experimentation and analysis to understand what factors lead to this difference.\n\n## References, and additional resources\n\nWe used the following resources and papers to learn about neural networks and FPGAs:\n\n 1. \"Deep Residual Learning for Image Recognition\", by He, et al.\n 2. \"Distilling the Knowledge in a Neural Network\", by Hinton, et al.\n 3. https://keras.io/examples/vision/knowledge_distillation/\n 4. https://pp4fpgas.readthedocs.io/en/latest/\n 5. \"Deep Learning with Python\", by François Chollet\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgemarcano%2Ffpga_resnet_compression","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgemarcano%2Ffpga_resnet_compression","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgemarcano%2Ffpga_resnet_compression/lists"}