{"id":17429786,"url":"https://github.com/marph91/pocket-cnn","last_synced_at":"2025-09-04T02:37:38.551Z","repository":{"id":105040184,"uuid":"247461890","full_name":"marph91/pocket-cnn","owner":"marph91","description":"CNN-to-FPGA-framework for small CNN, written in VHDL and Python","archived":false,"fork":false,"pushed_at":"2021-06-08T17:51:10.000Z","size":7647,"stargazers_count":22,"open_issues_count":5,"forks_count":7,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-29T03:41:36.799Z","etag":null,"topics":["cnn","cnn-architecture","deep-learning","fpga","fully-convolutional-networks","hardware","image-processing","onnx","python","vhdl"],"latest_commit_sha":null,"homepage":"","language":"VHDL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/marph91.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-03-15T12:27:48.000Z","updated_at":"2025-03-19T02:35:55.000Z","dependencies_parsed_at":null,"dependency_job_id":"4abea056-cdbd-4e41-a6ce-7154b532dd29","html_url":"https://github.com/marph91/pocket-cnn","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marph91%2Fpocket-cnn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marph91%2Fpocket-cnn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marph91%2Fpocket-cnn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marph91%2Fpocket-cnn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/marph91","download_url":"https://codeload.github.com/marph91/pocket-cnn/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249165947,"owners_count":21223352,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cnn","cnn-architecture","deep-learning","fpga","fully-convolutional-networks","hardware","image-processing","onnx","python","vhdl"],"created_at":"2024-10-17T07:09:13.147Z","updated_at":"2025-04-15T22:31:01.636Z","avatar_url":"https://github.com/marph91.png","language":"VHDL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pocket-cnn\n\n[![testsuite](https://github.com/marph91/pocket-cnn/workflows/testsuite/badge.svg)](https://github.com/marph91/pocket-cnn/actions?query=workflow%3Atestsuite)\n[![codecov](https://codecov.io/gh/marph91/pocket-cnn/branch/master/graph/badge.svg)](https://codecov.io/gh/marph91/pocket-cnn)\n[![synthesis](https://github.com/marph91/pocket-cnn/workflows/synthesis/badge.svg)](https://github.com/marph91/pocket-cnn/actions?query=workflow%3Asynthesis)\n[![script_style](https://github.com/marph91/pocket-cnn/workflows/check_scripts/badge.svg)](https://github.com/marph91/pocket-cnn/actions?query=workflow%3Acheck_scripts)\n[![vhdl_style](https://github.com/marph91/pocket-cnn/workflows/vhdl_style/badge.svg)](https://github.com/marph91/pocket-cnn/actions?query=workflow%3Avhdl_style)\n\npocket-cnn is a framework to map small Convolutional Neural Networks (CNN) fully on a FPGA. There is no communication outside the FPGA needed, except of providing the image and reading the result.\n\nNote that weights and activations are fixed to 8 bit, due to ONNX limitations. For binary neural networks in hardware, you can check out [pocket-bnn](https://github.com/marph91/pocket-bnn).\n\n## Installation and Usage\n\nBefore using the framework, the `PYTHONPATH` has to be extended by `path/to/pocket-cnn/code/python_tools`. A complete end-to-end example can be found at the [example folder](examples/end_to_end/README.md).\n\nFor running the tests, [ghdl](https://github.com/ghdl/ghdl), [vunit](https://github.com/vunit/vunit), [onnx](https://github.com/onnx/onnx) and [fpbinary](https://github.com/smlgit/fpbinary) are required. For debugging, [gtkwave](https://github.com/gtkwave/gtkwave) and [netron](https://github.com/lutzroeder/netron) are useful. To run the full testsuite, simply execute:\n\n```bash\ncd code/vhdl/sim/vunit\npython3 run_all.py\n```\n\n### Workflows\n\nThis section describes the different workflows. In the end-to-end example, an ONNX model gets trained with pytorch, quantized and processed until it's ready for synthesis by pocket-cnn.\nThere are some other workflows. At first, the quantized ONNX model can be directly trained in the CNN framework. This requires that the framework supports the quantization of pocket-cnn. The rest of the workflow is the same as described above.\nAnother branch in the workflow is to create ONNX models manually. This doesn't have any relevance for real models, but is really useful for testing. I. e. small ONNX models can be created fastly without any training. This allows to test (i. e. simulate) multiple configurations extensively. The test models are defined in the [model zoo](code/python_tools/cnn_onnx/model_zoo.py).\n\n![workflow](doc/images/workflow.svg)\n\n### Supported layers\n\n| Layer | Properties | Limitations |\n| :---: | :--- | :--- |\n| Convolution | \u003cul\u003e\u003cli\u003eKernel: 1x1, 2x2, 3x3, 5x5\u003c/li\u003e\u003cli\u003eStride: 1, 2, 3\u003c/li\u003e\u003c/ul\u003e | Quantization of the activations and weights: Scale has to be power-of-two, zero point has to be zero. |\n| Maximum Pooling | \u003cul\u003e\u003cli\u003eKernel: 2x2, 3x3\u003c/li\u003e\u003cli\u003eStride: 1, 2, 3\u003c/li\u003e\u003c/ul\u003e | - |\n| Global Average Pooling | - | The averaging factor is quantized to the 16 bit fixed point value of `1 / height * width`. |\n| Zero Padding | - | The padding has to be the same at each edge. |\n| (Leaky) ReLU | - | Leaky ReLU has a fixed alpha of 0.125 |\n\n### Limitations\n\nBefore using the framework, you should be aware of several limitations:\n\n- It is not complete. There might be several bugs and things missing. Please open an issue.\n- There will be a different accuracy and loss due to the 8 bit quantization.\n- Only small CNN can be synthesized, because the weights get mapped to LUT.\n- Only a [subset of layers](#supported-layers) is supported.\n\n### Interface\n\nMost of the toplevel generics are describing the CNN architecture. They get derived from the ONNX model and don't need to be specified manually by the user. A table, containing the most important toplevel generics and signals, can be found [here](doc/toplevel_interface.md). The communication protocol is similar in all submodules of this design.\n\n## Architecture\n\npocket-cnn accepts ONNX models as input. Each convolution layer of the CNN gets converted to a processing element (PE). This is the central element of the hardware design.\nThe convolution in a PE can be preceeded by *zero padding* and followed by *ReLU* and/or *maximum pooling*. See also [pe.vhd](code/vhdl/src/pe.vhd). Below are a few common configurations.\n\n![processing_element](doc/images/processing_element.svg)\n\nThe python framework takes care of converting the ONNX model into the VHDL representation. The VHDL toplevel can be found at [top.vhd](code/vhdl/src/top.vhd). The toplevel structure is illustrated in the following image.\n\n![toplevel](doc/images/toplevel.svg)\n\nMore details about the specific modules can be found [here](doc/modules.md).\n\n## TODO\n\nCan be found at the [documentation folder](doc/todo.md) and in the issues.\n\n## Related work\n\n- Haddoc2: \u003chttps://github.com/DreamIP/haddoc2\u003e, \u003chttps://arxiv.org/pdf/1712.04322.pdf\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarph91%2Fpocket-cnn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarph91%2Fpocket-cnn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarph91%2Fpocket-cnn/lists"}