{"id":47329512,"url":"https://github.com/d-becking/nncodec2","last_synced_at":"2026-03-17T20:02:14.555Z","repository":{"id":300991741,"uuid":"1007198156","full_name":"d-becking/nncodec2","owner":"d-becking","description":"Fraunhofer HHI implementation of the Neural Network Coding (NNC) Standard","archived":false,"fork":false,"pushed_at":"2025-07-03T23:20:14.000Z","size":7476,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-10-27T17:33:48.410Z","etag":null,"topics":["cabac","efficient-deep-learning","entropy-coding","neural-network-compression","nnc","nncodec"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/d-becking.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.md","dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-23T16:00:00.000Z","updated_at":"2025-08-15T14:28:30.000Z","dependencies_parsed_at":"2025-06-24T16:47:10.847Z","dependency_job_id":null,"html_url":"https://github.com/d-becking/nncodec2","commit_stats":null,"previous_names":["d-becking/nncodec2"],"tags_count":15,"template":false,"template_full_name":null,"purl":"pkg:github/d-becking/nncodec2","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d-becking%2Fnncodec2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d-becking%2Fnncodec2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d-becking%2Fnncodec2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d-becking%2Fnncodec2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/d-becking","download_url":"https://codeload.github.com/d-becking/nncodec2/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d-becking%2Fnncodec2/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30630026,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-17T17:32:55.572Z","status":"ssl_error","status_checked_at":"2026-03-17T17:32:38.732Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cabac","efficient-deep-learning","entropy-coding","neural-network-compression","nnc","nncodec"],"created_at":"2026-03-17T20:02:13.568Z","updated_at":"2026-03-17T20:02:14.544Z","avatar_url":"https://github.com/d-becking.png","language":"Python","readme":"\u003cdiv align=\"center\"\u003e\n\n\u003cimg src=\"https://github.com/fraunhoferhhi/nncodec/assets/65648299/69b41b38-19ed-4c45-86aa-2b2cd4d835f7\" width=\"660\"/\u003e\n\n# A Software Implementation of the ISO/IEC 15938-17 Neural Network Coding (NNC) Standard\n\n\u003c/div\u003e\n\n\n## Table of Contents\n\n- [Information](#information)\n- [Quick Start](#quick-start)\n- [Installation](#installation)\n- [NNCodec Usage](#nncodec-usage)\n  - [Tensor Coding](#coding-tensors-in-ai-based-media-processing-taimp)\n  - [Neural Network Coding](#coding-neural-networks-and-neural-network-updates)\n  - [Federated Learning](#federated-learning-with-nncodec)\n  - [Paper Results](#paper-results)\n- [Citation and Publications](#citation-and-publications)\n- [License](#license)\n\n## Information\n\nThis repository hosts a beta version of NNCodec 2.0, which incorporates new compression tools for incremental neural \nnetwork data, as introduced in the second edition of the NNC standard. It also supports coding \n\"Tensors in AI-based Media Processing\" (TAIMP), addressing recent MPEG requirements for coding individual tensors rather \nthan entire neural networks or differential updates to a base neural network.\n\nThe repository also includes a novel use case demonstrating federated learning (FL) for tiny language models in \ntelecommunications.\n\nThe official NNCodec 1.0 git repository, which served as the foundation for this project, can be found here:\n\n[![Conference](https://img.shields.io/badge/fraunhoferhhi-nncodec-green)](https://github.com/fraunhoferhhi/nncodec)\n\nIt also contains a [Wiki-Page](https://github.com/fraunhoferhhi/nncodec/wiki) providing further information on NNCodec. \n\nUpon approval, this second version will update the official git repository.\n\n### The Fraunhofer Neural Network Encoder/Decoder (NNCodec)\n\nNNCodec is an efficient implementation of NNC ([Neural Network Coding ISO/IEC 15938-17](https://www.iso.org/standard/85545.html)), \nthe first international standard for compressing (incremental) neural network data.\nIt provides the following main features:\n\n- Standard-compliant encoder/decoder including, e.g., DeepCABAC, quantization, and sparsification\n- Built-in support for common deep learning frameworks (e.g., PyTorch)\n- Integrated support for data-driven compression tools on common datasets (ImageNet, CIFAR, PascalVOC)\n- Federated AI support via [*Flower*](https://flower.ai), a prominent and widely used framework\n- Compression pipelines for:\n  - Neural Networks (NN)\n  - Tensors (TAIMP)\n  - Federated Learning (FL)\n\n## Quick Start\n\nInstall and run a tensor compression example:\n\n```bash\npip install nncodec\npython example/tensor_coding.py\n```\n\n## Installation\n\n### Requirements\n\n- Python \u003e= 3.8 with working pip\n- Windows: Visual Studio 2015 Update 3 or later\n\n### Package Installation from PyPI\n\nNNCodec 2.0 supports pip installation:\n```bash\npip install nncodec\n```\nwill install packages from  `install_requires` list in [setup.py](https://github.com/d-becking/nncodec2/blob/master/setup.py) \n\n### Package Installation from Source\nTo install NNCodec from source, we recommend creating a virtual Python environment and install via pip from the root of \nthe cloned repository:\n\n```bash\npython3 -m venv env\nsource env/bin/activate\npip install --upgrade pip\npip install -e .\n```\n\n## NNCodec Usage\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"https://github.com/user-attachments/assets/564b9d02-a706-459a-a8bb-241d2ec4608f\" width=\"660\"/\u003e\n\u003c/div\u003e\n\nNNCodec 2.0, as depicted above, includes three main pipelines:\n- One for tensorial data in AI-based media processing (e.g., function coefficients, feature maps, ...),\n  ```python\n  from nncodec.tensor import encode, decode\n  ```\n- one for coding entire neural networks (or their differential updates), and\n  ```python\n  from nncodec.nn import encode, decode\n  ```\n- one for federated learning scenarios.\n  ```python\n  from nncodec.fl import NNClient, NNCFedAvg\n  ```\n\n### Coding Tensors in AI-based Media Processing (TAIMP)\n\nThe [tensor_coding.py](https://github.com/d-becking/nncodec2/blob/master/example/tensor_coding.py) script provides\nencoding and decoding examples of random tensors.\nThe first example codes a random _PyTorch_ tensor (which could also be an integer tensor or a _numpy_ array):\n\n```python\nexample_tensor = torch.randn(256, 64, 64) # torch.randint(0, 255, (3, 3, 32, 32)) # example 8-bit uint tensor\nbitstream = encode(example_tensor, args_dict) \ndec_tensor = torch.tensor(decode(bitstream, args_dict[\"tensor_id\"]))\n```\nhere, `args_dict` is a python dictionary that specifies the encoding configuration. The default configuration is:\n\n```python\nargs_dict = { 'approx_method': 'uniform', # Quantization method ['uniform' or 'codebook']\n              'qp': -32, # main quantization parameter (QP)\n              'nonweight_qp': -75, # QP for non-weights, e.g., 1D or BatchNorm params (default: -75, i.e., fine quantization)\n              'use_dq': True, # enables dependent scalar / Trellis-coded quantization\n              'bitdepth': None, # Optional: integer-aligned bitdepth for limited precision [1, 31] bit; note: overwrites QPs.\n              'quantize_only': False, # if True encode() returns quantized parameter instead of bitstream \n              'tca': False, # enables Temporal Context Adaptation (TCA)\n              'row_skipping': True, # enables skipping tensor rows from arithmetic coding if entirely zero\n              'sparsity': 0.0, # introduces mean- \u0026 std-based unstructured sparsity [0.0, 1.0]\n              'struct_spars_factor': 0.0, # introduces structured per-channel sparsity (based on channel means); requires sparsity \u003e 0.0\n              'job_identifier': 'TAIMP_coding', # Name extension for generated *.nnc bitstream files and for logging\n              'results': '.', # path where results / bitstreams shall be stored\n              'tensor_id': '0', # identifier for tensor\n              'tensor_path': None, # path to tensor to be encoded\n              'compress_differences': False, # if True bitstream represents a differential update of a base tensor; set automatically if TCA enabled\n              'verbose': True # print stdout process information.\n             }\n```\n\nAn exemplary minimal config:\n\n```python\nargs_dict = {\n  'approx_method': 'uniform', 'bitdepth': 4, 'use_dq': True, 'sparsity': 0.5, 'struct_spars_factor': 0.9, 'tensor_id': '0'\n}\n```\n\nThe second example targets incremental tensor coding with the coding tool Temporal Context Adaptation (TCA). \nRunning tensor_coding.py with `--incremental` updates 50% of the example tensor's elements for `num_increments`\niterations and stores the previously decoded, co-located tensor in `approx_param_base`.\n\n`approx_param_base` must be initialized with \n```python\napprox_param_base = {\"parameters\": {}, \"put_node_depth\": {}, \"device_id\": 0, \"parameter_id\": {}} \n```\n\n### Coding Neural Networks and Neural Network Updates\n\nThe [nn_coding.py](https://github.com/d-becking/nncodec2/blob/master/example/nn_coding.py) script provides\nencoding and decoding examples of entire neural networks (NN) (`--uc=0`), incremental full NN (`--uc=1`) \nand incremental differential dNN (`--uc=2`).\n\n**Minimal example:** In its most simple form, an NN's parameters can be represented as a python dictionary of float32 or int32 numpy arrays:\n```python\nfrom nncodec.nn import encode, decode\nimport numpy as np\n\nmodel = {f\"parameter_{i}\": np.random.randn(np.random.randint(1, 36), \n                                           np.random.randint(1, 303)).astype(np.float32) for i in range(5)}\nbitstream = encode(model)\nrec_mdl_params = decode(bitstream)\n```\nHyperparameters can be inserted (like in the `nncodec.tensor` pipeline above) by passing an `args_dict`\nto `encode()` containing one or more configurations, e.g.,\n\n```python\nbitstream = encode(model, args={'qp': -24, 'use_dq': True, 'sparsity': 0.4})\n```\nor instead of a `qp` also a `bitdepth` can be used:\n```python\nbitstream = encode(model, args={'bitdepth': 4, 'use_dq': True, 'sparsity': 0.4})\n```\n\n**Example CLI:** For coding an actual NN, we included a _ResNet-56_ model pre-trained on _CIFAR-100_. Additionally, all \n_torchvision_ models can be coded out of the box. To see a list of all available models, execute:\n```bash\npython example/nn_coding.py --help\n```\n\nThe following example codes the _mobilenet_v2_ model from torchvision:\n```bash\npython example/nn_coding.py --model=mobilenet_v2\n```\n\nThe following example codes _ResNet-56_ and tests the model's performance afterward:\n```bash\npython example/nn_coding.py --dataset_path=\u003cyour_path\u003e --dataset=cifar100 --model=resnet56 --model_path=./models/ResNet56_CIF100.pt \n```\n\nTraining a randomly initialized _ResNet-56_ from scratch and code the incremental updates with temporal context adaptation (TCA) is achieved by:\n```bash\npython example/nn_coding.py --uc=1 --dataset_path=\u003cyour_path\u003e --model=resnet56 --model_rand_int --dataset=cifar100 --tca\n```\nFor **coding incremental differences** with respect to the base model, i.e., \u003cimg src=\"https://latex.codecogs.com/svg.image?\\Delta NN^{(e=1)} = NN^{(e=1)} - NN^{(e=2)}\" alt=\"dNN\"/\u003e,\nset `--uc=2`. \n\n`--max_batches` can be used to decrease the number of batches used per train epoch.\nOther available hyperparameters and coding tools like `--sparsity`, `--use_dq`, `--opt_qp`, `--bitdepth`, `--approx_method=codebook`, and others are described in [nn_coding.py](https://github.com/d-becking/nncodec2/blob/master/example/nn_coding.py).\n\n\n\n### Federated Learning with NNCodec\n\nThe [nnc_fl.py](https://github.com/d-becking/nncodec2/blob/master/example/nnc_fl.py) file implements a base script for communication-efficient\nFederated Learning with NNCodec. It imports the `NNClient` and `NNCFedAvg` classes — specialized NNC-[*Flower*](https://flower.ai) objects — that \nare responsible for establishing and handling the compressed FL environment.\n\n#### Important: Install Flower before using, e.g., by issuing:\n```bash\npip install -U \"flwr[simulation]\u003e=1.5\"\n```\n\nThe default configuration launches FL with two _ResNet-56_ clients learning the _CIFAR-100_ classification task. The _CIFAR_ dataset\nis automatically downloaded if not available under `--dataset_path` (~170MB).\n```bash\npython example/nnc_fl.py --dataset_path=\u003cyour_path\u003e --model_rand_int --epochs=30 --compress_upstream --compress_downstream --err_accumulation --compress_differences\n```\n\nMain coding tools and hyperparameter settings for coding are:\n```bash\n--qp   'Quantization parameter (QP) for NNs (default: -32)'\n--diff_qp   'Quantization parameter for dNNs. Defaults to QP if unspecified (default: None)'\n--nonweight_qp  'QP for non-weights, e.g., 1D or BatchNorm params (default: -75)'\n--opt_qp  'Enables layer-wise QP modification based on relative layer size within NN'\n--use_dq  'Enables dependent scalar / Trellis-coded quantization'\n--bitdepth 'Optional: integer-aligned bitdepth for limited precision [1, 31] bit; note: overwrites QPs.'\n--bnf  'Enables incremental BatchNorm Folding (BNF)'\n--sparsity  'Introduces mean- \u0026 std-based unstructured sparsity [0.0, 1.0] (default: 0.0)'\n--struct_spars_factor 'Introduces structured per-channel sparsity (based on channel means); requires sparsity \u003e 0 (default: 0.9)'\n--row_skipping  'Enables skipping tensor rows from arithmetic coding that are entirely zero'\n--tca 'Enables Temporal Context Adaptation (TCA)'\n```\n\nAdditional important hyperparameters for FL (among others in [nnc_fl.py](https://github.com/d-becking/nncodec2/blob/master/example/nnc_fl.py)):\n```bash\n--compress_differences  'Weight differences wrt. to base model (dNN) are compressed, otherwise full base models (NN) are communicated'\n--model_rand_int 'If set, model is randomly initialized, i.e., w/o loading pre-trained weights'\n--num_clients 'Number of clients in FL scenario (default: 2)'\n--compress_upstream 'Compression of clients-to-server communication'\n--compress_downstream 'Compression of server-to-clients communication'\n--err_accumulation  'If set, quantization errors are locally accumulated (\"residuals\") and added to NN update prior to compression'\n```\nSection [Paper results](#EuCNC-2025-Poster-Session) (EuCNC) below introduces an additional use case and implementation of NNCodec 2.0 FL with tiny language models collaboratively learning feature predictions in cellular data.\n\n### Logging results using Weights \u0026 Biases\n\nWe used Weights \u0026 Biases (wandb) for experimental results logging. Enable `--wandb` if you want to use it. Add your wandb key and optionally an experiment identifier for the run:\n\n```bash\n--wandb --wandb_key=\"my_key\" --wandb_run_name=\"my_project\"\n```\n\n#### Important: Install wandb before using, e.g., by issuing:\n```bash\npip install wandb\n```\n\n\n## Paper results\n\n\n- ### EuCNC 2025 Poster Session  \n  [![Conference](https://img.shields.io/badge/EuCNC-Paper-blue)](https://arxiv.org/abs/2504.01947)\n\n  We presented **\"Efficient Federated Learning Tiny Language Models for Mobile Network Feature Prediction\"** at the Poster Session I of the 2025 Joint European Conference on Networks and Communications \u0026 6G Summit (EuCNC/6G Summit).\n    \n  **TL;DR** -  This work introduces a communication-efficient Federated Learning (FL) framework for training tiny language models (TLMs) that collaboratively learn to predict mobile network features (such as ping, SNR or frequency band) across five geographically distinct regions from the Berlin V2X dataset. Using NNCodec, the framework reduces communication overhead by over 99% with minimal performance degradation, enabling scalable FL deployment across autonomous mobile network cells.\n  \u003cimg src=\"https://github.com/user-attachments/assets/4fba1aca-50ca-492f-901b-d601cc20874c\" width=\"750\" /\u003e \u003cbr\u003e\n\n  To reproduce the experimental results and evaluate NNCodec in the telco FL setting described above, execute:\n\n  ```bash\n  python example/nnc_fl.py --dataset=V2X --dataset_path=\u003cyour_path\u003e/v2x --model=tinyllama --model_rand_int \\\n  --num_clients=5 --epochs=30 --compress_upstream --compress_downstream --err_accumulation --compress_differences \\\n  --qp=-18 --batch_size=8 --max_batches=300 --max_batches_test=150 --sparsity=0.8 --struct_spars_factor=0.9 \\\n  --TLM_size=1 --tca --tokenizer_path=./example/tokenizer/telko_tokenizer.model\n  ```\n\n  The pre-tokenized [Berlin V2X dataset](https://ieee-dataport.org/open-access/berlin-v2x) can be downloaded here: https://datacloud.hhi.fraunhofer.de/s/CcAeHRoWRqe5PiQ\n  and the pre-trained Sentencepiece Tokenizer is included in this repository at [telko_tokenizer.model](https://github.com/d-becking/nncodec2/blob/master/example/tokenizer/).\n  \n  Resulting bitstreams and the best performing global TLM of all communication rounds will be stored in a `results` directory (with path set via `--results`).\n  To evaluate this model, execute:\n\n  ```bash\n  python example/eval.py --model_path=\u003cyour_path\u003e/best_tinyllama_.pt --batch_size=1 --dataset=V2X \\\n  --dataset_path=\u003cyour_path\u003e/v2x --model=tinyllama --TLM_size=1 --tokenizer_path=./example/tokenizer/telko_tokenizer.model\n  ```\n\n\n- ### ICML 2023 Neural Compression Workshop\n  [![Conference](https://img.shields.io/badge/ICML-Paper-blue)](https://openreview.net/forum?id=5VgMDKUgX0)\n\n  Our paper titled **\"NNCodec: An Open Source Software Implementation of the Neural Network Coding \n  ISO/IEC Standard\"** was awarded a Spotlight Paper at the ICML 2023 Neural Compression Workshop.\n\n    **TL;DR** -  The paper presents NNCodec 1.0, analyses its coding tools with respect to the principles of information theory and gives comparative results for a broad range of neural network architectures.\n    The code for reproducing the experimental results of the paper and a software demo are available \nhere:\n  \n  [![Conference](https://img.shields.io/badge/ICML-Code-red)](https://github.com/d-becking/nncodec-icml-2023-demo)\n    \n\n\n## Citation and Publications\nIf you use NNCodec in your work, please cite:\n```\n@inproceedings{becking2023nncodec,\ntitle={{NNC}odec: An Open Source Software Implementation of the Neural Network Coding {ISO}/{IEC} Standard},\nauthor={Daniel Becking and Paul Haase and Heiner Kirchhoffer and Karsten M{\\\"u}ller and Wojciech Samek and Detlev Marpe},\nbooktitle={ICML 2023 Workshop Neural Compression: From Information Theory to Applications},\nyear={2023},\nurl={https://openreview.net/forum?id=5VgMDKUgX0}\n}\n```\n### Additional Publications (chronological order)\n- D. Becking et al., **\"Neural Network Coding of Difference Updates for Efficient Distributed Learning Communication\"**, IEEE Transactions on Multimedia, vol. 26, pp. 6848–6863, 2024, doi: 10.1109/TMM.2024.3357198, Open Access\n- H. Kirchhoffer et al. **\"Overview of the Neural Network Compression and Representation (NNR) Standard\"**, IEEE Transactions on Circuits and Systems for Video Technology, pp. 1-14, July 2021, doi: 10.1109/TCSVT.2021.3095970, Open Access\n- P. Haase et al. **\"Encoder Optimizations For The NNR Standard On Neural Network Compression\"**, 2021 IEEE International Conference on Image Processing (ICIP), 2021, pp. 3522-3526, doi: 10.1109/ICIP42928.2021.9506655.\n- K. Müller et al. **\"Ein internationaler KI-Standard zur Kompression Neuronaler Netze\"**, FKT- Fachzeitschrift für Fernsehen, Film und Elektronische Medien, pp. 33-36, September 2021\n- S. Wiedemann et al., **\"DeepCABAC: A universal compression algorithm for deep neural networks\"**, in IEEE Journal of Selected Topics in Signal Processing, doi: 10.1109/JSTSP.2020.2969554.\n\n## License\n\nPlease see [LICENSE.txt](./LICENSE.txt) file for the terms of the use of the contents of this repository.\n\nFor more information and bug reports, please contact: [nncodec@hhi.fraunhofer.de](mailto\\:nncodec@hhi.fraunhofer.de)\n\n**Copyright (c) 2019-2025, Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. \u0026 The NNCodec Authors.**\n\n**All rights reserved.**\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fd-becking%2Fnncodec2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fd-becking%2Fnncodec2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fd-becking%2Fnncodec2/lists"}