{"id":14958173,"url":"https://github.com/necla-ml/diff-jpeg","last_synced_at":"2025-10-24T07:30:21.322Z","repository":{"id":192056806,"uuid":"684170741","full_name":"necla-ml/Diff-JPEG","owner":"necla-ml","description":"Official and maintained implementation of the paper \"Differentiable JPEG: The Devil is in the Details\" [WACV 2024].","archived":false,"fork":false,"pushed_at":"2023-12-30T16:50:09.000Z","size":25227,"stargazers_count":83,"open_issues_count":1,"forks_count":5,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-09-26T14:22:44.039Z","etag":null,"topics":["differentiable-codec","differentiable-jpeg","image-codec","image-compression","jpeg","pytorch","ste","surrogate","wacv"],"latest_commit_sha":null,"homepage":"https://christophreich1996.github.io/differentiable_jpeg/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/necla-ml.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-08-28T15:45:48.000Z","updated_at":"2024-09-23T14:27:48.000Z","dependencies_parsed_at":"2023-09-02T10:07:35.267Z","dependency_job_id":"97c42cd8-354f-44c2-8133-933c858d8475","html_url":"https://github.com/necla-ml/Diff-JPEG","commit_stats":null,"previous_names":["necla-ml/diff-jpeg"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/necla-ml%2FDiff-JPEG","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/necla-ml%2FDiff-JPEG/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/necla-ml%2FDiff-JPEG/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/necla-ml%2FDiff-JPEG/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/necla-ml","download_url":"https://codeload.github.com/necla-ml/Diff-JPEG/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219866292,"owners_count":16555826,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["differentiable-codec","differentiable-jpeg","image-codec","image-compression","jpeg","pytorch","ste","surrogate","wacv"],"created_at":"2024-09-24T13:16:24.791Z","updated_at":"2025-10-24T07:30:14.922Z","avatar_url":"https://github.com/necla-ml.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Differentiable JPEG: The Devil is in the Details\n\n[![arXiv](https://img.shields.io/badge/cs.CV-arXiv%3A2309.06978-B31B1B.svg)](https://arxiv.org/abs/2309.06978)\n[![License](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause)\n[![Framework](https://img.shields.io/badge/PyTorch-%23EE4C2C.svg?\u0026logo=PyTorch\u0026logoColor=white)](https://pytorch.org/)\n\n**[Christoph Reich](https://christophreich1996.github.io)\n, [Biplob Debnath](https://www.nec-labs.com/research/integrated-systems/people/biplob-debnath/)\n, [Deep Patel](https://deepsworld.github.io)\n\u0026 [Srimat Chakradhar](https://www.nec-labs.com/research/integrated-systems/people/srimat-t-chakradhar/)**\u003cbr/\u003e\n\n## | [Project Page](https://christophreich1996.github.io/differentiable_jpeg/) | [Paper](https://arxiv.org/abs/2309.06978) | [Poster](https://christophreich1996.github.io/pdfs/WACV_Poster.pdf) | [Talk](https://christophreich1996.github.io/images/WACV_Talk_2.mp4) |\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"github/first_fig.png\"  alt=\"1\" width = 791px height = 176px \u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  This repository includes the \u003cb\u003eofficial\u003c/b\u003e and \u003cb\u003emaintained\u003c/b\u003e implementation of the differentiable JPEG approach proposed in the paper \u003ca href=\"https://arxiv.org/abs/2308.12116\"\u003eDifferentiable JPEG: The Devil is in the Details\u003c/a\u003e.\n\u003c/p\u003e\n\n## Abstract\n\n*JPEG remains one of the most widespread lossy image coding methods. However, the non-differentiable nature of JPEG\nrestricts the application in deep learning pipelines. Several differentiable approximations of JPEG have recently been\nproposed to address this issue. This paper conducts a comprehensive review of existing differentiable JPEG approaches\nand identifies critical details that have been missed by previous methods. To this end, we propose a novel\ndifferentiable JPEG approach, overcoming previous limitations. Our approach is differentiable w.r.t. the input image,\nthe JPEG quality, the quantization tables, and the color conversion parameters. We evaluate the forward and backward\nperformance of our differentiable JPEG approach against existing methods. Additionally, extensive ablations are\nperformed to evaluate crucial design choices. Our proposed differentiable JPEG resembles the (non-differentiable)\nreference implementation best, significantly surpassing the recent-best differentiable approach by 3.47dB (PSNR) \non average. For strong compression rates, we can even improve PSNR by 9.51dB. Strong adversarial attack results\nare yielded by our differentiable JPEG, demonstrating the effective gradient approximation.*\n\n**If you use our differentiable JPEG or find this research useful in your work, please cite our paper:**\n\n```bibtex\n@inproceedings{Reich2024,\n    author={Reich, Christoph and Debnath, Biplob and Patel, Deep and Chakradhar, Srimat},\n    title={{Differentiable JPEG: The Devil is in the Details}},\n    booktitle={{WACV}},\n    year={2024}\n}\n```\n\n## Installation\n\nOur differentiable JPEG implementation can be installed as a Python package by running:\n\n```shell script\npip install git+https://github.com/necla-ml/Diff-JPEG\n```\n\nAll dependencies are listed in [requirements.txt](requirements.txt).\n\n## Usage\n\nWe offer both a functional and class ([nn.Module](https://pytorch.org/docs/stable/generated/torch.nn.Module.html))\nimplementation of our differentiable JPEG approach. Note beyond the examples provided here we also have\nan [example.py](example.py) file.\n\nThe following example showcases the use of the functional implementation.\n\n```python\nimport torch\nimport torchvision\nfrom torch import Tensor\n\nfrom diff_jpeg import diff_jpeg_coding\n\n# Load test image and reshape to [B, 3, H, W]\nimage: Tensor = torchvision.io.read_image(\"test_images/test_image.png\").float()[None]\n# Init JPEG quality\njpeg_quality: Tensor = torch.tensor([2.0])\n# Perform differentiable JPEG coding\nimage_coded: Tensor = diff_jpeg_coding(image_rgb=image, jpeg_quality=jpeg_quality)\n```\n\nIn the following code example, the class ([nn.Module](https://pytorch.org/docs/stable/generated/torch.nn.Module.html))\nimplementation is used.\n\n```python\nimport torch\nimport torch.nn as nn\nimport torchvision\nfrom torch import Tensor\n\nfrom diff_jpeg import DiffJPEGCoding\n\n# Init module\ndiff_jpeg_coding_module: nn.Module = DiffJPEGCoding()\n# Load test image and reshape to [B, 3, H, W]\nimage: Tensor = torchvision.io.read_image(\"test_images/test_image.png\").float()[None]\n# Init JPEG quality\njpeg_quality: Tensor = torch.tensor([19.04])\n# Perform differentiable JPEG coding\nimage_coded: Tensor = diff_jpeg_coding_module(image_rgb=image, jpeg_quality=jpeg_quality)\n```\n\n### STE Variant\n\nTo utilize the proposed straight-through estimator (STE) variant just set the `ste: bool = True` parameter.\n\n```python\n# Perform differentiable JPEG coding\nimage_coded: Tensor = diff_jpeg_coding(image_rgb=image, jpeg_quality=jpeg_quality, ste=True)\n```\n\n```python\n# Init module\ndiff_jpeg_coding_module: nn.Module = DiffJPEGCoding(ste=True)\n```\n\n### Custom Quantization Tables\n\nBoth the `diff_jpeg_coding` function and the forward function of `DiffJPEGCoding` offer the option to use custom\nquantization tables. Just use the `quantization_table_y: Optional[Tensor]` and `quantization_table_c: Optional[Tensor]`\nparameter. Both parameters are required to be a `torch.Tensor` of the shape `[8, 8]`. If no quantization table is\ngiven (or set to `None`), the respective standard JPEG quantization tables are utilized.\n\nHere we provide two examples of using a custom quantization table.\n\n```python\nimport torch\nimport torchvision\nfrom torch import Tensor\n\nfrom diff_jpeg import diff_jpeg_coding\n\n# Load test image and reshape to [B, 3, H, W]\nimage: Tensor = torchvision.io.read_image(\"test_images/test_image.png\").float()[None]\n# Init JPEG quality\njpeg_quality: Tensor = torch.tensor([2.0])\n# Perform differentiable JPEG coding\nimage_coded: Tensor = diff_jpeg_coding(\n    image_rgb=image,\n    jpeg_quality=jpeg_quality,\n    quantization_table_y=torch.randint(low=1, high=256, size=(8, 8)),\n    quantization_table_c=torch.randint(low=1, high=256, size=(8, 8)),\n)\n```\n\n```python\nimport torch\nimport torch.nn as nn\nimport torchvision\nfrom torch import Tensor\n\nfrom diff_jpeg import DiffJPEGCoding\n\n# Init module\ndiff_jpeg_coding_module: nn.Module = DiffJPEGCoding()\n# Load test image and reshape to [B, 3, H, W]\nimage: Tensor = torchvision.io.read_image(\"test_images/test_image.png\").float()[None]\n# Init JPEG quality\njpeg_quality: Tensor = torch.tensor([19.04])\n# Perform differentiable JPEG coding\nimage_coded: Tensor = diff_jpeg_coding_module(\n    image_rgb=image,\n    jpeg_quality=jpeg_quality,\n    quantization_table_y=torch.randint(low=1, high=256, size=(8, 8)),\n    quantization_table_c=torch.randint(low=1, high=256, size=(8, 8)),\n)\n```\n\n## Issues\n\nIf you encounter any issues with this implementation please open a GitHub issue!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnecla-ml%2Fdiff-jpeg","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnecla-ml%2Fdiff-jpeg","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnecla-ml%2Fdiff-jpeg/lists"}