{"id":21490020,"url":"https://github.com/aimaster-dev/image-compression-using-vae","last_synced_at":"2025-04-23T10:26:28.990Z","repository":{"id":254045944,"uuid":"845324667","full_name":"aimaster-dev/image-compression-using-vae","owner":"aimaster-dev","description":null,"archived":false,"fork":false,"pushed_at":"2024-08-21T03:09:18.000Z","size":4288,"stargazers_count":5,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-29T23:22:08.917Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aimaster-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-21T03:07:35.000Z","updated_at":"2025-01-10T20:42:04.000Z","dependencies_parsed_at":"2024-08-21T06:15:18.700Z","dependency_job_id":null,"html_url":"https://github.com/aimaster-dev/image-compression-using-vae","commit_stats":null,"previous_names":["aimaster-dev/image-compression-using-vae"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aimaster-dev%2Fimage-compression-using-vae","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aimaster-dev%2Fimage-compression-using-vae/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aimaster-dev%2Fimage-compression-using-vae/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aimaster-dev%2Fimage-compression-using-vae/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aimaster-dev","download_url":"https://codeload.github.com/aimaster-dev/image-compression-using-vae/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250414672,"owners_count":21426633,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-23T14:29:45.465Z","updated_at":"2025-04-23T10:26:28.966Z","avatar_url":"https://github.com/aimaster-dev.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Lossy Image Compression using Hierarchical VAEs\n\nThis repository contains the authors' implementation of several deep learning-based methods related to lossy image compression.\n\n- [Models](#models)\n- [Results](#results)\n- [Install](#install)\n- [Usage (compress, decompress, train, evaluation)](#usage)\n- [Licenses](#license)\n\n## Models\n### Implemented Methods (Pre-Trained Models Available)\n- **Lossy Image Compression with Quantized Hierarchical VAEs** [[arxiv](https://arxiv.org/abs/2208.13056)] [[cvf](https://openaccess.thecvf.com/content/WACV2023/html/Duan_Lossy_Image_Compression_With_Quantized_Hierarchical_VAEs_WACV_2023_paper.html)] [[ieee](https://ieeexplore.ieee.org/document/10030851)]\n    - Published at WACV 2023,[ ***Best Algorithms Paper Award***](https://wacv2023.thecvf.com/node/174)\n    - Abstract: a 12-layer VAE model named QRes-VAE. Good compression performance.\n    - \\[Code \u0026 pre-trained models\\]: [lossy-vae/lvae/models/qres](lvae/models/qresvae)\n- **QARV: Quantization-Aware ResNet VAE for Lossy Image Compression** [[arxiv](https://arxiv.org/abs/2302.08899)] [[ieee](https://ieeexplore.ieee.org/document/10274142)]\n    - Published at TPAMI 2023\n    - Abstract: an improved version of the previous model; **Variable rate, faster decoding, better performance.**\n    - \\[Code \u0026 pre-trained models\\]: [lossy-vae/lvae/models/qarv](lvae/models/qarv)\n- **An Improved Upper Bound on the Rate-Distortion Function of Images** [[arxiv](https://arxiv.org/abs/2309.02574)] [[ieee](https://ieeexplore.ieee.org/document/10222199)]\n    - Published at ICIP 2023\n    - Abstract: a 15-layer VAE model used to estimate the information R(D) function. This model proves that -30% BD-rate w.r.t. VTM is theoretically achievable.\n    - \\[Code \u0026 pre-trained models\\]: [lossy-vae/lvae/models/rd](lvae/models/rd)\n\n### Features\n**Progressive coding:** our models learn *a deep hierarchy of* latent variables and compress/decompress images in a *coarse-to-fine* fashion. This feature comes from the hierarchical nature of ResNet VAEs.\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/24869582/187014268-405851e8-b8a5-47e3-b28d-7b5d4ac20316.png\" width=\"756\" height=\"300\"\u003e\n\u003c/p\u003e\n\n**Compression performance**: our models are powerful in terms of both rate-distortion and decoding speed. Please see the results section below.\n\n\n## Results\n### Bpp-PSNR results in JSON format\n- Kodak images: [lossy-vae/results/kodak](results/kodak)\n- Tecknick TESTIMAGES RGB 1200x1200: [lossy-vae/results/tecnick-rgb-1200](results/tecnick-rgb-1200)\n- CLIC 2022 test set: [lossy-vae/results/clic2022-test](results/clic2022-test)\n\nNotes on metric computation:\n- Bpp and PSNR are first computed for each image and then averaged over all images in a dataset.\n- Bpp is the saved file size (in bits) divided by # of image pixels.\n- PSNR is computed in RGB space (not YUV).\n\n### Encoding/decoding latency on CPU/GPU, and BD-rate\n\u003cdiv align=\"center\"\u003e\n\n| Model Name  | CPU* Enc. | CPU* Dec. | 3080 ti Enc. | 3080 ti Dec. | BD-rate* (lower is better) |\n| :---------: | :-------: | :-------: | :----------: | :----------: | :------------------------: |\n|  `qres34m`  |  0.899s   |  0.441s   |    0.116s    |    0.083s    |          -3.95 %           |\n| `qarv_base` |  0.757s   |  0.295s   |    0.096s    |    0.063s    |          -7.26 %           |\n\n\u003c/div\u003e\n\n*Time is the latency to encode/decode a 512x768 image, averaged over 24 Kodak images. Tested in plain PyTorch (v1.13 + CUDA 11.7) code, ie, no mixed-precision, torchscript, ONNX/TensorRT, etc. \\\n*CPU is Intel 10700k. \\\n*BD-rate is w.r.t. [VTM 18.0](https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/-/tree/VTM-18.0), averaged on three common test sets (Kodak, Tecnick TESTIMAGES, and CLIC 2022 test set).\n\n\n\n## Install\n**Requirements**:\n- Python\n- PyTorch \u003e= 1.9 : https://pytorch.org/get-started/locally\n- tqdm : `conda install tqdm`\n- CompressAI : https://github.com/InterDigitalInc/CompressAI\n- **timm \u003e= 0.8.0** : https://github.com/huggingface/pytorch-image-models\n\n**Download and Install**:\n1. Download the repository;\n2. Modify the dataset paths in `lossy-vae/lvae/paths.py`.\n3. [Optional] pip install the repository in development mode:\n```\ncd /pasth/to/lossy-vae\npython -m pip install -e .\n```\n\n\n## Usage\n### Get pre-trained weights\n```python\nfrom lvae import get_model\nmodel = get_model('qarv_base', pretrained=True) # weights are downloaded automatically\nmodel.eval()\nmodel.compress_mode(True) # initialize entropy coding\n```\n\n### Compress images\nEncode an image:\n```python\nmodel.compress_file('/path/to/image.png', '/path/to/compressed.bits')\n```\n\nDecode an image:\n```python\nim = model.decompress_file('/path/to/compressed.bits')\n# im is a torch.Tensor of shape (1, 3, H, W). RGB. pixel values in [0, 1].\n```\n\n\n### Datasets\n**COCO**\n1. Download the COCO dataset \"2017 Train images [118K/18GB]\" from https://cocodataset.org/#download\n2. Unzip the images anywhere, e.g., at `/path/to/datasets/coco/train2017`\n3. Edit `lossy-vae/lvae/paths.py` such that\n```\nknown_datasets['coco-train2017'] = '/path/to/datasets/coco/train2017'\n```\n\n**Kodak** ([link](http://r0k.us/graphics/kodak)),\n**Tecnick TESTIMAGES** ([link](https://testimages.org/)),\nand **CLIC** ([link](http://compression.cc/))\n```\npython scripts/download-dataset.py --name kodak         --datasets_root /path/to/datasets\n                                          clic2022-test\n                                          tecnick\n```\nThen, edit `lossy-vae/lvae/paths.py` such that `known_datasets['kodak'] = '/path/to/datasets/kodak'`, and similarly for other datasets.\n\n**Custom Dataset**\n1. Prepare a folder containing images. The folder should contain only images (may contain subfolders).\n2. Edit `lossy-vae/lvae/paths.py` such that `known_datasets['custom-name'] = '/path/to/my-dataset'`, where `custom-name` is the name of your dataset, and `/path/to/my-dataset` is the path to the folder containing images.\n3. Then, you can use `custom-name` as the dataset name in the training/evaluation scripts.\n\n### Training and evaluation scripts\nTraining and evaluation scripts vary from model to model. For example, `qres34m` uses fixed-rate train/eval scheme, while `qarv_base` uses variable-rate train/eval scheme. \\\nDetailed training/evaluation instructions are provided in each model's subfolder (see the section [Models](#models)).\n\n\n## License\nCode in this repository is freely available for non-commercial use.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faimaster-dev%2Fimage-compression-using-vae","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faimaster-dev%2Fimage-compression-using-vae","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faimaster-dev%2Fimage-compression-using-vae/lists"}