{"id":21135247,"url":"https://github.com/cloneofsimo/imagenet.int8","last_synced_at":"2025-10-12T16:26:28.057Z","repository":{"id":236422132,"uuid":"792581726","full_name":"cloneofsimo/imagenet.int8","owner":"cloneofsimo","description":null,"archived":false,"fork":false,"pushed_at":"2024-04-27T01:37:53.000Z","size":3365,"stargazers_count":37,"open_issues_count":3,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-27T01:11:11.168Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cloneofsimo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-27T01:37:48.000Z","updated_at":"2025-01-06T04:13:15.000Z","dependencies_parsed_at":"2024-04-27T02:42:16.774Z","dependency_job_id":null,"html_url":"https://github.com/cloneofsimo/imagenet.int8","commit_stats":null,"previous_names":["cloneofsimo/imagenet.int8"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloneofsimo%2Fimagenet.int8","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloneofsimo%2Fimagenet.int8/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloneofsimo%2Fimagenet.int8/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloneofsimo%2Fimagenet.int8/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cloneofsimo","download_url":"https://codeload.github.com/cloneofsimo/imagenet.int8/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248691910,"owners_count":21146466,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-20T06:46:54.532Z","updated_at":"2025-10-12T16:26:23.021Z","avatar_url":"https://github.com/cloneofsimo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Imagenet.int8: Entire Imagenet dataset in 5GB\n\n\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"contents/vae.png\" alt=\"small\" width=\"800\"\u003e\n\u003c/p\u003e\n\n*original, reconstructed from float16, reconstructed from uint8*\n\n\u003ca href='https://huggingface.co/datasets/cloneofsimo/imagenet.int8'\u003e\u003cimg src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Dataset-blue'\u003e\u003c/a\u003e\n\n\nFind 138 GB of imagenet dataset too bulky? Did you know entire imagenet actually just fits inside the ram of apple watch?\n\n* Resized, Center-croped to 256x256\n* VAE compressed with [SDXL's VAE](https://huggingface.co/stabilityai/sdxl-vae)\n* Further quantized to int8 near-lossless manner, compressing the entire training dataset of 1,281,167 images down to just 5GB!\n\nIntroducing Imagenet.int8, the new MNIST of 2024. After the great popularity of the [Latent Diffusion](https://arxiv.org/abs/2112.10752) (Thank you stable diffusion!), its *almost* the standard to use VAE version of the imagenet for diffusion-model training. As you might know, lot of great diffusion research is based on latent variation of the imagenet. \n\nThese include: \n\n* [DiT](https://arxiv.org/abs/2212.09748)\n* [Improving Traning Dynamics](https://arxiv.org/abs/2312.02696v1)\n* [SiT](https://arxiv.org/abs/2401.08740)\n* [U-ViT](https://openaccess.thecvf.com/content/CVPR2023/html/Bao_All_Are_Worth_Words_A_ViT_Backbone_for_Diffusion_Models_CVPR_2023_paper.html)\n* [Min-SNR](https://openaccess.thecvf.com/content/ICCV2023/html/Hang_Efficient_Diffusion_Training_via_Min-SNR_Weighting_Strategy_ICCV_2023_paper.html)\n* [MDT](https://openaccess.thecvf.com/content/ICCV2023/papers/Gao_Masked_Diffusion_Transformer_is_a_Strong_Image_Synthesizer_ICCV_2023_paper.pdf)\n\n... but so little material online on the actual preprocessed dataset. I'm here to fix that. One thing I noticed was that latent doesn't have to be full precision! Indeed, they can be as small as int-8, and it doesn't hurt!\n\nSo clearly, it doesn't make sense to download entire Imagenet and process with VAE everytime. Just download this, `to('cuda')` the entire dataset just to flex, and call it a day.😌\n\n(BTW If you think you'll need higher precision, you can always further fine-tune your model on higher precision. But I doubt that.)\n\n\n# How do I use this?\n\nFirst download this. You can use `huggingface-cli` for that. \n\n```bash\n# Pro tip : use `hf_transfer` to get faster download speed.\npip install hf_transfer\nexport HF_HUB_ENABLE_HF_TRANSFER=True\n# actual download script. \nhuggingface-cli download --repo-type dataset cloneofsimo/imagenet.int8 --local-dir ./vae_mds\n```\n\n\n\nThen, you need to install [streaming dataset](https://github.com/mosaicml/streaming) to use this. The dataset is MDS format.\n\n```bash\npip install mosaicml-streaming\n```\n\nThen, you can very simply use the dataset like this:\n\n(for more info on using Mosaic's StreamingDataset and MDS format, [reference here](https://docs.mosaicml.com/projects/streaming/en/stable/index.html))\n\n```python\nfrom streaming.base.format.mds.encodings import Encoding, _encodings\nimport numpy as np\nfrom typing import Any\nimport torch\nfrom streaming import StreamingDataset\n\nclass uint8(Encoding):\n    def encode(self, obj: Any) -\u003e bytes:\n        return obj.tobytes()\n\n    def decode(self, data: bytes) -\u003e Any:\n        x=  np.frombuffer(data, np.uint8).astype(np.float32)\n        return (x / 255.0 - 0.5) * 24.0\n\n_encodings[\"uint8\"] = uint8\n\nremote_train_dir = \"./vae_mds\" # this is the path you installed this dataset.\nlocal_train_dir = \"./local_train_dir\"\n\ntrain_dataset = StreamingDataset(\n    local=local_train_dir,\n    remote=remote_train_dir,\n    split=None,\n    shuffle=True,\n    shuffle_algo=\"naive\",\n    num_canonical_nodes=1,\n    batch_size = 32\n)\n\ntrain_dataloader = torch.utils.data.DataLoader(\n    train_dataset,\n    batch_size=32,\n    num_workers=3,\n)\n```\n\nBy default, batch will have three attributes: `vae_output`, `label`, `label_as_text`.\n\nThats the dataloader! Now, below is the example usage. Notice how you have to reshape the data back to `(B, 4, 32, 32)` as they are decoded flattened.\n\n```python\n###### Example Usage. Decode back the 5th image. BTW shuffle plz\nfrom diffusers.models import AutoencoderKL\nfrom diffusers.image_processor import VaeImageProcessor\n\nmodel = \"stabilityai/your-stable-diffusion-model\"\nvae = AutoencoderKL.from_pretrained(\"stabilityai/sdxl-vae\").to(\"cuda:0\")\n\nbatch = next(iter(train_dataloader))\n\ni = 5\nvae_latent = batch[\"vae_output\"].reshape(-1, 4, 32, 32)[i:i+1].cuda().float()\nidx = batch[\"label\"][i]\ntext_label = batch['label_as_text'][i]\n\nprint(f\"idx: {idx}, text_label: {text_label}, latent: {vae_latent.shape}\")\n# idx: 402, text_label: acoustic guitar, latent: torch.Size([1, 4, 32, 32])\n\n# example decoding\nx = vae.decode(vae_latent.cuda()).sample\nimg = VaeImageProcessor().postprocess(image = x.detach(), do_denormalize = [True, True])[0]\nimg.save(\"5th_image.png\")\n```\n\nEnjoy!\n\n# Citations\n\nIf you find this material helpful, consider citation!\n\n```bibtex\n@misc{imagenet_int8,\n  author       = {Simo Ryu},\n  title        = {Imagenet.int8: Entire Imagenet dataset in 5GB},\n  year         = 2024,\n  publisher    = {Hugging Face Datasets},\n  url          = {https://huggingface.co/datasets/cloneofsimo/imagenet.int8},\n  note         = {Entire Imagenet dataset compressed to 5GB using VAE and quantized with int8}\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloneofsimo%2Fimagenet.int8","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcloneofsimo%2Fimagenet.int8","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloneofsimo%2Fimagenet.int8/lists"}