{"id":29093295,"url":"https://github.com/nvidia-ai-iot/clip-distillation","last_synced_at":"2025-10-13T15:01:59.003Z","repository":{"id":185442387,"uuid":"659448336","full_name":"NVIDIA-AI-IOT/clip-distillation","owner":"NVIDIA-AI-IOT","description":"Zero-label image classification via OpenCLIP knowledge distillation","archived":false,"fork":false,"pushed_at":"2023-09-12T04:46:33.000Z","size":2129,"stargazers_count":92,"open_issues_count":4,"forks_count":14,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-05-14T00:14:41.453Z","etag":null,"topics":["clip","distillation","inference","jetson","knowledge","nvidia","qat","sparsity","tensorrt"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NVIDIA-AI-IOT.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-06-27T21:27:52.000Z","updated_at":"2024-05-07T08:48:22.000Z","dependencies_parsed_at":"2023-09-09T18:01:15.858Z","dependency_job_id":null,"html_url":"https://github.com/NVIDIA-AI-IOT/clip-distillation","commit_stats":null,"previous_names":["nvidia-ai-iot/clip-distillation"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/NVIDIA-AI-IOT/clip-distillation","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA-AI-IOT%2Fclip-distillation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA-AI-IOT%2Fclip-distillation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA-AI-IOT%2Fclip-distillation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA-AI-IOT%2Fclip-distillation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NVIDIA-AI-IOT","download_url":"https://codeload.github.com/NVIDIA-AI-IOT/clip-distillation/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA-AI-IOT%2Fclip-distillation/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262396521,"owners_count":23304448,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clip","distillation","inference","jetson","knowledge","nvidia","qat","sparsity","tensorrt"],"created_at":"2025-06-28T08:07:48.965Z","updated_at":"2025-10-13T15:01:58.883Z","avatar_url":"https://github.com/NVIDIA-AI-IOT.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CLIP Knowledge Distillation\n\n\n\u003cimg src=\"assets/overview.png\" height=\"320\"/\u003e\n\nThis repository contains code and instructions that enable you\nto create your own customized image classification models\nwith zero-labeled data, by performing knowledge distillation of OpenCLIP models.\n\nEven if you don't need an image classifier directly, you may find this project helpful as inspiration for how you can use knowledge distillation to optimized models for inference, or as an example of how to train models with quantization aware training and structured sparsity for inference on NVIDIA Jetson. \n\nThis project includes,\n\n1. Scripts to search and download relevant data from the LAION database to use for distillation\n2. Scripts to distil any OpenCLIP model to any Pytorch image models (timm) CNN model.\n    - Supports Quantization Aware Training (QAT) for downstream INT8 inference\n    - Supports training to enforce 2:4 structured sparsity with the ASP library\n3. Scripts to run inference with NVIDIA TensorRT\n    - Supports INT8 model\n    - Supports acceleration of 2:4 structured sparse models on certain NVIDIA Jetson platforms, like NVIDIA Jetson Orin Nano.\n\nTo get started, follow the instructions below.\n\n\u003e If you're new to the subject, check out our tutorial [jetson-intro-to-distillation](https://github.com/NVIDIA-AI-IOT/jetson-intro-to-distillation)\n\u003e for an introduction to knowledge distillation!\n\n\n## Instructions\n\n1. [Step 1 - Search and download relevant unlabeled images to use for distillation](#step-1)\n2. [Step 2 - Pre-compute OpenCLIP embeddings](#step-2)\n3. [Step 3 - Train the student CNN model to mimic the OpenCLIP model](#step-3)\n4. [Step 4 - Run inference using the distilled model](#step-4)\n5. [Step 5 (advanced) - Train a student model with structured sparsity](#step-5)\n6. [Step 6 (advanced) - Train a student with Quantization aware training and INT8 precision](#step-6)\n7. [Next Steps](#next-steps)\n\n\u003ca name=\"step-1\"\u003e\u003c/a\u003e\n\n## Step 1 - Search and download images with CLIP filtering\n\n### Search for relevant image URLs in the LAION database using CLIP filtering\n\nThe first thing we need to do when distilling a model, is obtain data to use for distillation. \n\nFor this task, we'll look for relevant images by searching the LAION database.  We've provided a script to make this simple.\n\nTo search for relevant images, first create a file ``data/text_prompts.txt`` with the text prompts to query.\n\nEach prompt should exist on it's own line.\n\n```txt\na dog\na cat\n```\n\nNext, call the script to query the images that match the text prompts.\n\n```bash\npython3 search_clip_images.py \\\n    \"data/text_prompts.txt\" \\\n    \"data/image_urls.txt\" \\\n    -n 5000 \\\n    -m 10000 \\\n    --max_workers 2 \\\n    --append\n```\n\nThis will output a file ``data/image_urls.txt`` that contains the URLs of images matching our text prompt queries.\n\nFor the full set of arguments please type\n\n```bash\npython3 search_clip_images.py --help\n```\n\n### Download images from URL file\n\nNow that we've found relevant images to use for distillation, we need to download them.\n\nTo do so, we call the following script to download images to an output folder.\n\n```bash\npython3 download_images.py \\\n    \"data/image_urls.txt\" \\\n    \"data/images\" \\\n    --max_workers 32 \\\n    --timeout 2\n```\n\nThis script will download images to the folder ``data/images``.  Each image\nwill be given a unique filename base on it's URL.  \n\nFor the full set of arguments please type\n\n```bash\npython3 download_images.py --help\n```\n\n\u003ca name=\"step-2\"\u003e\u003c/a\u003e\n\n## Step 2 - Compute OpenCLIP embeddings\n\nThe images we downloaded above will be used as inputs to our teacher and student models during distillation.  Unfortunately, it can be slow to execute the teacher during training.  \n\nTo speed up this process, we'll pre-compute the outputs of our teacher model so we don't need to execute the teacher model during training.\n\nTo do this, call the ``compute_openclip_embeddings.py`` script as follows,\n\n```bash\npython3 compute_openclip_embeddings.py \\\n    data/images \\\n    data/embeddings \\\n    --batch_size 16 \\\n    --num_workers 8 \\\n    --model_name ViT-B-32 \\\n    --pretrained laion2b_s34b_b79k\n```\n\nThis will write the ouptput embeddings to the folder ``data/embeddings``, with filenames that match the image filenames, except for the file extensions.\n\n\u003e Note: For available model names and pretrained weight identifiers please reference [OpenCLIP Repo](https://github.com/mlfoundations/open_clip/blob/fb72f4db1b17133befd6c67c9cf32a533b85a321/src/open_clip/pretrained.py#L227).\n\nFor the full set of arguments please type\n\n```bash\npython3 compute_openclip_embeddings.py --help\n```\n\n\u003ca name=\"step-3\"\u003e\u003c/a\u003e\n\n## Step 3 - Train the student CNN model to mimic the OpenCLIP model\n\nNow that we have the data to use for knowledge distillation, we can perform the distillation (student model training) by calling the ``distil_model_embeddings.py`` script as follows.\n\n```bash\npython3 distil_model_embeddings.py \\\n    resnet18 \\\n    data/images \\\n    data/embeddings \\\n    data/models/resnet18 \\\n    --output_dim 512 \\\n    --pretrained\n```\n\nThis will output model checkpoints and information to ``data/models/resnet18``.\n\nThe distilled model we use in this example is resnet18.  This model is highly optimized by TensorRT, and we can readily apply other optimizations like reduced precision and structured sparsity during training.  Please see the additional steps below for more information.\n\nFor the full set of arguments please type\n\n```bash\npython3 distil_model_embeddings.py --help\n```\n\n\n\u003ca name=\"step-4\"\u003e\u003c/a\u003e\n\n## Step 4 - Run inference using the distilled model\n\n### Compute text embeddings\n\nDuring distillation, we trained our student model to match the *features* of our open-clip model.  However, we're interested in creating a classification model.\n\nTo create the zero-shot classification model, we need to generate text embeddings from the text prompts that describe our class labels.\n\nTo do so, we use the pre-trained OpenCLIP text encoder.\n\nWe call the ``compute_openclip_text_embeddings.py`` script to create the text embeddings.\n\n```bash\npython3 compute_openclip_text_embeddings.py \\\n    data/text_prompts.txt \\\n    data/text_embeddings.npy \\\n    --model_name ViT-B-32\n```\n\nIn this instance, we used the same text prompts we used for image search as our text prompts for classification.  \n\n### Predict single image with PyTorch\n\nNow that we have computed th text prompts for our image classes, we can perform image classification with our PyTorch model as follows:\n\n```bash\npython3 predict_pytorch.py \\\n    resnet18 \\\n    data/models/resnet18/checkpoint.pth \\\n    data/text_embeddings.npy \\\n    assets/cat.jpg \\\n    --text_prompts data/text_prompts.txt\n```\n\n### Live demo with camera\n\nWe can similarily perform inference on a live camera feed as follows:\n\n```bash\npython3 demo_pytorch.py \\\n    resnet18 \\\n    data/models/resnet18/checkpoint.pth \\\n    data/text_embeddings.npy \\\n    --text_prompts data/text_prompts.txt \\\n    --camera_device 0\n```\n\n## Step 5 (advanced) - Train a student model with structured sparsity\n\nThe training script offers the ability to train for [structured sparsity](https://developer.nvidia.com/blog/accelerating-inference-with-sparsity-using-ampere-and-tensorrt/).  This can offer additional acceleration when deploying the model on applicable NVIDIA Jetson platforms with TensorRT.\n\n### Train the model with structured sparsity\n\n```bash\npython3 distil_model_embeddings.py \\\n    resnet18 \\\n    data/images \\\n    data/embeddings \\\n    data/models/resnet18_sparse \\\n    --output_dim 512 \\\n    --pretrained \\\n    --init_checkpoint data/models/resnet18/checkpoint.pth \\\n    --use_asp \\\n    --num_epochs 25\n```\n\n### Predict with PyTorch\n\n```bash\npython3 predict_pytorch.py \\\n    resnet18 \\\n    data/models/resnet18_sparse/checkpoint.pth \\\n    data/text_embeddings.npy \\\n    assets/cat.jpg \\\n    --text_prompts data/text_prompts.txt \\\n    --use_asp\n```\n\n### Demo with PyTorch\n\n```bash\npython3 demo_pytorch.py \\\n    resnet18 \\\n    data/models/resnet18_sparse/checkpoint.pth \\\n    data/text_embeddings.npy \\\n    --text_prompts data/text_prompts.txt \\\n    --camera_device 0 \\\n    --use_asp\n```\n\n### Export to ONNX\n\n```bash\npython3 export_onnx.py \\\n    resnet18 \\\n    data/models/resnet18_sparse/checkpoint.pth \\\n    data/onnx/resnet18_sparse.onnx \\\n    --use_asp\n```\n\n\n\u003ca name=\"step-6\"\u003e\u003c/a\u003e\n\n## Step 6 (advanced) - Train a student with Quantization aware training and INT8 precision\n\nIn addition to structured sparsity, another technique we can use for additional performance is by using reduced INT8 precision.  Quantization aware training is a technique to minimize quantization errors introduced when deploying with INT8 precision.  It does so by applying quantization during the model forward pass during training.  This allows the model to adapt to quantization errors during training.  It also allows us to avoid the need for calibration when using post-training quantization.\n\nTo distil the model with quantization aware training, follow theses steps\n\n### Train the model with quantization aware training (QAT)\n\n```bash\npython3 distil_model_embeddings.py \\\n    resnet18 \\\n    data/images \\\n    data/embeddings \\\n    data/models/resnet18_qat \\\n    --output_dim 512 \\\n    --pretrained \\\n    --init_checkpoint data/models/resnet18/checkpoint.pth \\\n    --use_qat \\\n    --num_epochs 25\n```\n\n### Predict with PyTorch\n\n```bash\npython3 predict_pytorch.py \\\n    resnet18 \\\n    data/models/resnet18_sparse/checkpoint.pth \\\n    data/text_embeddings.npy \\\n    assets/cat.jpg \\\n    --text_prompts data/text_prompts.txt \\\n    --use_qat\n```\n\n### Demo with PyTorch\n\n```bash\npython3 demo_pytorch.py \\\n    resnet18 \\\n    data/models/resnet18_sparse/checkpoint.pth \\\n    data/text_embeddings.npy \\\n    --text_prompts data/text_prompts.txt \\\n    --camera_device 0 \\\n    --use_qat\n```\n\n### Export to ONNX\n\n```bash\npython3 export_onnx.py \\\n    resnet18 \\\n    data/models/resnet18_qat/checkpoint.pth \\\n    data/onnx/resnet18_qat.onnx \\\n    --use_qat\n```\n\n\u003ca name=\"next-steps\"\u003e\u003c/a\u003e\n\n## Next steps\n\nWe hope you found this project helpful and that you were able to train your own image classification model, without using any labeled data.  \n\nAs a next step, we recommend reading through the source code to see how we used knoweldge distillation in this project.  We also recommend reading the source code to see how you can train a model with the convenient libraries in PyTorch for quantization aware training and structured sparsity, for more optimized inference on Jetson.\n\nIf you have any questions, or run into any issues, please let us know by opening an issue on GitHub!","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnvidia-ai-iot%2Fclip-distillation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnvidia-ai-iot%2Fclip-distillation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnvidia-ai-iot%2Fclip-distillation/lists"}