{"id":22283609,"url":"https://github.com/ashok-arjun/zero-shot-sketch-based-image-retrieval","last_synced_at":"2025-07-28T21:32:47.936Z","repository":{"id":37666545,"uuid":"283417889","full_name":"ashok-arjun/Zero-Shot-Sketch-Based-Image-Retrieval","owner":"ashok-arjun","description":"Zero-shot sketch-based image retrieval using a domain adversarial neural network","archived":false,"fork":false,"pushed_at":"2022-11-22T09:48:36.000Z","size":18689,"stargazers_count":18,"open_issues_count":3,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2023-03-05T07:42:50.935Z","etag":null,"topics":["computer-vision","deep-learning","domain-adversarial","gradient-reversal-layer","image-retrieval","machine-learning","pytorch","sketch-based-image-retrieval","zero-shot-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ashok-arjun.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-07-29T06:29:15.000Z","updated_at":"2022-12-23T11:56:36.000Z","dependencies_parsed_at":"2022-09-09T08:20:08.660Z","dependency_job_id":null,"html_url":"https://github.com/ashok-arjun/Zero-Shot-Sketch-Based-Image-Retrieval","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashok-arjun%2FZero-Shot-Sketch-Based-Image-Retrieval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashok-arjun%2FZero-Shot-Sketch-Based-Image-Retrieval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashok-arjun%2FZero-Shot-Sketch-Based-Image-Retrieval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashok-arjun%2FZero-Shot-Sketch-Based-Image-Retrieval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ashok-arjun","download_url":"https://codeload.github.com/ashok-arjun/Zero-Shot-Sketch-Based-Image-Retrieval/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227960024,"owners_count":17847726,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","deep-learning","domain-adversarial","gradient-reversal-layer","image-retrieval","machine-learning","pytorch","sketch-based-image-retrieval","zero-shot-learning"],"created_at":"2024-12-03T16:41:21.872Z","updated_at":"2024-12-03T16:41:22.367Z","avatar_url":"https://github.com/ashok-arjun.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src=\"https://img.shields.io/badge/python%20-%2314354C.svg?\u0026style=for-the-badge\u0026logo=python\u0026logoColor=white\"/\u003e \u003cimg src=\"https://img.shields.io/badge/PyTorch%20-%23EE4C2C.svg?\u0026style=for-the-badge\u0026logo=PyTorch\u0026logoColor=white\" /\u003e\n\n# Zero Shot Sketch-Based Image Retrieval\n\nThe problem of retrieving images from a large-database using ambiguous sketches has been addressed. This problem has been addressed in the **zero-shot scenario**, where the test sketches/images are from **unseen classes** and the deep feature extractor's output embedding distances of the sketch and the image have been used to retrieve the top k closest images from the image database of the unseen classes.\n\nThe standard **triplet loss** has been used, along with a **domain loss**, which is trained to differentiate between sketches and images.\n\nThe embedding is passed in to **gradient reversal layer** \u003csup\u003e1\u003c/sup\u003e, and then into a domain-classifier network, and then into the domain loss. The **gradient reversal layer** acts as an **identity layer in the forward pass**, and **multiplies the gradient by -1 in the backward pass** which results in the main network learning a **domain-agnostic representation** i.e. to fool the domain-classifier network.\n\n# Architecture Overview\n\n![](docs/zs-sbir-architecture.png)\n\n\u003csub\u003e† Diagram created by Arjun Ashok using [app.diagrams.net](http://app.diagrams.net)\u003c/sub\u003e \n\n# Results on unseen(zero-shot) classes\n\nThe below table presents a few qualitative results of our model on unseen test classes. The table presents the top 5 results from left to right.\n  \n| Query Sketch 1  |\n|:---------------:|\n|![](docs/examples/1/n03512147_1442-5.png)|\n\n| Retrieved Image 1 | Retrieved Image 2 | Retrieved Image 3 | Retrieved Image 4 | Retrieved Image 5 |\n|:-----------------:|:-----------------:|:-----------------:|:-----------------:|:-----------------:|\n|![](docs/examples/1/ext_620.jpg)|![](docs/examples/1/ext_479.jpg)|![](docs/examples/1/ext_441.jpg)|![](docs/examples/1/ext_437.jpg)|![](docs/examples/1/n03512147_44302.jpg)|\n\n| Query Sketch 2  |\n|:---------------:|\n|![](docs/examples/2/n02958343_10092-1.png)|\n\n| Retrieved Image 1 | Retrieved Image 2 | Retrieved Image 3 | Retrieved Image 4 | Retrieved Image 5 |\n|:-----------------:|:-----------------:|:-----------------:|:-----------------:|:-----------------:|\n|![](docs/examples/2/ext_201.jpg)|![](docs/examples/2/n02958343_13615.jpg)|![](docs/examples/2/n04166281_6690.jpg)|![](docs/examples/2/ext_389.jpg)|![](docs/examples/2/n04166281_241.jpg)|\n\n# Instructions\n\u003cdetails\u003e\n\u003csummary\u003e\n  \u003cb\u003eInstallation\u003c/b\u003e\n\u003c/summary\u003e\n\nPlease execute the following command to install the required libraries:\n\n```\npip install -r requirements.txt\n```\n\n\u003c/details\u003e\n\u003cdetails\u003e\n\u003csummary\u003e\n\u003cb\u003eData\u003c/b\u003e\n\u003c/summary\u003e\n\nExecute ```bash download_data.sh```\n\n\u003c/details\u003e\n\u003cdetails\u003e\n\n\u003csummary\u003e\n\u003cb\u003eTraining\u003c/b\u003e\n\u003c/summary\u003e\n\nThe file ```train.py``` can be invoked with the following arguments:\n\n```\nusage: train.py [-h] --data_dir DATA_DIR --batch_size BATCH_SIZE\n                --checkpoint_dir CHECKPOINT_DIR --epochs EPOCHS\n                [--domain_loss_ratio DOMAIN_LOSS_RATIO]\n                [--triplet_loss_ratio TRIPLET_LOSS_RATIO]\n                [--grl_threshold_epoch GRL_THRESHOLD_EPOCH]\n                [--print_every PRINT_EVERY]\n\nTraining of SBIR\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --data_dir DATA_DIR   Data directory path. Directory should contain two\n                        folders - sketches and photos, along with 2 .txt files\n                        for the labels\n  --batch_size BATCH_SIZE\n                        Batch size to process the train sketches/photos\n  --checkpoint_dir CHECKPOINT_DIR\n                        Directory to save checkpoints\n  --epochs EPOCHS       Number of epochs\n  --domain_loss_ratio DOMAIN_LOSS_RATIO\n                        Domain loss weight\n  --triplet_loss_ratio TRIPLET_LOSS_RATIO\n                        Triplet loss weight\n  --grl_threshold_epoch GRL_THRESHOLD_EPOCH\n                        Threshold epoch for GRL lambda\n  --print_every PRINT_EVERY\n                        Logging interval in iterations\n```\n\nIt is advised to use a GPU for training. The code automatically detects and uses a GPU, if available.\n \n\u003c/details\u003e\n\n\u003cdetails\u003e\n\n\u003csummary\u003e\n\u003cb\u003eInference\u003c/b\u003e\n\u003c/summary\u003e\n\nThe file ```evaluate.py``` can be invoked with the following args:\n\n```\nusage: evaluate.py [-h] [--model MODEL] --data DATA [--num_images NUM_IMAGES]\n                   [--num_sketches NUM_SKETCHES] [--batch_size BATCH_SIZE]\n                   [--output_dir OUTPUT_DIR]\n\nEvaluation of SBIR\n\narguments:\n  -h, --help            show this help message and exit\n  --model MODEL         Model checkpoint path\n  --data DATA           Data directory path. Directory should contain two\n                        folders - sketches and photos, along with 2 .txt files\n                        for the labels\n  --num_images NUM_IMAGES\n                        Number of random images to output for every\n                        sketch\n  --num_sketches NUM_SKETCHES\n                        Number of random sketches to output\n  --batch_size BATCH_SIZE\n                        Batch size to process the test sketches/photos\n  --output_dir OUTPUT_DIR\n                        Directory to save output sketch and images\n```\n\nIt is advised to use a GPU for evaluation. The code automatically detects and uses a GPU, if available.\n\n\u003c/details\u003e\n\n# References\n\n1. Ganin, Yaroslav et al. \"Domain-Adversarial Training Of Neural Networks\". Journal of Machine Learning Research, 2016, pp. 1-35, url:http://jmlr.org/papers/v17/15-239.html\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fashok-arjun%2Fzero-shot-sketch-based-image-retrieval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fashok-arjun%2Fzero-shot-sketch-based-image-retrieval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fashok-arjun%2Fzero-shot-sketch-based-image-retrieval/lists"}