Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/apple/ml-no-token-left-behind
https://github.com/apple/ml-no-token-left-behind
Last synced: 3 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/apple/ml-no-token-left-behind
- Owner: apple
- License: other
- Created: 2022-05-31T17:20:26.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2022-11-25T10:03:27.000Z (about 2 years ago)
- Last Synced: 2024-09-29T00:40:57.214Z (3 months ago)
- Language: Jupyter Notebook
- Size: 62.3 MB
- Stars: 137
- Watchers: 10
- Forks: 13
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
- awesome-generative-ai - apple/ml-no-token-left-behind - Aided Image Classification and Generation (Image Synthesis / AI Tools for Searching)
README
# PyTorch Implementation of [No Token Left Behind: Explainability-Aided Image Classification and Generation](https://arxiv.org/abs/2204.04908)
## Usage
### 1. Notebook for spatially conditioned image generation
### 2. Notebook for image editing
### 3. Notebook for image generation
### 4. Prompt engineering running instructions
First, follow [DATASETS.md](external/CoOp/DATASETS.md) to install the datasets.
Create the required enviromnet with
```
conda env create -f external/CoOp/dassl_env.yml
conda activate dassl
pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 -f https://download.pytorch.org/whl/torch_stable.html
```
Then clone and install dassl under 'external' direrctory:
```
cd external/Dassl.pytorch/
python setup.py develop
cd ../../
```
To run the experiment please run:
```
python external/CoOp/train.py --root --trainer CoOp --dataset-config-file --config-file external/CoOp/configs/trainers/CoOp/_ep50.yaml --output-dir --model-dir --seed 1 DATASET.NUM_SHOTS 1 TRAINER.COOP.EXPL_WEIGHT TRAINER.COOP.CSC False TRAINER.COOP.RETURN_EXPL_SCORE True TRAINER.COOP.CLASS_TOKEN_POSITION middle TRAINER.COOP.N_CTX 16
```
## Citation
```
@misc{Paiss2022NoTL,
url = {https://arxiv.org/abs/2204.04908},
author = {Paiss, Roni and Chefer, Hila and Wolf, Lior},
title = {No Token Left Behind: Explainability-Aided Image Classification and Generation},
publisher = {arXiv},
year = {2022}
}
```
## Acknowledements
* Image manipulation code is based on [StyleCLIP](https://github.com/orpatashnik/StyleCLIP)
* Image generation code is based on [FuseDream](https://github.com/gnobitab/FuseDream)
* Image generation with spatial conditioning code is based on [VQGAN+CLIP](https://colab.research.google.com/drive/1ZAus_gn2RhTZWzOWUpPERNC0Q8OhZRTZ#scrollTo=VA1PHoJrRiK9) and [VQGAN](https://github.com/CompVis/taming-transformers)
* Prompt engineering code is based on [CoOp](https://github.com/KaiyangZhou/CoOp) and [Dassl](https://github.com/KaiyangZhou/Dassl.pytorch)
* Explainability method code is based on [Transformer-MM-Explainability](https://github.com/hila-chefer/Transformer-MM-Explainability)
## License
This sample code is released under the [LICENSE](LICENSE) terms.