Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/mhamilton723/featup

Official code for "FeatUp: A Model-Agnostic Frameworkfor Features at Any Resolution" ICLR 2024
https://github.com/mhamilton723/featup

Last synced: 29 days ago
JSON representation

Official code for "FeatUp: A Model-Agnostic Frameworkfor Features at Any Resolution" ICLR 2024

Host: GitHub
URL: https://github.com/mhamilton723/featup
Owner: mhamilton723
License: mit
Created: 2024-02-22T19:43:56.000Z (9 months ago)
Default Branch: main
Last Pushed: 2024-06-28T23:45:46.000Z (5 months ago)
Last Synced: 2024-10-15T05:24:19.838Z (29 days ago)
Language: Jupyter Notebook
Size: 38.1 MB
Stars: 1,353
Watchers: 18
Forks: 78
Open Issues: 51
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

        # FeatUp: A Model-Agnostic Framework for Features at Any Resolution

###  ICLR 2024

[![Website](https://img.shields.io/badge/FeatUp-%F0%9F%8C%90Website-purple?style=flat)](https://aka.ms/featup) [![arXiv](https://img.shields.io/badge/arXiv-2403.10516-b31b1b.svg)](https://arxiv.org/abs/2403.10516) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/mhamilton723/FeatUp/blob/main/example_usage.ipynb)

[![Huggingface](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-FeatUp-orange)](https://huggingface.co/spaces/mhamilton723/FeatUp) 

[![Huggingface](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Paper%20Page-orange)](https://huggingface.co/papers/2403.10516)

[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/featup-a-model-agnostic-framework-for/feature-upsampling-on-imagenet)](https://paperswithcode.com/sota/feature-upsampling-on-imagenet?p=featup-a-model-agnostic-framework-for)

[Stephanie Fu*](https://stephanie-fu.github.io/),

[Mark Hamilton*](https://mhamilton.net/),

[Laura Brandt](https://people.csail.mit.edu/lebrandt/),

[Axel Feldman](https://feldmann.nyc/),

[Zhoutong Zhang](https://ztzhang.info/),

[William T. Freeman](https://billf.mit.edu/about/bio)

*Equal Contribution.

![FeatUp Overview Graphic](https://mhamilton.net/images/website_hero_small-p-1080.jpg)

*TL;DR*:FeatUp improves the spatial resolution of any model's features by 16-32x without changing their semantics.

https://github.com/mhamilton723/FeatUp/assets/6456637/8fb5aa7f-4514-4a97-aebf-76065163cdfd

## Contents

   * [Install](#install)

   * [Using Pretrained Upsamplers](#using-pretrained-upsamplers)

   * [Fitting an Implicit Upsampler](#fitting-an-implicit-upsampler-to-an-image)

   * [Coming Soon](coming-soon)

   * [Citation](#citation)

   * [Contact](#contact)

## Install

### Pip

For those just looking to quickly use the FeatUp APIs install via:

```shell script

pip install git+https://github.com/mhamilton723/FeatUp

```

### Local Development

To install FeatUp for local development and to get access to the sample images install using the following:

```shell script

git clone https://github.com/mhamilton723/FeatUp.git

cd FeatUp

pip install -e .

```

## Using Pretrained Upsamplers

To see examples of pretrained model usage please see our [Collab notebook](https://colab.research.google.com/github/mhamilton723/FeatUp/blob/main/example_usage.ipynb). We currently supply the following pretrained versions of FeatUp's JBU upsampler:

| Model Name | Checkpoint                                                                                                                       | Checkpoint (No LayerNorm)                                                                                                                  | Torch Hub Repository | Torch Hub Name |

|------------|----------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------|----------------------|----------------|

| DINO       | [Download](https://marhamilresearch4.blob.core.windows.net/feature-upsampling-public/pretrained/dino16_jbu_stack_cocostuff.ckpt) | [Download](https://marhamilresearch4.blob.core.windows.net/feature-upsampling-public/pretrained/no_norm/dino16_jbu_stack_cocostuff.ckpt)   | mhamilton723/FeatUp  | dino16         |

| DINO v2    | [Download](https://marhamilresearch4.blob.core.windows.net/feature-upsampling-public/pretrained/dinov2_jbu_stack_cocostuff.ckpt) | [Download](https://marhamilresearch4.blob.core.windows.net/feature-upsampling-public/pretrained/no_norm/dinov2_jbu_stack_cocostuff.ckpt)   | mhamilton723/FeatUp  | dinov2         |

| CLIP       | [Download](https://marhamilresearch4.blob.core.windows.net/feature-upsampling-public/pretrained/clip_jbu_stack_cocostuff.ckpt)   | [Download](https://marhamilresearch4.blob.core.windows.net/feature-upsampling-public/pretrained/no_norm/clip_jbu_stack_cocostuff.ckpt)     | mhamilton723/FeatUp  | clip           |

| MaskCLIP   | n/a                                                                                                                              | [Download](https://marhamilresearch4.blob.core.windows.net/feature-upsampling-public/pretrained/no_norm/maskclip_jbu_stack_cocostuff.ckpt) | mhamilton723/FeatUp  | maskclip       |

| ViT        | [Download](https://marhamilresearch4.blob.core.windows.net/feature-upsampling-public/pretrained/vit_jbu_stack_cocostuff.ckpt)      | [Download](https://marhamilresearch4.blob.core.windows.net/feature-upsampling-public/pretrained/no_norm/vit_jbu_stack_cocostuff.ckpt)      | mhamilton723/FeatUp  | vit            |

| ResNet50   | [Download](https://marhamilresearch4.blob.core.windows.net/feature-upsampling-public/pretrained/resnet50_jbu_stack_cocostuff.ckpt) | [Download](https://marhamilresearch4.blob.core.windows.net/feature-upsampling-public/pretrained/no_norm/resnet50_jbu_stack_cocostuff.ckpt) | mhamilton723/FeatUp  | resnet50       |

For example, to load the FeatUp JBU upsampler for the DINO backbone without an additional LayerNorm on the spatial features:

```python

upsampler = torch.hub.load("mhamilton723/FeatUp", 'dino16', use_norm=False)

```

To load upsamplers trained on backbones with additional LayerNorm operations which makes training and transfer learning a bit more stable:

```python

upsampler = torch.hub.load("mhamilton723/FeatUp", 'dino16')

```

## Fitting an Implicit Upsampler to an Image

To train an implicit upsampler for a given image and backbone first clone the repository and install it for 

[local development](#local-development). Then run

```python

cd featup

python train_implicit_upsampler.py

```

Parameters for this training operation can be found in the [implicit_upsampler config file](featup/configs/implicit_upsampler.yaml).

## Local Gradio Demo

To run our [HuggingFace Spaces hosted FeatUp demo](https://huggingface.co/spaces/mhamilton723/FeatUp) locally first install FeatUp for local development. Then  run:

```shell

python gradio_app.py

```

Wait a few seconds for the demo to spin up, then navigate to [http://localhost:7860/](http://localhost:7860/) to view the demo.

## Coming Soon:

- Training your own FeatUp joint bilateral upsampler

- Simple API for Implicit FeatUp training

## Citation

```

@inproceedings{

    fu2024featup,

    title={FeatUp: A Model-Agnostic Framework for Features at Any Resolution},

    author={Stephanie Fu and Mark Hamilton and Laura E. Brandt and Axel Feldmann and Zhoutong Zhang and William T. Freeman},

    booktitle={The Twelfth International Conference on Learning Representations},

    year={2024},

    url={https://openreview.net/forum?id=GkJiNn2QDF}

}

```

## Contact

For feedback, questions, or press inquiries please contact [Stephanie Fu](mailto:[email protected]) and [Mark Hamilton](mailto:[email protected])