https://github.com/woctezuma/steam-dinov2
Retrieve Steam games with similar store banners, with Facebook's DINOv2.
https://github.com/woctezuma/steam-dinov2
dino dino-v2 dinov2 discovery facebook-dino facebook-dino-v2 facebook-dinov2 fb-dino fb-dino-v2 fb-dinov2 game games meta-ai-dino-v2 meta-ai-dinov2 meta-dino-v2 meta-dinov2 steam steam-game steam-games
Last synced: about 1 month ago
JSON representation
Retrieve Steam games with similar store banners, with Facebook's DINOv2.
- Host: GitHub
- URL: https://github.com/woctezuma/steam-dinov2
- Owner: woctezuma
- License: mit
- Created: 2023-04-18T08:06:37.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2024-12-10T10:57:35.000Z (5 months ago)
- Last Synced: 2025-03-27T12:35:19.656Z (about 2 months ago)
- Topics: dino, dino-v2, dinov2, discovery, facebook-dino, facebook-dino-v2, facebook-dinov2, fb-dino, fb-dino-v2, fb-dinov2, game, games, meta-ai-dino-v2, meta-ai-dinov2, meta-dino-v2, meta-dinov2, steam, steam-game, steam-games
- Language: Jupyter Notebook
- Homepage: https://ai.facebook.com/blog/dino-v2-computer-vision-self-supervised-learning/
- Size: 74.2 KB
- Stars: 2
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Steam DINOv2: match banners with Meta AI's DINOv2
This repository contains Python code to retrieve Steam games with similar store banners, using [Meta AI's DINOv2][fb-dinov2-demo].
Image similarity is assessed by the cosine similarity between image features encoded by DINOv2.
![Similar vertical banners][wiki-cover]
## Data
Data consists of **vertical** Steam banners (`library_600x900.jpg` at 300x450 resolution).
### Pre-processing
DINOv2 follows torchvision's [pre-processing pipeline][dinov2-pre-process] for classification:
- [resize][resize] to 256 resolution, i.e. the **smallest** edge of the image will match this number,
- [center-crop][centercrop] at 224 resolution, i.e. a **square** crop is made,
- normalize intensity.
```python
transforms_list = [
transforms.Resize(resize_size, interpolation=interpolation),
transforms.CenterCrop(crop_size),
MaybeToTensor(),
make_normalize_transform(mean=mean, std=std),
]
```
Therefore, downloaded images can be resized to 256x384 resolution before being stored to disk.
As discussed in [this Github issue][dinov2-pre-process-issue], the crop of DINOv2 can be made less agressive by resizing to 224 instead of 256 resolution.
In this case, downloaded images can be resized to 224x336 resolution before being stored to disk.
### Snapshot
A snapshot of image data was downloaded **on July 20, 2023** and stored as [a Github release][input-data-github-release].
If you wish to re-create this data snapshot, run [`download_steam_images.ipynb`][input-data-colab-notebook].
[![Open In Colab][colab-badge]][input-data-colab-notebook]
### Filtering (optional)
If you wish to filter the image dataset, identify issues with [`find_dataset_issues.ipynb`][cleanvision-colab-notebook].
[![Open In Colab][colab-badge]][cleanvision-colab-notebook]
## Usage
Run [`match_steam_images.ipynb`][match-colab-notebook].
[![Open In Colab][colab-badge]][match-colab-notebook]
> [!Note]
> For the same appID, there can exist some differences between matches computed on the fly and pre-computed matches, because matches are obtained based on features extracted from images resized with different interpolation algorithms:
> - for on-the-fly matching, images are resized with [`transforms.InterpolationMode.BICUBIC`][dinov2-bicubic-interpolation],
> - for pre-computed matches, images were resized by [`img2dataset`][img2dataset-downscale-interpolation] with [`cv2.INTER_AREA`][opencv-interpolation-flags], as suggested [in the doc][opencv-resize] of OpenCV for downscale interpolation.
## Results
For each game in the top 100 most played games in the past 2 weeks, the 10 most similar games are retrieved with:
- [ViT-S/14 distilled][wiki-results-ViTS],
- [ViT-B/14 distilled][wiki-results-ViTB],
- [ViT-L/14 distilled][wiki-results-ViTL].
AppIDs for all these apps can be found in JSON files in [`data/similar_to_top_100/`](data/similar_to_top_100/).
> [!Note]
> The linked pages contain a lot of images and might be slow to load depending on your Internet bandwidth.
### Examples obtained with ViT-L
#### Same visual theme
Basket ball:
Cartoony monsters:
Colorful:
Far West:
Paladins in anime style:
#### Same character
Armed persons with helmets:
Dinosaurs:
Knights:
Robots:
#### Same item
Bows:
Cars:
Castles:
Farms:
Tanks:
Tools:
#### Same franchise
Borderlands:
Cities (with skyscrapers):
Hollow Knight:
Left 4 Dead (with hands):
Monster Hunter (with dragons):
Mount & Blade (with horses):
Naraka:
Payday:
The Witcher (with Geralt of Rivia):
Truck simulators:
War planes:
War vehicles:
### Comparison of ViT-S, ViT-B and ViT-L
Each model performs reasonably well in terms of image retrieval.
#### ViT-S
The following results are obtained with ViT-S.
Cars in *Rocket League*:
Cars in *Wallpaper Engine*:
Dinosaurs:
Farms:
Persons standing in a forest:
Robots:
Sailing:
Tanks:
Interestingly, the results for *The Binding of Isaac* may be better than with ViT-L.
The first match for *Sekiro* reveals a similarity between Wolf in *Sekiro* and Mitsurugi in *Soul Calibur*.
Most matches for *Naraka* include a sword-like weapon with a blue glow:
#### ViT-B
The following results are obtained with ViT-B.
Bows:
Cartoon characters with a white mask:
Colorful:
Dinosaurs:
Interestingly, the first match for *Monster Hunter* is correctly in the same franchise.
## References
- A [feature extractor][feature-extractor] for Github repositories which include a `hubconf.py` file.
- A [feature matcher][feature-matcher] based on the `faiss` library.
- [`match-steam-banners`][github-match-steam-banners]: retrieve games with similar store banners.
- Facebook's DINO:
- [Official blog post][fb-dino-blog]
- [Official Github repository][fb-dino-code]
- [Caron, Mathilde, et al. *Emerging Properties in Self-Supervised Vision Transformers*. arXiv 2021.][fb-dino-paper]
- Facebook's DINOv2:
- [Official demo][fb-dinov2-demo]
- [Official blog post][fb-dinov2-blog]
- [Official Github repository][fb-dinov2-code]
- [Oquab, Maxime, et al. *DINOv2: Learning Robust Visual Features without Supervision*. arXiv 2023.][fb-dinov2-paper]
[wiki-cover]:
[dinov2-pre-process]:
[fb-dino-blog]:
[fb-dino-code]:
[fb-dino-paper]:
[fb-dinov2-demo]:
[fb-dinov2-blog]:
[fb-dinov2-code]:
[fb-dinov2-paper]:
[resize]:
[centercrop]:
[dinov2-pre-process-issue]:
[feature-extractor]:
[feature-matcher]:
[github-match-steam-banners]:
[dinov2-bicubic-interpolation]:
[opencv-interpolation-flags]:
[img2dataset-downscale-interpolation]:
[opencv-resize]:
[wiki-results-ViTS]:
[wiki-results-ViTB]:
[wiki-results-ViTL]:
[input-data-github-release]:
[input-data-colab-notebook]:
[cleanvision-colab-notebook]:
[match-colab-notebook]:
[colab-badge]: