{"id":15677133,"url":"https://github.com/arthurzucker/recvisproject","last_synced_at":"2026-03-04T23:02:41.050Z","repository":{"id":37685969,"uuid":"443783020","full_name":"ArthurZucker/RecvisProject","owner":"ArthurZucker","description":"In this project, we propose to study Vision Transformers trained using the Barlow Twins self-supervised method, and compare the results with DINO. We demonstrate the effectiveness of the Barlow Twins method by showing that networks pretrained on the small PASCAL VOC 2012 dataset are able to generalize well. Authors: Apavou Clément \u0026 Zucker Arthur","archived":false,"fork":false,"pushed_at":"2023-10-03T23:34:29.000Z","size":11478,"stargazers_count":15,"open_issues_count":2,"forks_count":2,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-05-07T00:45:22.175Z","etag":null,"topics":["computer-vision","pytorch","self-supervised-learning","semantic-segmentation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ArthurZucker.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-02T14:13:57.000Z","updated_at":"2024-10-17T03:15:24.000Z","dependencies_parsed_at":"2024-10-23T13:04:06.428Z","dependency_job_id":null,"html_url":"https://github.com/ArthurZucker/RecvisProject","commit_stats":{"total_commits":277,"total_committers":5,"mean_commits":55.4,"dds":"0.40794223826714804","last_synced_commit":"24cf91daa55e7fecb7637575e9613a0cb8595ab1"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ArthurZucker/RecvisProject","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArthurZucker%2FRecvisProject","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArthurZucker%2FRecvisProject/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArthurZucker%2FRecvisProject/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArthurZucker%2FRecvisProject/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ArthurZucker","download_url":"https://codeload.github.com/ArthurZucker/RecvisProject/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArthurZucker%2FRecvisProject/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273842872,"owners_count":25177921,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-05T02:00:09.113Z","response_time":402,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","pytorch","self-supervised-learning","semantic-segmentation"],"created_at":"2024-10-03T16:08:36.735Z","updated_at":"2026-03-04T23:02:41.037Z","avatar_url":"https://github.com/ArthurZucker.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Study of the Emerging Properties of Self-Supervised Vision Transformers and Semantic Segmentation \n\n\u003e Authors: [Apavou Clément](https://github.com/clementapa) \u0026 [Zucker Arthur](https://github.com/ArthurZucker)\n\n![Python](https://img.shields.io/badge/Python-green.svg?style=plastic)\n![PyTorch](https://img.shields.io/badge/PyTorch-orange.svg?style=plastic)\n![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet.svg?style=plastic)\n\n# Abstract \n\nSelf-supervised learning using transformers has shown interesting emerging properties and learn rich embeddings without annotations. Most recently, Barlow Twins proposed an elegant self-supervised learning technique using a ResNet-50 backbone which achieved competitive results when fine-tuned on downstream tasks. In this paper, we propose to study Vision Transformers trained using the Barlow Twins self-supervised method, and compare the results with. We demonstrate the effectiveness of the Barlow Twins method by showing that networks pretrained on the small PASCAL VOC 2012 dataset are able to generalize well while requiring less training and computing power than the DINO method. Finally, we propose to leverage self-supervised vision transformers and their semantically rich attention maps for semantic segmentation tasks.\n\n|\u003cimg alt=\"dino attention map with patch size 8\" src=\"/assets/attention_patch8_dino.png\" width=\"100%\" title=\"Dino pretrained attention maps with a patch size of 8. Credits to Apavou \u0026 Zucker\" \u003e| \u003cimg alt=\"dino attention map with patch size 16\" src=\"/assets/attention_patch16_dino.png\" width=\"100%\" title=\"Dino pretrained attention maps with a patch size of 16. Credits to Apavou \u0026 Zucker\" \u003e|\n|---|---|\n|Dino pretrained attention maps with a patch size of 8. Credits to Apavou \u0026 Zucker | Dino pretrained attention maps with a patch size of 16. Credits to Apavou \u0026 Zucker |\n\n\n\u003cp align=\"center\"\u003e\n  \u003cimg alt=\"effective receptive fields\" src=\"/assets/erf.png\" width=\"100%\" title=\"Effective receptive fields of 8 layers of the deeplabv3 architecture. Credits to Apavou \u0026 Zucker\" \u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n  \"Effective receptive fields of 8 layers of the deeplabv3 architecture. Credits to Apavou \u0026 Zucker\"\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg alt=\"Cross correlation matrix\" src=\"assets/brisk-valley-111-ccM.png\" width=\"45%\" title=\"Cross correlation matrix obtained by training a ViT with Barlow Twins, initialized with DINO weights. Refer to brisk-valley-111 experiment. Credits to Apavou \u0026 Zucker\" \u003e \n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n  Cross correlation matrix obtained by training a ViT with Barlow Twins, initialized with DINO weights. Refer to brisk-valley-111 experiment. Credits to Apavou \u0026 Zucker\n\u003c/p\u003e\n\n\n\n|\u003cimg alt=\"predictions pretrain\" src=\"assets/pretrained_method_predictions.png\" width=\"100%\" title=\"Predictions for a set of 4 images from the validation dataset using 4 different ResNet-50 backbones and a deeplabv3 head. Credits to Apavou \u0026 Zucker\" \u003e| \u003cimg alt=\"predictions heads\" src=\"/assets/predictions_heads.png\" width=\"100%\" title=\"Predictions for a set of 4 images from the validation dataset using a ViT-S/8 backbones and 4 different heads. Credits to Apavou \u0026 Zucker\" \u003e|\n|---|---|\n|Predictions for a set of 4 images from the validation dataset using 4 different ResNet-50 backbones and a deeplabv3 head. Credits to Apavou \u0026 Zucker | Predictions for a set of 4 images from the validation dataset using a ViT-S/8 backbones and 4 different heads. Credits to Apavou \u0026 Zucker |\n\n\n# Project report \n\nYou can find the complete project report in the repository or click [here](FPR_Apavou_Zucker.pdf). Our slides are also available [here](https://docs.google.com/presentation/d/1MvE78E8pb4XEIMQxZLkBZnLMXvVNC8E-m7rnxW5wa8Q/edit?usp=sharing).\n\n# Experiments\n\nOurs experiments are available on wandb : [![](https://github.com/wandb/assets/blob/main/wandb-github-badge-gradient.svg)](https://wandb.ai/recvis/sem-seg?workspace=user-clementapa)\n\n# Setting up the environment \n\nWe exported the required packages in a `requirement.txt` file that can be used as follows : \n```\npip install -r requirements.txt\n```\n\n# Training \n\nRefer to the [Barlow Twins Wiki](\nhttps://github.com/ArthurZucker/RecvisProject/wiki/Barlow-Twins-Training) and the [Semantic Segmentation Wiki ]() for more details\n\n# Contributions \n\nWe implemented the global structure and the Barlow Twins method from scratch in PyTorch Lightning, our visualization of the attention maps is inspired from the official [DINO repository](https://github.com/facebookresearch/dino). Our `trainer` module takes care of initializing the lightning module and the datamodule, both of which can be chosen in our configuration file (`config/hparams.py`). *simple parsing* package extracts and parses the configuration file and allows us to switch between the two tasks: Barlow Twins training and Semantic Segmentation fine-tuning. We used the very practical Weights \\\u0026 Biases (wandb) library to log all of our experiments.\n\n# Visualizations \nWe implemented two very efficient and easy-to use callbacks to visualize the effective receptive fields and the attention maps at train and validation time. \nExamples are shown  above. Both rely on pytorch `hooks` and provide more interpretation to the training. Both were implemented from scratch, and the visualization of the effective receptive fields is based on the theory from [Understanding the Effective Receptive Field in Deep Convolutional Neural Networks](https://arxiv.org/abs/1701.04128).\n\nWe also logged the evolution of the cross-correlation matrix which is fare more interpretable than the value of the loss. As various training showed, a decreasing loss can have a cross-correlation matrix far from the identity. We used a heatmap to represent the empirical cross correlation matrix were values close to 1 are red and values close to zeros are cyan blue. \n\n# Acknowledgments\n\nOur implementation relies on `pytorch lightning`, and thus requires its installation. We also use the `rich` library for nicer progress bars and the very handy `wandb` library to visualize our experiments.  \n\nWe used the following implementations for the backbones and heads : \n\n\n+ [DeepLabV3](https://pytorch.org/hub/pytorch_vision_deeplabv3_resnet101/) from pytorch vision model\n+ [ViT](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/vision_transformer.py) from pytorch image models\n+ [ViT](https://github.com/lucidrains/vit-pytorc) from vit-pytorch\n+ [SETR](https://github.com/gupta-abhay/setr-pytorh) adapted from setr-pytorch\n\n\nIn `model/fix_tim/vision_transformer` the vision transformer returns every token in the forward pass (while only the cls token is usually returned). We use this to obtain more features for the semantic segmentation task. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farthurzucker%2Frecvisproject","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farthurzucker%2Frecvisproject","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farthurzucker%2Frecvisproject/lists"}