{"id":51237077,"url":"https://github.com/dabane-ghassan/int-lab-book","last_synced_at":"2026-06-28T21:11:53.823Z","repository":{"id":56751495,"uuid":"354756936","full_name":"dabane-ghassan/int-lab-book","owner":"dabane-ghassan","description":"Foveated Spatial Transformers","archived":false,"fork":false,"pushed_at":"2022-02-10T14:42:55.000Z","size":507687,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-04-16T03:53:54.801Z","etag":null,"topics":["bio-inspired-vision","computer-vision","convolutional-neural-networks","deep-learning","foveated-spatial-transformer","machine-learning","spatial-transformer-networks","stn"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dabane-ghassan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-04-05T07:41:27.000Z","updated_at":"2024-01-06T18:47:13.000Z","dependencies_parsed_at":"2022-08-16T01:40:11.430Z","dependency_job_id":null,"html_url":"https://github.com/dabane-ghassan/int-lab-book","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dabane-ghassan/int-lab-book","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dabane-ghassan%2Fint-lab-book","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dabane-ghassan%2Fint-lab-book/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dabane-ghassan%2Fint-lab-book/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dabane-ghassan%2Fint-lab-book/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dabane-ghassan","download_url":"https://codeload.github.com/dabane-ghassan/int-lab-book/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dabane-ghassan%2Fint-lab-book/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34904067,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-28T02:00:05.809Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bio-inspired-vision","computer-vision","convolutional-neural-networks","deep-learning","foveated-spatial-transformer","machine-learning","spatial-transformer-networks","stn"],"created_at":"2026-06-28T21:11:52.982Z","updated_at":"2026-06-28T21:11:53.819Z","avatar_url":"https://github.com/dabane-ghassan.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![forthebadge](https://forthebadge.com/images/badges/made-with-python.svg)](https://forthebadge.com)\n[![forthebadge](https://forthebadge.com/images/badges/open-source.svg)](https://forthebadge.com)\n[![forthebadge](https://forthebadge.com/images/badges/for-robots.svg)](https://forthebadge.com)\n\n# Project\n\n\u003e In contrast with computer vision, biological vision is characterized by an anisotropic sensor (**The Retina**) as well as the ability to **move** the eyesight to   different locations in the visual scene through ocular **saccades**. To better understand how the human eye analyzes visual scenes, a bio-inspired artificial  vision  model was recently suggested by ***Daucé et al (2020) \u003csup\u003e1\u003c/sup\u003e***.The goal of this master’s internship would be to compare the results obtained by Daucé et   al with some of the more classical attentional computer vision models like the ***Spatial transformer network \u003csup\u003e2\u003c/sup\u003e*** where the visual input undergoes a foveal deformation.\n\n# Computational graph of a foveated spatial transformer network\n- This module is used in the POLO_ATN network.\n![foveated st module](foveated_st.png)\n\n# Results\n\n## The Generic Spatial Transformer Network Vs. The What pathway\u003csup\u003e1\u003c/sup\u003e\n\n### Exploring the 28x28 Noisy MNIST dataset. \n\n\u003e Taking a look at a few examples from the dataset:\n\n![28x28 noisy no shift](figures/noisy_no_shift_28x28_data.png)\n\n### STN_28x28\n- ***Spatial Transformer: 2 convolutional layers in localization network (ConvNet), grid sampler without downscaling (28x28 pixels) \u0026#8594; \u003cimg src=\"https://latex.codecogs.com/gif.latex?\\bold{\\theta}\"/\u003e (affine transformations) = 6 parameters***\n\n\n- Training for 160  epochs with SGD, learning rate of 0.01 without decay, Each 10 epochs, increment the shift standard deviation by 1 [0, 15].\n\n\u003e Training statistics:\n\n![training stn 28x28](figures/loss_acc_training_stn_28x28.png)\n\n### Performance\n\n- **Overall results**: *Central* accuracy of **88%** and *general* accuracy of **43%**, compared to **84%** and **34%** in the generic what pathway, respectively.\n\n\u003e Accuracy map comparaison with the generic what pathway from the paper with the same training parameters:\n\nSpatial Transformer Network             |  Generic What pathway \u003csup\u003e1\u003c/sup\u003e\n:-------------------------:|:-------------------------:\n![acc map stn](figures/stn_28x28_accuracy_map.png)  |  ![acc map what](figures/what_map.png)\n\n\n\u003e A test on a noisy dataset with a shift standard deviation = 7\n\n![results](figures/transforms_28x28.png)\n\n## Spatial Transformer Networks Vs. The What/Where pathway\u003csup\u003e1\u003c/sup\u003e\n\n### Exploring the 128x128 Noisy MNIST dataset \u003csup\u003e1\u003c/sup\u003e.\n\n\u003e Taking a look at a few examples:\n\n![128x128 noisy shift dataset](figures/data_128x128_noisy_no_shift_.png)\n\n### STN_128x128 \n- ***Spatial Transformer: 4 convolutional layers in localization network (ConvNet), grid sampler without downscaling (128x128 pixels) \u0026#8594; \u003cimg src=\"https://latex.codecogs.com/gif.latex?\\bold{\\theta}\"/\u003e  (affine transformations) = 6 parameters***\n\u003e Training for 110 epochs with an initial learning rate of 0.01 that decays by a factor of 10 every 30 epochs, each 10 epochs increase the standard deviation of the eccentricity, last 20 epochs vary the contrast.\n\n![training stn 128x128](figures/acc_training_stn_128x128.png)\n\n\u003e After transformation with a STN:\n\n![transformed 128x128](figures/preliminary_128x128.png)\n\n\u003e Performance when the contrast varies between 30-70% and the digit is shifted by 40 pixels (the maximum amount):\n\n![contrast 128x128](figures/contrast_128x128.png)\n\n\n\n### ATN\n\n- ***Spatial Transformer: 4 convolutional layers in localization network (ConvNet), grid sampler with downscaling (28x28 pixels) \u0026#8594; \u003cimg src=\"https://latex.codecogs.com/gif.latex?\\bold{\\theta}\"/\u003e  (attention) = 3 parameters***\n\n\u003e Training for 110 epochs with an initial learning rate of 0.01 that decays by a half every 10 epochs, each 10 epochs increase the standard deviation of the eccentricity, last 20 epochs vary the contrast.\n\n![training stn 128x128](figures/acc_training_atn.png)\n\n\u003e After transformation with a ATN (STN parametrized for attention), the digit is shifted by 40 pixels to check if the network can catch it:\n\n![transformed atn_128x128](figures/atn_attention.png)\n\n\u003e Performance when the contrast is 30 and the digit is shifted by 40 pixels (the maximum amount):\n\n![contrast 128x128](figures/atn_attention_0.3.png)\n\n### POLO_ATN\n\n- ***Spatial Transformer: 2 fully-connected layers in localization network (FeedForward Net), grid sampler with downscaling (28x28 pixels) \u0026#8594; \u003cimg src=\"https://latex.codecogs.com/gif.latex?\\bold{\\theta}\"/\u003e  (fixed attention) = 2 parameters***\n\n\u003e Polar-Logarithmic compression: the filters were placed on [theta=8, eccentricity=6, azimuth=16], on 768 dimensions, providing a compression of \n~**95%**, the original what/where model had 2880 filters, with a lesser compression rate of ~**83%**.\n\n![polo_transformed_dataset](figures/polo_dataset.png)\n\n\u003e Training for 110 epochs with an initial learning rate of 0.005 that decays by a half every 10 epochs, each 10 epochs increase the standard deviation of the eccentricity, last 20 epochs vary the contrast.\n\n![training polo_atn](figures/acc_training_polo_atn.png)\n\n\u003e After transformation with a POLO-ATN, the digit is shifted by 40 pixels to check if the network can catch it:\n\n![transformed polo_atn](figures/polo_atn_attention.png)\n\n### Benchmark\n\n\u003e Accuracy comparison of spatial transformer networks with the What/Where model, in function of contrast and eccentricity of the digit on the screen.\n\n![benchmarks](figures/benchmark.png)\n\n# References\n\n[*[1] Emmanuel Daucé, Pierre Albiges, Laurent U. Perrinet; A dual foveal-peripheral visual processing model implements efficient saccade selection. Journal of Vision 2020;20(8):22.*](https://jov.arvojournals.org/article.aspx?articleid=2770680)\n\n[*[2] Max Jaderberg, Karen Simonyan, Andrew Zisserman, Koray Kavukcuoglu; Spatial Transformer Networks. arXiv:1506.02025*](https://arxiv.org/abs/1506.02025)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdabane-ghassan%2Fint-lab-book","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdabane-ghassan%2Fint-lab-book","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdabane-ghassan%2Fint-lab-book/lists"}