Master Assignment
Exploring Inhibition in Vision Transformers
Type: Master CS
Period: TBD
Student: (Unassigned)
Supervisory team: Prof. George Azzopardi, dr. Nicola Strisciuglio, Peter van der Wal
If you are interested please contact :
Objective:
Investigate how Inhibited Self-Attention (ISA), inspired by inhibitory biological neural circuits, can further improve the scalability, robustness and efficiency of Vision Transformers (ViTs).
Description:
Standard Transformer attention is purely excitatory, summing positive token contributions. Our recent work (currently under submission) introduces an inhibitory component by utilizing negative attention scores, which significantly improves focus on objects of interest. This project aims to build upon our Inhibited Self-Attention (ISA) framework. A draft of this paper will be provided as a solid starting point, ensuring the student can build upon the existing ISA framework without needing to develop it from scratch. While we propose several promising research directions below, the student is free to develop and propose their own original ideas within this research theme. Possible directions include:
- Enhancing Model Depth: Investigate how ISA prevents attention score shift and potential attention collapse in very deep ViTs, enabling deeper and more efficient network architectures.
- Optimizing for Efficiency: Improve the computational performance of ISA and explore alternative formulations, such as Inhibited Sigmoid Self-Attention, to balance effectiveness with speed.
- Boosting OOD Robustness: Systematically evaluate how ISA's inherent properties—improved object focus and reduced shortcut learning—translate to superior performance on Near-OOD, Far-OOD, and Covariate-Shifted benchmarks (e.g., SSB-hard, ImageNet-C).
Skills Gained:
- Deep learning with PyTorch and Transformers
- Attention visualization and analysis
- Experiment design and evaluation
Expected Outcome:
A solid understanding of inhibitory mechanisms in deep transformers, with reproducible experiments demonstrating their impact. Results could contribute to scientific publications.