This paper introduces Geometric Positional Embedding (GeoPE), a novel framework that uses quaternions and 3D rotations to restore the natural spatial topology in Vision Transformers. GeoPE addresses the limitations of existing positional embeddings by geometrically coupling spatial dimensions, leading to improved performance in various vision tasks and enhanced shape bias.

This paper explores the marginal benefits of pre-training and intermediate fine-tuning on a small 5M-parameter Vision Transformer for semantic segmentation. It reveals that while pre-training and fine-tuning provide benefits with diminishing returns, inappropriate intermediate fine-tuning can harm downstream performance.