This paper introduces Geometric Positional Embedding (GeoPE), a novel framework that uses quaternions and 3D rotations to restore the natural spatial topology in Vision Transformers. GeoPE addresses the limitations of existing positional embeddings by geometrically coupling spatial dimensions, leading to improved performance in various vision tasks and enhanced shape bias.