This paper explores the marginal benefits of pre-training and intermediate fine-tuning on a small 5M-parameter Vision Transformer for semantic segmentation. It reveals that while pre-training and fine-tuning provide benefits with diminishing returns, inappropriate intermediate fine-tuning can harm downstream performance.