Pyramid Sparse Attention (PSA) introduces multi-level pooled key-value representations to address the quadratic complexity and information loss challenges of existing sparse attention mechanisms. By dynamically allocating finer pooling levels to critical blocks and coarser levels to less important ones, PSA significantly expands the receptive field and preserves contextual information, outperforming current baselines in video understanding and generation tasks.