Research2h ago
Apple Identifies Outlier Tokens in Diffusion Transformers
High-norm outlier tokens appear in both the encoder and denoiser of RAE-DiT pipelines. These tokens attract disproportionate attention but carry minimal local information. Apple researchers found this phenomenon persists especially in intermediate layers. This discovery helps developers refine attention mechanisms to prevent these outliers from degrading image generation quality and stability.