Research1h ago
Apple Identifies Outlier Tokens in Diffusion Transformers
High-norm tokens in Diffusion Transformers attract disproportionate attention while carrying minimal local information. Researchers at Apple found these outliers in both the encoder and denoiser of RAE-DiT pipelines. This phenomenon occurs primarily in intermediate layers. Addressing these anomalies helps developers stabilize attention mechanisms and improve the efficiency of generative image models.