Research1h ago
Apple Identifies Outlier Tokens In Diffusion Transformers
High-norm outlier tokens appear in both the encoder and denoiser of Representation Autoencoder pipelines. These tokens attract disproportionate attention but carry limited local information. Apple researchers found this phenomenon persists in intermediate layers of Diffusion Transformers. This discovery helps developers refine attention mechanisms to prevent these outliers from degrading image generation quality.