Research2h ago
Apple Identifies Outlier Tokens In Diffusion Transformers
High-norm outlier tokens appear in both encoders and denoisers within Representation Autoencoder-DiT pipelines. These tokens attract disproportionate attention but carry minimal local information. Apple researchers found this phenomenon persists across intermediate layers. Addressing these outliers improves training stability and representation quality for practitioners building high-resolution image generation models.