Byte Models Outperform Subword Tokenization At Scale | dailyai.report
25 stories from today
Research
2d ago
Byte Models Outperform Subword Tokenization At Scale
Raw byte sequences outperform traditional subword models as parameter sizes scale. Researchers used token-superposition training and hash embeddings to prove that longer sequences provide more computation per parameter. This challenges the industry assumption that tokenizers are necessary for efficiency.
The Signal
Practitioners can now consider flat Transformers to achieve lower optimal loss without complex preprocessing.