Before large language models (LLMs) can process text, they split it into chunks such as words or word fragments. The LLM behind ChatGPT, for example, splits “LMU München” into the three chunks “LM”, “U” and “München”, so it never directly sees the individual letters in “München.” This step, known as subword tokenization, makes LLMs efficient but creates limitations, including reduced character-level understanding.
Models that instead read text byte by byte — using the basic units in which computers store text — avoid some of these problems. In practice, however, they have not yet become a viable alternative to LLMs based on subword tokenization.
In a new study published in the journal Nature, researchers from LMU Munich, the Allen Institute for AI, the University of Cambridge, the University of Washington and Imperial College London have developed a method that converts existing LLMs into byte-level models without retraining them from scratch.
The method, called “byteification,” requires less than 1% of the training typically needed for such a model and largely preserves the performance of the original LLM while adding the benefits of byte-level processing. The resulting models outperform all previously published byte-level LLMs of comparable size.
For enterprise architects and IT leaders, the research points to a potentially more efficient approach to improving AI systems without the expense and complexity of building entirely new models. Byte-level processing could also prove valuable in enterprise environments where AI must interpret source code, structured data and other information requiring precise character-level understanding.
“We believe that byteification, and byte-level LLMs more generally, have the potential to overcome some long-standing shortcomings of LLMs,” said Valentin Hofmann, junior professor for information and language processing using AI methods at LMU Munich and last author of the study.
“Traditional LLMs struggle with tasks that require character-level capabilities, such as spelling a word backwards. Our byteified models are substantively better at this.”
These capabilities extend beyond academic applications.
“A good representation of the low-level structure of text is critical in many areas of science, for example when working with code or biological sequences,” Hofmann explained.
From an enterprise architecture and IT governance perspective, the findings may eventually influence how organizations evaluate and adapt foundational AI models, particularly as they seek to balance performance, computational costs and reliability. The ability to enhance existing models with relatively limited additional training could be especially relevant to organizations seeking greater flexibility in their AI infrastructure.
The researchers hope byteification will make byte-level models a practical alternative to today’s LLMs and open new research directions. The models, code and training data are publicly available.