Chapter 2How does it work?
Tokens
Models read tokens, which are word-pieces. Tokens are what fill the context window, and what usage is counted in.
Tokens. Text is chopped into word-pieces called tokens; each becomes a number.1 (Try it below.)
Notes (4)
- 1Rico Sennrich, Barry Haddow, and Alexandra Birch, “Neural Machine Translation of Rare Words with Subword Units,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2016), 1715–1725, https://doi.org/10.18653/v1/P16-1162.
- 2Aeree Cho, Grace C. Kim, Alexander Karpekov, Alec Helbling, Zijie J. Wang, Seongmin Lee, Benjamin Hoover, and Duen Horng (Polo) Chau, “TRANSFORMER EXPLAINER: Interactive Learning of Text-Generative Models,” Proceedings of the AAAI Conference on Artificial Intelligence 39, no. 28 (April 11, 2025): 29625–29627, https://doi.org/10.1609/aaai.v39i28.35347.
- 3Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei, “Scaling Laws for Neural Language Models” (preprint, arXiv, January 23, 2020), https://doi.org/10.48550/arXiv.2001.08361.
- 4Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, et al., “Training Compute-Optimal Large Language Models,” in Advances in Neural Information Processing Systems 35 (2022), 30016–30030, https://doi.org/10.52202/068431-2176.
Chapter 2 · page 2 of 3
Turn pages with the ← and → keys.