steve sparks
10 of 42
ContentsPart II · Chapter 5

Chapter 5From early models to today

From counting to transformers

Language models went from counting letters, to reading word by word, to reading everything at once. Then they grew huge, learned to chat, learned to think, and learned to act.

The line from 1948 to today

Illustration: real, sourced dates; the line itself is a picture

Today's AI is the latest step in more than 75 years of "predict the next symbol," and each era added one new ability.

1948–1996 Counting symbols

Guess the next letter or word from the ones before it.

  1. 1948Shannon predicts English. Claude Shannon builds English-like text by picking each letter or word by how likely it is to follow the previous ones.12
  2. 1951People as predictors. Shannon has people guess English text one letter at a time, to measure how predictable it is.32
  3. 1966ELIZA (a contrast). Joseph Weizenbaum's ELIZA chats by matching keywords to hand-written rules. No prediction and no learning: a useful contrast with what came later.45

1997–2016 Neural networks read in order

Words become numbers; networks read one word at a time.

  1. 1997LSTM. Networks that can carry memory across long sequences.6
  2. 2003Neural language model. A neural network learns word meanings as numbers while predicting the next word.7
  3. 2013word2vec. Word arithmetic: king − man + woman lands near queen.89
  4. 2014Seq2seq and attention. Networks translate whole sentences, and 'attention' lets them look back at the right words.1011

2017–2021 Transformers and scale

Read everything at once, which makes huge models possible.

  1. June 12, 2017Attention Is All You Need. Eight researchers drop step-by-step reading entirely. The transformer reads a whole passage at once.12
  2. 2018GPT-1 and BERT. Pretrain on lots of text, then fine-tune for a task.1314
  3. Feb. 14, 2019GPT-2. Fluent paragraphs. OpenAI holds back the full model at first over misuse concerns.15
  4. Jan. 23, 2020Scaling laws. Bigger models, more data and more compute make prediction error fall predictably.16
  5. May 22, 2020Retrieval-augmented generation. Look documents up first, then answer from them.17
  6. May 28, 2020GPT-3. 175 billion parameters. Learns new tasks from a few examples in the prompt.18

Dates and descriptions from each source; the era names and boundaries are this page's own grouping. ELIZA is shown as a contrast: it is famous, but it doesn't predict anything.

Notes (18)
  1. 1C. E. Shannon, “A Mathematical Theory of Communication,” Bell System Technical Journal 27, no. 3 (July 1948): 379–423; 27, no. 4 (October 1948): 623–656, https://doi.org/10.1002/j.1538-7305.1948.tb01338.x.
  2. 2Steve Sparks, “From Shannon's bits to Claude's tokens,” stevesparks.dev, 2026, https://stevesparks.com/retro/shannon/ (accessed October 26, 2026).
  3. 3C. E. Shannon, “Prediction and Entropy of Printed English,” Bell System Technical Journal 30, no. 1 (January 1951): 50–64, https://doi.org/10.1002/j.1538-7305.1951.tb01366.x.
  4. 4Joseph Weizenbaum, “ELIZA—A Computer Program for the Study of Natural Language Communication Between Man and Machine,” Communications of the ACM 9, no. 1 (January 1966): 36–45, https://doi.org/10.1145/365153.365168.
  5. 5Steve Sparks, “ELIZA (1966), playable,” stevesparks.dev, 2026, https://stevesparks.com/arcade/eliza (accessed October 26, 2026).
  6. 6Sepp Hochreiter and Jürgen Schmidhuber, “Long Short-Term Memory,” Neural Computation 9, no. 8 (November 1997): 1735–1780, https://doi.org/10.1162/neco.1997.9.8.1735.
  7. 7Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin, “A Neural Probabilistic Language Model,” Journal of Machine Learning Research 3 (February 2003): 1137–1155, https://www.jmlr.org/papers/v3/bengio03a.html.
  8. 8Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean, “Efficient Estimation of Word Representations in Vector Space” (preprint, arXiv, January 16, 2013), https://doi.org/10.48550/arXiv.1301.3781.
  9. 9Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig, “Linguistic Regularities in Continuous Space Word Representations,” in Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Atlanta, Georgia: Association for Computational Linguistics, June 2013), 746–751, https://aclanthology.org/N13-1090/.
  10. 10Ilya Sutskever, Oriol Vinyals, and Quoc V. Le, “Sequence to Sequence Learning with Neural Networks” (preprint, arXiv, September 10, 2014), https://doi.org/10.48550/arXiv.1409.3215.
  11. 11Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate” (preprint, arXiv, September 1, 2014), https://doi.org/10.48550/arXiv.1409.0473.
  12. 12Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin, “Attention Is All You Need” (preprint, arXiv, June 12, 2017), https://doi.org/10.48550/arXiv.1706.03762.
  13. 13A. Radford et al., “Improving Language Understanding by Generative Pre-Training,” OpenAI, 2018, https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf (accessed October 26, 2026).
  14. 14Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (2019), 4171–4186, https://doi.org/10.18653/v1/n19-1423.
  15. 15OpenAI, “Better language models and their implications,” February 14, 2019, https://openai.com/index/better-language-models/ (accessed October 26, 2026).
  16. 16Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei, “Scaling Laws for Neural Language Models” (preprint, arXiv, January 23, 2020), https://doi.org/10.48550/arXiv.2001.08361.
  17. 17Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (preprint, arXiv, May 22, 2020), https://doi.org/10.48550/arXiv.2005.11401.
  18. 18Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, et al., “Language Models are Few-Shot Learners” (preprint, arXiv, May 28, 2020), https://doi.org/10.48550/arXiv.2005.14165.

Chapter 5 · page 1 of 3

Turn pages with the ← and → keys.

Cite this page

https://aifromthepartsup.com/from-early-models-to-today/1/

Using this book

© 2026 Steve Sparks. All rights reserved. You're welcome to quote short passages and cite any page, with a link back. To reuse diagrams or larger parts, ask through GitHub.

Quoted material and figures from other sources remain their owners'.