steve sparks

Retro Lab · History

From Shannon's bits to Claude's tokens

In 1948, Claude Shannon generated English-like text by hand, picking each letter or word according to how often it follows the ones before.1 Today's large language models make the same kind of prediction, learned from vast amounts of text.2

Random letters1948, zero order
XFOML RXKHRJFFJUJ ZLPWCFWKCYJ
Letter frequencies1948, first order
OCRO HLI RGWR NMIELWIS EU LL NBNESEBYA TH EEI
Three-letter patterns1948, third order
IN NO IST LAT WHEY CRATICT FROURE BIRS GROCID
Word pairs1948, second-order words
THE HEAD AND IN FRONTAL ATTACK ON AN ENGLISH WRITER
A modern language modelToday, learned from vast text
Shannon measured information in …

Illustrative probabilities, not real model output. The model scores every possible next token, then samples one.

Specimens excerpted from Shannon's "A Mathematical Theory of Communication" (1948). He made the first two with a book of random numbers and a table of letter frequencies, and the rest by opening printed books at random, one letter or word at a time.1

Three threads from 1948 to today

Shannon did not invent language models. He supplied the ideas they are built and measured on. Each thread below starts in his papers and runs through later work into how models like Claude are trained.

Prediction

Language can be modeled as guessing what comes next.

  1. 1948

    Shannon generates English-like text from letter and word statistics, and concludes that a complex enough random process can represent a language source.1

  2. 1951

    In "Prediction and Entropy of Printed English," a test subject guesses a passage letter by letter, and the guesses measure how predictable English is.5

  3. 2003

    Bengio and colleagues train a neural network to predict the next word, learning word embeddings along the way.6

  4. 2017

    The Transformer's attention mechanism trains far more in parallel than earlier networks,7 which made next-token prediction practical at very large scale.8

  5. Today

    Language models write each response one token at a time, sampling from a learned probability distribution over their vocabulary.910

Measurement

Information has a unit, and predictability can be counted in it.

  1. 1948

    Shannon defines entropy and names its unit the bit, a word he credits to John Tukey.111

  2. 1951

    With up to 100 letters of context, English carries roughly one bit per letter, within Shannon's estimated bounds of about 0.6 to 1.3.5

  3. n-gram era

    Cross-entropy and perplexity, both built on Shannon entropy, become standard yardsticks for comparing language models.6

  4. 2020

    Scaling-laws research, four of whose ten authors later co-founded Anthropic, shows cross-entropy loss falling predictably with model size, data and compute, and notes the trend must level off because language itself has non-zero entropy.812

  5. Today

    Pretraining means minimizing cross-entropy: how surprised the model is by the actual next token, measured in Shannon's units.8

Compression

A better predictor is a better compressor, and the reverse.

  1. 1948

    The source coding theorem: no code can average fewer bits per symbol than the source's entropy, and good codes can come arbitrarily close.1

  2. 1994

    Philip Gage's Byte Pair Encoding compresses files by repeatedly replacing the most common pair of bytes with a single unused byte.13

  3. 2006

    The Hutter Prize rewards compressing Wikipedia, treating compression as a measurable stand-in for intelligence.14

  4. 2016

    Sennrich and colleagues adapt BPE to split words into subword tokens. Its descendants are how most LLMs tokenize text.1510

  5. 2023

    Google DeepMind's "Language Modeling Is Compression" shows the 70B Chinchilla model, trained mostly on text, compressing image patches to 43% and audio to 16% of their raw size, better than PNG (58%) and FLAC (30%), not counting the size of the model itself.16

Shannon's own workLater researchModern LLMs

The same experiment, at a different scale

Shannon, 1951
Claude today

Shannon's subject predicted the next letter from up to 100 letters of preceding text.5 According to Anthropic's documentation, current Claude models can draw on up to a million tokens,17 at least ten thousand times as much text. Bars are not to exact scale; at true scale Shannon's would be invisible.

The groundwork beneath it all

Shannon's other contributions shaped the hardware AI runs on and the field of AI itself.

1937

Logic in circuits

His MIT master's thesis showed Boolean algebra can describe switching circuits.1811 Every chip running a model today is, at bottom, a vast switching circuit.

1950

Search and evaluation

"Programming a Computer for Playing Chess" laid out look-ahead search scored by an evaluation function.19

1950

A machine that learns

Theseus, his relay-controlled mechanical mouse, found its way through a maze by trial and error and remembered the route.202111

1955

Proposing the field

Shannon co-authored the 1955 Dartmouth proposal, whose title used the term "artificial intelligence" and which set the goal of finding "how to make machines use language."22

For how today's models use that groundwork, part by part, see the book AI, from the Parts Up.

In Shannon's words

"reproducing at one point either exactly or approximately a message selected at another point"

The fundamental problem of communication, 1948, p. 3791

His definition of the problem. A language model is, in a sense, an engine for the "approximately" half.

"of the order of one bit per letter"

Prediction and Entropy of Printed English, 1951, p. 505

His estimate of how much new information each letter of English carries once context is known.

"a point at which this statistical approach is going to break down"

Discussion of "The Redundancy of English," Macy Conference on Cybernetics, 1950 (published 1951), p. 13923

Shannon doubted statistics could capture the long-range structure of language. Modern LLMs test that doubt at a scale he never saw.

"it has perhaps been ballooned to an importance beyond its actual accomplishments"

The Bandwagon, 1956, p. 3, on information theory's popularity24

Shannon cautioned against overselling his own theory. It remains good advice for how we talk about AI.

Read more

On my reading list

A Mind at Play

Jimmy Soni and Rob Goodman's biography, How Claude Shannon Invented the Information Age (New York: Simon & Schuster, 2017).11 The story behind everything on this page, from the thesis to the unicycles. It's in my books list.

Links go to English Wikipedia, for further reading only; the claims on this page rest on the sources below.

Sources

Research papers

  1. 1C. E. Shannon, “A Mathematical Theory of Communication,” Bell System Technical Journal 27, no. 3 (July 1948): 379–423; 27, no. 4 (October 1948): 623–656, https://doi.org/10.1002/j.1538-7305.1948.tb01338.x. Cited at pages 379, 380, 388–389, 389, 392–393, 401–403.
  2. 2Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, et al., “Language Models are Few-Shot Learners” (preprint, arXiv, May 28, 2020), https://doi.org/10.48550/arXiv.2005.14165.
  3. 5C. E. Shannon, “Prediction and Entropy of Printed English,” Bell System Technical Journal 30, no. 1 (January 1951): 50–64, https://doi.org/10.1002/j.1538-7305.1951.tb01366.x. Cited at pages 50, 54–57, 57, 64.
  4. 6Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin, “A Neural Probabilistic Language Model,” Journal of Machine Learning Research 3 (February 2003): 1137–1155, https://www.jmlr.org/papers/v3/bengio03a.html. Cited at pages 1137, 1141.
  5. 7Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin, “Attention Is All You Need” (preprint, arXiv, June 12, 2017), https://doi.org/10.48550/arXiv.1706.03762.
  6. 8Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei, “Scaling Laws for Neural Language Models” (preprint, arXiv, January 23, 2020), https://doi.org/10.48550/arXiv.2001.08361. Cited at pages 1, 6, 17.
  7. 9Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi, “The Curious Case of Neural Text Degeneration” (preprint, arXiv, April 22, 2019), https://doi.org/10.48550/arXiv.1904.09751.
  8. 10Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever, Language Models are Unsupervised Multitask Learners (San Francisco: OpenAI, February 2019), https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf (accessed October 26, 2026). Cited at page 4.
  9. 15Rico Sennrich, Barry Haddow, and Alexandra Birch, “Neural Machine Translation of Rare Words with Subword Units,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2016), 1715–1725, https://doi.org/10.18653/v1/P16-1162. Cited at pages 1715–1717.
  10. 16Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, Marcus Hutter, and Joel Veness, “Language Modeling Is Compression,” in International Conference on Learning Representations (ICLR 2024) (2024), https://doi.org/10.48550/arXiv.2309.10668.
  11. 18Claude E. Shannon, “A Symbolic Analysis of Relay and Switching Circuits,” Transactions of the American Institute of Electrical Engineers 57, no. 12 (December 1938): 713–723, https://doi.org/10.1109/T-AIEE.1938.5057767. Cited at page 713.
  12. 19Claude E. Shannon, “Programming a Computer for Playing Chess,” The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 41, no. 314 (March 1950): 256–275, https://doi.org/10.1080/14786445008521796. Cited at §§3–4.
  13. 24Claude E. Shannon, “The Bandwagon,” IRE Transactions on Information Theory 2, no. 1 (March 1956): 3, https://doi.org/10.1109/TIT.1956.1056774. Cited at page 3.

Books

  1. 11Jimmy Soni and Rob Goodman, A Mind at Play: How Claude Shannon Invented the Information Age (New York: Simon & Schuster, 2017). Cited at pages 39, 141, 203–204.

Oral histories, museums and archives

  1. 20Claude E. Shannon, “Presentation of a Maze-Solving Machine,” in Cybernetics: Circular Causal and Feedback Mechanisms in Biological and Social Systems. Transactions of the Eighth Conference, March 15–16, 1951, New York, N.Y. (New York: Josiah Macy, Jr. Foundation, 1952), 173–180, https://archive.org/details/cybernetics0000hein_i4t4. Cited at page 173.
  2. 21Deborah Halber, “Shannon collection shows wit and whimsy,” MIT News, May 30, 2007, https://news.mit.edu/2007/shannon-0530 (accessed October 26, 2026).
  3. 22J. McCarthy, M. L. Minsky, N. Rochester, and C. E. Shannon, A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence (John McCarthy's archive, Stanford University, August 31, 1955), https://www-formal.stanford.edu/jmc/history/dartmouth/dartmouth.html (accessed October 26, 2026).
  4. 23Claude E. Shannon, “The Redundancy of English,” in Cybernetics: Circular Causal and Feedback Mechanisms in Biological and Social Systems. Transactions of the Seventh Conference, March 23–24, 1950, New York, N.Y. (New York: Josiah Macy, Jr. Foundation, 1951), 123–158, https://archive.org/details/cybernetics0000hein. Cited at page 139.

Anthropic

  1. 12Anthropic, “Leadership at Anthropic,” n.d., https://www.anthropic.com/company/leadership (accessed October 26, 2026).
  2. 17Anthropic, “Context windows,” Claude Platform Docs, n.d., https://platform.claude.com/docs/en/build-with-claude/context-windows (accessed October 26, 2026).

Data and explainers

  1. 14Marcus Hutter, “Hutter Prize for Lossless Compression of Human Knowledge,” prize.hutter1.net, 2006 (updated 2020), http://prize.hutter1.net/ (accessed October 26, 2026).

Press

  1. 3Kevin Roose, “Inside the White-Hot Center of A.I. Doomerism,” The New York Times, July 11, 2023, https://www.nytimes.com/2023/07/11/technology/anthropic-ai-claude-chatbot.html (accessed October 26, 2026).
  2. 4Gideon Lewis-Kraus, “What Is Claude? Anthropic Doesn't Know, Either,” The New Yorker, February 9, 2026, https://www.newyorker.com/magazine/2026/02/16/what-is-claude-anthropic-doesnt-know-either (accessed October 26, 2026).
  3. 13Philip Gage, “A New Algorithm for Data Compression,” The C Users Journal 12, no. 2 (February 1994): 23–38, https://jacobfilipp.com/DrDobbs/articles/CUJ/1994/9402/gage/gage.htm.

An educational overview of Shannon's influence on modern language models. Not affiliated with or endorsed by Anthropic.