Chapter 1
Language Models

Technological progress tends not to be smooth. Sometimes, a single development unlocks a large amount of potential, resulting in a burst of activity in which many applications are found. Enthusiastic technologists rapidly explore a space of new inventions and ways of working, opened by a single advance in a fundamental field. In the world of data science, we are living through just such a period-the invention of the transformer neural network. Coinciding with sufficient computational power, this invention has allowed us to build machines that seem to have unlimited capabilities to learn deep semantic relationships from text data alone. This fundamental transformation has unlocked near perfect language translation, the ability to generate web pages from text descriptions, and the new commercial world of general-purpose Artificial Intelligence (AI) systems that can respond to a very wide variety of requests. This is all possible only because the foundation models they are built upon can consume the library of data on the Internet and build rich semantic representations of various human natural languages, programming languages, and other data structures.

Many tasks that can be accomplished with these language models were previously done by dedicated data science teams, collecting data and crafting algorithms for that specific purpose. In many instances, they still perform that work whenever the task requires higher levels of accuracy or security than the general-purpose AI systems permit. Nevertheless, the abilities of language models allow us to improve many aspects of the data science workflow, from data collection, prototyping, and even scripting solutions. In this book, we explore those opportunities, beginning with the most fundamental utility of the representation of text as semantic vectors, through to using models to write code or experiment with agent-based AI systems. However, we must start with a discussion of what a language model is and the gradual development of the technology that has unlocked this possibility.

What Is a Language Model?

The term language model emerged from linguistics to describe the attempt to model the statistical structure of language production. In its most simplistic form, this model could be a finite-state automaton that captures the probability of the next text character, given the current text character. I show this idea schematically in Figure 1.1, using the limited English alphabet. The state consists of the last character that was printed and determines the probability distribution over characters, with a new character sampled from this distribution. This new character is output and then used to update the state of the system.

Figure 1.1: A simple statistical language model.

As you might imagine, such a machine is incapable of learning the subtleties of human language. However, as shown in the following example output, you can see that certain common vowel-to-consonant pairings are routinely generated. It isn't English, but it has language-like properties.

This output was generated by a simple script that opens a dataset consisting of three short stories by the science fiction writer Phillip K. Dick. It cleans the text to remove new lines, underscores, and a variety of characters used to separate chapters and sections. Then it builds a statistical model that simply learns the probability of the next character from the current character. In building this simple model, we now have multiple key ideas of language models that will recur throughout the book.

The first idea is that text data must be broken into pieces known as tokens in a process called tokenization. These tokens could be the individual letters in the text, but in many instances, they are multi-letter compounds that depend on the language and text domain. The complete list of possible tokens that a model can recognize is called the vocabulary. In many instances, the model will have a special token for coping with any unrecognized token. Unrecognized tokens may occur, as the model is being used in text that extends beyond the domain of the training data.

When a block of text is tokenized into a list of tokens, the order of the tokens from the original string is maintained. The simple idea of tokenization is to determine the most granular representation of text, and the individual token identities in the list determine how the text is encoded for the model.

Text can be arbitrary length, so it follows that tokenized text becomes an arbitrary length list of tokens. However, in practice, modern language models operate with a finite length input, called the context window. In this very simple model, the context window is one, because it looks at only one token to determine the probability of the next token. However, we could extend this very easily. I have modified the script so that it builds a new set of probabilities based on all possible consecutive token pairs, to model the probability of the next token that follows the current pair. An example output of this new model is shown here:

I seeded this model with the input pair "Th," as those were the first two characters of the previous model, which it extended into the word "They." Already you can see that the output looks slightly more like English. There is no such thing as an "inkly bunkepler," but it sounds like the kind of thing we might find in a science fiction novel. We can extend the context window again to three characters and generate output one more time, which results in the following:

This was seeded with the three characters "The" to follow on from the previous model. Now you can see that the text is getting very close to resembling English language; in fact, many of the words in this salad of characters are genuine English words.

We could continue this process of gradually growing the context window and would generally see that the quality improves. This is because we are starting to capture the statistical patterns of words or other language structures. But at some point, the data will become so sparse that it will be difficult to learn with this simple probability model. Simple statistical language models were stuck at this point of development for a very long time. Soon I explain how modern language models overcome these difficulties, but at this stage it's important to recognize that models generally have a context window that you need to work within, and the atomic units they operate on are called tokens.

If you try to input text longer than this input limitation, the text will be truncated to the maximum length defined by the context window. Considerable research goes into finding methods that extend the size of context windows. However, at this time of writing, they remain a fundamental limitation, because running efficient and economical language models requires reducing the amount of text processing.

note

Tokens and Tokenizers

Text is converted to a sequence of tokens by a tokenizer. The tokenizer determines the complete vocabulary of the model. The model and tokenizer are intrinsically linked: If a token is not recognized by the tokenizer, then it is not recognized by the model. When working with any language model, you must understand the nature and limitations of the tokenizer.

Representing Meaning

The previous section described how you could gradually build up a simple language model, by learning increasingly complicated probability distributions based on a larger context window of tokens. Two key ideas should have emerged: First, the more awareness the model has of previous characters, the better you can model a language. Second, as the context window grows, so too does the number of parameters. So, the counteracting force is that you need exponentially increasing data to learn the model. It is, of course, arguable that this approach would never succeed in perfectly modeling language, because it doesn't form any kind of meaning abstraction from language; it merely models empirical probabilities of characters. However, regardless of that theoretical distinction, this approach will always fail practically, because we cannot possibly generate datasets large enough to learn the character probabilities of all possible sequences. What is required to solve this problem is some way of learning an internal representation of the language's structure, such that we can generalize to sentences not seen in the training data.

Learning internal representations is the purpose of the hidden layers of neural networks. Typically, they contain fewer nodes and parameters than the entire input space, thus forcing the model to learn some form of abstractions over the training data. You can improve the simple language models introduced in the previous section by including a layer of hidden nodes and imposing a learning process where the parameters are slowly adjusted using the training data. Nevertheless, these models tend to fall short of generating language that resembles human ability. Why do they fall short?

One main reason appears to be that words are, in fact, very flexible tools. We can use numerous words in multiple ways, and...