1
The Concept of Probability

Source: Keystone-France / Getty Images.

Regarded as the founder of modern probability theory, Kolmogorov was a Soviet mathematician whose work was also influential in several other scientific areas, notably in topology, constructive logic, classical mechanics, mathematical ecology, and algorithmic information theory.

He earned his Doctor of Philosophy (PhD) degree from Moscow State University in 1929, and two years later, he was appointed a professor in that university. In his book, Foundations of the Theory of Probability, which was published in 1933 and which remains a classic text to this day, he built up probability theory from fundamental axioms in a rigorous manner, comparable to Euclid's axiomatic development of geometry.

1.1 Chance Experiments - Sample Spaces

In this chapter, we present the main ideas and the theoretical background to understand what probability is and provide some illustrations of the way it is used to tackle problems in everyday life. It is rather difficult to try to answer the question "what is probability?" in a single sentence. However, from our experience and the use of this word in common language, we understand that it is a way to deal with uncertainty in our lives. In fact, probability theory has been referred to as "the science of uncertainty"; although intuitively most people associate probability with the degree of belief that something may happen, probability theory goes far beyond that as it attempts to formalize uncertainty in a way that is universally accepted and is also subject to rigorous mathematical treatment.

Since the idea of uncertainty is paramount when we discuss probability, we shall first introduce a concept that is broad enough to deal with uncertainties in a wide-ranging context when we consider practical applications. A chance experiment or a random experiment is any process which leads to an outcome that is not known beforehand. So tossing a coin, selecting a person at random and asking their age, or testing the lifetime of a new machine are all examples of random experiments.

Definition 1.1

A sample space of a chance experiment is the set of all possible outcomes that may appear in a realization of this experiment. The elements of are called sample points for this experiment. A subset of is called an event.

An event , consisting of a single sample point, i.e. a single outcome , is called an elementary event. We use capital letters , and so on to denote events.1 If an event consists of more than one outcome, then it is called a compound event.

The following simple examples illustrate the above concepts.

Example 1.1

Perhaps the simplest example of a chance experiment is tossing a coin. There are two possible outcomes - Heads (denoted by and Tails (denoted by . In this notation, the sample space of the experiment is

If we toss two coins instead, there are four possible outcomes, represented by the pairs . The sample space for this experiment is thus

Here, the symbol means that both coins land Heads, while means that the first coin lands Heads and the second lands Tails. Note in particular that we treat the two events and as distinguishable, rather than combining them into a single event. The main reason for this is that the events and are elementary events, while the event "one coin lands Heads and the other lands Tails," which contains both and , is no longer an elementary event. As we will see later on, when we assign probabilities to the events of a sample space, it is much easier to work with elementary events, since in many cases such events are equally likely, and so it is reasonable the same probability to be assigned to each of them.

Consider now the experiment of tossing three coins. The sample space consists of triplets of the form , and so on. Since for each coin toss there are two outcomes, for three coins the number of possible outcomes is . More explicitly, the sample space for this experiment is

Each of the eight elements of this set forms an elementary event. Note that for events which are not elementary, it is sometimes easier to express them in words, by describing a certain property shared by all elements of the event we consider, rather than by listing all its elements (which may become inconvenient if these elements are too many). For instance, let be the event "exactly two Heads appear when we toss three coins." Then,

On the other hand, the event

could be described in words as "all three coin outcomes are the same."

Example 1.2

Another very simple experiment consists of throwing a single die. The die may land on any face with a number on it, where takes the values . Therefore, this experiment has sample space

The elementary events are the sets

Any other event may again be described either by listing the sample points in it, such as

or, in words, by expressing a certain property of its elements. For instance, the event

may be expressed as "the outcome of the die is an even integer."

For the experiments we considered in the above two examples, the number of sample points was finite in each case. For instance, in Example 1.2, the sample space has six elements, while in the experiment of throwing three coins there are eight sample points as given in 1.1. Such sample spaces which contain a finite number of elements (possible outcomes) are called finite sample spaces. It is obvious that any event, i.e. a subset of the sample space, in this case has also finitely many elements.

When dealing with finite sample spaces, the process of enumerating their elements, or the elements of events in such spaces, is often facilitated by the use of tree diagrams. Figure 1.1 depicts such a diagram which corresponds to the experiment of tossing three coins, as considered in Example 1.1.

Figure 1.1 Tree diagram for the experiment of tossing three coins.

From the "root" of the tree, two segments start, each representing an outcome ( and , resp.) of the first coin toss. Thus, at the first stage, i.e. after the first throw, there are two nodes. From each of these, in turn, two further segments start corresponding to the two outcomes of the second toss. At the end of the second stage (after the second toss of the coin), there are four nodes. Finally, each of these is associated with two further nodes, which are shown at the next level (end of the three coin tosses). The final column in Figure 1.1 shows the eight possible outcomes for this experiment, i.e. it contains all the elements of the sample space . Each outcome can be traced by connecting the endpoint to the root and writing down the corresponding three-step tree route.

Example 1.3 (An unlimited sequence of coin tosses)

Let us consider the experiment of tossing a coin until "Tails" appear for the first time. In this case, our sample space consists of sequences like ; that is, The event "Tails appear for the first time at the fifth trial" is then the elementary event

while the set

has as its elements all outcomes where Tails appear in the first three tosses. So the event can be described by saying "the experiment is terminated within the first three coin tosses." Finally, the event "there are at least four tosses until the experiment is terminated" corresponds to the set (event)

In the previous example, the sample space has infinitely many points. In particular, and since these points can be enumerated, we speak of a countably infinite sample space. Examples of sets with countably2 many points are the set of integers, the set of positive integers, the set of rationals, etc. When a sample space is countably infinite, the events of that sample space may have either finitely many elements (e.g. the event in Example 1.3) or infinitely many elements (e.g. the event in Example 1.3).

In contrast, a set whose points cannot be enumerated, is called an uncountable set; typical examples of such sets are intervals and unions of intervals on the real line. To illustrate this, we consider the following example.

Example 1.4

In order to monitor the quality of light bulbs that are produced by a manufacturing line, we select a bulb at random, plug it in and record the length of time (in hours) until it fails. In principle, this time length may take any nonnegative real value (however, this presupposes we can take an infinitely accurate measurement of time!). Therefore, the sample space for the experiment whose outcome is the life duration of the bulb is

The subset of

describes the event "the life time of the light bulb does not exceed 500 hours," while the event...