Chapter 1
Preliminaries

1.1 Basics of Probability

1.1.1 Introduction

In this chapter, we introduce some basics of probability that will be needed in the later chapters. We also take the liberty in stating some theorems without presenting proofs and emphasize that the contents of this chapter, by no means, represent all topics of probability that deserve a detailed discussion.

A chance or random experiment is an experiment whose outcomes or results of its performance are uncertain. A set of outcomes is called an event. The set of all possible outcomes is referred to as the sample space, denoted by . Thus, an event is a subset of the sample space and an element of the sample space is a sample point.

Two events and are referred to as mutually exclusive if their intersection is empty. A set is called a partition of if the events are mutually exclusive, such that .

Probability of an event E, denoted by , indicates a number between 0 and 1 (inclusive), describing the likelihood of occurrence of the event E. There are two particular events: an event with probability 1 (referred to as almost sure event) and another with probability 0 (referred to as null or impossible event).

For a finite sample space with n elements, if all outcomes have the same chance to occur, each member is assigned a probability of 1/n and the sample space is called equiprobable. In case of an infinite sample space, elements are with uniform measure.

If a chance experiment is repeated, the chance of occurrence of an outcome is the ratio of the number of occurrences of the outcome to the total number of repetitions. Hence, for a sample space with n equiprobable points, the probability of an event with k points is k/n, referred to as relative frequency, and it is an approximation of the probability of the event. In other words, for a sample space with equiprobable sample points, if E is an event with k points, the probability of the event E is given by

The number of elements of the event E is referred to as the "size" of E. Thus, probability of the event E may be defined as

The triplet is called the probability space associated with the random experiment, where

1. a. is the sample space, that is, the set of all outcomes of the random experiment.
2. b. is the set function containing all possible events drawn from , which has the structure of the field. This means that satisfies the following conditions:
   1. i. the empty set is in ,
   2. ii. if , then the complement of E is also in , and
   3. iii. if , then .
3. c. P is the probability (measure) of an event. In fact, P is a function that associates a number for each element E of with the following properties (called axioms of probability):
   1. Axiom 1 , for each event E is .
   2. Axiom 2 .
   3. Axiom 3 For any sequence of mutually exclusive events (disjoint sets, that is, if ) in ,

1.1.2 Conditional Probability

For the probability space , let B be an event (that is, ) and P(B) > 0. Then, given the probability of an event A, the conditional probability of B, denoted by , defined on , is given by

If , then is not defined.

It should be noted that conditional probability exhibits properties similar to ordinary probability, but restricted to a smaller space.

One of the concepts often needed is the "independence" of events. We offer the definition for two events that can be easily expanded. Hence, we have the following:

Two events A and B are independent if and only if

In other words, occurrence of one does not affect the chance of occurrence of the other. Relation (1.4) can be expanded for an arbitrary number of events. Hence, the arbitrary family of events , where is the set of natural numbers, is independent if

for every finite subset of indices .

As a consequence of Equation (1.4), it can be easily proved that if two events A and B are independent and then

and conversely, if and Equation (1.6) holds to be true, then A and B are independent.

As another application of independence, the following is called the multiplicative law. Although we offer the definition for only two events, it can be expanded for any finite number of events. Thus, for any two events A and B with conditional probability or and as long as or is nonnegative, we have

Another property of conditional probability that can be easily verified called the law of total probability or total probability theorem is that if is a partition of the sample space , that is, n mutually exclusive events whose sum is unity, then for any given arbitrary event E, we have

Using Equation (1.8) and the conditional probability, a very important theorem called the Bayes' theorem can be proved. It can be stated as follows: If E is an event and a partition of the sample space , then

Relation (1.9) is referred to as Bayes' formula. The conditional probability is called the posterior probability. The original probability of is called the prior probability of .

1.2 Discrete Random Variables and Distributions

Although sample points are numerical in many practical problems, there are nonnumerical in many others. For practical purposes, a numerical sample space is more desirable. The tool to quantify a sample space to a numerical one is the random variable.

Before defining a random variable, we define a countable set. A set is called countable if it has the same number of elements (cardinality) as some subset of the set of natural numbers . In case of a finite subset of , the set is sometimes referred to as finitely countable or at most countable. For the case of the same cardinality as , it is sometimes referred to as countably infinite. A set that is not countable is called uncountable or denumerable. Throughout the book, we may use any of the terms as may be appropriate. Thus for us, the set of natural numbers , the set of natural numbers including zero , the set of integers , and the set of rational numbers (reactions) are all countable or infinitely countable sets. However, is at most countable or finitely countable. We note that all infinitely countable sets are of the same size, infinite. However, the set of real numbers and an interval are not countable. The latter was first proved by George Cantor using the diagonal argument method.

Thus, a random variable is defined as a function (mapping) that assigns a numerical value (or a set of values) to a sample point. Hence, If X is a random variable, it assigns a value to each outcome in . If the function X is on an at most countable sample space into the set of real numbers, , the random variable X is called a discrete random variable. Thus, a random variable is discrete if it takes at most countably many values. In other words, if there is a finitely countable set of real numbers, say A, such that .

For the sake of convenience, we may, allow a discrete random variable to assume positive infinity, as one of its values such as waiting time for an event to occur for the first time. This is because the event may never occur.

We leave it as an exercise for the reader to prove the following properties of discrete random variables. If X and Y are two discrete random variables, then , , and are also random variables, for the last one, provided Y is nonzero.

Probability distribution of a discrete random variable indicates the assignment of probabilities over the entire values of a random variable.

It is important to note that probabilities are nonnegative real numbers and total assignment of probabilities must be 1. Hence, these two simple properties establish two conditions for a function to be a probability distribution function.

Let the discrete random variable X be defined on a sample space with a typical element x. Then, the probability mass function (pmf) of X, denoted by , is defined as . If no specific value is given, we denote it by .The pmf is sometimes described by a table or matrix. For instance, if X is a random variable with the general vale x and specific values , that are assigned probabilities , then the pmf can be written as in Table 1.1.

Table 1.1...