Introduction

What is Probability? In essence:

Mathematical modeling of random events and phenomena. It is fundamentally different from modeling deterministic events and functions, which constitutes the traditional study of Mathematics.

However, the study of probability uses concepts and notions straight from Mathematics; in fact Measure Theory and Potential Theory are expressions of abstract mathematics generalizing the Theory of Probability.

Like so many other branches of mathematics, the development of probability theory has been stimulated by the variety of its applications. In turn, each advance in the theory has enlarged the scope of its influence. Mathematical statistics is one important branch of applied probability; other applications occur in such widely different fields as genetics, biology, psychology, economics, finance, engineering, mechanics, optics, thermodynamics, quantum mechanics, computer vision, geophysics,etc. In fact I compel the reader to find one area in today's science where no applications of probability theory can be found.

Early history

In the XVII-th century the first notions of Probability Theory appeared. More precisely, in 1654 Antoine Gombaud Chevalier de Méré, a French nobleman with an interest in gaming and gambling questions, was puzzled by an apparent contradiction concerning a popular dice game. The game consisted of throwing a pair of dice 24 times; the problem was to decide whether or not to bet even money on the occurrence of at least one "double six" during the 24 throws. A seemingly well-established gambling rule led de Méré to believe that betting on a double six in 24 throws would be profitable (based on the payoff of the game). However, his own calculations based on many repetitions of the 24 throws indicated just the opposite. Using modern probability language de Méré was trying to establish if such an event has probability greater than 0.5 (we are looking at this question in example 1.7). Puzzled by this and other similar gambling problems he called on the famous mathematician BlaisePascal. This, in turn led to an exchange of letters between Pascal and another famous French mathematician Pierre de Fermat. This is the first known documentation of the fundamental principles of the theory of probability. Before this famous exchange of letters, a few other simple problems on games of chance had been solved in the XV-th and XVI-th centuries by Italian mathematicians; however, no general principles had been formulated before this famous correspondence.

In 1655 during his first visit to Paris, the Dutch scientist Christian Huygens learned of the work on probability carried out in this correspondence. On his return to Holland in 1657, Huygens wrote a small work De Ratiociniis in Ludo Aleae, the first printed work on the calculus of probabilities. It was a treatise on problems associated with gambling. Because of the inherent appeal of games of chance, probability theory soon became popular, and the subject developed rapidly during the XVIII-th century.

The XVIII-th century

The major contributors during this period were Jacob Bernoulli (1654-1705) and Abraham de Moivre (1667-1754). Jacob (Jacques) Bernoulli was a Swiss mathematician who was the first to use the term integral. He was the first mathematician in the Bernoulli family, a family of famous scientists of the XVIII-th century. Jacob Bernoulli's most original work was Ars Conjectandi published in Basel in 1713, eight years after his death. The work was incomplete at the time of his death but it still was a work of the greatest significance in the development of the Theory of Probability. De Moivre was a French mathematician who lived most of his life in England1. De Moivre pioneered the modern approach to the Theory of Probability, in his work The Doctrine of Chance: A Method of Calculating the Probabilities of Events in Play in the year 1718. A Latin version of the book had been presented to the Royal Society and published in the Philosophical Transactions in 1711. The definition of statistical independence appears in this book for the first time. The Doctrine of Chance appeared in new expanded editions in 1718, 1738 and 1756. The birthday problem (example 1.12) appeared in the 1738 edition, the gambler's ruin problem (example 1.11) in the 1756 edition. The 1756 edition of The Doctrine of Chance contained what is probably de Moivre's most significant contribution to probability, namely the approximation of the binomial distribution with the normal distribution in the case of a large number of trials - which is now known by most probability textbooks as "The First Central Limit Theorem" (we will discuss this theorem in Chapter 4). He understood the notion of standard deviation and is the first to write the normal integral (and the distribution density). In Miscellanea Analytica (1730) he derives Stirling's formula (wrongly attributed to Stirling) which he uses in his proof of the central limit theorem. In the second edition of the book in 1738 de Moivre gives credit to Stirling for an improvement to the formula. De Moivrewrote:

"I desisted in proceeding farther till my worthy and learned friend Mr James Stirling, who had applied after me to that inquiry, [discovered that ]."

De Moivre also investigated mortality statistics and the foundation of the theory of annuities. In 1724 he published one of the first statistical applications to finance Annuities on Lives, based on population data for the city of Breslau. In fact, in A History of the Mathematical Theory of Probability (London, 1865), Isaac Todhunter says that probability:

... owes more to [de Moivre] than any other mathematician, with the single exception of Laplace.

De Moivre died in poverty. He did not hold a university position despite his influential friends Leibnitz, Newton, and Halley, and his main income came from tutoring.

De Moivre, like Cardan (Girolamo Cardano), predicted the day of his own death. He discovered that he was sleeping 15 minutes longer each night and summing the arithmetic progression, calculated that he would die on the day when he slept for 24 hours. He was right!

The XIX-th century

This century saw the development and generalization of the early Probability Theory. Pierre-Simon de Laplace (1749-1827) published Théorie Analytique des Probabilités in 1812. This is the first fundamental book in probability ever published (the second being Kolmogorov's 1933 monograph). Before Laplace, probability theory was solely concerned with developing a mathematical analysis of games of chance. The first edition was dedicated to Napoleon-le-Grand, but the dedication was removed in later editions!2

The work consisted of two books and a second edition two years later saw an increase in the material by about 30 per cent. The work studies generating functions, Laplace's definition of probability, Bayes rule (so named by Poincaré many years later), the notion of mathematical expectation, probability approximations, a discussion of the method of least squares, Buffon's needle problem, andinverse Laplace transform. Later editions of the Théorie Analytique des Probabilités also contains supplements which consider applications of probability to determine errors in observations arising in astronomy, the other passion of Laplace.

On the morning of Monday 5 March 1827, Laplace died. Few events would cause the Academy to cancel a meeting but they did so on that day as a mark of respect for one of the greatest scientists of all time.

Century XX and modern times

Many scientists have contributed to the theory since Laplace's time; among the most important are Chebyshev, Markov, von Mises, and Kolmogorov.

One of the difficulties in developing a mathematical theory of probability has been to arrive at a definition of probability that is precise enough for use in mathematics, yet comprehensive enough to be applicable to a wide range of phenomena. The search for a widely acceptable definition took nearly three centuries and was marked by much controversy. The matter was finally resolved in the 20th century by treating probability theory on an axiomatic basis. In 1933, a monograph by the Russian giant mathematician Andrey Nikolaevich Kolmogorov (1903-1987) outlined an axiomatic approach that forms the basis for the modern theory. In 1925, the year he started his doctoral studies, Kolmogorov published his first paper with Khinchin on the probability theory. The paper contains, among other inequalities about partial series of random variables, the three series theorem which provides important tools for stochastic calculus. In 1929, when he finished his doctorate, he already had published 18 papers. Among them were versions of the strong law of large numbers and the law of iterated logarithm.

In 1933, two years after his appointment as a professor at Moscow University, Kolmogorov published Grundbegriffe der Wahrscheinlichkeitsrechnung his most fundamental book. In it he builds up probability theory in a rigorous way from fundamental axioms in a way comparable with Euclid's treatment of geometry. He gives a rigorous definition of the conditional expectation which later became fundamental for the definition of Brownian motion, stochastic integration, and Mathematics of Finance. (Kolmogorov's monograph is available in English translation as Foundations of Probability Theory,...