Introduction to Probability Modeling

William R. Leo    Astral Geneva, Switzerland

1.1 Probability in Experimental Science

Historically, the invention of probability theory is generally attributed to Pascal and Femiat, who first developed it to treat a narrow domain of problems connected with games of chance. Progress in the ensuing centuries, however, has widened its applications such that today almost all disciplines, be they the physical sciences, engineering, the social sciences, economics, etc., are touched in some way or another. In physics, for example, probability appears in an extremely fundamental role in quantum theory and in statistical mechanics, where it is used to describe the ultimate behavior of matter. Notwithstanding its importance to modern physical theory, however, probability also plays an almost equally fundamental role on the experimental side—for it can be said that probability is one of the truly basic tools of the experimental sciences, without which many results would not have been possible.

Modem science, of course, is based on active experimentation; that is, observation and measurement. This is the fundamental means by which new knowledge is acquired. But it becomes evident to anyone who has ever performed experiments that all measurements are fraught with uncertainty or “errors,” which limit the conclusions that can be drawn. (The term errors here is the one used most often by scientists to refer to experimental unceilainties, and its meaning should not be confused with that of “making a mistake. ”) The questions that inevitably arise then are these: How is one to handle these uncertainties? Can one still draw any real conclusions from the data? How much confidence can one have in the results? Moreover, given the data, how does one compare them to results from other experiments? Are they different or consistent? Here the problem is not so much the final result itself but the errors incurred. This implies that a common quantitative measure and “language” are necessary to describe the errors. Finally, given that there are errors in all measurements, is it possible to plan and design one’s experiment so that it will give meaningful results? Here again, a quantitative procedure is necessary.

The answer to these questions comes from recognizing that measurement and observational errors are in fact random phenomena and that they can be modelled and interpreted using the theory of probability. The consequence is that “uncertainty,” an intuitive notion, can now be quantified and treated with the mathematical apparatus already developed in probability and in statistics. Ultimately, this allows the confidence one can have in the result, and therefore the meaningfulness of the experiment, to be gauged in a quantitative way. At the same time, the mathematical theory provides a standard framework in which different measurements or observations of the same phenomenon can be compared in a consistent manner.

The notion of probability is familiar to most people, however, the mathematical theory of probability and its application are much less so. Indeed, nowhere in mathematics are more errors (i.e., mistakes!) in reasoning made more often than in probability theory (see, for example, Chapters 1–2 of [1] and Section VI, [2]). Applying probability in a consistent manner requires, in fact, a rather radical change in conceptual thinking.

History gives us some interesting illustrations of this, one of which is the work of Tobias Mayer, who in the mid-18th century successfully treated the problem of the libration of the moon. The term libration refers to slow, oscillatory movements made by the moon’s body such that parts of its face appear and disappear from view. There are, in fact, several movements along different axes that, over an extended period of time, allow about 60% of the moon’s surface to be seen from the earth. The causes of these effects were known at the time of Tobias Mayer, but the problem was to account for these movements either by a mathematical equation or a set of lunar tables. Indeed, a solution was of particular commercial and military value for detailed knowledge of the moon’s different motions could serve as a navigational aide to ships at sea.

Thus, in 1748, Johann Tobias Mayer, an already well-known cartographer and practical astronomer, undertook the study of the libration problem. After making observations of the moon for more than a year, Mayer came up with a method for determining various characteristics of these movements. A critical pm of this solution was to find the relationship between the natural coordinate system of the moon as defined by its axis of rotation and equator and the astronomer’s system defined by the plane of the earth’s orbit about the sun (ecliptic). To do this, Mayer focused his observations on the position of the crater Manilius and derived a relation between the two reference systems involving six parameters, three of which were unknown constants and three of which varied with the moon’s motion, but could be measured. To solve for these unknowns, therefore, Mayer needed only to make three day’s worth of observations. He did better than this, however, and made observations over a period of 27 days.

Keeping in mind that the discovery of the least squares principle (see Chapter 9) was still a half-century off, Mayer was thus faced with the dilemma of having 27 inconsistent equations for his three unknowns; that is, if he attempted to solve for the unknowns by just taking three of the equations, different results would be obtained depending on which three equations he chose. The inconsistencies, of course, were due to the observational errors, an effect well known to the scientists and mathematicians of the time, but for which no one had a solution. It was here that Mayer came up with a remarkable idea. From his own practical experience, Mayer knew intuitively that adding observations made under similar conditions could actually reduce the errors incurred. Using this notion, he divided his equations into three groups of nine similar equations, added the equations in each group and solved for his unknowns. Simple1 as that!

The point, however, is that Mayer essentially thought of his errors as random phenomena, although he had no clear concept of probability, so that adding data could actually lead to a cancellation of the errors and a reduction in the overall uncertainty. This was totally contrary to the view of the times, which focused on the maximum possible error (as opposed to the probable error) that could occur when data were manipulated. From this viewpoint, adding data would only increase the final errors, since it wouldjust cumulate the maximum possible error of each datum. Such a large final error is possible under the probabilistic view but highly improbable.

Interestingly enough history also provides a “control experiment” to prove the point. Only one year prior to Mayer’s work, Leonhard Euler, undoubtedly one of the greatest mathematicians of all time, reported his work on the problem of “inequalities” in the orbits of Jupiter and Samm. These effects were thought to be due to the two planets' mutual attraction, an instance of what today is called the three-body problem. Formulating a mathematical equation for the longitudinal position of Saturn that took into account the attraction of both the sun and Jupiter, Euler sought to provide some test of his calculation by using some actual data (not of his own taking). After making a number of approximations to linearize his formula, he ended up with 75 equations for six unknowns. He thus faced a problem analytically similar to Tobias Mayer’s, although on a somewhat larger scale.

Here, however, Euler failed to make the conceptual jump that Mayer made. Indeed, whereas Mayer made his own measurements and thus had an intuitive feeling for the uncertainties involved, Euler, a pure mathematician dealing with data that he did not take, had no basis for even imagining the random nature of measurement errors. Euler was thus left to grope for a solution that he never found. His theoretical work, was nevertheless a significant contribution to celestial mechanics, and in recognition, he was awarded the 1748 prize offered by the Academy of Sciences in Paris. Further details conceming Tobias Mayer’s and Leonhard Euler’s work may be found in [3].

These historical accounts illustrate the paradigm shift that the application of probability theory required in its early stages, but even today dealing with probability theory still requires a conceptual change that is underestimated by most people. Indeed, to think probabilistically essentially requires giving up certainty in everything that is done, as we will see later on. This is also complicated by a good deal of confusion over the meaning of certain terms—beginning with the expressions probability and probability theory themselves. We will attempt to clarify these points in the next sections.

1.2 Defining Probability

In applying probability theory it is necessary to distinguish between two distinct problems: the modeling of random processes...