AO Trauma - Statistics and Data Management (eBook)

2 Errors and uncertainty

2 Errors and uncertainty

1 Introduction

2 Descriptions of uncertainty

2.1 Accuracy and precision

2.2 Randomization

2.3 Types of error

2.4 Comparison and contrast

3 Sources of variability

3.1 The measurement

3.2 The observer/reader

3.3 The subject

3.4 The population

4 Distributions

4.1 Normally distributed data

4.2 Skewed data

4.3 Other distributions

5 Standard deviation versus standard error

5.1 Standard deviation

5.2 Standard error

6 Summary

2 Errors and uncertainty

1 Introduction

Uncertainty makes life interesting and challenging. You may have worked out a well-structured plan for your day at the hospital or your private office, but this can be messed up within minutes because of sudden events or because you missed an appointment or task. This does not mean that your time was wasted. You may achieve very different, equally important results, make new discoveries, and enhance your knowledge because you followed a direction slightly different from that intended.

The same is true for a scientific experiment, whether it is laboratory or clinical. Uncertainty corresponds to the unexplained variability of observations. Forecasts and predictions (not only of weather) are susceptible to an enormous number of variables, and you never know if you considered all of them during the planning phase of your project. Of course, if any observation in biomedicine was entirely predictable, we would not need scientists anymore.

In clinical practice, you may have made a certain observation in a distinct setting for the first, second, and third time. This makes you believe a specific association or rule, but the fourth time you observe the exact opposite of what you had expected. Certain findings, though impressive and breathtaking, may occur simply by chance. The famous philosopher Karl Popper was of the opinion that a hypothesis cannot be proved due to the fact that we do not have access to an infinite amount of information.

2 Descriptions of uncertainty

2.1 Accuracy and precision

Uncertainty, variability, and error are integral parts of science. Un-avoidable as they are, and in some instances desirable, they should be expressed and handled in a qualitative and quantitative manner.

We may be inclined to assume that a point estimate derived from a clinical study (eg, a change in functional scores, bone healing rates) reflects an absolute truth.

However, study findings represent a summary, aggregate, average, or extract from a sample of patients. On an individual basis, results may differ dramatically from subject to subject, or within a subject at different time points. To communicate information, we often need to abstract these results, and to abandon individual observations in favor of the sample mean.

All scientific work generates a likely range of observations that supports or weakens a theory, compatible or incompatible with a hypothesis.

It is important to know how close the range of observations and the extreme values are to the average. Fig 2-1 illustrates this by a set of studies investigating quality of life after fracture treatment, using the physical component score (PCS) of the short-form 36 health survey (SF-36). This global measure of physical health is standardized to the population norm. The norm value is 50; values below that indicate a health status worse than the norm, values above indicate a health status better than the norm. Two studies came up with mean values of 50, but with very different distributions. We trust the estimate of a single study more if it is surrounded by many similar values and very few outliers, as in study 2.

The accuracy of an estimate is high, if it comes close to the truth, with many similar values and few outliers.

It is also important to know whether the estimate is replicable, ie, if repeated studies come up with similar trends, or show a heterogeneous or random pattern. This refers to the precision of an estimate.

The precision of an estimate is high if repeated studies come up with similar trends.

Fig 2-1a–b Different studies determining the health-related quality of life after fracture treatment.

a Studies 1 and 2 have similar mean values, indicating restoration of physical function to norm values. Study 1 has a wide distribution of values, making the estimate inaccurate. Study 2 shows high accuracy of the estimate because the distribution of values is narrow.

b Thirty further studies, each of two different fracture treatments. Repeated studies indicated by a solid dot consistently come up with almost similar results in one treatment group. The precision of this treatment effect is high. In contrast, highly variable results are observed with studies in the other treatment group indicated by circles. It is uncertain whether an observation can be reproduced in a subsequent trial.

Imagine a new femoral nail intended to be inserted through the trochanteric tip. Fig 2-2 indicates the entry points achieved by four different surgeons during the first clinical trial of the new product.

Surgeon A created a series of entry points at the trochanteric tip quite close to each other. He achieved both high accuracy (low variability) and precision (in aiming the correct entry point).

Surgeon B inserted all nails through the trochanteric fossa. He achieved high precision, but low accuracy, since all insertions were made away from the correct entry point. There may be two different reasons for this: failure and systematic error, or bias. First, he may have not read the instruction manual and failed to use the implant correctly because he did not know how. Second, his usual access route and patient positioning may conflict with the tip entry. He may insert a nail through the fossa blindly, but still needs to adapt his technique to the new implant. Until he realizes this, the shape of the new rod may cause problems in the distal part of the femur and worse outcomes compared to the established implant—not because of inadequate hardware, but due to surgeon-related bias. The surgeon may also have mistakenly entered the nail through the fossa, despite having planned to target the tip.

Surgeon C attempted to enter the medullary canal through the trochanteric tip, but made drill holes within a larger area than sur geon A. On average, all nails may have been inserted accurately, but with low precision.

Finally, surgeon D requires the assistance of an experienced colleague, since all entry points were placed away from the correct site, and somewhere within the trochanteric fossa.

Situation B is critical and underlines the importance of bias. You may observe astonishingly high complication rates (such as distal cracks or malalignment), and conclude that the new implant requires improvement; however, the true reason for unfavorable outcomes must be looked for elsewhere.

Fig 2-2 Entry points of a new tip-entry femoral nail achieved by four different surgeons.

Surgeon A: both high accuracy (low variability) and precision (in aiming the correct entry point).

Surgeon B: high precision, but low accuracy (all insertions away from the correct entry point).

Surgeon C: nails may have been inserted accurately, but with low precision.

Surgeon D: all entry points away from the correct side with high variability.

Systematic error, or bias, should always be considered as a likely explanation of unexpected findings.

Any investment to explore and minimize bias in a clinical trial is valuable and pays off in the long run.

In a clinical study that compares two or more treatment interventions, there is only one way to remove bias—to randomly allocate patients to study arms.

2.2 Randomization

There are many objections to randomized controlled trials (RCTs) in trauma and orthopaedic surgery, most of which are unfounded. If you have two treatments, and do not know which performs best in a typical clinical setting (eg, in a certain type of fracture), there is no easier and better way to do this than by an RCT.

Fractures of the distal tibia may be suitable for minimally invasive plate osteosynthesis and nailing. Imagine that you had performed ORIF by plate in 20 patients, and by nail fixation in the subsequent 20 patients, all of whom had similar fracture types. After 1 year, you observe nonunions in 1 and 5 patients, as shown in Table 2-1.

Unfortunately, although the patients’ age, gender, and even the duration of surgery were well balanced, there were clearly more smokers in the nailing group. It is thus unclear whether the intervention or the smoking influenced the higher rate of nonunion.

You may now consider including only nonsmokers to avoid this bias. After 1 year, you now observe higher nonunion rates in the plating group. Unfortunately, two patients cheated you about their smoking habits, and another resumed smoking after years of abstinence shortly after discharge from the hospital. You also realized that there were an unequal number of diabetics in your study.

The list of potential confounders is almost endless, and can only contain those that are known and measurable. There may be distinct genetic factors that contribute...