Testing and Tuning Market Trading Systems (eBook)

Timothy Masters received a PhD in mathematical statistics with a specialization in numerical computing. Since then he has continuously worked as an independent consultant for government and industry. His early research involved automated feature detection in high-altitude photographs while he developed applications for flood and drought prediction, detection of hidden missile silos, and identification of threatening military vehicles. Later he worked with medical researchers in the development of computer algorithms for distinguishing between benign and malignant cells in needle biopsies. For the last twenty years he has focused primarily on methods for evaluating automated financial market trading systems. He has authored five books on practical applications of predictive modeling: Practical Neural Network Recipes in C++ (Academic Press, 1993); Signal and Image Processing with Neural Networks (Wiley, 1994); Advanced Algorithms for Neural Networks (Wiley, 1995); Neural, Novel, and Hybrid Algorithms for Time Series Prediction (Wiley, 1995); Data Mining Algorithms in C++ (Apress, 2018); Assessing and Improving Prediction and Classification (Apress, 2018); Deep Belief Nets in C++ and CUDA C: Volume 1 (Apress, 2018); and Deep Belief Nets in C++ and CUDA C: Volume 2 (Apress, 2018).

Table of Contents 4
About the Author 8
About the Technical Reviewer 9
Chapter 1: Introduction 10
The Target Audience, and Overview of Contents 10
What’s in This Book 10
What’s Not in This Book 12
About Trading Systems 13
Market Prices and Returns 14
Two Types of Automated Trading Systems 15
The Agony of Believing the Computer 16
Future Leak Is More Dangerous Than You May Think 16
The Percent Wins Fallacy 17
Chapter 2: Pre-optimization Issues 19
Assessing and Improving Stationarity 19
The STATN Program 21
Improving Location Stationarity by Oscillating 25
Extreme Stationarity Induction 27
Measuring Indicator Information with Entropy 28
Computing the Relative Entropy of an Indicator 30
Entropy Impacts Predictive Model Quality 32
Improving the Entropy of an Indicator 33
Monotonic Tail-Only Cleaning 37
The ENTROPY Program 39
Chapter 3: Optimization Issues 42
Regularizing a Linear Model 42
Overview of the Regularized Model 43
Beta Adjustment with Guaranteed Convergence 47
Differential Case Weighting 48
Rapid Computation with Covariance Updates 49
Preparatory Code 50
Outline of the Beta Optimization Process 53
Code for Beta Optimization 55
Descending a Lambda Path 62
Optimizing Lambda with Cross Validation 66
The CD_MA Program 70
Making a Linear Model Nonlinear 74
Differential Evolution: A Universal Nonlinear Optimizer 76
The DIFF_EV.CPP Routine for Differential Evolution 82
Chapter 4: Post-optimization Issues 97
Cheap Bias Estimates 97
The StocBias Class 98
Cheap Parameter Relationships 102
Parameter Sensitivity Curves 114
Putting It All Together Trading OEX 118
The Trading System 118
Linking Criterion Routines 121
Application to Trading OEX 123
Chapter 5: Estimating Future Performance I: Unbiased Trade Simulation 127
In-Sample and Out-of-Sample Performance 127
The TrnBias Program to Demonstrate Training Bias 129
Selection Bias 130
Interlude: What Does Unbiased Really Mean? 131
Selection Bias, Continued 132
The SelBias Program 133
Walkforward Analysis 135
Future Leak by Unobvious IS/OOS Overlap 137
Error Variance Inflation with Multiple-Bar Lookaheads 138
The General Walkforward Algorithm 140
C++ Code for the Algorithm 141
Date-Dependent Walkforward 143
Exploring Walkforward Blunders 143
Testing Robustness Against Nonstationarity 148
Cross-Validation Analysis 149
Unobvious IS/OOS Overlap 149
The Fully General Cross-Validation Algorithm 150
C++ Code for the General Algorithm 152
Cross Validation Can Have Pessimistic Bias 156
Cross Validation Can Have Optimistic Bias 156
Cross Validation Does Not Reflect Real Life 156
Special Precautions for Algorithmic Trading 157
Converting Unknown-Lookahead Systems to Single-Bar 161
Unbounded Lookback Can Subtly Happen 162
Comparing Cross Validation with Walkforward: XVW 162
Computationally Symmetric Cross Validation 164
The CSCV Algorithm: Intuition and General Statement 166
What Does This Test Actually Measure? 169
C++ Code for the CSCV Superiority Test 170
An Example with SPX 175
Nested Walkforward Analysis 178
The Nested Walkforward Algorithm 180
A Practical Application of Nested Walkforward 185
An Example Using S& P 100 Components
Cross Validation Nested Inside Walkforward 194
Chapter 6: Estimating Future Performance II: Trade Analysis 198
Handling Dynamic Trading Systems 198
Unknown Lookahead to Single Bars, Revisited 199
Profit per Bar? Per Trade? Per Time? 200
Analyzing Completed Trade Returns Is Problematic 201
The PER_WHAT Program 203
A Lower Bound for Mean Future Returns 214
Brief Digression: Hypothesis Tests 215
So, How Do We Use This Probability? 217
Parametric P-Values 221
Parametric Confidence Intervals 223
Lower Confidence Bounds and Hypothesis Tests 227
Bootstrap Confidence Intervals 227
The Pivot and Percentile Methods 228
The BCa Bootstrap Algorithm 230
The BOOT_CONF.CPP Subroutines 232
The BOUND_MEAN Program and Results with SPX 237
Beware of Bootstrapping Ratios 243
Bounding Future Returns 246
Deriving a Lower Bound from Empirical Quantiles 247
Confidence in the Computed Lower Bound 249
What About an Upper Bound on Future Returns? 252
The CONFTEST Program: Overview 253
The CONFTEST Program: Code 256
The BND_RET Program 262
Bounding Drawdown 267
Intuition Gone Wrong 268
Bootstrapping Drawdown Bounds 270
The DRAWDOWN Program 272
Experiments with the DRAWDOWN Program 282
The CHOOSER_DD Program 284
Chapter 7: Permutation Tests 287
Overview of Permutation Testing 287
Testing a Fully Specified Trading System 289
Testing the Training Process 290
Walkforward Testing a Trading System Factory 291
Permutation Testing of Predictive Models 293
The Permutation Testing Algorithm 295
Extending the Algorithm for Selection Bias 296
Partitioning Total Return of a Trading System 298
Essential Permutation Algorithms and Code 302
Simple Permutation 302
Permuting Simple Market Prices 303
Permuting Multiple Markets with an Offset 305
Permuting Price Bars 309
Example: P-Value and Partitioning 314
Example: Training with Next Bar Returns 316
Example: Permuting Multiple Markets 320
Index 323