Multiblock Data Fusion in Statistics and Machine Learning

Age K. Smilde is a Professor of Biosystems Data Analysis at the Swammerdam Institute for Life Sciences at the University of Amsterdam. He also holds a part-time position at the Department of Machine Intelligence of Simula Metropolitan Center for Digital Engineering in Oslo, Norway. His research interest is multiblock data analysis and its implementation in different fields of life sciences. He is currently the Editor-in-Chief of the Journal of Chemometrics. Tormod Næs is a Senior Scientist at Nofima, a food research institute in Norway. He is also currently employed as adjoint professor at the Department of Food Science, University of Copenhagen, Denmark and as extraordinary professor at University of Stellenbosch, South Africa. His main research interest is multivariate analysis with special emphasis on applications in sensory science and spectroscopy. Kristian Hovde Liland is an Associate Professor with a top scientist scholarship in Data Science at the Norwegian University of Life Sciences and works in the areas of chemometrics, data analysis and machine learning. His main research is in linear prediction modelling, spectroscopy, and the transition between chemometrics and machine learning.

Foreword xiii

Preface xv

List of Figures xvii

List of Tables xxxi

Part I Introductory Concepts and Theory 1

1 Introduction 3

1.1 Scope of the Book 3

1.2 Potential Audience 4

1.3 Types of Data and Analyses 5

1.3.1 Supervised and Unsupervised Analyses 5

1.3.2 High-, Mid- and Low-level Fusion 5

1.3.3 Dimension Reduction 7

1.3.4 Indirect Versus Direct Data 8

1.3.5 Heterogeneous Fusion 8

1.4 Examples 8

1.4.1 Metabolomics 8

1.4.2 Genomics 11

1.4.3 Systems Biology 13

1.4.4 Chemistry 13

1.4.5 Sensory Science 15

1.5 Goals of Analyses 16

1.6 Some History 17

1.7 Fundamental Choices 17

1.8 Common and Distinct Components 19

1.9 Overview and Links 20

1.10 Notation and Terminology 21

1.11 Abbreviations 22

2 Basic Theory and Concepts 25

2.i General Introduction 25

2.1 Component Models 25

2.1.1 General Idea of Component Models 25

2.1.2 Principal Component Analysis 26

2.1.3 Sparse PCA 30

2.1.4 Principal Component Regression 31

2.1.5 Partial Least Squares 32

2.1.6 Sparse PLS 36

2.1.7 Principal Covariates Regression 37

2.1.8 Redundancy Analysis 38

2.1.9 Comparing PLS, PCovR and RDA 38

2.1.10 Generalised Canonical Correlation Analysis 38

2.1.11 Simultaneous Component Analysis 39

2.2 Properties of Data 39

2.2.1 Data Theory 39

2.2.2 Scale-types 42

2.3 Estimation Methods 44

2.3.1 Least-squares Estimation 44

2.3.2 Maximum-likelihood Estimation 45

2.3.3 Eigenvalue Decomposition-based Methods 47

2.3.4 Covariance or Correlation-based Estimation Methods 47

2.3.5 Sequential Versus Simultaneous Methods 48

2.3.6 Homogeneous Versus Heterogeneous Fusion 50

2.4 Within- and Between-block Variation 52

2.4.1 Definition and Example 52

2.4.2 MAXBET Solution 54

2.4.3 MAXNEAR Solution 54

2.4.4 PLS2 Solution 55

2.4.5 CCA Solution 55

2.4.6 Comparing the Solutions 56

2.4.7 PLS, RDA and CCA Revisited 56

2.5 Framework for Common and Distinct Components 60

2.6 Preprocessing 63

2.7 Validation 64

2.7.1 Outliers 64

2.7.1.1 Residuals 64

2.7.1.2 Leverage 66

2.7.2 Model Fit 67

2.7.3 Bias-variance Trade-off 69

2.7.4 Test Set Validation 70

2.7.5 Cross-validation 72

2.7.6 Permutation Testing 75

2.7.7 Jackknife and Bootstrap 76

2.7.8 Hyper-parameters and Penalties 77

2.8 Appendix 78

3 Structure of Multiblock Data 87

3.i General Introduction 87

3.1 Taxonomy 87

3.2 Skeleton of a Multiblock Data Set 87

3.2.1 Shared Sample Mode 88

3.2.2 Shared Variable Mode 88

3.2.3 Shared Variable or Sample Mode 88

3.2.4 Shared Variable and Sample Mode 89

3.3 Topology of a Multiblock Data Set 90

3.3.1 Unsupervised Analysis 90

3.3.2 Supervised Analysis 93

3.4 Linking Structures 95

3.4.1 Linking Structure for Unsupervised Analysis 95

3.4.2 Linking Structures for Supervised Analysis 96

3.5 Summary 98

4 Matrix Correlations 99

4.i General Introduction 99

4.1 Definition 99

4.2 Most Used Matrix Correlations 101

4.2.1 Inner Product Correlation 101

4.2.2 GCD coefficient 101

4.2.3 RV-coefficient 102

4.2.4 SMI-coefficient 102

4.3 Generic Framework of Matrix Correlations 104

4.4 Generalised Matrix Correlations 105

4.4.1 Generalised RV-coefficient 105

4.4.2 Generalised Association Coefficient 106

4.5 Partial Matrix Correlations 108

4.6 Conclusions and Recommendations 110

4.7 Open Issues 111

Part II Selected Methods for Unsupervised and Supervised Topologies 113

5 Unsupervised Methods 115

5.i General Introduction 115

5.ii Relations to the General Framework 115

5.1 Shared Variable Mode 117

5.1.1 Only Common Variation 117

5.1.1.1 Simultaneous Component Analysis 117

5.1.1.2 Clustering and SCA 123

5.1.1.3 Multigroup Data Analysis 125

5.1.2 Common, Local, and Distinct Variation 126

5.1.2.1 Distinct and Common Components 127

5.1.2.2 Multivariate Curve Resolution 130

5.2 Shared Sample Mode 133

5.2.1 Only Common Variation 133

5.2.1.1 SUM-PCA 133

5.2.1.2 Multiple Factor Analysis and STATIS 135

5.2.1.3 Generalised Canonical Analysis 136

5.2.1.4 Regularised Generalised Canonical Correlation Analysis 139

5.2.1.5 Exponential Family SCA 140

5.2.1.6 Optimal-scaling 143

5.2.2 Common, Local, and Distinct Variation 146

5.2.2.1 Joint and Individual Variation Explained 146

5.2.2.2 Distinct and Common Components 147

5.2.2.3 PCA-GCA 148

5.2.2.4 Advanced Coupled Matrix and Tensor Factorisation 153

5.2.2.5 Penalised-ESCA 156

5.2.2.6 Multivariate Curve Resolution 158

5.3 Generic Framework 159

5.3.1 Framework for Simultaneous Unsupervised Methods 159

5.3.1.1 Description of the Framework 159

5.3.1.2 Framework Applied to Simultaneous Unsupervised Data Analysis Methods 161

5.3.1.3 Framework of Common/Distinct Applied to Simultaneous Unsupervised Multiblock Data Analysis Methods 161

5.4 Conclusions and Recommendations 162

5.5 Open Issues 164

6 ASCA and Extensions 167

6.i General Introduction 167

6.ii Relations to the General Framework 167

6.1 ANOVA-Simultaneous Component Analysis 168

6.1.1 The ASCA Method 168

6.1.2 Validation of ASCA 176

6.1.2.1 Permutation Testing 176

6.1.2.2 Back-projection 178

6.1.2.3 Confidence Ellipsoids 178

6.1.3 The ASCA+ and LiMM-PCA Methods 181

6.2 Multilevel-SCA 182

6.3 Penalised-ASCA 183

6.4 Conclusions and Recommendations 185

6.5 Open Issues 186

7 Supervised Methods 187

7.i General Introduction 187

7.ii Relations to the General Framework 187

7.1 Multiblock Regression: General Perspectives 188

7.1.1 Model and Assumptions 188

7.1.2 Different Challenges and Aims 188

7.2 Multiblock PLS Regression 190

7.2.1 Standard Multiblock PLS Regression 190

7.2.2 MB-PLS Used for Classification 194

7.2.3 Sparse Multiblock PLS Regression (sMB-PLS) 196

7.3 The Family of SO-PLS Regression Methods (Sequential and Orthogonalised PLS Regression) 199

7.3.1 The SO-PLS Method 199

7.3.2 Order of Blocks 202

7.3.3 Interpretation Tools 202

7.3.4 Restricted PLS Components and their Application in SO-PLS 203

7.3.5 Validation and Component Selection 204

7.3.6 Relations to ANOVA 205

7.3.7 Extensions of SO-PLS to Handle Interactions Between Blocks 212

7.3.8 Further Applications of SO-PLS 215

7.3.9 Relations Between SO-PLS and ASCA 215

7.4 Parallel and Orthogonalised PLS (PO-PLS) Regression 217

7.5 Response Oriented Sequential Alternation 222

7.5.1 The ROSA Method 222

7.5.2 Validation 225

7.5.3 Interpretation 225

7.6 Conclusions and Recommendations 228

7.7 Open Issues 229

Part III Methods for Complex Multiblock Structures 231

8 Complex Block Structures; with Focus on L-Shape Relations 233

8.i General Introduction 233

8.ii Relations to the General Framework 234

8.1 Analysis of L-shape Data: General Perspectives 235

8.2 Sequential Procedures for L-shape Data Based on PLS/PCR and ANOVA 236

8.2.1 Interpretation of X1, Quantitative X2-data, Horizontal Axis First 236

8.2.2 Interpretation of X1, Categorical X2-data, Horizontal Axis First 238

8.2.3 Analysis of Segments/Clusters of X1 Data 240

8.3 The L-PLS Method for Joint Estimation of Blocks in L-shape Data 246

8.3.1 The Original L-PLS Method, Endo-L-PLS 247

8.3.2 Exo- Versus Endo-L-PLS 250

8.4 Modifications of the Original L-PLS Idea 252

8.4.1 Weighting Information from X3 and X1 in L-PLS Using a Parameter α252

8.4.2 Three-blocks Bifocal PLS 253

8.5 Alternative L-shape Data Analysis Methods 254

8.5.1 Principal Component Analysis with External Information 254

8.5.2 A Simple PCA Based Procedure for Using Unlabelled Data in Calibration 255

8.5.3 Multivariate Curve Resolution for Incomplete Data 256

8.5.4 An Alternative Approach in Consumer Science Based on Correlations Between X3 and X1 257

8.6 Domino PLS and More Complex Data Structures 258

8.7 Conclusions and Recommendations 258

8.8 Open Issues 260

Part IV Alternative Methods for Unsupervised and Supervised Topologies 261

9 Alternative Unsupervised Methods 263

9.i General Introduction 263

9.ii Relationship to the General Framework 263

9.1 Shared Variable Mode 263

9.2 Shared Sample Mode 265

9.2.1 Only Common Variation 265

9.2.1.1 DIABLO 265

9.2.1.2 Generalised Coupled Tensor Factorisation 266

9.2.1.3 Representation Matrices 267

9.2.1.4 Extended PCA 272

9.2.2 Common, Local, and Distinct Variation 273

9.2.2.1 Generalised SVD 273

9.2.2.2 Structural Learning and Integrative Decomposition 273

9.2.2.3 Bayesian Inter-battery Factor Analysis 275

9.2.2.4 Group Factor Analysis 276

9.2.2.5 OnPLS 277

9.2.2.6 Generalised Association Study 278

9.2.2.7 Multi-Omics Factor Analysis 278

9.3 Two Shared Modes and Only Common Variation 281

9.3.1 Generalised Procrustes Analysis 282

9.3.2 Three-way Methods 282

9.4 Conclusions and Recommendations 283

9.4.1 Open Issues 284

10 Alternative Supervised Methods 287

10.i General Introduction 287

10.ii Relations to the General Framework 287

10.1 Model and Focus 288

10.2 Extension of PCovR 288

10.2.1 Sparse Multiblock Principal Covariates Regression, Sparse PCovR 288

10.2.2 Multiway Multiblock Covariates Regression 289

10.3 Multiblock Redundancy Analysis 292

10.3.1 Standard Multiblock Redundancy Analysis 292

10.3.2 Sparse Multiblock Redundancy Analysis 294

10.4 Miscellaneous Multiblock Regression Methods 295

10.4.1 Multiblock Variance Partitioning 296

10.4.2 Network Induced Supervised Learning 296

10.4.3 Common Dimensions for Multiblock Regression 298

10.5 Modifications and Extensions of the SO-PLS Method 298

10.5.1 Extensions of SO-PLS to Three-Way Data 298

10.5.2 Variable Selection for SO-PLS 299

10.5.3 More Complicated Error Structure for SO-PLS 299

10.5.4 SO-PLS Used for Path Modelling 300

10.6 Methods for Data Sets Split Along the Sample Mode, Multigroup Methods 304

10.6.1 Multigroup PLS Regression 304

10.6.2 Clustering of Observations in Multiblock Regression 306

10.6.3 Domain-Invariant PLS, DI-PLS 307

10.7 Conclusions and Recommendations 308

10.8 Open Issues 309

Part V Software 311

11 Algorithms and Software 313

11.1 Multiblock Software 313

11.2 R package multiblock 313

11.3 Installing and Starting the Package 314

11.4 Data Handling 314

11.4.1 Read From File 314

11.4.2 Data Pre-processing 315

11.4.3 Re-coding Categorical Data 316

11.4.4 Data Structures for Multiblock Analysis 317

11.4.4.1 Create List of Blocks 317

11.4.4.2 Create data.frame of Blocks 317

11.5 Basic Methods 318

11.5.1 Prepare Data 319

11.5.2 Modelling 319

11.5.3 Common Output Elements Across Methods 319

11.5.4 Scores and Loadings 320

11.6 Unsupervised Methods 321

11.6.1 Formatting Data for Unsupervised Data Analysis 321

11.6.2 Method Interfaces 322

11.6.3 Shared Sample Mode Analyses 322

11.6.4 Shared Variable Mode 322

11.6.5 Common Output Elements Across Methods 323

11.6.6 Scores and Loadings 324

11.6.7 Plot From Imported Package 325

11.7 ANOVA Simultaneous Component Analysis 325

11.7.1 Formula Interface 325

11.7.2 Simulated Data 325

11.7.3 ASCA Modelling 325

11.7.4 ASCA Scores 326

11.7.5 ASCA Loadings 326

11.8 Supervised Methods 327

11.8.1 Formatting Data for Supervised Analyses 327

11.8.2 Multiblock Partial Least Squares 328

11.8.2.1 MB-PLS Modelling 328

11.8.2.2 MB-PLS Summaries and Plotting 328

11.8.3 Sparse Multiblock Partial Least Squares 328

11.8.3.1 Sparse MB-PLS Modelling 328

11.8.3.2 Sparse MB-PLS Plotting 329

11.8.4 Sequential and Orthogonalised Partial Least Squares 330

11.8.4.1 SO-PLS Modelling 330

11.8.4.2 Måge Plot 331

11.8.4.3 SO-PLS Loadings 332

11.8.4.4 SO-PLS Scores 333

11.8.4.5 SO-PLS Prediction 334

11.8.4.6 SO-PLS Validation 334

11.8.4.7 Principal Components of Predictions 336

11.8.4.8 CVANOVA 336

11.8.5 Parallel and Orthogonalised Partial Least Squares 337

11.8.5.1 PO-PLS Modelling 337

11.8.5.2 PO-PLS Scores and Loadings 338

11.8.6 Response Optimal Sequential Alternation 339

11.8.6.1 ROSA Modelling 339

11.8.6.2 ROSA Loadings 340

11.8.6.3 ROSA Scores 340

11.8.6.4 ROSA Prediction 340

11.8.6.5 ROSA Validation 341

11.8.6.6 ROSA Image Plots 342

11.8.7 Multiblock Redundancy Analysis 343

11.8.7.1 MB-RDA Modelling 343

11.8.7.2 MB-RDA Loadings and Scores 343

11.9 Complex Data Structures 344

11.9.1 L-PLS 344

11.9.1.1 Simulated L-shaped Data 344

11.9.1.2 Exo-L-PLS 344

11.9.1.3 Endo-L-PLS 344

11.9.1.4 L-PLS Cross-validation 345

11.9.2 SO-PLS-PM 345

11.9.2.1 Single SO-PLS-PM Model 346

11.9.2.2 Multiple Paths in an SO-PLS-PM Model 346

11.10 Software Packages 347

11.10.1 R Packages 347

11.10.2 MATLAB Toolboxes 348

11.10.3 Python 349

11.10.4 Commercial Software 349               

References 351

Index 373