Electronic Structure Calculations on Graphics Processing Units

Ross C. Walker, San Diego Supercomputer Center and Department of Chemistry and Biochemistry, University of California San Diego Dr. Walker is an Assistant Research Professor at the San Diego Supercomputer Center, an Adjunct Assistant Professor in the Department of Chemistry and Biochemistry at the University of California San Diego, and an NVIDIA CUDA fellow. He leads a team of scientists that develop advanced techniques for molecular dynamics (MD) simulations aimed at improving drug and biocatalyst design. Aspects of his work that are of particular relevance for the proposed book include the development of quantum mechanics (QM) and quantum mechanics/molecular mechanics (QM/MM) methods for MD simulations, and the development of a widely used GPU accelerated MD code with funding from the National Science Foundation program SI2 (Software Infrastructure for Sustained Innovation). These methods, including the GPU accelerated MD code, are integrated into the AMBER MD software package that is used worldwide. Over the course of the last years Dr. Walker has given presentations and lectured on multiple occasions about GPU acceleration of MD codes and scientific applications. Dr. Walker's research is documented in over 30 peer-reviewed journal articles and multiple collected works. In 2010 Dr. Walker co-authored with Dr. Goetz a book chapter that reviews the use of GPU accelerators in quantum chemistry. Andreas W. Goetz, San Diego Supercomputer Center, University of California San Diego Dr. Goetz is an Assistant Project Scientist at the San Diego Supercomputer Center with strong expertise in method and scientific software development for quantum chemistry and molecular dynamics simulations on high performance computing platforms. He is a contributing author of the ADF (Amsterdam Density Functional) software for DFT calculations and the AMBER MD software package. Over the last years, Dr. Goetz has given various contributed and invited presentations of his work at renowned universities and international conferences. Dr. Goetz has also organized and taught workshops demonstrating the use of the software he develops. His research is documented in 21 peer-reviewed journal articles and 1 book contribution.

List of Contributors xiii

Preface xvii

Acknowledgments xix

Glossary xxi

Abbreviations xxv

1. Why Graphics Processing Units 1”
Perri Needham, Andreas W. Götz and Ross C. Walker

1.1 A Historical Perspective of Parallel Computing 1

1.2 The Rise of the GPU 5

1.3 Parallel Computing on Central Processing Units 7

1.4 Parallel Computing on Graphics Processing Units 12

1.5 GPU-Accelerated Applications 15

References 19

2. GPUs: Hardware to Software 23
Perri Needham, Andreas W. Götz and Ross C. Walker

2.1 Basic GPU Terminology 24

2.2 Architecture of GPUs 24

2.3 CUDA Programming Model 26

2.4 Programming and Optimization Concepts 30

2.5 Software Libraries for GPUs 34

2.6 Special Features of CUDA-Enabled GPUs 35

References 36

3. Overview of Electronic Structure Methods 39
Andreas W. Götz

3.1 Introduction 39

3.2 Hartree–Fock Theory 42

3.3 Density Functional Theory 46

3.4 Basis Sets 49

3.5 Semiempirical Methods 53

3.6 Density Functional Tight Binding 56

3.7 Wave Function-Based Electron Correlation Methods 57

Acknowledgments 60

References 61

4. Gaussian Basis Set Hartree–Fock, Density Functional Theory, and Beyond on GPUs 67
Nathan Luehr, Aaron Sisto and Todd J. Martínez

4.1 Quantum Chemistry Review 68

4.2 Hardware and CUDA Overview 72

4.3 GPU ERI Evaluation 73

4.4 Integral-Direct Fock Construction on GPUs 78

4.5 Precision Considerations 88

4.6 Post-SCF Methods 91

4.7 Example Calculations 93

4.8 Conclusions and Outlook 97

References 98

5. GPU Acceleration for Density Functional Theory with Slater-Type Orbitals 101
Hans van Schoot and Lucas Visscher

5.1 Background 101

5.2 Theory and CPU Implementation 102

5.3 GPU Implementation 105

5.4 Conclusion 112

References 113

6. Wavelet-Based Density Functional Theory on Massively Parallel Hybrid Architectures 115
Luigi Genovese, Brice Videau, Damien Caliste, Jean-François Méhaut, Stefan Goedecker and Thierry Deutsch

6.1 Introductory Remarks on Wavelet Basis Sets for Density Functional Theory Implementations 115

6.2 Operators in Wavelet Basis Sets 117

6.3 Parallelization 123

6.4 GPU Architecture 124

6.5 Conclusions and Outlook 132

References 133

7. Plane-Wave Density Functional Theory 135
Maxwell Hutchinson, Paul Fleurat-Lessard, Ani Anciaux-Sedrakian, Dusan Stosic, Jeroen Bédorf and Sarah Tariq

7.1 Introduction 135

7.2 Theoretical Background 136

7.3 Implementation 143

7.4 Optimizations 148

7.5 Performance Examples 151

7.6 Exact Exchange with Plane Waves 159

7.7 Summary and Outlook 165

Acknowledgments 165

References 165

Appendix A: Definitions and Conventions 168

Appendix B: Example Kernels 168

8. GPU-Accelerated Sparse Matrix–Matrix Multiplication for Linear Scaling Density Functional Theory 173
Ole Schütt, Peter Messmer, Jürg Hutter and Joost VandeVondele

8.1 Introduction 173

8.2 Software Architecture for GPU-Acceleration 177

8.3 Maximizing Asynchronous Progress 180

8.4 Libcusmm: GPU Accelerated Small Matrix Multiplications 183

8.5 Benchmarks and Conclusions 186

Acknowledgments 189

References 189

9. Grid-Based Projector-Augmented Wave Method 191
Samuli Hakala, Jussi Enkovaara, Ville Havu, Jun Yan, Lin Li, Chris O’Grady

and Risto M. Nieminen

9.1 Introduction 191

9.2 General Overview 193

9.3 Using GPUs in Ground-State Calculations 196

9.4 Time-Dependent Density Functional Theory 202

9.5 Random Phase Approximation for the Correlation Energy 203

9.6 Summary and Outlook 207

Acknowledgments 208

References 208

10. Application of Graphics Processing Units to Accelerate Real-Space Density Functional Theory and Time-Dependent Density Functional Theory Calculations 211
Xavier Andrade and Alán Aspuru-Guzik

10.1 Introduction 212

10.2 The Real-Space Representation 213

10.3 Numerical Aspects of the Real-Space Approach 214

10.4 General GPU Optimization Strategy 216

10.5 Kohn–Sham Hamiltonian 217

10.6 Orthogonalization and Subspace Diagonalization 221

10.7 Exponentiation 222

10.8 The Hartree Potential 223

10.9 Other Operations 224

10.10 Numerical Performance 225

10.11 Conclusions 228

10.12 Computational Methods 228

Acknowledgments 229

References 229

11. Semiempirical Quantum Chemistry 239
Xin Wu, Axel Koslowski and Walter Thiel

11.1 Introduction 239

11.2 Overview of Semiempirical Methods 240

11.3 Computational Bottlenecks 241

11.4 Profile-Guided Optimization for the Hybrid Platform 244

11.5 Performance 249

11.6 Applications 251

11.7 Conclusion 252

Acknowledgement 253

References 253

12. GPU Acceleration of Second-Order Møller–Plesset Perturbation Theory with Resolution of Identity 259
Roberto Olivares-Amaya, Adrian Jinich, Mark A. Watson and Alán Aspuru-Guzik

12.1 Møller–Plesset Perturbation Theory with Resolution of Identity Approximation (RI-MP2) 259

12.2 A Mixed-Precision Matrix Multiplication Library 263

12.3 Performance of Accelerated RI-MP2 266

12.4 Example Applications 270

12.5 Conclusions 273

References 273

13. Iterative Coupled-Cluster Methods on Graphics Processing Units 279
A. Eugene DePrince III, Jeff R. Hammond and C. David Sherrill

13.1 Introduction 279

13.2 Related Work 280

13.3 Theory 281

13.4 Algorithm Details 284

13.5 Computational Details 287

13.6 Results 290

13.7 Conclusions 295

Acknowledgments 296

References 296

14. Perturbative Coupled-Cluster Methods on Graphics Processing Units: Single- and Multi-Reference Formulations 301
Wenjing Ma, Kiran Bhaskaran-Nair, Oreste Villa, Edoardo Aprà, Antonino Tumeo, Sriram Krishnamoorthy and Karol Kowalski

14.1 Introduction 302

14.2 Overview of Electronic Structure Methods 303

14.3 NWChem Software Architecture 308

14.4 GPU Implementation 309

14.5 Performance 315

14.6 Outlook 319

Acknowledgments 320

References 320

Index 327