The Principles of Deep Learning Theory

Daniel A. Roberts was cofounder and CTO of Diffeo, an AI company acquired by Salesforce; a research scientist at Facebook AI Research; and a member of the School of Natural Sciences at the Institute for Advanced Study in Princeton, NJ. He was a Hertz Fellow, earning a PhD from MIT in theoretical physics, and was also a Marshall Scholar at Cambridge and Oxford Universities. Sho Yaida is a research scientist at Meta AI. Prior to joining Meta AI, he obtained his PhD in physics at Stanford University and held postdoctoral positions at MIT and at Duke University. At Meta AI, he uses tools from theoretical physics to understand neural networks, the topic of this book. Boris Hanin is an Assistant Professor at Princeton University in the Operations Research and Financial Engineering Department. Prior to joining Princeton in 2020, Boris was an Assistant Professor at Texas A&M in the Math Department and an NSF postdoc at MIT. He has taught graduate courses on the theory and practice of deep learning at both Texas A&M and Princeton.

Preface; 0. Initialization; 1. Pretraining; 2. Neural networks; 3. Effective theory of deep linear networks at initialization; 4. RG flow of preactivations; 5. Effective theory of preactivations at initializations; 6. Bayesian learning; 7. Gradient-based learning; 8. RG flow of the neural tangent kernel; 9. Effective theory of the NTK at initialization; 10. Kernel learning; 11. Representation learning; ∞. The end of training; ε. Epilogue; A. Information in deep learning; B. Residual learning; References; Index.