Deep Learning

A hands-on and intuitive guide to the foundations of modern deep learning In Deep Learning: Principles and Implementations, distinguished researcher and professor Weidong “Will” Kuang delivers an up-to-date exploration of how major deep learning algorithms and architectures are formalized and developed from mathematical equations. The book bridges theory and practice and covers a wide range of fundamental topics, including linear regression, logistic regression, basic neural networks, convolution neural networks, as well as other basic and advanced subjects in the field. The author provides intuitive introductions to each subject and presents the development of algorithms and architectures from basic mathematical concepts. Along the way, he relies on straightforward math to keep the topics accessible for non-mathematicians and accompanies his explanations with tested Python sample code you can apply in your own work. You’ll also find: Thorough introductions to both linear and logistic regression, offering a solid foundation and insight into neural networks Comprehensive explorations of neural networks, computer vision, natural language processing, generative models, and reinforcement learning Practical exercises that students and practitioners can use to apply and develop the concepts found in the book Balanced treatments of the mathematics, algorithms, architecture, and code that serve as the foundations of a complete understanding of deep learning Perfect for undergraduate and graduate students with an interest in deep learning, Deep Learning: Principles and Implementations will also benefit practicing software engineers, faculty, and researchers whose work involves deep learning and related topics.

Preface xv Mathematical Notation xxi 1 Introduction to Deep Learning 1 1.1 Introduction 1 1.2 Types of Machine Learning 2 1.3 Data Representation in Machine Learning 6 1.4 An Overview of Deep Learning 8 1.5 Resources for Deep Learning 14 2 Linear Regression 19 2.1 Linear Regression with Single Feature 19 2.2 Linear Regression with Multiple Features 25 2.3 Linear Models for Regression 28 2.4 Linear Regression – a Probabilistic Perspective View 31 2.5 An Example: House Price Prediction 35 2.6 Summary and Further Reading 41 3 Classification and Logistic Regression 45 3.1 Logistic Regression 45 3.2 Performance Metrics for Classification 52 3.3 Implementation of Logistic Regression in Python 56 3.4 Summary 61 4 Basics of Neural Networks 67 4.1 A Simplest Neural Network: A Logistic Regression Unit 67 4.2 From Regression to Neural Networks 69 4.3 Neural Network Representation: Feedforward Propagation 72 4.4 Activation Functions 73 4.5 Network Training: Backward Propagation 76 4.6 Multi-class Classification: Softmax and Cross-Entropy Loss 79 4.7 Practice in Python 82 4.8 Summary and Further Reading 100 5 Practical Considerations in Neural Networks 107 5.1 Multiple-Layer Neural Networks 108 5.2 Generalization and Model Selection 111 5.3 Regularization 115 5.4 Weight Initialization 119 5.5 Mini-batch Gradient Descent 122 5.6 Normalization 124 5.7 Adam Optimization 129 5.8 Gradient Checking 132 5.9 Examples in Python 133 5.10 Summary and Further Reading 166 6 Introduction to PyTorch 171 6.1 Why PyTorch? 171 6.2 Tensors 172 6.3 Data Representation Using Tensors 184 6.4 Linear Regression Using PyTorch 189 6.5 Neural Networks Using PyTorch 198 6.6 Summary and Further Reading 203 7 Convolutional Neural Networks 205 7.1 Architecture of Convolutional Neural Networks 205 7.2 Convolution Layer 207 7.3 Pooling Layer and Fully Connected Layer 212 7.4 Backpropagation in CNNs (Optional) 214 7.5 Batch Normalization for CNNs 222 7.6 Implement CNNs in PyTorch 223 7.7 Summary and Further Reading 234 8 Classic Architectures of CNNs 239 8.1 Datasets 239 8.2 AlexNet 243 8.3 VGG: Networks Using Blocks 246 8.4 GoogLeNet 249 8.5 ResNet 250 8.6 Pretrained Models 253 8.7 Summary and Further Reading 265 9 Object Detection – YOLO 269 9.1 Introduction 269 9.2 YOLO (v1) 270 9.3 YOLO (v2) 276 9.4 YOLO (v3) 280 9.5 Implementation of YOLO v3 Using Pre-trained Model 289 9.6 A Metric for Object Detection: mAP 310 9.7 Summary and Further Reading 314 10 Introduction to Probabilistic Generative Models 319 10.1 Generative Models with Latent Variables 320 10.2 EM Algorithm 323 10.3 Variational Auto-encoder (VAE) 333 10.4 VAE on MNIST Dataset in PyTorch 340 10.5 Summary and Further Reading 348 11 Generative Adversarial Networks 351 11.1 Mathematical Description of the Original GAN 351 11.2 Implementation of GANs 354 11.3 Practical Issues with the Original GAN 362 11.4 Conditional GAN 362 11.5 InfoGAN 364 11.6 Wasserstein GAN 367 11.7 CycleGAN 372 11.8 f-GANs 375 11.9 Example: Deep Convolutional GAN on MNIST Dataset 378 11.10 Summary and Further Reading 394 12 Diffusion Models 399 12.1 Revisit Variational Auto-Encoder 399 12.3 Score-Based Generative Modeling 409 12.4 Denoising Diffusion Implicit Models for Acceleration 417 12.5 Guidance 421 12.6 Implementation of a Simple Diffusion Model on MNIST Dataset 424 12.7 Summary and Further Reading 436 13 Word Embedding 439 13.1 Introduction to Natural Language Processing 439 13.2 Word2vec 442 13.3 Hierarchical Softmax in Word2vec 452 13.4 Negative Sampling in Word2vec 459 13.5 GloVe 463 13.6 Implementation of a Skip-Gram Model by PyTorch 464 13.7 Summary and Further Reading 472 14 Recurrent Neural Networks 475 14.1 Introduction to Sequence Models 475 14.2 Basic RNNs 476 14.3 Long Short-Term Memory 482 14.4 Practical RNN Architectures 486 14.5 Sequence-to-Sequence Learning: An Application of RNNs 490 14.6 Attention Mechanism in Encoder-Decoder Architectures 494 14.7 BLEU: A Metric of Machine Translation 496 14.8 Implementations of RNNs Using PyTorch 499 14.9 Summary and Further Reading 504 15 Transformer 509 15.1 Bahdanau Attention Mechanism 510 15.2 Attention Mechanism 512 15.3 Transformer Architecture 514 15.4 Bert 520 15.5 Generative Pre-trained Transformer (GPT) 526 15.6 Implementation of a Transformer in PyTorch 529 15.7 Summary and Further Reading 543 16 Introduction to Reinforcement Learning 547 16.1 Definition of Markov Decision Process 547 16.2 Policy, Value Function, and Bellman Equation 550 16.3 Dynamic Programming for MDPs 557 16.4 Monte Carlo Learning 560 16.5 Temporal Difference Learning 564 16.6 Implementation of Q-Learning for a Mountain Car Task 568 16.7 Summary and Further Reading 575 17 Deep Q-Learning 579 17.1 Value Function Approximation 579 17.2 Basic Deep Q-Network 582 17.3 Double Deep Q-Network 585 17.4 Implementation of DQN for Mountain Car-v 0 586 17.5 Summary and Further Reading 595 18 Policy Gradient Methods 601 18.1 Introduction to Policy-Based Methods 601 18.2 Policy Gradient Theorem 602 18.3 REINFORCE Algorithm 605 18.4 Actor-Critic Methods 609 18.5 Policy Optimization Methods 613 18.6 Deep Deterministic Policy Gradient (DDPG) 623 18.7 Soft Actor-Critic Algorithm 627 18.8 On-Policy and Off-Policy 631 18.9 Implementations of Policy Gradient Algorithms in Python 633 18.10 Summary and Further Reading 652 Exercises 653 References 656 Appendix A Mathematics in Machine Learning 657 Index 717