Momentum vs Normalization in Deep learning -Part 2 – Day 34

Comparing Momentum and Normalization in Deep Learning: A Mathematical Perspective Momentum and normalization are two pivotal techniques in deep learning that enhance the efficiency and stability of training. This article explores the mathematics behind these methods, provides examples with and without these techniques, and demonstrates why they are beneficial for deep learning models.  Comparing Momentum and Normalization Momentum: Smoothing and Accelerating Convergence Momentum is an optimization technique that modifies the standard gradient descent by adding a velocity term to the update rule. This velocity term is a running average of past gradients, which helps the optimizer to continue moving in directions where gradients are consistently pointing, thereby accelerating convergence and reducing oscillations. Mathematical Formulation: Without Momentum (Standard Gradient Descent): With Momentum: Here, is the momentum coefficient (typically around 0.9), and accumulates the gradients to provide smoother and more directed updates. Example with and Without Momentum: Consider a simple quadratic loss function , starting with , a learning rate , and for momentum. Without Momentum: Iteration 1: Gradient at : Update: Iteration 2: Gradient at : Update: With Momentum: Iteration 1: Gradient at : Velocity update: Update: Iteration 2: Gradient at : Velocity update: Update: Why Momentum is Better: Faster Convergence:…