Batch normalisation part 2 – day 26

Introduction to Batch Normalization Batch normalization is a widely used technique in deep learning that significantly improves the performance and stability of neural networks. Introduced by Sergey Ioffe and Christian Szegedy in 2015, this technique addresses the issues of vanishing and exploding gradients that can occur during training, particularly in deep networks. Why Batch Normalization? In deep learning, as data propagates through the layers of a neural network, it can lead to shifts in the distribution of inputs to layers deeper in the network—a phenomenon known as internal covariate shift. This shift can cause issues such as vanishing gradients, where gradients become too small, slowing down the training process, or exploding gradients, where they become too large, leading to unstable training. Traditional solutions like careful initialization and lower learning rates help, but they don’t entirely solve these problems. What is Batch Normalization? Batch normalization (BN) mitigates these issues by normalizing the inputs of each layer within a mini-batch, ensuring that the inputs to a given layer have a consistent distribution. This normalization happens just before or after the activation function of each hidden layer. Here’s a step-by-step breakdown of how batch normalization works: Zero-Centering and Normalization: \[ \mu_B = \frac{1}{m_B}...

Membership Required

You must be a member to access this content.

View Membership Levels

Already a member? Log in here
Read More
24

3 Types of Gradient Decent Types : Batch, Stochastic & Mini-Batch _ Day 8

Understanding Gradient Descent: Batch, Stochastic, and Mini-Batch Understanding Gradient Descent: Batch, Stochastic, and Mini-Batch Learn the key differences between Batch Gradient Descent, Stochastic Gradient Descent, and Mini-Batch Gradient Descent, and how to apply them in your machine learning models. Batch Gradient Descent Batch Gradient Descent uses the entire dataset to calculate the gradient of the cost function, leading to stable, consistent steps toward an optimal solution. It is computationally expensive, making it suitable for smaller datasets where high precision is crucial. Formula: \[\theta := \theta – \eta \cdot \frac{1}{m} \sum_{i=1}^{m} \nabla_{\theta} J(\theta; x^{(i)}, y^{(i)})\] \(\theta\) = parameters \(\eta\) = learning rate \(m\) = number of training examples \(\nabla_{\theta} J(\theta; x^{(i)}, y^{(i)})\) = gradient of the cost function Stochastic Gradient Descent (SGD) Stochastic Gradient Descent updates parameters using each training example individually. This method can quickly adapt to new patterns, potentially escaping local minima more effectively than Batch Gradient Descent. It is particularly useful for large datasets and online learning environments. Formula: \[\theta := \theta – \eta \cdot \nabla_{\theta} J(\theta; x^{(i)}, y^{(i)})\] \(\theta\) = parameters \(\eta\) = learning rate \(\nabla_{\theta} J(\theta; x^{(i)}, y^{(i)})\) = gradient of the cost function for a single training example Mini-Batch Gradient Descent Mini-Batch Gradient Descent is...

Membership Required

You must be a member to access this content.

View Membership Levels

Already a member? Log in here
Read More
16