Mathematical Explanation behind SGD Algorithm in Machine Learning _ day 5

In our previous blog post – on day 4 – we have talked about using the SGD algorithm for the MNIST dataset. But what is Stochastic Gradient Descent? Stochastic Gradient Descent (SGD) is an iterative method for optimizing an objective function that is written as a sum of differentiable functions. It’s a variant of the traditional gradient descent algorithm but with a twist: instead of computing the gradient of the whole dataset, it approximates the gradient using a single data point or a small batch of data points. This makes SGD much faster and more scalable, especially for large datasets. Why is SGD Important? Efficiency: By updating the parameters using only a subset of data, SGD reduces computation time, making it faster than batch gradient descent for large datasets. Online Learning: SGD can be used in online learning scenarios where the model is updated continuously as new data comes in. Convergence: Although SGD introduces more noise into the optimization process, this can help in escaping local minima and finding a better global minimum. The SGD Algorithm The goal of SGD is to minimize an objective function $J(\theta)$ with respect to the parameters $\theta$. Here’s the general procedure: Initialize: Randomly initialize…