Understanding Dropout in Neural Networks with a Real Numerical Example In deep learning, overfitting is a common problem where a model performs extremely well on training data but fails to generalize to unseen data. One popular solution is dropout, which randomly deactivates neurons during training, making the model more robust. In this section, we will demonstrate dropout with a simple example using numbers and explain how dropout manages weights during training. What is Dropout? Dropout is a regularization technique used in neural networks to prevent overfitting. In a neural network, neurons are connected between layers, and dropout randomly turns off a subset of those neurons during the training phase. When dropout is applied, each neuron has a probability \( p \) of being “dropped out” (i.e., set to zero). For instance, if \( p = 0.5 \), each neuron has a 50% chance of being dropped for a particular training iteration. Importantly, dropout does not remove neurons or weights permanently. Instead, it temporarily deactivates them during training, and they may be active again in future iterations. Let’s walk through a numerical example to see how dropout works in action and how weights are managed during the dropout process. Numerical Example: How Dropout Works Consider a simple neural network with 4 input neurons and 1 output neuron. The input neurons are fully connected to the output neuron, meaning there are 4 weights (one for each input neuron). We will apply dropout with a dropout rate \( p = 0.5 \) and see how the weights are updated. Suppose the input neurons have the following activations \( a_1, a_2, a_3, a_4 \): The weights associated with these neurons are: To compute the output \( z \) of this layer without dropout, we calculate the weighted sum of the activations: Substituting the values: Now, let’s apply dropout with \( p = 0.5 \). This means that each neuron has a 50% chance of being dropped. Step 1: Applying Dropout We randomly “drop out” two neurons. Let’s say we drop out and . These neurons will be ignored (set to 0) in this iteration: The new weighted sum becomes: With dropout, the output \( z_{\text{dropout}} \) is significantly lower because two neurons were dropped from the calculation. What Happens to the Weights During Dropout? It’s important to note that the weights associated with the dropped neurons (i.e., \( w_2 \) and \( w_4 \)) are not removed from the network. They are temporarily ignored for this particular training iteration. In the next iteration, the dropout is applied again, but different neurons may be dropped (or not dropped). The model doesn’t permanently remove any neuron or weight—dropout simply deactivates them at random for different iterations. For example, in one training iteration, neurons \( a_2 \) and \( a_4 \) were dropped, but in the next iteration, \( a_1 \) and \( a_3 \) might be dropped. This ensures that no specific neuron becomes too influential in making predictions, which helps the model generalize better to unseen data. During the testing phase (or validation), dropout is not applied. All neurons are active, and the weights are used as they were trained. Thus, the weights are never permanently removed, but rather they are used or ignored at random during training. Step 2: Scaling During Training To ensure the network doesn’t lose too much information due to dropout, we scale the remaining neurons by a factor of \( \frac{1}{1-p} \). In this case, with \( p = 0.5 \), we scale the remaining neurons by \( 2 \). Thus, the updated activations are: Now, the weighted sum becomes: The output of the neuron after applying dropout and scaling is \( z_{\text{scaled}} = 0.38 \). Step 3: Testing Phase (Without Dropout) During the testing phase, we do not apply dropout. Instead, all neurons are used, and we scale the activations by to account for the dropout that occurred during training. Without dropout, we would return to the original computation: But since the neurons were trained with dropout, we adjust for the…
Thank you for reading this post, don't forget to subscribe!