Introduction

Regularization techniques are essential in machine learning to prevent overfitting and improve the generalization of models. These techniques add constraints or penalties to the model to reduce its complexity. In this blog, we will explore various regularization methods, their mathematical definitions, and their effects during the forward and backward passes.

L1 and L2 Regularization

L1 Regularization (Lasso)

1. Definition:
   Least Absolute Shrinkage and Selection Operator (Lasso) regression adds the absolute value of the magnitude of coefficients as a penalty term to the loss function. It encourages sparsity in the model by driving some coefficients to zero.
   • Why L1 pushes weights to zero: L1 regularization creates a sparse weight matrix, meaning many of the weights become zero. This is because the absolute value function has a constant gradient, which applies an equal force to reduce the magnitude of all weights. Over time, this force can zero out weights, especially those that contribute less to the loss function.
2. Regularization Term:
   $$
   \lambda \sum_{i} |w_i|
   $$
3. Equation:
   $$
   \text{Loss} = \text{Loss}_{\text{original}} + \lambda \sum_{i} |w_i|
   $$
4. Forward Pass:
   • The regularization term $(\lambda \sum_{i} |w_i|)$ is added to the loss function.
5. Backward Pass:
   • The gradient update includes the derivative of the regularization term, which is $(\lambda \text{sign}(w_i))$.
6. Original Paper:
   Tibshirani, R. (1996). Regression Shrinkage and Selection via the Lasso. Journal of the Royal Statistical Society: Series B (Statistical Methodology) https://academic.oup.com/jrsssb/article/58/1/267/7027929

L2 Regularization (Ridge)

1. Definition:
   Ridge regression adds the squared magnitude of coefficients as a penalty term to the loss function, shrinking coefficients without driving them to zero.
   • Why L2 does not push weights to zero: L2 regularization tends to shrink weights but does not necessarily zero them out. This is because the squared term grows faster than the absolute value term, leading to a gradient that is proportional to the weight. Smaller weights are shrunk less aggressively, while larger weights are penalized more. This leads to overall weight shrinkage but not sparsity.
2. Regularization Term:
   $$
   \lambda \sum_{i} w_i^2
   $$
3. Equation:
   $$
   \text{Loss} = \text{Loss}_{\text{original}} + \lambda \sum_{i} w_i^2
   $$
4. Forward Pass:
   • The regularization term $(\lambda \sum_{i} w_i^2)$ is added to the loss function.
5. Backward Pass:
   • The gradient update includes the derivative of the regularization term, which is $(2\lambda w_i)$.
6. Original Paper:
   Hoerl, A. E., & Kennard, R. W. (1970). Ridge Regression: Biased Estimation for Nonorthogonal Problems. Technometrics, 12(1), 55-67 https://www.semanticscholar.org/paper/Ridge-Regression%3A-Biased-Estimation-for-Problems-Hoerl-Kennard/1473110f6c33b483251ade10b79416d3efee2da4?p2df

Elastic Net Regularization

1. Definition:
   Elastic Net combines both $L1$ and $L2$ regularization penalties, balancing between feature selection ($L1$) and coefficient shrinkage ($L2$).
2. Regularization Term:
   $$
   \lambda_1 \sum_{i} |w_i| + \lambda_2 \sum_{i} w_i^2
   $$
3. Equation:
   $$
   \text{Loss} = \text{Loss}_{\text{original}} + \lambda_1 \sum_{i} |w_i| + \lambda_2 \sum_{i} w_i^2
   $$
4. Forward Pass:
   • Both regularization terms are added to the loss function.
5. Backward Pass:
   • The gradient update includes the derivatives of both regularization terms: $(\lambda_1 \text{sign}(w_i) + 2\lambda_2 w_i)$.
6. Original Paper:
   Zou, H., & Hastie, T. (2005). Regularization and Variable Selection via the Elastic Net. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67(2), 301-320 https://academic.oup.com/jrsssb/article/67/2/301/7109482?login=false

Weight Decay

1. Definition:
   Weight decay is a technique used to prevent overfitting by adding a penalty term to the loss function. This penalty term is proportional to the square of the weights (or the L2 norm of the weights). The idea is that by adding this penalty, the model is discouraged from learning excessively large weights, which can lead to overfitting.
2. Equation:
   $$
   \text{Loss} = \text{Loss}_{\text{original}} + \lambda \sum_{i} w_i^2
   $$
3. Forward Pass:
   • The regularization term $(\lambda \sum_{i} w_i^2)$ is added to the loss function.
4. Backward Pass:
   • Weights are decayed directly in the weight update rule: $(w_i = w_i – \eta \frac{\partial L}{\partial w_i} – \eta \lambda w_i)$.
5. Original Paper:
   Krogh, A., & Hertz, J. A. (1992). A Simple Weight Decay Can Improve Generalization. Advances in Neural Information Processing Systems (NIPS), 4, 950-957 https://proceedings.neurips.cc/paper_files/paper/1991/hash/8eefcfdf5990e441f0fb6f3fad709e21-Abstract.html

Dropout Regularization

1. Definition:
   Dropout randomly sets a subset of neurons to zero during each training iteration, preventing the network from relying too heavily on specific neurons and promoting more robust learning. It is particularly effective in preventing overfitting in large neural networks.
2. Mathematical Formulation:
   During training, each neuron’s output is kept with probability $( p )$ and dropped (set to zero) with probability $( 1 – p )$. The remaining neurons are scaled by $( \frac{1}{p} )$ to maintain the same expected output during training and testing.
   $$
   Y = \frac{X \cdot M}{1 – p}
   $$
   where $( M )$ is the dropout mask, and $( p )$ is the dropout rate.
3. Forward Pass:
   • Creating the Mask: A binary mask is created with the same shape as the input $( X )$. Each element in the mask is a random value between 0 and 1. If the value is greater than the dropout rate, it is set to 1; otherwise, it is set to 0, effectively “dropping out” certain neurons by setting their activations to zero.
   • Scaling: The activations are scaled by $( \frac{1}{p} )$ to maintain the same expected output during training and testing. This is because during training, only a fraction $( p )$ of the neurons are active on average, so we need to scale the activations to compensate for this reduction.
4. Backward Pass:
   • Applying the Mask: During the backward pass, the same mask is applied to the gradients, ensuring that the gradients flow only through the neurons that were active during the forward pass.
5. Original Paper:
   • Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, 15, 1929-1958 http://jmlr.org/papers/v15/srivastava14a.html

Data Augmentation

1. Definition:
   Data augmentation techniques are used to artificially increase the size of a training dataset by creating modified versions of existing data. This can involve applying transformations such as rotation, scaling, flipping, and more.
2. Forward Pass:
   • Transformed data is fed into the model during training.
3. Backward Pass:
   • The standard backpropagation process is applied to the augmented data.
4. Original Paper:
   Simard, P. Y., Steinkraus, D., & Platt, J. C. (2003). Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis. Proceedings of the International Conference on Document Analysis and Recognition (ICDAR), 958-962 https://www.microsoft.com/en-us/research/publication/best-practices-for-convolutional-neural-networks-applied-to-visual-document-analysis/

CutMix and MixUp

CutMix

1. Definition:
   CutMix combines two images by cutting out a rectangular region from one image and pasting it into another image. The labels are also mixed proportionally to the area of the pasted region.
2. Example:
   Image A: A dog
   Image B: A cat
   CutMix Result: An image with a rectangular region showing part of the cat pasted onto the dog image.
   Label: A weighted combination of the dog and cat labels, e.g., 0.7 dog and 0.3 cat.

MixUp

1. Definition:
   MixUp creates a new training example by linearly combining two images and their labels
2. Example:
   Image A: A dog
   Image B: A cat
   MixUp Result: A new image that is a linear blend of the dog and cat images.
   Label: A weighted combination of the dog and cat labels, e.g., 0.6 dog and 0.4 cat.

Early Stopping

1. Definition:
   Early stopping monitors model performance on a validation set during training and stops training when performance degrades, preventing overfitting.
2. Forward Pass:
   • Model evaluation on the validation set at each epoch.
3. Backward Pass:
   • Training is stopped when validation performance no longer improves.
4. Original Paper:
   Prechelt, L. (1998). Early Stopping – But When? Neural Networks: Tricks of the Trade. Springer, Berlin, Heidelberg https://link.springer.com/chapter/10.1007/3-540-49430-8_3

Batch Normalization

1. Definition:
   Batch normalization normalizes the inputs of each layer to have zero mean and unit variance, including learnable parameters for scaling and shifting the normalized values.
2. Equation:
   $$
   \hat{x} = \frac{x – \mu}{\sqrt{\sigma^2 + \epsilon}}
   $$
   $$
   y = \gamma \hat{x} + \beta
   $$
3. Forward Pass:
   • Compute mean and variance of the mini-batch.
   • Normalize inputs and apply scaling and shifting.
4. Backward Pass:
   • Gradients are calculated for both the normalized values and the learnable parameters (scaling and shifting).
5. Original Paper:
   Ioffe, S., & Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv preprint arXiv:1502.03167 https://arxiv.org/abs/1502.03167

Layer Normalization

1. Definition:
   Layer normalization is similar to batch normalization but normalizes across the features within each training example.
2. Equation:
   $$
   \hat{x} = \frac{x – \mu}{\sqrt{\sigma^2 + \epsilon}}
   $$
   $$
   y = \gamma \hat{x} + \beta
   $$
3. Forward Pass:
   • Compute mean and variance across the features of each training example.
   • Normalize and apply scaling and shifting.
4. Backward Pass:
   • Gradients are calculated for the normalized values and the learnable parameters.
5. Original Paper:
   Ba, J. L., Kiros, J. R., & Hinton, G. E. (2016). Layer Normalization. arXiv preprint arXiv:1607.06450 https://arxiv.org/abs/1607.06450

DropConnect

1. Definition:
   DropConnect is similar to dropout but instead of dropping neurons, individual weights are set to zero randomly.
2. Equation:
   $$
   W = W \cdot M
   $$
   where $( M )$ is a mask applied to the weights.
3. Forward Pass:
   • Randomly set weights to zero and scale remaining weights.
4. Backward Pass:
   • Gradients flow through the remaining weights.
5. Original Paper:
   Wan, L., Zeiler, M., Zhang, S., LeCun, Y., & Fergus, R. (2013). Regularization of Neural Networks using DropConnect. Proceedings of the 30th International Conference on Machine Learning (ICML) http://proceedings.mlr.press/v28/wan13.html

Max Norm Regularization

1. Definition:
   Max norm regularization constrains the norm of the weights to be less than or equal to a predefined value.
2. Equation:
   $$
   |w|_2 \leq c
   $$
   where $( c )$ is a predefined maximum value.
3. Forward Pass:
   • Apply constraint after each weight update.
4. Backward Pass:
   • Standard backpropagation followed by norm constraint enforcement.
5. Original Paper:
   Srebro, N., & Shraibman, A. (2005). Rank, Trace-Norm and Max-Norm. Learning Theory https://link.springer.com/chapter/10.1007/11503415_37

Stochastic Depth

1. Definition:
   Stochastic depth randomly skips entire layers during training.
2. Forward Pass:
   • Layers are randomly skipped according to a predefined probability.
3. Backward Pass:
   • Gradients are only propagated through the active layers.
4. Original Paper:
   Huang, G., Sun, Y., Liu, Z., Sedra, D., & Weinberger, K. Q. (2016). Deep Networks with Stochastic Depth. arXiv preprint arXiv:1603.09382 https://arxiv.org/abs/1603.09382

Label Smoothing

1. Definition:
   Label Smoothing is a regularization technique that softens the labels during training. Instead of assigning a probability of 1 to the correct class and 0 to others, it assigns a slightly lower probability to the correct class and distributes the remaining probability among the incorrect classes. It prevents the model from becoming too confident in its predictions, making it less sensitive to input perturbations. By avoiding overly confident predictions, the model’s weights tend to be smaller and more evenly distributed, which can improve generalization.
2. Equation:
   $$
   \tilde{y} = (1 – \epsilon) y + \frac{\epsilon}{K}
   $$
   where $( \epsilon )$ is the smoothing parameter, $( y )$ is the one-hot encoded label, and $( K )$ is the number of classes.
3. Forward Pass:
   • Modify the labels before computing the loss.
4. Backward Pass:
   • Compute gradients with respect to the smoothed labels.
5. Original Paper:
   Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). Rethinking the Inception Architecture for Computer Vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://arxiv.org/abs/1512.00567

Conclusion

Regularization techniques are vital in machine learning to enhance model generalization and prevent overfitting. Each method has its unique advantages and is suited for different types of models and data. Understanding and correctly applying these regularization methods can significantly improve the performance and robustness of machine learning models.