optimization. – Ghazi Hudeihed

Tag Archives: optimization.

Blog

Optimization Techniques for Deep Learning: Enhancing Performance and Efficiency

Introduction Training deep neural networks presents several challenges related to memory constraints, computational resources, and convergence issues. This document explores advanced techniques that address these challenges, including optimization algorithms like Stochastic Gradient Descent (SGD), SGD with Momentum, Adam, LARS, and LAMB, as well as methods such as gradient accumulation and activation checkpointing. Optimizing the Loss…

July 27, 2024 Ghazi Hudeihed

Activation Checkpointing Adam Optimizer Adaptive Learning Rates Computational Efficiency Deep Learning Gradient Accumulation Gradient Descent LAMB LARS Machine Learning Memory Optimization Model Training Neural Networks optimization.Weight Decay

Blog, Machine Learning

Regularization Techniques to Improve Model Generalization

Introduction In our last discussion, we explored dropout regularization techniques, which involve randomly setting a fraction of the activations to zero during training. This helps prevent overfitting by encouraging the network to learn redundant representations and improving generalization. Today, we will extend our focus to other regularization methods, including L1 and L2 regularization, label smoothing,…

July 9, 2024 Ghazi Hudeihed

active learning CNNs CutMix Data Augmentation Deep Learning Dropout feature selection generalization image classification L1 Regularization L2 Regularization Label Smoothing Loss Function Machine Learning Mean Squared Error medical diagnostics MixUp model confidence model uncertainty Monte Carlo dropout Neural Networks optimization.Overfitting Regularization segmentation masks Stochastic Gradient Descent Weight Decay weight updates