https://github.com/amirjahantab/regularization
An effective way to avoid (or at least to reduce) overfitt
https://github.com/amirjahantab/regularization
ai data-science deep-learning machine-learning regularization regularization-hyperparameters
Last synced: 3 months ago
JSON representation
An effective way to avoid (or at least to reduce) overfitt
- Host: GitHub
- URL: https://github.com/amirjahantab/regularization
- Owner: amirjahantab
- Created: 2024-07-17T19:04:55.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2024-07-18T06:55:07.000Z (11 months ago)
- Last Synced: 2025-01-11T20:15:23.002Z (5 months ago)
- Topics: ai, data-science, deep-learning, machine-learning, regularization, regularization-hyperparameters
- Language: Jupyter Notebook
- Homepage:
- Size: 1.89 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Regularization in Machine Learning
## Overview
Regularization is a crucial technique in machine learning used to prevent overfitting by adding a penalty to the loss function. Overfitting occurs when a model learns the noise in the training data instead of the actual signal, leading to poor generalization on new, unseen data. Regularization methods aim to make the model simpler, enhancing its ability to generalize.
## Types of Regularization
### L1 Regularization (Lasso)
- **Description**: Adds the absolute value of the coefficients as a penalty term to the loss function.
- **Effect**: Encourages sparsity, meaning it tends to set some coefficients to zero, effectively selecting a simpler model with fewer features.
- **Formula**: \( \text{Loss} = \text{Original Loss} + \lambda \sum |w_i| \)
### L2 Regularization (Ridge)
- **Description**: Adds the square of the coefficients as a penalty term to the loss function.
- **Effect**: Encourages smaller coefficients overall but does not enforce sparsity.
- **Formula**: \( \text{Loss} = \text{Original Loss} + \lambda \sum w_i^2 \)
### Elastic Net
- **Description**: Combines both L1 and L2 regularization penalties.
- **Effect**: Balances between sparsity of L1 and small coefficients of L2.
- **Formula**: \( \text{Loss} = \text{Original Loss} + \lambda_1 \sum |w_i| + \lambda_2 \sum w_i^2 \)
## When to Use Regularization
- **Overfitting**: When the model performs well on training data but poorly on validation/test data.
- **Complex Models**: When using models that have a large number of parameters or features.
- **Feature Selection**: When you want to perform feature selection (mainly L1 regularization).
## Regularization in Practice
In practice, regularization is applied by selecting an appropriate value for the regularization parameter (λ). This parameter determines the weight of the penalty term. Choosing the right λ often involves cross-validation to find the optimal balance between underfitting and overfitting.
## Learn with Jupyter Notebook
To dive deeper into the implementation and impact of regularization techniques in machine learning, explore my [Jupyter Notebook on Regularization](https://github.com/amirjahantab/Regularization/blob/main/Regularization.ipynb). The notebook contains detailed explanations, visualizations, and code examples to help you understand and apply regularization methods effectively.
## Conclusion
Regularization is a powerful tool in the machine learning arsenal, helping to create models that generalize well to new data by preventing overfitting. By using L1, L2, or Elastic Net regularization, you can control model complexity and improve performance.