{"id":47457200,"url":"https://github.com/stabgan/awesome-loss-functions","last_synced_at":"2026-05-25T00:11:46.523Z","repository":{"id":344444300,"uuid":"1181649439","full_name":"stabgan/awesome-loss-functions","owner":"stabgan","description":"📚 350+ loss functions across 25+ AI subdomains — classification, GANs, diffusion, LLM alignment, RL, contrastive learning, audio, video, time series, and more. Chronologically ordered with paper links, math formulas, and implementations.","archived":false,"fork":false,"pushed_at":"2026-03-14T18:44:08.000Z","size":72,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-15T04:06:21.676Z","etag":null,"topics":["awesome","awesome-list","computer-vision","contrastive-learning","deep-learning","diffusion-models","gan","generative-models","knowledge-distillation","llm-alignment","loss-functions","machine-learning","nlp","object-detection","paper-list","pytorch","reinforcement-learning","self-supervised-learning","survey","time-series"],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stabgan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"contributing.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-14T12:40:22.000Z","updated_at":"2026-03-14T19:21:05.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/stabgan/awesome-loss-functions","commit_stats":null,"previous_names":["stabgan/awesome-loss-functions"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/stabgan/awesome-loss-functions","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stabgan%2Fawesome-loss-functions","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stabgan%2Fawesome-loss-functions/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stabgan%2Fawesome-loss-functions/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stabgan%2Fawesome-loss-functions/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stabgan","download_url":"https://codeload.github.com/stabgan/awesome-loss-functions/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stabgan%2Fawesome-loss-functions/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31504897,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-07T03:10:19.677Z","status":"ssl_error","status_checked_at":"2026-04-07T03:10:13.982Z","response_time":105,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["awesome","awesome-list","computer-vision","contrastive-learning","deep-learning","diffusion-models","gan","generative-models","knowledge-distillation","llm-alignment","loss-functions","machine-learning","nlp","object-detection","paper-list","pytorch","reinforcement-learning","self-supervised-learning","survey","time-series"],"created_at":"2026-03-24T00:00:26.576Z","updated_at":"2026-05-25T00:11:46.477Z","avatar_url":"https://github.com/stabgan.png","language":null,"funding_links":[],"categories":["ML Sub-fields","Other Lists"],"sub_categories":["TeX Lists"],"readme":"# Awesome Loss Functions [![Awesome](https://awesome.re/badge.svg)](https://awesome.re)\n\n[![License: CC0-1.0](https://img.shields.io/badge/License-CC0_1.0-lightgrey.svg)](https://creativecommons.org/publicdomain/zero/1.0/)\n[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](http://makeapullrequest.com)\n[![Contributions](https://img.shields.io/badge/contributions-welcome-orange.svg)](https://github.com/stabgan/awesome-loss-functions/issues)\n\nA comprehensive, chronologically ordered collection of loss functions across all subdomains of deep learning and machine learning — with paper links, one-line descriptions, mathematical formulations, and implementation references.\n\n**350+ loss functions. 25+ categories. Every subdomain of AI.**\n\n\u003e If this resource helps your research or engineering work, please consider giving it a ⭐\n\n---\n\n## What's New\n\n- 🔊 **Audio, Music \u0026 Speech Generation** — WaveNet to Stable Audio, 19 losses\n- 🎬 **Video Generation \u0026 Understanding** — VGAN to VideoPoet, 20 losses\n- ⏳ **Time Series Forecasting** — Pinball Loss to TimesFM, 23 losses\n- 🧠 **Continual \u0026 Lifelong Learning** — EWC to EASE, 18 methods\n- ⚖️ **Calibration, Fairness \u0026 Bias Mitigation** — Brier Score to Group DRO, 18 losses\n- 🛡️ **Adversarial Robustness \u0026 OOD Detection** — FGSM-AT to CIDER, 22 losses\n- 🔍 **Anomaly Detection \u0026 Multi-Modal Learning** — Deep SVDD to ImageBind, 17 losses\n- 🖼️ **Image-to-Image Translation** — Total Variation to DoveNet, 16 losses\n- 📐 **Semi-Supervised Learning** — Pseudo-Label to SoftMatch, 12 losses\n- 🎯 **Optical Flow, Video \u0026 Pose** — Horn-Schunck to SEA-RAFT, 33 losses\n\n---\n\n## Contents\n\n**Core Categories (inline)**\n\n- [Loss Selection Guide](#-loss-selection-guide)\n- [Key Mathematical Formulations](#-key-mathematical-formulations)\n- [Classification](#classification)\n- [Regression](#regression)\n- [Segmentation](#segmentation)\n- [Object Detection (Bounding Box)](#object-detection-bounding-box)\n- [Generative Models — GANs](#generative-models--gans)\n- [Generative Models — VAEs](#generative-models--vaes)\n- [Generative Models — Diffusion \u0026 Flow](#generative-models--diffusion--flow)\n- [Reconstruction \u0026 Perceptual](#reconstruction--perceptual)\n- [Image Super-Resolution \u0026 Restoration](#image-super-resolution--restoration)\n- [Contrastive \u0026 Self-Supervised Learning](#contrastive--self-supervised-learning)\n- [Metric Learning \u0026 Face Recognition](#metric-learning--face-recognition)\n- [NLP \u0026 Language Modeling](#nlp--language-modeling)\n- [LLM Alignment (RLHF / DPO)](#llm-alignment-rlhf--dpo)\n- [Sequence-to-Sequence \u0026 Speech](#sequence-to-sequence--speech)\n- [Reinforcement Learning](#reinforcement-learning)\n- [Knowledge Distillation](#knowledge-distillation)\n- [Regularization](#regularization)\n- [3D Vision \u0026 Point Clouds](#3d-vision--point-clouds)\n- [Depth Estimation](#depth-estimation)\n- [Medical Imaging](#medical-imaging)\n- [Graph Neural Networks](#graph-neural-networks)\n- [Recommendation Systems](#recommendation-systems)\n- [Multi-Task Learning](#multi-task-learning)\n- [Uncertainty Estimation](#uncertainty-estimation)\n- [Domain Adaptation](#domain-adaptation)\n\n**Extended Categories (separate files)**\n\n- [Audio, Music \u0026 Speech Generation](sections/audio-music-speech.md) — 19 losses\n- [Video Generation \u0026 Understanding](sections/video-generation-understanding.md) — 20 losses\n- [Time Series Forecasting](sections/time-series-forecasting.md) — 23 losses\n- [Continual \u0026 Lifelong Learning](sections/continual-lifelong-learning.md) — 18 methods\n- [Calibration, Fairness \u0026 Bias Mitigation](sections/calibration-fairness.md) — 18 losses\n- [Adversarial Robustness \u0026 OOD Detection](sections/adversarial-robustness-ood.md) — 22 losses\n- [Anomaly Detection \u0026 Multi-Modal Learning](sections/anomaly-detection-and-multimodal.md) — 17 losses\n- [Image-to-Image Translation \u0026 Style Transfer](sections/image-to-image-translation.md) — 16 losses\n- [Semi-Supervised Learning \u0026 Self-Training](sections/semi-supervised-learning.md) — 12 losses\n- [Optical Flow, Video Prediction \u0026 Pose Estimation](sections/temporal-motion.md) — 33 losses\n\n**Resources**\n\n- [Survey Papers](#survey-papers)\n- [Key Implementation Libraries](#key-implementation-libraries)\n\n---\n\n## 🧭 Loss Selection Guide\n\nNot sure which loss to use? Here's a quick decision framework:\n\n| Task | Default Choice | Class Imbalance | Noisy Labels | Need Calibration |\n|------|---------------|-----------------|--------------|------------------|\n| Binary Classification | BCE | Focal Loss | SCE / GCE | Focal + Temp. Scaling |\n| Multi-class Classification | Cross-Entropy | Class-Balanced CE | Label Smoothing | Label Smoothing |\n| Semantic Segmentation | CE + Dice | Focal Tversky | — | — |\n| Object Detection (box) | Smooth L1 + Focal | Focal Loss | — | — |\n| Object Detection (IoU) | CIoU / GIoU | — | — | — |\n| Image Generation (GAN) | Hinge / Non-Saturating | — | — | — |\n| Image Generation (Diffusion) | DDPM (ε-prediction) | — | — | — |\n| Super-Resolution | L1 + Perceptual + GAN | — | — | — |\n| Self-Supervised (vision) | InfoNCE / DINO | — | — | — |\n| Face Recognition | ArcFace / AdaFace | Sub-center ArcFace | ElasticFace | — |\n| Language Modeling | Cross-Entropy (NTP) | — | — | — |\n| LLM Alignment | DPO / SimPO | — | — | — |\n| Speech Recognition | CTC / RNN-T | — | — | — |\n| RL (value-based) | DQN / Double DQN | — | — | — |\n| RL (policy-based) | PPO | — | — | — |\n| Regression | MSE / Huber | — | Huber | NLL w/ variance |\n| Metric Learning | Triplet / Proxy Anchor | — | — | — |\n| Medical Segmentation | Dice + Boundary | Tversky / Focal Tversky | — | — |\n| 3D Reconstruction | Chamfer + Normal | — | — | — |\n| Depth Estimation | Scale-Invariant | — | — | — |\n| Time Series | MSE / Quantile | — | Huber | CRPS |\n| Continual Learning | EWC / DER++ | — | — | — |\n| Fairness | Group DRO | — | — | — |\n\n---\n\n## 📐 Key Mathematical Formulations\n\n**Cross-Entropy Loss**\n\n$$\n\\mathcal{L}_{CE} = -\\sum_{c=1}^{C} y_c \\log(\\hat{y}_c)\n$$\n\n**Binary Cross-Entropy**\n\n$$\n\\mathcal{L}_{BCE} = -[y \\log(\\hat{y}) + (1-y) \\log(1-\\hat{y})]\n$$\n\n**Focal Loss**\n\n$$\n\\mathcal{L}_{FL} = -\\alpha_t (1 - p_t)^\\gamma \\log(p_t)\n$$\n\n**Dice Loss**\n\n$$\n\\mathcal{L}_{Dice} = 1 - \\frac{2 \\sum_i p_i g_i}{\\sum_i p_i + \\sum_i g_i}\n$$\n\n**Triplet Loss**\n\n$$\n\\mathcal{L}_{Triplet} = \\max(0, \\|f_a - f_p\\|_2 - \\|f_a - f_n\\|_2 + \\alpha)\n$$\n\n**InfoNCE / Contrastive Loss**\n\n$$\n\\mathcal{L}_{InfoNCE} = -\\log \\frac{\\exp(\\text{sim}(z_i, z_j) / \\tau)}{\\sum_{k=1}^{2N} \\mathbb{1}_{[k \\neq i]} \\exp(\\text{sim}(z_i, z_k) / \\tau)}\n$$\n\n**KL Divergence**\n\n$$\nD_{KL}(P \\| Q) = \\sum_x P(x) \\log \\frac{P(x)}{Q(x)}\n$$\n\n**DDPM Loss (simplified)**\n\n$$\n\\mathcal{L}_{DDPM} = \\mathbb{E}_{t, x_0, \\epsilon} \\left[ \\| \\epsilon - \\epsilon_\\theta(x_t, t) \\|^2 \\right]\n$$\n\n**DPO Loss**\n\n$$\n\\mathcal{L}_{DPO} = -\\log \\sigma \\left( \\beta \\log \\frac{\\pi_\\theta(y_w|x)}{\\pi_{ref}(y_w|x)} - \\beta \\log \\frac{\\pi_\\theta(y_l|x)}{\\pi_{ref}(y_l|x)} \\right)\n$$\n\n**IoU Loss**\n\n$$\n\\mathcal{L}_{IoU} = 1 - \\frac{|B_p \\cap B_{gt}|}{|B_p \\cup B_{gt}|}\n$$\n\n**ArcFace Loss**\n\n$$\n\\mathcal{L}_{ArcFace} = -\\log \\frac{e^{s \\cos(\\theta_{y_i} + m)}}{e^{s \\cos(\\theta_{y_i} + m)} + \\sum_{j \\neq y_i} e^{s \\cos \\theta_j}}\n$$\n\n**Wasserstein Distance (WGAN)**\n\n$$\n\\mathcal{L}_{WGAN} = \\mathbb{E}_{x \\sim p_{data}}[D(x)] - \\mathbb{E}_{z \\sim p_z}[D(G(z))]\n$$\n\n---\n\n## Classification\n\n**0/1 Loss** (1950) — The theoretical misclassification indicator; 1 if prediction ≠ label, 0 otherwise. Non-differentiable, foundational to learning theory.\n📄 *Statistical Decision Functions* — Wald, A.\n\n**Cross-Entropy Loss / Log Loss / Negative Log-Likelihood** (1948) — Measures divergence between predicted probability distribution and true labels; the default loss for multi-class classification.\n📄 [A Mathematical Theory of Communication](https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf) — Shannon, C.E.\n💻 [`torch.nn.CrossEntropyLoss`](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html)\n\n**Binary Cross-Entropy** (1958) — Cross-entropy specialized for two-class or multi-label problems; operates on each output independently.\n📄 Derived from logistic regression — Cox, D.R. (1958)\n💻 [`torch.nn.BCEWithLogitsLoss`](https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html)\n\n**Hinge Loss / SVM Loss** (1995) — Maximizes the margin between classes; the core loss behind Support Vector Machines.\n📄 [Support-Vector Networks](https://link.springer.com/article/10.1007/BF00994018) — Cortes, C. \u0026 Vapnik, V.\n💻 [`torch.nn.MultiMarginLoss`](https://pytorch.org/docs/stable/generated/torch.nn.MultiMarginLoss.html)\n\n**Knowledge Distillation Loss / Soft Cross-Entropy** (2015) — Trains a student network to mimic a teacher by matching softened output distributions.\n📄 [Distilling the Knowledge in a Neural Network](https://arxiv.org/abs/1503.02531) — Hinton, G., Vinyals, O. \u0026 Dean, J.\n💻 [`torch.nn.KLDivLoss`](https://pytorch.org/docs/stable/generated/torch.nn.KLDivLoss.html)\n\n**Large-Margin Softmax Loss (L-Softmax)** (2016) — Introduces angular margin constraints into softmax for intra-class compactness and inter-class separability.\n📄 [Large-Margin Softmax Loss for Convolutional Neural Networks](https://arxiv.org/abs/1612.02295) — Liu, W., Wen, Y., Yu, Z. \u0026 Yang, M.\n💻 [wy1iu/LargeMargin_Softmax_Loss](https://github.com/wy1iu/LargeMargin_Softmax_Loss)\n\n**Center Loss** (2016) — Penalizes distance of features from learned class centers, improving discriminative feature learning.\n📄 [A Discriminative Feature Learning Approach for Deep Face Recognition](https://kpzhang93.github.io/papers/eccv2016.pdf) — Wen, Y., Zhang, K., Li, Z. \u0026 Qiao, Y.\n💻 [KaiyangZhou/pytorch-center-loss](https://github.com/KaiyangZhou/pytorch-center-loss)\n\n**Label Smoothing** (2016) — Replaces hard one-hot targets with soft targets, preventing overconfident predictions and improving generalization.\n📄 [Rethinking the Inception Architecture for Computer Vision](https://arxiv.org/abs/1512.00567) — Szegedy, C. et al.\n💻 [`torch.nn.CrossEntropyLoss(label_smoothing=...)`](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html)\n\n**Sparsemax Loss** (2016) — Sparse alternative to softmax that assigns exactly zero probability to irrelevant classes.\n📄 [From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification](https://arxiv.org/abs/1602.02068) — Martins, A.F.T. \u0026 Astudillo, R.F.\n💻 [deep-spin/entmax](https://github.com/deep-spin/entmax)\n\n**Focal Loss** (2017) — Down-weights well-classified examples to focus training on hard negatives; designed for extreme class imbalance.\n📄 [Focal Loss for Dense Object Detection](https://arxiv.org/abs/1708.02002) — Lin, T.-Y., Goyal, P., Girshick, R., He, K. \u0026 Dollár, P.\n💻 [AdeelH/pytorch-multi-class-focal-loss](https://github.com/AdeelH/pytorch-multi-class-focal-loss)\n\n**Generalized Cross-Entropy (GCE)** (2018) — Noise-robust loss interpolating between MAE and cross-entropy via a tunable parameter q.\n📄 [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://arxiv.org/abs/1805.07836) — Zhang, Z. \u0026 Sabuncu, M.R.\n💻 [AlanChou/Truncated-Loss](https://github.com/AlanChou/Truncated-Loss)\n\n**Complement Objective Training (COT)** (2019) — Augments cross-entropy with a complement objective that neutralizes non-target class probabilities.\n📄 [Complement Objective Training](https://arxiv.org/abs/1903.01182) — Chen, H.-Y. et al.\n💻 [henry8527/COT](https://github.com/henry8527/COT)\n\n**Class-Balanced Loss** (2019) — Re-weights loss by the effective number of samples per class for long-tailed distributions.\n📄 [Class-Balanced Loss Based on Effective Number of Samples](https://arxiv.org/abs/1901.05555) — Cui, Y. et al.\n💻 [vandit15/Class-balanced-loss-pytorch](https://github.com/vandit15/Class-balanced-loss-pytorch)\n\n**Symmetric Cross-Entropy (SCE)** (2019) — Combines standard CE with reverse CE for robustness to label noise.\n📄 [Symmetric Cross Entropy for Robust Learning with Noisy Labels](https://arxiv.org/abs/1908.06112) — Wang, Y. et al.\n\n**Bi-Tempered Logistic Loss** (2019) — Two temperature parameters bound the loss (handling mislabeled data) and produce heavy-tailed softmax (handling outliers).\n📄 [Robust Bi-Tempered Logistic Loss Based on Bregman Divergences](https://arxiv.org/abs/1906.03361) — Amid, E. et al.\n💻 [google/bi-tempered-loss](https://github.com/google/bi-tempered-loss)\n\n**Taylor Cross-Entropy Loss** (2020) — Taylor series expansion of CE creating a noise-robust loss.\n📄 [Can Cross Entropy Loss Be Robust to Label Noise?](https://www.ijcai.org/proceedings/2020/305) — Feng, L. et al.\n\n**Asymmetric Loss (ASL)** (2021) — Different focusing levels for positive and negative samples in multi-label classification.\n📄 [Asymmetric Loss For Multi-Label Classification](https://arxiv.org/abs/2009.14119) — Ben-Baruch, E. et al.\n💻 [Alibaba-MIIL/ASL](https://github.com/Alibaba-MIIL/ASL)\n\n**Poly Loss** (2022) — Views loss functions as polynomial expansions and adjusts leading coefficients; generalizes CE and focal loss.\n📄 [PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functions](https://arxiv.org/abs/2204.12511) — Leng, Z. et al.\n💻 [abhuse/polyloss-pytorch](https://github.com/abhuse/polyloss-pytorch)\n\n## Regression\n\n**Mean Absolute Error (MAE) / L1 Loss** (~1757) — Penalizes absolute differences; robust to outliers but non-smooth gradient at zero.\n📄 Attributed to Boscovich, R.J. (1757)\n💻 [`torch.nn.L1Loss`](https://pytorch.org/docs/stable/generated/torch.nn.L1Loss.html)\n\n**Mean Squared Error (MSE) / L2 Loss** (~1805) — Penalizes squared differences; sensitive to outliers. The method of least squares.\n📄 Legendre, A.-M. (1805); Gauss, C.F. (1809)\n💻 [`torch.nn.MSELoss`](https://pytorch.org/docs/stable/generated/torch.nn.MSELoss.html)\n\n**Huber Loss** (1964) — MSE for small errors, MAE for large errors. Robust to outliers with smooth gradients near zero.\n📄 [Robust Estimation of a Location Parameter](https://projecteuclid.org/journals/annals-of-mathematical-statistics/volume-35/issue-1/Robust-Estimation-of-a-Location-Parameter/10.1214/aoms/1177703732.full) — Huber, P.J.\n💻 [`torch.nn.HuberLoss`](https://pytorch.org/docs/stable/generated/torch.nn.HuberLoss.html)\n\n**Tukey's Biweight Loss** (1974) — Redescending M-estimator that completely rejects gross outliers beyond a threshold.\n📄 [The Fitting of Power Series, Meaning Polynomials, Illustrated on Band-Spectroscopic Data](https://www.tandfonline.com/doi/abs/10.1080/00401706.1974.10489171) — Beaton, A.E. \u0026 Tukey, J.W.\n\n**Quantile Loss / Pinball Loss** (1978) — Asymmetrically penalizes over/under-predictions for quantile regression and uncertainty estimation.\n📄 [Regression Quantiles](https://people.eecs.berkeley.edu/~jordan/sail/readings/koenker-bassett.pdf) — Koenker, R. \u0026 Bassett, G.\n\n**Smooth L1 Loss** (2015) — L2 for small errors, L1 for large errors (Huber with δ=1); standard for bounding box regression.\n📄 [Fast R-CNN](https://arxiv.org/abs/1504.08083) — Girshick, R.\n💻 [`torch.nn.SmoothL1Loss`](https://pytorch.org/docs/stable/generated/torch.nn.SmoothL1Loss.html)\n\n**Wing Loss** (2018) — Amplifies small-to-medium range errors for facial landmark localization.\n📄 [Wing Loss for Robust Facial Landmark Localisation with Convolutional Neural Networks](https://arxiv.org/abs/1711.06753) — Feng, Z.-H. et al.\n\n**Balanced L1 Loss** (2019) — Rebalances inlier vs. outlier loss contributions in object detection regression.\n📄 [Libra R-CNN: Towards Balanced Learning for Object Detection](https://arxiv.org/abs/1904.02701) — Pang, J. et al.\n💻 [OceanPang/Libra_R-CNN](https://github.com/OceanPang/Libra_R-CNN)\n\n**Adaptive Wing Loss** (2019) — Adapts curvature based on ground truth heatmap values for face alignment.\n📄 [Adaptive Wing Loss for Robust Face Alignment via Heatmap Regression](https://arxiv.org/abs/1904.07399) — Wang, X. et al.\n💻 [protossw512/AdaptiveWingLoss](https://github.com/protossw512/AdaptiveWingLoss)\n\n**Log-Cosh Loss** (2022) — Approximates Huber loss using log(cosh(x)); twice differentiable everywhere.\n📄 [Statistical Properties of the Log-Cosh Loss Function Used in Machine Learning](https://arxiv.org/abs/2208.04564) — Chen, K. et al.\n\n## Segmentation\n\n**Sensitivity-Specificity Loss** (2015) — Weighted combination of sensitivity and specificity for extreme class imbalance in lesion segmentation.\n📄 [Deep Convolutional Encoder Networks for Multiple Sclerosis Lesion Segmentation](https://doi.org/10.1007/978-3-319-24574-4_1) — Brosch et al.\n\n**Dice Loss** (2016) — Directly optimizes the Dice coefficient (F1 score); robust to class imbalance.\n📄 [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://arxiv.org/abs/1606.04797) — Milletari, F. et al.\n💻 [JunMa11/SegLossOdyssey](https://github.com/JunMa11/SegLossOdyssey)\n\n**Generalized Dice Loss (GDL)** (2017) — Per-class volume weighting for multi-class segmentation with highly imbalanced labels.\n📄 [Generalised Dice Overlap as a Deep Learning Loss Function for Highly Unbalanced Segmentations](https://arxiv.org/abs/1707.03237) — Sudre, C.H. et al.\n\n**Tversky Loss** (2017) — Tunable α/β parameters controlling the FP/FN trade-off; useful for small lesion segmentation.\n📄 [Tversky Loss Function for Image Segmentation Using 3D Fully Convolutional Deep Networks](https://arxiv.org/abs/1706.05721) — Salehi, S.S.M. et al.\n\n**Lovász-Softmax Loss** (2018) — Tractable convex surrogate for directly optimizing the Jaccard index (IoU).\n📄 [The Lovász-Softmax Loss: A Tractable Surrogate for the Optimization of the Intersection-Over-Union Measure](https://arxiv.org/abs/1705.08790) — Berman, M. et al.\n💻 [bermanmaxim/LovaszSoftmax](https://github.com/bermanmaxim/LovaszSoftmax)\n\n**Exponential Logarithmic Loss** (2018) — Combines exponentially weighted focal-style Dice and CE for very small structures.\n📄 [3D Segmentation with Exponential Logarithmic Loss for Highly Unbalanced Object Sizes](https://arxiv.org/abs/1809.00076) — Wong et al.\n\n**Asymmetric Similarity Loss** (2018) — Asymmetric Fβ-score-based similarity to balance precision and recall.\n📄 [Asymmetric Loss Functions and Deep Densely Connected Networks for Highly Imbalanced Medical Image Segmentation](https://arxiv.org/abs/1803.11078) — Hashemi et al.\n\n**Focal Tversky Loss** (2019) — Focal-style exponent on Tversky loss to focus on hard, misclassified regions.\n📄 [A Novel Focal Tversky Loss Function with Improved Attention U-Net for Lesion Segmentation](https://arxiv.org/abs/1810.07842) — Abraham, N. \u0026 Khan, N.M.\n\n**Boundary Loss** (2019) — Distance metric on contour space rather than region overlap; effective for highly unbalanced tasks.\n📄 [Boundary Loss for Highly Unbalanced Segmentation](https://arxiv.org/abs/1812.07032) — Kervadec, H. et al.\n💻 [LIVIAETS/boundary-loss](https://github.com/LIVIAETS/boundary-loss)\n\n**Hausdorff Distance Loss** (2019) — Directly optimizes the Hausdorff distance between predicted and ground-truth boundaries.\n📄 [Reducing the Hausdorff Distance in Medical Image Segmentation with Convolutional Neural Networks](https://arxiv.org/abs/1904.10030) — Karimi, D. \u0026 Salcudean, S.E.\n\n**Combo Loss** (2019) — Weighted combination of modified CE and Dice loss for input and output class imbalance.\n📄 [Combo Loss: Handling Input and Output Imbalance in Multi-Organ Segmentation](https://arxiv.org/abs/1805.02798) — Taghanaki et al.\n\n**Region Mutual Information (RMI) Loss** (2019) — Maximizes mutual information between predicted and ground-truth label regions.\n📄 [Region Mutual Information Loss for Semantic Segmentation](https://arxiv.org/abs/1910.12037) — Zhao et al.\n💻 [ZJULearning/RMI](https://github.com/ZJULearning/RMI)\n\n**Topological Loss** (2019) — Uses persistent homology to enforce correct topological structure in segmentation.\n📄 [Topology-Preserving Deep Image Segmentation](https://arxiv.org/abs/1906.05404) — Hu et al.\n💻 [HuXiaoling/TopoLoss](https://github.com/HuXiaoling/TopoLoss)\n\n**Log-Cosh Dice Loss** (2020) — Log-cosh smoothing on Dice loss for smoother gradients and stable training.\n📄 [A Survey of Loss Functions for Semantic Segmentation](https://arxiv.org/abs/2006.14822) — Jadon, S.\n\n**clDice** (2021) — Topology-preserving loss for tubular structures; computes Dice on skeletonized centerlines.\n📄 [clDice — A Novel Topology-Preserving Loss Function for Tubular Structure Segmentation](https://arxiv.org/abs/2003.07311) — Shit et al.\n💻 [jocpae/clDice](https://github.com/jocpae/clDice)\n\n**Unified Focal Loss** (2022) — Hierarchical framework generalizing Dice-based and CE-based losses with focal modulation.\n📄 [Unified Focal Loss: Generalising Dice and Cross Entropy-Based Losses to Handle Class Imbalanced Medical Image Segmentation](https://arxiv.org/abs/2102.04525) — Yeung et al.\n💻 [mlyg/unified-focal-loss](https://github.com/mlyg/unified-focal-loss)\n\n## Object Detection (Bounding Box)\n\n**Smooth L1 Loss** (2015) — Piecewise L2/L1 loss; standard for bounding box regression.\n📄 [Fast R-CNN](https://arxiv.org/abs/1504.08083) — Girshick, R.\n\n**IoU Loss** (2016) — Directly regresses Intersection-over-Union between predicted and ground-truth boxes.\n📄 [UnitBox: An Advanced Object Detection Network](https://arxiv.org/abs/1608.01471) — Yu et al.\n\n**Focal Loss** (2017) — Modulating factor (1−pₜ)^γ down-weights easy negatives in dense detection.\n📄 [Focal Loss for Dense Object Detection](https://arxiv.org/abs/1708.02002) — Lin, T.-Y. et al.\n💻 [facebookresearch/detectron2](https://github.com/facebookresearch/detectron2)\n\n**Bounded IoU Loss** (2018) — Upper-bounds IoU change per coordinate for stable high-IoU refinement.\n📄 [Improving Object Localization with Fitness NMS and Bounded IoU Loss](https://arxiv.org/abs/1711.00164) — Tychsen-Smith \u0026 Petersson\n\n**GIoU Loss** (2019) — Extends IoU with a penalty based on the smallest enclosing box; enables gradient flow for non-overlapping boxes.\n📄 [Generalized Intersection over Union](https://arxiv.org/abs/1902.09630) — Rezatofighi et al.\n\n**DIoU Loss** (2020) — Adds normalized center-point distance penalty to IoU for faster convergence.\n📄 [Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression](https://arxiv.org/abs/1911.08287) — Zheng et al.\n💻 [Zzh-tju/DIoU](https://github.com/Zzh-tju/DIoU)\n\n**CIoU Loss** (2020) — Extends DIoU with aspect ratio consistency penalty for complete geometric alignment.\n📄 [Distance-IoU Loss](https://arxiv.org/abs/1911.08287) — Zheng et al.\n\n**Alpha-IoU Loss** (2021) — Power parameter α amplifies loss and gradient for high-quality anchors.\n📄 [Alpha-IoU: A Family of Power Intersection over Union Losses](https://arxiv.org/abs/2110.13675) — He et al.\n💻 [Jacobi93/Alpha-IoU](https://github.com/Jacobi93/Alpha-IoU)\n\n**EIoU Loss** (2022) — Decomposes CIoU penalty into separate width/height terms.\n📄 [Focal and Efficient IOU Loss for Accurate Bounding Box Regression](https://arxiv.org/abs/2101.08158) — Zhang et al.\n\n**SIoU Loss** (2022) — Angle-aware penalty considering vector direction between predicted and target boxes.\n📄 [SIoU Loss: More Powerful Learning for Bounding Box Regression](https://arxiv.org/abs/2205.12740) — Gevorgyan\n\n**WIoU Loss** (2023) — Dynamic non-monotonic focusing mechanism based on outlier degree.\n📄 [Wise-IoU: Bounding Box Regression Loss with Dynamic Focusing Mechanism](https://arxiv.org/abs/2301.10051) — Tong et al.\n💻 [Instinct323/Wise-IoU](https://github.com/Instinct323/Wise-IoU)\n\n**MPDIoU Loss** (2023) — Bounding box similarity via minimum point distances between corners.\n📄 [MPDIoU: A Loss for Efficient and Accurate Bounding Box Regression](https://arxiv.org/abs/2307.07662) — Ma \u0026 Xu\n\n**Inner-IoU Loss** (2023) — IoU through auxiliary inner bounding boxes with a scaling factor.\n📄 [Inner-IoU: More Effective Intersection over Union Loss with Auxiliary Bounding Box](https://arxiv.org/abs/2311.02877) — Zhang et al.\n\n## Generative Models — GANs\n\n**Minimax / Original GAN Loss** (2014) — Discriminator maximizes, generator minimizes binary cross-entropy in a two-player minimax game.\n📄 [Generative Adversarial Nets](https://arxiv.org/abs/1406.2661) — Goodfellow et al.\n\n**Non-Saturating GAN Loss** (2014) — Generator maximizes log(D(G(z))) instead of minimizing log(1−D(G(z))), providing stronger early gradients.\n📄 [Generative Adversarial Nets](https://arxiv.org/abs/1406.2661) — Goodfellow et al.\n\n**Feature Matching Loss** (2016) — Generator matches expected feature statistics at an intermediate discriminator layer.\n📄 [Improved Techniques for Training GANs](https://arxiv.org/abs/1606.03498) — Salimans et al.\n\n**Least Squares GAN Loss (LSGAN)** (2017) — L2 objective minimizing Pearson χ² divergence for more stable training.\n📄 [Least Squares Generative Adversarial Networks](https://arxiv.org/abs/1611.04076) — Mao et al.\n\n**Wasserstein Loss (WGAN)** (2017) — Earth Mover's distance providing meaningful gradients even for non-overlapping distributions.\n📄 [Wasserstein GAN](https://arxiv.org/abs/1701.07875) — Arjovsky, M. et al.\n💻 [martinarjovsky/WassersteinGAN](https://github.com/martinarjovsky/WassersteinGAN)\n\n**WGAN-GP** (2017) — Gradient penalty replacing weight clipping for better Lipschitz constraint enforcement.\n📄 [Improved Training of Wasserstein GANs](https://arxiv.org/abs/1704.00028) — Gulrajani et al.\n\n**Hinge Loss GAN** (2017) — Max-margin formulation with bounded gradients; used in BigGAN, SAGAN.\n📄 [Geometric GAN](https://arxiv.org/abs/1705.02894) — Lim \u0026 Ye\n📄 [Spectral Normalization for GANs](https://arxiv.org/abs/1802.05957) — Miyato et al.\n\n**Spectral Normalization** (2018) — Constrains spectral norm of weight matrices to stabilize discriminator training.\n📄 [Spectral Normalization for Generative Adversarial Networks](https://arxiv.org/abs/1802.05957) — Miyato et al.\n\n**R1 Regularization** (2018) — Zero-centered gradient penalty on real data for local convergence guarantees.\n📄 [Which Training Methods for GANs do actually Converge?](https://arxiv.org/abs/1801.04406) — Mescheder et al.\n💻 [NVlabs/stylegan2-ada-pytorch](https://github.com/NVlabs/stylegan2-ada-pytorch)\n\n**Relativistic GAN Loss (RaGAN)** (2018) — Discriminator estimates probability that real data is more realistic than fake.\n📄 [The Relativistic Discriminator](https://arxiv.org/abs/1807.00734) — Jolicoeur-Martineau, A.\n\n**Mode Seeking Loss** (2019) — Maximizes image/latent distance ratio to encourage diverse mode exploration.\n📄 [Mode Seeking Generative Adversarial Networks for Diverse Image Synthesis](https://arxiv.org/abs/1903.05628) — Mao et al.\n\n**Path Length Regularization** (2020) — Consistent Jacobian norm across latent space for smooth interpolations.\n📄 [Analyzing and Improving the Image Quality of StyleGAN](https://arxiv.org/abs/1912.04958) — Karras et al.\n💻 [NVlabs/stylegan2-ada-pytorch](https://github.com/NVlabs/stylegan2-ada-pytorch)\n\n**LeCam Regularization** (2021) — LeCam divergence-based stabilization under limited data.\n📄 [Regularizing Generative Adversarial Networks under Limited Data](https://arxiv.org/abs/2104.03310) — Tseng et al.\n💻 [google/lecam-gan](https://github.com/google/lecam-gan)\n\n**Projected GAN Loss** (2021) — Multi-scale discrimination in projected feature space from pretrained networks.\n📄 [Projected GANs Converge Faster](https://arxiv.org/abs/2111.01007) — Sauer et al.\n\n## Generative Models — VAEs\n\n**ELBO / VAE Loss** (2013) — Reconstruction loss + KL divergence regularizer pushing posterior toward prior.\n📄 [Auto-Encoding Variational Bayes](https://arxiv.org/abs/1312.6114) — Kingma, D.P. \u0026 Welling, M.\n💻 [AntixK/PyTorch-VAE](https://github.com/AntixK/PyTorch-VAE)\n\n**β-VAE Loss** (2017) — Upweights KL divergence (β \u003e 1) for more disentangled latent representations.\n📄 [β-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework](https://openreview.net/forum?id=Sy2fzU9gl) — Higgins et al.\n\n**VQ-VAE Loss** (2017) — Reconstruction + vector quantization commitment loss + codebook loss for discrete latents.\n📄 [Neural Discrete Representation Learning](https://arxiv.org/abs/1711.00937) — van den Oord et al.\n\n**WAE Loss** (2018) — Penalized Wasserstein distance using MMD or adversarial regularization on latent space.\n📄 [Wasserstein Auto-Encoders](https://arxiv.org/abs/1711.01558) — Tolstikhin et al.\n\n## Generative Models — Diffusion \u0026 Flow\n\n**Denoising Score Matching** (2011) — Training a denoising autoencoder equals matching the score function of noise-perturbed data.\n📄 [A Connection Between Score Matching and Denoising Autoencoders](https://doi.org/10.1162/NECO_a_00142) — Vincent, P.\n\n**Score Matching with Langevin Dynamics (NCSN)** (2019) — Noise-conditional score network across multiple noise scales with annealed Langevin sampling.\n📄 [Generative Modeling by Estimating Gradients of the Data Distribution](https://arxiv.org/abs/1907.05600) — Song, Y. \u0026 Ermon, S.\n\n**DDPM Loss** (2020) — Simplified variational bound: predict the noise added at each diffusion step via weighted MSE.\n📄 [Denoising Diffusion Probabilistic Models](https://arxiv.org/abs/2006.11239) — Ho, J. et al.\n\n**Variational Diffusion Loss** (2021) — Continuous-time variational lower bound with learnable noise schedule.\n📄 [Variational Diffusion Models](https://arxiv.org/abs/2107.00630) — Kingma et al.\n\n**v-prediction Loss** (2022) — Predicts velocity v = α·ε − σ·x for improved numerical stability and progressive distillation.\n📄 [Progressive Distillation for Fast Sampling of Diffusion Models](https://arxiv.org/abs/2202.00512) — Salimans, T. \u0026 Ho, J.\n\n**Rectified Flow Loss** (2022) — Learns straight-line ODE trajectories between noise and data distributions.\n📄 [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://arxiv.org/abs/2209.03003) — Liu et al.\n\n**Flow Matching Loss** (2023) — Simulation-free training for continuous normalizing flows; regresses vector fields of conditional probability paths.\n📄 [Flow Matching for Generative Modeling](https://arxiv.org/abs/2210.02747) — Lipman et al.\n💻 [facebookresearch/flow_matching](https://github.com/facebookresearch/flow_matching)\n\n**Consistency Loss** (2023) — Self-consistency along the probability flow ODE for high-quality one-step generation.\n📄 [Consistency Models](https://arxiv.org/abs/2303.01469) — Song et al.\n💻 [OpenAI/consistency_models](https://github.com/openai/consistency_models)\n\n## Reconstruction \u0026 Perceptual\n\n**SSIM Loss** (2004) — Structural similarity using luminance, contrast, and structure comparisons; used as 1−SSIM.\n📄 [Image Quality Assessment: From Error Visibility to Structural Similarity](https://ieeexplore.ieee.org/document/1284395) — Wang et al.\n💻 [VainF/pytorch-msssim](https://github.com/VainF/pytorch-msssim)\n\n**Style Loss (Gram Matrix)** (2015) — Matches Gram matrices of CNN feature maps for texture/style transfer.\n📄 [A Neural Algorithm of Artistic Style](https://arxiv.org/abs/1508.06576) — Gatys et al.\n\n**Perceptual Loss / VGG Loss** (2016) — L2 distance between deep feature representations of generated and target images.\n📄 [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://arxiv.org/abs/1603.08155) — Johnson et al.\n\n**LPIPS** (2018) — Learned perceptual metric using calibrated deep features; correlates better with human perception than SSIM/PSNR.\n📄 [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://arxiv.org/abs/1801.03924) — Zhang et al.\n💻 [richzhang/PerceptualSimilarity](https://github.com/richzhang/PerceptualSimilarity)\n\n## Image Super-Resolution \u0026 Restoration\n\n**Charbonnier Loss** (1994) — Differentiable approximation to L1 (√(x²+ε²)); robust to outliers, smooth at zero.\n📄 [Two Deterministic Half-Quadratic Regularization Algorithms for Computed Imaging](https://ieeexplore.ieee.org/document/413553) — Charbonnier et al.\n\n**MS-SSIM Loss** (2003) — Multi-scale SSIM evaluating structural similarity across multiple resolutions.\n📄 [Multi-Scale Structural Similarity for Image Quality Assessment](https://ieeexplore.ieee.org/document/1292216) — Wang et al.\n\n**SRGAN Loss** (2017) — Adversarial loss + VGG perceptual content loss for photo-realistic 4× super-resolution.\n📄 [Photo-Realistic Single Image Super-Resolution Using a GAN](https://arxiv.org/abs/1609.04802) — Ledig et al.\n\n**Contextual Loss** (2018) — Feature-level context matching without spatial alignment; enables training with non-aligned data.\n📄 [The Contextual Loss for Image Transformation with Non-Aligned Data](https://arxiv.org/abs/1803.02077) — Mechrez et al.\n\n**ESRGAN Loss** (2018) — Relativistic average discriminator + pre-activation VGG perceptual loss for enhanced texture recovery.\n📄 [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://arxiv.org/abs/1809.00219) — Wang et al.\n💻 [xinntao/ESRGAN](https://github.com/xinntao/ESRGAN)\n\n**Focal Frequency Loss** (2021) — Adaptively focuses on hard-to-synthesize frequencies in the Fourier domain.\n📄 [Focal Frequency Loss for Image Reconstruction and Synthesis](https://arxiv.org/abs/2012.12821) — Jiang et al.\n💻 [EndlessSora/focal-frequency-loss](https://github.com/EndlessSora/focal-frequency-loss)\n\n## Contrastive \u0026 Self-Supervised Learning\n\n**Contrastive Loss** (2005) — Pairwise loss pulling similar pairs together and pushing dissimilar pairs apart by a margin.\n📄 [Learning a Similarity Metric Discriminatively, with Application to Face Verification](https://www.researchgate.net/publication/4156225) — Chopra, Hadsell, LeCun\n\n**N-pair Loss** (2016) — Generalizes triplet loss by simultaneously pushing away negatives from N−1 classes.\n📄 [Improved Deep Metric Learning with Multi-class N-pair Loss Objective](https://papers.nips.cc/paper/6200) — Sohn, K.\n\n**InfoNCE / CPC Loss** (2018) — Noise-contrastive estimation maximizing mutual information between latent representations.\n📄 [Representation Learning with Contrastive Predictive Coding](https://arxiv.org/abs/1807.03748) — van den Oord et al.\n💻 [RElbers/info-nce-pytorch](https://github.com/RElbers/info-nce-pytorch)\n\n**MoCo Loss** (2020) — InfoNCE with momentum-updated encoder and dynamic dictionary queue.\n📄 [Momentum Contrast for Unsupervised Visual Representation Learning](https://arxiv.org/abs/1911.05722) — He et al.\n💻 [facebookresearch/moco](https://github.com/facebookresearch/moco)\n\n**NT-Xent / SimCLR Loss** (2020) — Normalized temperature-scaled cross-entropy over cosine similarities of augmented pairs.\n📄 [A Simple Framework for Contrastive Learning of Visual Representations](https://arxiv.org/abs/2002.05709) — Chen et al.\n\n**BYOL Loss** (2020) — MSE between L2-normalized predictions and targets; learns without negative pairs via momentum teacher.\n📄 [Bootstrap Your Own Latent](https://arxiv.org/abs/2006.07733) — Grill et al.\n\n**SwAV Loss** (2020) — Swapped prediction contrasting cluster assignments from different augmented views.\n📄 [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://arxiv.org/abs/2006.09882) — Caron et al.\n💻 [facebookresearch/swav](https://github.com/facebookresearch/swav)\n\n**Supervised Contrastive Loss (SupCon)** (2020) — Extends self-supervised contrastive loss with label information to pull same-class embeddings together.\n📄 [Supervised Contrastive Learning](https://arxiv.org/abs/2004.11362) — Khosla et al.\n💻 [HobbitLong/SupContrast](https://github.com/HobbitLong/SupContrast)\n\n**Barlow Twins Loss** (2021) — Cross-correlation matrix close to identity; reduces redundancy between embedding dimensions.\n📄 [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://arxiv.org/abs/2103.03230) — Zbontar et al.\n💻 [facebookresearch/barlowtwins](https://github.com/facebookresearch/barlowtwins)\n\n**DINO Loss** (2021) — Self-distillation via cross-entropy between sharpened softmax outputs of student and momentum-teacher.\n📄 [Emerging Properties in Self-Supervised Vision Transformers](https://arxiv.org/abs/2104.14294) — Caron et al.\n💻 [facebookresearch/dino](https://github.com/facebookresearch/dino)\n\n**SimSiam Loss** (2021) — Negative cosine similarity with stop-gradient; no negatives, momentum, or large batches needed.\n📄 [Exploring Simple Siamese Representation Learning](https://arxiv.org/abs/2011.10566) — Chen \u0026 He\n\n**CLIP Loss** (2021) — Symmetric cross-entropy over image-text cosine similarities aligning visual and language representations.\n📄 [Learning Transferable Visual Models From Natural Language Supervision](https://arxiv.org/abs/2103.00020) — Radford et al.\n💻 [mlfoundations/open_clip](https://github.com/mlfoundations/open_clip)\n\n**VICReg Loss** (2022) — Variance + invariance + covariance regularization preventing collapse without negatives.\n📄 [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://arxiv.org/abs/2105.04906) — Bardes et al.\n💻 [facebookresearch/vicreg](https://github.com/facebookresearch/vicreg)\n\n**Decoupled Contrastive Loss** (2022) — Removes positive term from InfoNCE denominator, eliminating negative-positive coupling.\n📄 [Decoupled Contrastive Learning](https://arxiv.org/abs/2110.06848) — Yeh et al.\n\n**DINOv2 Loss** (2023) — DINO self-distillation + iBOT masked image modeling + Sinkhorn centering at scale.\n📄 [DINOv2: Learning Robust Visual Features without Supervision](https://arxiv.org/abs/2304.07193) — Oquab et al.\n💻 [facebookresearch/dinov2](https://github.com/facebookresearch/dinov2)\n\n**SigLIP Loss** (2023) — Pairwise sigmoid loss replacing softmax for efficient batch-parallel language-image pre-training.\n📄 [Sigmoid Loss for Language Image Pre-Training](https://arxiv.org/abs/2303.15343) — Zhai et al.\n\n## Metric Learning \u0026 Face Recognition\n\n**Triplet Loss** (2015) — Minimizes anchor-positive distance while maximizing anchor-negative distance by a margin.\n📄 [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://arxiv.org/abs/1503.03832) — Schroff et al.\n💻 [KevinMusgrave/pytorch-metric-learning](https://github.com/KevinMusgrave/pytorch-metric-learning)\n\n**Lifted Structured Loss** (2016) — Mines all positive and negative pairs in a batch simultaneously.\n📄 [Deep Metric Learning via Lifted Structured Feature Embedding](https://arxiv.org/abs/1511.06452) — Oh Song et al.\n\n**SphereFace / A-Softmax** (2017) — Multiplicative angular margin on a hypersphere for discriminative face features.\n📄 [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://arxiv.org/abs/1704.08063) — Liu et al.\n\n**Proxy-NCA Loss** (2017) — Data-to-proxy comparisons with one learnable proxy per class; dramatically faster convergence.\n📄 [No Fuss Distance Metric Learning Using Proxies](https://arxiv.org/abs/1703.07464) — Movshovitz-Attias et al.\n\n**CosFace / LMCL** (2018) — Cosine margin penalty on target logit in normalized softmax.\n📄 [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://arxiv.org/abs/1801.09414) — Wang et al.\n💻 [deepinsight/insightface](https://github.com/deepinsight/insightface)\n\n**ArcFace** (2019) — Additive angular margin with clear geodesic distance interpretation.\n📄 [ArcFace: Additive Angular Margin Loss for Deep Face Recognition](https://arxiv.org/abs/1801.07698) — Deng et al.\n💻 [deepinsight/insightface](https://github.com/deepinsight/insightface)\n\n**Multi-Similarity Loss** (2019) — Mines and weights pairs using self-similarity, relative similarity, and negative similarity.\n📄 [Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning](https://arxiv.org/abs/1904.06627) — Wang et al.\n\n**SoftTriple Loss** (2019) — Multiple centers per class bridging proxy-based and triplet-based losses.\n📄 [SoftTriple Loss: Deep Metric Learning Without Triplet Sampling](https://arxiv.org/abs/1909.05235) — Qian et al.\n\n**Circle Loss** (2020) — Unified pair similarity optimization with self-paced weighting.\n📄 [Circle Loss: A Unified Perspective of Pair Similarity Optimization](https://arxiv.org/abs/2002.10857) — Sun et al.\n\n**Proxy Anchor Loss** (2020) — Proxies as anchors associated with all batch data; fast convergence.\n📄 [Proxy Anchor Loss for Deep Metric Learning](https://arxiv.org/abs/2003.13911) — Kim et al.\n💻 [tjddus9597/Proxy-Anchor-CVPR2020](https://github.com/tjddus9597/Proxy-Anchor-CVPR2020)\n\n**Sub-center ArcFace** (2020) — Multiple sub-centers per class for noisy label handling.\n📄 [Sub-center ArcFace: Boosting Face Recognition by Large-Scale Noisy Web Faces](https://arxiv.org/abs/1801.07698) — Deng et al.\n\n**AdaFace** (2022) — Adaptive margin emphasizing hard or easy samples based on image quality.\n📄 [AdaFace: Quality Adaptive Margin for Face Recognition](https://arxiv.org/abs/2204.00964) — Kim et al.\n💻 [mk-minchul/AdaFace](https://github.com/mk-minchul/AdaFace)\n\n**ElasticFace** (2022) — Random margin values from a normal distribution each iteration for flexible separability.\n📄 [ElasticFace: Elastic Margin Loss for Deep Face Recognition](https://arxiv.org/abs/2109.09416) — Boutros et al.\n💻 [fdbtrs/ElasticFace](https://github.com/fdbtrs/ElasticFace)\n\n## NLP \u0026 Language Modeling\n\n**Cross-Entropy / Next Token Prediction** — Standard autoregressive LM loss; foundation of GPT and all causal LMs.\n📄 [Language Models are Unsupervised Multitask Learners](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) — Radford et al. (GPT-2, 2019)\n\n**Masked Language Model (MLM) Loss** (2019) — Masks 15% of tokens and predicts from bidirectional context. Introduced pre-train/fine-tune for NLU.\n📄 [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805) — Devlin et al.\n\n**Replaced Token Detection (RTD)** (2020) — Discriminator classifies every token as original or replaced; loss defined over all tokens for better sample efficiency.\n📄 [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators](https://arxiv.org/abs/2003.10555) — Clark et al.\n\n**Sentence Order Prediction (SOP)** (2020) — Predicts whether two consecutive segments are in correct or swapped order.\n📄 [ALBERT: A Lite BERT for Self-supervised Learning](https://arxiv.org/abs/1909.11942) — Lan et al.\n\n**Span Corruption Loss** (2020) — Masks contiguous spans; encoder-decoder reconstructs only missing spans. All NLP tasks as text-to-text.\n📄 [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/abs/1910.10683) — Raffel et al.\n\n**Mixture of Denoisers (MoD)** (2022) — Unifies causal LM, prefix LM, and span corruption into a single pre-training objective.\n📄 [UL2: Unifying Language Learning Paradigms](https://arxiv.org/abs/2205.05131) — Tay et al.\n\n## LLM Alignment (RLHF / DPO)\n\n**PPO Loss / RLHF** (2017/2022) — Clipped surrogate objective for aligning LLMs with human preferences via a learned reward model.\n📄 [Proximal Policy Optimization Algorithms](https://arxiv.org/abs/1707.06347) — Schulman et al.\n📄 [Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155) — Ouyang et al.\n💻 [huggingface/trl](https://github.com/huggingface/trl)\n\n**Reward Model Loss / Bradley-Terry** (2022) — Cross-entropy on pairwise human preferences for training scalar reward models.\n📄 [Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155) — Ouyang et al.\n\n**SLiC-HF Loss** (2023) — Contrastive ranking loss calibrating sequence likelihoods to human preferences.\n📄 [SLiC-HF: Sequence Likelihood Calibration with Human Feedback](https://arxiv.org/abs/2305.10425) — Zhao et al.\n\n**DPO Loss** (2023) — Closed-form policy optimization directly from preference pairs; no separate reward model or RL loop.\n📄 [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://arxiv.org/abs/2305.18290) — Rafailov et al.\n💻 [huggingface/trl — DPOTrainer](https://github.com/huggingface/trl)\n\n**IPO Loss** (2023) — Squared loss on preference margins avoiding overfitting to Bradley-Terry assumption.\n📄 [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://arxiv.org/abs/2310.12036) — Azar et al.\n\n**CPO Loss** (2024) — Contrastive preference loss without reference model for machine translation.\n📄 [Contrastive Preference Optimization](https://arxiv.org/abs/2401.08417) — Xu et al.\n\n**KTO Loss** (2024) — Kahneman-Tversky prospect theory applied to alignment; works from binary (good/bad) feedback.\n📄 [KTO: Model Alignment as Prospect Theoretic Optimization](https://arxiv.org/abs/2402.01306) — Ethayarajh et al.\n💻 [huggingface/trl — KTOTrainer](https://github.com/huggingface/trl)\n\n**GRPO Loss** (2024) — Group Relative Policy Optimization; estimates advantages from sampled output groups, eliminating the critic model.\n📄 [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://arxiv.org/abs/2402.03300) — Shao et al.\n\n**ORPO Loss** (2024) — Odds-ratio penalty added to SFT loss; combines instruction tuning and preference alignment in one stage.\n📄 [ORPO: Monolithic Preference Optimization without Reference Model](https://arxiv.org/abs/2403.07691) — Hong et al.\n💻 [huggingface/trl — ORPOTrainer](https://github.com/huggingface/trl)\n\n**SimPO Loss** (2024) — Reference-free preference optimization using length-normalized average log probability as implicit reward.\n📄 [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://arxiv.org/abs/2405.14734) — Meng et al.\n💻 [princeton-nlp/SimPO](https://github.com/princeton-nlp/SimPO)\n\n**SPPO Loss** (2024) — Self-play preference optimization framing alignment as a two-player constant-sum game.\n📄 [Self-Play Preference Optimization for Language Model Alignment](https://arxiv.org/abs/2405.00675) — Wu et al.\n\n## Sequence-to-Sequence \u0026 Speech\n\n**CTC Loss** (2006) — Marginalizes over all valid alignments between input and output sequences; foundational for ASR.\n📄 [Connectionist Temporal Classification](https://www.cs.toronto.edu/~graves/icml_2006.pdf) — Graves et al.\n💻 [`torch.nn.CTCLoss`](https://pytorch.org/docs/stable/generated/torch.nn.CTCLoss.html)\n\n**RNN-T Loss** (2012) — Extends CTC with a prediction network conditioning on previous outputs for streaming transduction.\n📄 [Sequence Transduction with Recurrent Neural Networks](https://arxiv.org/abs/1211.3711) — Graves, A.\n💻 [`torchaudio.transforms.RNNTLoss`](https://pytorch.org/audio/stable/generated/torchaudio.transforms.RNNTLoss.html)\n\n**Scheduled Sampling Loss** (2015) — Gradually replaces ground-truth tokens with model predictions during training to mitigate exposure bias.\n📄 [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://arxiv.org/abs/1506.03099) — Bengio et al.\n\n**Sequence-Level Training / MIXER** (2016) — Directly optimizes BLEU/ROUGE using REINFORCE.\n📄 [Sequence Level Training with Recurrent Neural Networks](https://arxiv.org/abs/1511.06732) — Ranzato et al.\n\n**Minimum Risk Training** (2016) — Minimizes expected task-level loss (e.g., 1−BLEU) via sampling.\n📄 [Minimum Risk Training for Neural Machine Translation](https://arxiv.org/abs/1512.02433) — Shen et al.\n\n**Mel-Spectrogram Reconstruction Loss** (2017) — L1/L2 between predicted and target mel-spectrograms; primary TTS training objective.\n📄 [Tacotron: Towards End-to-End Speech Synthesis](https://arxiv.org/abs/1703.10135) — Wang et al.\n\n**Multi-Resolution STFT Loss** (2020) — Spectral convergence + log-magnitude STFT at multiple FFT sizes for neural vocoder training.\n📄 [Parallel WaveGAN](https://arxiv.org/abs/1910.11480) — Yamamoto et al.\n💻 [csteinmetz1/auraloss](https://github.com/csteinmetz1/auraloss)\n\n## Reinforcement Learning\n\n**TD Loss / Temporal Difference** (1988) — Bootstrapped value estimation updating predictions toward reward + discounted next-state value.\n📄 [Learning to Predict by the Methods of Temporal Differences](https://link.springer.com/article/10.1007/BF00115009) — Sutton, R.S.\n\n**Q-Learning Loss** (1989) — Off-policy TD control bootstrapping with max Q-value over next actions.\n📄 [Learning from Delayed Rewards](https://www.cs.rhul.ac.uk/~chrisw/new_thesis.pdf) — Watkins, C.J.C.H.\n\n**REINFORCE / Policy Gradient** (1992) — Monte Carlo policy gradient weighted by returns.\n📄 [Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning](https://link.springer.com/article/10.1007/BF00992696) — Williams, R.J.\n\n**DQN Loss** (2015) — Q-learning with deep networks, experience replay, and target networks.\n📄 [Human-level Control through Deep Reinforcement Learning](https://web.stanford.edu/class/psych209/Readings/MnihEtAlHassibis15NatureControlDeepRL.pdf) — Mnih et al.\n💻 [DLR-RM/stable-baselines3](https://github.com/DLR-RM/stable-baselines3)\n\n**Double DQN Loss** (2015) — Decouples action selection from evaluation to reduce overestimation bias.\n📄 [Deep Reinforcement Learning with Double Q-learning](https://arxiv.org/abs/1509.06461) — van Hasselt et al.\n\n**DDPG Loss** (2015) — Deterministic policy gradients for continuous control with experience replay.\n📄 [Continuous Control with Deep Reinforcement Learning](https://arxiv.org/abs/1509.02971) — Lillicrap et al.\n\n**GAE** (2015) — Exponentially-weighted multi-step TD errors for tunable bias-variance tradeoff.\n📄 [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://arxiv.org/abs/1506.02438) — Schulman et al.\n\n**A3C / A2C Loss** (2016) — Actor-critic with policy gradient + value function baseline + entropy bonus.\n📄 [Asynchronous Methods for Deep Reinforcement Learning](https://arxiv.org/abs/1602.01783) — Mnih et al.\n\n**Distributional RL / C51 Loss** (2017) — Models full return distribution using categorical projection over fixed atoms.\n📄 [A Distributional Perspective on Reinforcement Learning](https://arxiv.org/abs/1707.06887) — Bellemare et al.\n\n**PPO Clipped Surrogate Loss** (2017) — Clips probability ratio to prevent destructively large policy updates.\n📄 [Proximal Policy Optimization Algorithms](https://arxiv.org/abs/1707.06347) — Schulman et al.\n💻 [DLR-RM/stable-baselines3](https://github.com/DLR-RM/stable-baselines3)\n\n**HER Loss** (2017) — Relabels failed trajectories with achieved goals for sample-efficient sparse-reward learning.\n📄 [Hindsight Experience Replay](https://arxiv.org/abs/1707.01495) — Andrychowicz et al.\n\n**QR-DQN Loss** (2018) — Quantile regression approximating the return distribution with learnable quantile locations.\n📄 [Distributional Reinforcement Learning with Quantile Regression](https://arxiv.org/abs/1710.10044) — Dabney et al.\n\n**SAC Loss** (2018) — Maximum entropy actor-critic balancing exploration and exploitation automatically.\n📄 [Soft Actor-Critic](https://arxiv.org/abs/1801.01290) — Haarnoja et al.\n\n**TD3 Loss** (2018) — Clipped double-Q learning + delayed policy updates + target policy smoothing.\n📄 [Addressing Function Approximation Error in Actor-Critic Methods](https://arxiv.org/abs/1802.09477) — Fujimoto et al.\n\n**V-trace Loss** (2018) — Importance-weighted off-policy correction for scalable distributed RL (IMPALA).\n📄 [IMPALA: Scalable Distributed Deep-RL](https://arxiv.org/abs/1802.01561) — Espeholt et al.\n\n**Decision Transformer Loss** (2021) — RL as sequence modeling; autoregressive transformer conditioned on returns, trained with supervised loss.\n📄 [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://arxiv.org/abs/2106.01345) — Chen et al.\n💻 [kzl/decision-transformer](https://github.com/kzl/decision-transformer)\n\n## Knowledge Distillation\n\n**Knowledge Distillation / KD Loss** (2015) — Student matches softened output distribution of teacher via KL divergence at elevated temperature.\n📄 [Distilling the Knowledge in a Neural Network](https://arxiv.org/abs/1503.02531) — Hinton, Vinyals, Dean\n\n**FitNets / Hint Loss** (2015) — Student mimics intermediate feature representations of teacher.\n📄 [FitNets: Hints for Thin Deep Nets](https://arxiv.org/abs/1412.6550) — Romero et al.\n\n**Attention Transfer Loss** (2017) — Forces student to mimic spatial attention maps of teacher's intermediate layers.\n📄 [Paying More Attention to Attention](https://arxiv.org/abs/1612.03928) — Zagoruyko \u0026 Komodakis\n💻 [szagoruyko/attention-transfer](https://github.com/szagoruyko/attention-transfer)\n\n**Born-Again Networks** (2018) — Self-distillation where identical-architecture student outperforms teacher.\n📄 [Born Again Neural Networks](https://arxiv.org/abs/1805.04770) — Furlanello et al.\n\n**PKT / Probabilistic KD** (2018) — Matches probability distributions in feature space rather than raw representations.\n📄 [Learning Deep Representations with Probabilistic Knowledge Transfer](https://arxiv.org/abs/1803.10837) — Passalis \u0026 Tefas\n\n**Relational KD / RKD** (2019) — Transfers mutual relations (distances and angles) between examples.\n📄 [Relational Knowledge Distillation](https://arxiv.org/abs/1904.05068) — Park et al.\n\n**Self-Distillation Loss** (2019) — Deeper layers supervise shallower classifiers within the same network.\n📄 [Be Your Own Teacher](https://arxiv.org/abs/1905.08094) — Zhang et al.\n\n**CRD / Contrastive Representation Distillation** (2020) — Maximizes mutual information between teacher and student via contrastive objective.\n📄 [Contrastive Representation Distillation](https://arxiv.org/abs/1910.10699) — Tian et al.\n💻 [HobbitLong/RepDistiller](https://github.com/HobbitLong/RepDistiller)\n\n**ReviewKD** (2021) — Student's lower-level features guided by teacher's higher-level features through attention-based fusion.\n📄 [Distilling Knowledge via Knowledge Review](https://arxiv.org/abs/2104.09044) — Chen et al.\n💻 [dvlab-research/ReviewKD](https://github.com/dvlab-research/ReviewKD)\n\n**DKD / Decoupled KD** (2022) — Decouples KD into target-class and non-target-class components for independent weighting.\n📄 [Decoupled Knowledge Distillation](https://arxiv.org/abs/2203.08679) — Zhao et al.\n💻 [megvii-research/mdistiller](https://github.com/megvii-research/mdistiller)\n\n**DIST Loss** (2022) — Preserves inter-class relations and intra-class ranking rather than exact probability matching.\n📄 [Knowledge Distillation from A Stronger Teacher](https://arxiv.org/abs/2205.10536) — Huang et al.\n💻 [hunto/DIST_KD](https://github.com/hunto/DIST_KD)\n\n## Regularization\n\n**KL Divergence** (1951) — Measures information lost when approximating one distribution with another.\n📄 [On Information and Sufficiency](https://doi.org/10.1214/aoms/1177729694) — Kullback \u0026 Leibler\n\n**L2 Regularization / Weight Decay** (1970) — Penalizes sum of squared weights to prevent overfitting.\n📄 [Ridge Regression](https://doi.org/10.1080/00401706.1970.10488634) — Hoerl \u0026 Kennard\n\n**L1 Regularization / Lasso** (1996) — Penalizes sum of absolute weights, inducing sparsity.\n📄 [Regression Shrinkage and Selection via the Lasso](https://doi.org/10.1111/j.2517-6161.1996.tb02080.x) — Tibshirani, R.\n\n**Elastic Net** (2005) — Combines L1 and L2 for sparsity + grouping of correlated features.\n📄 [Regularization and Variable Selection via the Elastic Net](https://doi.org/10.1111/j.1467-9868.2005.00503.x) — Zou \u0026 Hastie\n\n**Dropout** (2014) — Randomly zeroes activations; implicit ensemble of exponentially many sub-networks.\n📄 [Dropout: A Simple Way to Prevent Neural Networks from Overfitting](https://jmlr.org/papers/v15/srivastava14a.html) — Srivastava et al.\n\n**Confidence Penalty** (2017) — Penalizes low-entropy (overconfident) output distributions.\n📄 [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://arxiv.org/abs/1701.06548) — Pereyra et al.\n\n**Mixup Loss** (2018) — Trains on convex combinations of example pairs and their labels.\n📄 [mixup: Beyond Empirical Risk Minimization](https://arxiv.org/abs/1710.09412) — Zhang et al.\n💻 [facebookresearch/mixup-cifar10](https://github.com/facebookresearch/mixup-cifar10)\n\n**Manifold Mixup** (2019) — Extends Mixup to hidden representations at random intermediate layers.\n📄 [Manifold Mixup: Better Representations by Interpolating Hidden States](https://arxiv.org/abs/1806.05236) — Verma et al.\n\n**CutMix Loss** (2019) — Cuts and pastes rectangular patches between images while mixing labels proportionally.\n📄 [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://arxiv.org/abs/1905.04899) — Yun et al.\n\n## 3D Vision \u0026 Point Clouds\n\n**Chamfer Distance** (2017) — Average nearest-neighbor distance between two point sets; fast and widely used.\n📄 [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://arxiv.org/abs/1612.00603) — Fan et al.\n💻 [facebookresearch/pytorch3d](https://github.com/facebookresearch/pytorch3d)\n\n**Earth Mover's Distance (EMD)** (2017) — Optimal transport distance with bijective matching; higher quality but more expensive than CD.\n📄 [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://arxiv.org/abs/1612.00603) — Fan et al.\n\n**Normal Consistency Loss** (2018) — Penalizes inconsistency of surface normals between adjacent mesh faces.\n📄 [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://arxiv.org/abs/1804.01654) — Wang et al.\n\n**Mesh Laplacian Smoothing Loss** (2018) — Penalizes vertex deviation from neighbor centroid to prevent self-intersections.\n📄 [Pixel2Mesh](https://arxiv.org/abs/1804.01654) — Wang et al.\n💻 [facebookresearch/pytorch3d](https://github.com/facebookresearch/pytorch3d)\n\n**SDF Loss (DeepSDF)** (2019) — Regresses signed distance values; zero level-set defines the 3D surface.\n📄 [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://arxiv.org/abs/1901.05103) — Park et al.\n💻 [facebookresearch/DeepSDF](https://github.com/facebookresearch/DeepSDF)\n\n**Occupancy Loss** (2019) — Binary CE on predicted occupancy probabilities for 3D reconstruction.\n📄 [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://arxiv.org/abs/1812.03828) — Mescheder et al.\n\n**NeRF Photometric Loss** (2020) — MSE between rendered and observed pixel colors via differentiable volume rendering.\n📄 [NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis](https://arxiv.org/abs/2003.08934) — Mildenhall et al.\n\n**3D Gaussian Splatting Loss** (2023) — L1 + D-SSIM for optimizing anisotropic 3D Gaussians for real-time radiance field rendering.\n📄 [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://arxiv.org/abs/2308.04079) — Kerbl et al.\n💻 [graphdeco-inria/gaussian-splatting](https://github.com/graphdeco-inria/gaussian-splatting)\n\n## Depth Estimation\n\n**Scale-Invariant Loss** (2014) — Log-space depth error minus mean shift; invariant to global scale ambiguity.\n📄 [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://arxiv.org/abs/1406.2283) — Eigen et al.\n\n**Berhu Loss (Reverse Huber)** (2016) — L1 for small residuals, L2 for large; robust depth regression.\n📄 [Deeper Depth Prediction with Fully Convolutional Residual Networks](https://arxiv.org/abs/1606.00373) — Laina et al.\n\n**Photometric Consistency Loss** (2017) — Self-supervised SSIM + L1 with left-right disparity consistency for monocular depth.\n📄 [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://arxiv.org/abs/1609.03677) — Godard et al.\n💻 [nianticlabs/monodepth2](https://github.com/nianticlabs/monodepth2)\n\n**Edge-Aware Smoothness Loss** (2017) — Locally smooth depth except at image edges, weighted by image gradients.\n📄 [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://arxiv.org/abs/1609.03677) — Godard et al.\n\n## Medical Imaging\n\n**Deep Supervision Loss** (2015) — Auxiliary losses at intermediate layers providing direct gradient paths.\n📄 [Deeply-Supervised Nets](https://arxiv.org/abs/1409.5185) — Lee et al.\n\n**Dice Loss** (2016) — Directly optimizes Dice coefficient for volumetric medical image segmentation.\n📄 [V-Net](https://arxiv.org/abs/1606.04797) — Milletari et al.\n\n**Generalized Dice Loss** (2017) — Per-class volume weighting for highly imbalanced multi-class segmentation.\n📄 [Generalised Dice Overlap as a Deep Learning Loss Function](https://arxiv.org/abs/1707.03237) — Sudre et al.\n\n**Tversky Loss** (2017) — Tunable FP/FN trade-off for small lesion segmentation.\n📄 [Tversky Loss Function for Image Segmentation](https://arxiv.org/abs/1706.05721) — Salehi et al.\n\n**Attention-Gated Loss** (2018) — Learned attention gates suppress irrelevant regions in skip connections.\n📄 [Attention U-Net: Learning Where to Look for the Pancreas](https://arxiv.org/abs/1804.03999) — Oktay et al.\n\n**Boundary / Surface Loss** (2019) — Distance metric on contour space for highly unbalanced medical segmentation.\n📄 [Boundary Loss for Highly Unbalanced Segmentation](https://arxiv.org/abs/1812.07032) — Kervadec et al.\n💻 [LIVIAETS/boundary-loss](https://github.com/LIVIAETS/boundary-loss)\n\n**Distance Map Penalized CE** (2019) — Weights CE by distance transform maps to focus on boundary regions.\n📄 [Distance Map Loss Penalty Term for Semantic Segmentation](https://arxiv.org/abs/1908.03679) — Caliva et al.\n\n## Graph Neural Networks\n\n**Variational Graph Auto-Encoder (VGAE) Loss** (2016) — Reconstruction BCE on adjacency matrix + KL divergence for unsupervised graph learning.\n📄 [Variational Graph Auto-Encoders](https://arxiv.org/abs/1611.07308) — Kipf \u0026 Welling\n\n**Node Classification Loss** (2017) — Standard cross-entropy per-node in semi-supervised graph settings.\n📄 [Semi-Supervised Classification with Graph Convolutional Networks](https://arxiv.org/abs/1609.02907) — Kipf \u0026 Welling\n💻 [pyg-team/pytorch_geometric](https://github.com/pyg-team/pytorch_geometric)\n\n**Deep Graph Infomax (DGI) Loss** (2019) — Maximizes mutual information between local node and global graph representations.\n📄 [Deep Graph Infomax](https://arxiv.org/abs/1809.10341) — Veličković et al.\n💻 [PetarV-/DGI](https://github.com/PetarV-/DGI)\n\n**Graph Matching Loss** (2019) — Attention-based cross-graph matching with margin-based pairwise loss.\n📄 [Graph Matching Networks for Learning the Similarity of Graph Structured Objects](https://arxiv.org/abs/1904.12787) — Li et al.\n\n**InfoGraph Loss** (2020) — Maximizes mutual information between graph-level and substructure-level representations.\n📄 [InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning](https://arxiv.org/abs/1908.01000) — Sun et al.\n💻 [sunfanyunn/InfoGraph](https://github.com/sunfanyunn/InfoGraph)\n\n**GraphCL Loss** (2020) — NT-Xent contrastive loss on augmented graph views for self-supervised graph learning.\n📄 [Graph Contrastive Learning with Augmentations](https://arxiv.org/abs/2010.13902) — You et al.\n💻 [Shen-Lab/GraphCL](https://github.com/Shen-Lab/GraphCL)\n\n**BGRL Loss** (2022) — Negative-sample-free self-supervised loss bootstrapping graph representations (inspired by BYOL).\n📄 [Large-Scale Representation Learning on Graphs via Bootstrapping](https://arxiv.org/abs/2102.06514) — Thakoor et al.\n💻 [nerdslab/bgrl](https://github.com/nerdslab/bgrl)\n\n## Recommendation Systems\n\n**ListNet Loss** (2007) — Listwise learning-to-rank using top-one probability distributions.\n📄 [Learning to Rank: From Pairwise Approach to Listwise Approach](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr-2007-40.pdf) — Cao et al.\n\n**ListMLE Loss** (2008) — Listwise loss based on likelihood of ground-truth permutation under Plackett-Luce model.\n📄 [Listwise Approach to Learning to Rank: Theory and Algorithm](https://dl.acm.org/doi/10.1145/1390156.1390306) — Xia et al.\n\n**BPR Loss** (2009) — Pairwise loss maximizing posterior probability that user prefers observed over unobserved items.\n📄 [BPR: Bayesian Personalized Ranking from Implicit Feedback](https://arxiv.org/abs/1205.2618) — Rendle et al.\n💻 [guoyang9/BPR-pytorch](https://github.com/guoyang9/BPR-pytorch)\n\n**Sampled Softmax Loss** (2015) — Approximates full softmax over large item vocabulary by sampling negatives.\n📄 [On Using Very Large Target Vocabulary for Neural Machine Translation](https://arxiv.org/abs/1412.2007) — Jean et al.\n\n**DirectAU Loss** (2022) — Directly optimizes alignment and uniformity on the hypersphere for collaborative filtering.\n📄 [Towards Representation Alignment and Uniformity in Collaborative Filtering](https://arxiv.org/abs/2206.12811) — Wang et al.\n💻 [THUwangcy/DirectAU](https://github.com/THUwangcy/DirectAU)\n\n## Multi-Task Learning\n\n**Uncertainty Weighting / Homoscedastic Uncertainty** (2018) — Learns task weights by modeling task-dependent uncertainty; noisy tasks auto-downweighted.\n📄 [Multi-Task Learning Using Uncertainty to Weigh Losses](https://arxiv.org/abs/1705.07115) — Kendall et al.\n💻 [median-research-group/LibMTL](https://github.com/median-research-group/LibMTL)\n\n**GradNorm** (2018) — Dynamically normalizes gradient magnitudes across tasks to balance training rates.\n📄 [GradNorm: Gradient Normalization for Adaptive Loss Balancing](https://arxiv.org/abs/1711.02257) — Chen et al.\n\n**MGDA** (2018) — Multi-objective optimization finding Pareto-optimal descent direction via Frank-Wolfe on task gradients.\n📄 [Multi-Task Learning as Multi-Objective Optimization](https://arxiv.org/abs/1810.04650) — Sener \u0026 Koltun\n\n**PCGrad** (2020) — Projects conflicting task gradients onto normal planes to reduce destructive interference.\n📄 [Gradient Surgery for Multi-Task Learning](https://arxiv.org/abs/2001.06782) — Yu et al.\n\n**CAGrad** (2021) — Minimizes average loss while maximizing worst-case local improvement across tasks.\n📄 [Conflict-Averse Gradient Descent for Multi-task Learning](https://arxiv.org/abs/2110.14048) — Liu et al.\n\n**Nash-MTL** (2022) — Nash bargaining game where tasks negotiate a joint update direction.\n📄 [Multi-Task Learning as a Bargaining Game](https://arxiv.org/abs/2202.01017) — Navon et al.\n💻 [AvivNavon/nash-mtl](https://github.com/AvivNavon/nash-mtl)\n\n## Uncertainty Estimation\n\n**NLL with Learned Variance** (1994) — Network predicts mean and variance; NLL naturally trades off accuracy and calibration.\n📄 [Estimating the Mean and Variance of the Target Probability Distribution](https://ieeexplore.ieee.org/document/374138) — Nix \u0026 Weigend\n\n**MC Dropout** (2016) — Dropout at test time as approximate Bayesian inference for uncertainty estimation.\n📄 [Dropout as a Bayesian Approximation](https://arxiv.org/abs/1506.02142) — Gal \u0026 Ghahramani\n\n**Deep Ensembles Loss** (2017) — Ensemble of networks with proper scoring rules + adversarial training for diversity.\n📄 [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://arxiv.org/abs/1612.01474) — Lakshminarayanan et al.\n\n**Evidential Deep Learning Loss** (2018) — Dirichlet prior over class probabilities; Bayes risk + KL divergence regularizer.\n📄 [Evidential Deep Learning to Quantify Classification Uncertainty](https://arxiv.org/abs/1806.01768) — Sensoy et al.\n\n## Domain Adaptation\n\n**Maximum Mean Discrepancy (MMD)** (2012) — Distribution distance in RKHS; aligns source and target features without adversarial training.\n📄 [A Kernel Two-Sample Test](https://www.jmlr.org/papers/v13/gretton12a/gretton12a.pdf) — Gretton et al.\n💻 [ZongxianLee/MMD_Loss.Pytorch](https://github.com/ZongxianLee/MMD_Loss.Pytorch)\n\n**Domain Adversarial Loss / DANN** (2016) — Gradient reversal layer training domain classifier adversarially for domain-invariant features.\n📄 [Domain-Adversarial Training of Neural Networks](https://arxiv.org/abs/1505.07818) — Ganin et al.\n💻 [fungtion/DANN](https://github.com/fungtion/DANN)\n\n**Deep CORAL Loss** (2016) — Aligns second-order statistics (covariance) of source and target deep features.\n📄 [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://arxiv.org/abs/1607.01719) — Sun \u0026 Saenko\n\n**Wasserstein Distance for DA** (2018) — Earth Mover's Distance as domain discrepancy measure with gradient penalty.\n📄 [Wasserstein Distance Guided Representation Learning for Domain Adaptation](https://arxiv.org/abs/1707.01217) — Shen et al.\n\n**Contrastive Domain Discrepancy (CDD)** (2019) — Class-aware alignment maximizing inter-class and minimizing intra-class discrepancy across domains.\n📄 [Contrastive Adaptation Network for Unsupervised Domain Adaptation](https://arxiv.org/abs/1901.00976) — Kang et al.\n\n---\n\n## Survey Papers\n\n- 📄 [A Comprehensive Survey of Loss Functions and Metrics in Deep Learning](https://arxiv.org/abs/2307.02694) — Terven et al. (2025)\n- 📄 [A Survey of Loss Functions for Semantic Segmentation](https://arxiv.org/abs/2006.14822) — Jadon (2020)\n- 📄 [Loss Functions in the Era of Semantic Segmentation: A Survey and Outlook](https://arxiv.org/abs/2312.05391) — Azad et al. (2023)\n\n## Key Implementation Libraries\n\n| Library | Focus | Link |\n|---------|-------|------|\n| PyTorch (built-in) | CE, BCE, MSE, Huber, CTC, KLDiv, etc. | [pytorch.org](https://pytorch.org/docs/stable/nn.html#loss-functions) |\n| pytorch-metric-learning | Triplet, Contrastive, ArcFace, ProxyNCA, etc. | [GitHub](https://github.com/KevinMusgrave/pytorch-metric-learning) |\n| SegLossOdyssey | Dice, Tversky, Boundary, Hausdorff, etc. | [GitHub](https://github.com/JunMa11/SegLossOdyssey) |\n| Hugging Face TRL | DPO, PPO, KTO, ORPO, SimPO, etc. | [GitHub](https://github.com/huggingface/trl) |\n| Stable-Baselines3 | DQN, PPO, SAC, TD3, A2C, etc. | [GitHub](https://github.com/DLR-RM/stable-baselines3) |\n| lightly | SimCLR, BYOL, MoCo, DINO, Barlow Twins, etc. | [GitHub](https://github.com/lightly-ai/lightly) |\n| insightface | ArcFace, CosFace, Sub-center ArcFace | [GitHub](https://github.com/deepinsight/insightface) |\n| open_clip | CLIP, SigLIP contrastive losses | [GitHub](https://github.com/mlfoundations/open_clip) |\n| PyTorch3D | Chamfer, mesh losses, point cloud losses | [GitHub](https://github.com/facebookresearch/pytorch3d) |\n| PyTorch Geometric | GNN losses, link prediction, node classification | [GitHub](https://github.com/pyg-team/pytorch_geometric) |\n| LibMTL | Uncertainty weighting, GradNorm, PCGrad, Nash-MTL | [GitHub](https://github.com/median-research-group/LibMTL) |\n| auraloss | Multi-Resolution STFT, mel losses | [GitHub](https://github.com/csteinmetz1/auraloss) |\n| BasicSR | Perceptual, SSIM, Charbonnier, GAN losses for SR | [GitHub](https://github.com/XPixelGroup/BasicSR) |\n| kornia | Focal, Dice, SSIM, and more | [GitHub](https://github.com/kornia/kornia) |\n| anomalib | Anomaly detection losses and methods | [GitHub](https://github.com/open-edge-platform/anomalib) |\n| Avalanche | Continual learning (EWC, SI, LwF, etc.) | [GitHub](https://github.com/ContinualAI/avalanche) |\n| GluonTS | Time series forecasting losses | [GitHub](https://github.com/awslabs/gluonts) |\n| audiocraft | Audio generation (EnCodec, MusicGen) | [GitHub](https://github.com/facebookresearch/audiocraft) |\n| AIF360 | Fairness and bias mitigation | [GitHub](https://github.com/Trusted-AI/AIF360) |\n\n---\n\n## Star History\n\nIf you find this useful, please star the repo — it helps others discover it.\n\n[![Star History Chart](https://api.star-history.com/svg?repos=stabgan/awesome-loss-functions\u0026type=Date)](https://star-history.com/#stabgan/awesome-loss-functions\u0026Date)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstabgan%2Fawesome-loss-functions","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstabgan%2Fawesome-loss-functions","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstabgan%2Fawesome-loss-functions/lists"}