{"id":29543148,"url":"https://github.com/nataliarodriguez-uc/auc-opt","last_synced_at":"2026-05-08T10:32:05.292Z","repository":{"id":302894022,"uuid":"1013869423","full_name":"nataliarodriguez-uc/auc-opt","owner":"nataliarodriguez-uc","description":"Constructing new surrogate loss for optimizing pairwise difference metrics for training contrastive learning. This repo is implementing the loss to train for AUC classification tasks.","archived":false,"fork":false,"pushed_at":"2026-02-18T20:31:31.000Z","size":1591,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-18T23:42:35.491Z","etag":null,"topics":["auc","cifar10","classification","contrastive-learning","deep-learning","julia","lagrange-interpolation","machine-learning","newtons-method","optimization","python","support-vector-machines"],"latest_commit_sha":null,"homepage":"","language":"Julia","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nataliarodriguez-uc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-07-04T15:48:54.000Z","updated_at":"2026-01-26T02:19:12.000Z","dependencies_parsed_at":"2025-07-04T17:04:19.567Z","dependency_job_id":null,"html_url":"https://github.com/nataliarodriguez-uc/auc-opt","commit_stats":null,"previous_names":["nataliarodriguez-uc/auc-opt"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/nataliarodriguez-uc/auc-opt","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nataliarodriguez-uc%2Fauc-opt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nataliarodriguez-uc%2Fauc-opt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nataliarodriguez-uc%2Fauc-opt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nataliarodriguez-uc%2Fauc-opt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nataliarodriguez-uc","download_url":"https://codeload.github.com/nataliarodriguez-uc/auc-opt/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nataliarodriguez-uc%2Fauc-opt/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32776598,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-08T08:22:46.396Z","status":"ssl_error","status_checked_at":"2026-05-08T08:22:45.650Z","response_time":54,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["auc","cifar10","classification","contrastive-learning","deep-learning","julia","lagrange-interpolation","machine-learning","newtons-method","optimization","python","support-vector-machines"],"created_at":"2025-07-17T13:03:00.090Z","updated_at":"2026-05-08T10:32:05.276Z","avatar_url":"https://github.com/nataliarodriguez-uc.png","language":"Julia","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Efficient Proximal Optimization for Non-Smooth AUC Maximization\n\n\u003e This project develops efficient training optimization algorithms for direct AUC maximization using proximal gradients and augmented Lagrangian methods.\n\n---\n\n## Overview\n\n### The Problem\n\nIn binary classification, **Area Under the ROC Curve (AUC)** is often the metric that matters especially in imbalanced settings like medical diagnosis, fraud detection, and information retrieval. However, most deep learning frameworks optimize cross-entropy loss (empirical loss functions), which doesn't directly correspond to AUC performance.\n\n**Why standard methods struggle with AUC:**\n- AUC measures ranking quality: all positive samples should score higher than negative samples\n- This requires evaluating **pairwise comparisons** between classes, not individual predictions\n- The true AUC objective uses non-differentiable indicator functions: $\\mathbb{1}(\\text{score}_{\\text{pos}} \u003e \\text{score}_{\\text{neg}})$\n- Naively computing all pairs leads to $O(n^2)$ complexity\n\n### Our Approach\n\nThis project develops **proximal optimization methods** specifically designed for direct AUC maximization. We introduce:\n\n1. **Piecewise linear surrogate loss** that approximates the indicator function while remaining tractable\n2. **Closed-form proximal operators** with γ-dependent solutions that exploit problem structure for computation efficiency\n3. **Augmented Lagrangian Method (ALM)** + **Semi-Smooth Newton (SSN)** framework for efficient solving\n4. **Controlled pairwise sampling** that reduces complexity from $O(n^2)$ to $O(n \\cdot k)$ where k is a subset of pairs per sample used\n\n**Key innovation**: Our methodology and analysis reveals that proximal operator parameters are proportional to the Augmented Lagrangian parameters ($\\sigma = 1 / \\gamma$). (Need more here...)\n\n### Connection to Contrastive Learning\n\n\n**Binary Contrastive Learning** (current implementation):\n- Pull similar samples (same class) together, push dissimilar samples (different classes) apart\n- When embeddings are 1-dimensional, this reduces to AUC optimization\n- **Validated** on CIFAR-10 binary classification\n\n**Multi-Class Extension** (theoretical framework):\n- **One-vs-Rest**: Apply binary AUC optimization for each class vs. all others\n- **Pairwise Decomposition**: Compare all class pairs $(C_i, C_j)$ separately\n- **Supervised Contrastive Loss**: Same pairwise structure—positive pairs (same class) vs. negative pairs (different classes)\n\nOur formulation **naturally handles multi-class** settings since:\n1. The pairwise difference $w^\\top(z_j - z_i)$ works for any pair of classes\n2. The proximal operators don't depend on number of classes\n3. The ALM framework scales to multiple comparison sets $S_i$\n\n---\n\n## Key Results\n\n### CIFAR-10 Binary Classification\n\n**Setup**: Binary classification on CIFAR-10 (2 classes), comparing Prox-SGD against standard baselines.\n\n| Dataset | Configuration | Prox AUC | LibAUC Baseline | Gap |\n|---------|--------------|----------|-----------------|-----|\n| Balanced | σ=1.0, 25 pairs, 10 batches | **95.27%** | 97.56% | -2.29% |\n| Imbalanced (1:9) | σ=1.0, 25 pairs, 10 batches | **97.75%** | 98.08% | -0.33% |\n\n**Observations**:\n- Method performs competitively on imbalanced data (within 0.33% of state-of-the-art)\n- Controlled sampling (50 pos + 50 neg pairs per batch) enables efficient computation\n- Room for improvement through hyperparameter tuning and sampling strategies\n\n### Synthetic SVM Experiments\n\n**Setup**: Controlled geometric configurations to isolate algorithm behavior.\n\n| Scenario | Dimensions (m×n) | Prox AUC | BCE AUC | LibAUC AUC |\n|----------|-----------------|----------|---------|------------|\n| Low separation, many samples | 1000×50 | **99.12%** | 99.88% | 99.30% |\n| High separation, few samples | 50×500 | **100%** | 100% | 0%* |\n| High separation, many samples | 1000×50 | **99.94%** | 100% | 100% |\n\n*LibAUC completely fails in high-dimensional, low-sample regime where our method achieves perfect AUC.\n\n**Key Finding**: $\\sigma = 1.0$ provides robust performance across all scenarios. Method excels when features \u003e\u003e samples, a challenging regime for standard approaches.\n\n---\n\n## Quick Start\n\n### Installation\n\n```bash\ngit clone https://github.com/nataliarodriguez-uc/auc-opt.git\ncd auc-opt\npip install -r requirements.txt\n```\n\n### Run Demo\n\ntbd \n\n### Basic Usage\n\ntbd\n\n---\n\n## Repository Structure\n\n```\nauc-opt/\n├── demos/                      # Demo notebooks and examples\n│   └── svm_example.ipynb       # SMV example notebook\n├── docs/                       # Detailed documentation\n│   ├── 1_overview.md           # Problem motivation \u0026 X-risk background\n│   ├── 2_algorithm.md          # Mathematical formulation\n│   ├── 3_optimization.md       # Proximal methods \u0026 ALM details\n│   ├── 4_experiments.md        # Experimental design \u0026 analysis\n│   └── 5_math_appendix.md      # Derivations \u0026 proofs\n├── src/                        # Core implementation\n│   ├── julia/                  # Julia optimization routines\n│   └── python/                 # Python implementation\n│       └── aucopt/             # Main package\n│           ├── data/           # Data loading utilities\n│           ├── eval/           # Evaluation metrics\n│           ├── optim/          # Optimization algorithms (ALM, SSN)\n│           └── __init__.py\n├── .gitignore\n├── README.md\n└── requirements.txt\n```\n\n---\n\n## Documentation\n\n**Methodology Details** \n- **[Overview](docs/1_overview.md)** - Problem motivation and the X-risk framework\n- **[Algorithm](docs/2_algorithm.md)** - From pairwise objectives to AUC formulation\n\n**Implementation details:**\n- **[Optimization Methods](docs/3_optimization.md)** - Proximal operators, ALM, and SSN solver details\n- **[Experiments](docs/4_experiments.md)** - Full experimental setup, results, and analysis\n- **[Math Appendix](docs/5_math_appendix.md)** - Complete derivations and proofs\n\n---\n\n## Optimization Highlights\n\n### Proximal Optimization Framework\n\n**Surrogate Loss Construction**:\n- Replace non-differentiable indicator $\\mathbb{1}(t \u003e 0)$ with piecewise linear $\\ell_\\delta(t) = \\min(1, \\max(0, t - \\delta))$\n- Retains theoretical properties (Fisher consistency) while enabling efficient computation\n\n**γ-Dependent Proximal Operators**:\n- Closed-form solutions for proximal mapping: $\\text{prox}_{\\gamma \\ell_\\delta}(x)$\n- Three regimes based on $\\gamma = 1/\\sigma$:\n  - $\\gamma \u003c 2$: Sharp transitions, stable convergence\n  - $\\gamma = 2$: Subdifferential at boundary\n  - $\\gamma \u003e 2$: Wider non-differentiable region\n- Empirical finding: $\\sigma = 1.0$ ($\\gamma = 1.0$) optimal across problem types\n\n**Augmented Lagrangian Decomposition**:\n- Introduce auxiliary variables $y_{ij} = w^\\top(z_j - z_i)$ for each pair\n- ALM framework: alternate between primal (SSN) and dual (Lagrange multiplier) updates\n- Exploits sparsity: only update pairs in active regions (not correctly classified)\n\n### Computational Efficiency\n\nRather than evaluating all $O(n^2)$ pairs:\n1. **Controlled sampling**: Fix batch size (e.g., 25 pos × 25 neg = 625 pairs)\n2. **Sparse structure**: Many pairs are correctly classified → zero gradient → skip updates\n3. **Block updates**: Only recompute active pairs each iteration\n\n**Result**: Effective complexity $O(n \\cdot k)$ where $k \\ll n$.\n\nSee [docs/3_optimization.md](docs/3_optimization.md) for complete mathematical details.\n\n---\n\n## Applications\n\nThis methodology applies to:\n\n**AUC-Based Classification** (current validation):\n- Medical diagnosis: Binary or multi-class with imbalanced classes\n- Fraud detection: Rare positive class vs. normal transactions  \n- Information retrieval: Binary or graded relevance judgments\n- Any classification problem where AUC is the target metric\n\n**Contrastive Learning** (framework supports, validation in progress):\n- Binary contrastive learning: Validated on CIFAR-10\n- Multi-class contrastive learning: Pairwise formulation naturally extends\n- Supervised contrastive loss: Same pairwise comparison structure\n\n**Ranking Objectives**:\n- Learning to rank with pairwise preferences\n- Precision@K optimization\n- Multi-class AUC via one-vs-rest or pairwise decomposition\n\n**TBD...**:\n- Large-scale self-supervised pretraining (SimCLR, MoCo scale)\n- High-dimensional embeddings beyond linear projections\n- Production deployment as PyTorch/TensorFlow loss function\n\n---\n\n## Current Status\n\n**Completed:**\n- Proximal operator derivation and implementation (γ-dependent cases)\n- ALM + SSN optimization framework\n- Synthetic SVM validation experiments\n- CIFAR-10 binary classification experiments\n- Comprehensive technical documentation\n\n**In Progress:**\n- Advanced sampling strategies (hard negative mining, curriculum sampling)\n- Hyperparameter sensitivity analysis\n- Extension to larger-scale experiments\n\n**Future Directions:**\n- Multi-class AUC via one-vs-rest\n- Integration with PyTorch as a custom loss function\n- Application to medical imaging datasets\n- Self-supervised contrastive learning formulation\n\n---\n\n## Citation\n\n```bibtex\n@software{rodriguez2026aucopt,\n  author = {Rodriguez Figueroa, Natalia A.},\n  title = {Proximal Methods for AUC Optimization},\n  year = {2026},\n  url = {https://github.com/nataliarodriguez-uc/auc-opt}\n}\n```\n\n---\n\n## References\n\n**Theoretical Foundations:**\n- Yang, T. (2023). *Algorithmic foundations of empirical X-risk minimization*\n- Khanh, P. D., Mordukhovich, B. S., \u0026 Phat, V. T. (2022). *A generalized Newton method for subgradient systems*\n- Li, X., Sun, D., \u0026 Toh, K.-C. (2018). *A highly efficient semismooth Newton augmented Lagrangian method for solving LASSO problems*\n- Tian, L., \u0026 So, A. M.-C. (2022). *Computing d-stationary points of ρ-margin loss SVM*\n  \n---\n\n## Authors\n\n**Natalia A. Rodriguez Figueroa**  \nIndustrial Engineering \u0026 Operations Research  \nUniversity of California, Berkeley  \nAdvisor: Dr. Ying Cui\n\n📧 Email: natalia_rodriguezuc@berkeley.edu  \n🔗 [GitHub Profile](https://github.com/nataliarodriguez-uc)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnataliarodriguez-uc%2Fauc-opt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnataliarodriguez-uc%2Fauc-opt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnataliarodriguez-uc%2Fauc-opt/lists"}