https://github.com/alessiochen/multi-class-logistic-regression-with-optimization-methods

Implementation of Trust Region and Gradient Descent methods for Multinomial Regression
https://github.com/alessiochen/multi-class-logistic-regression-with-optimization-methods

armijo armijo-condition cauchy-point line-search mlr multiclass-logistic-regression optimization optimization-algorithms trust-region trust-region-optimization

Last synced: 3 days ago
JSON representation

Implementation of Trust Region and Gradient Descent methods for Multinomial Regression

• Host: GitHub
• URL: https://github.com/alessiochen/multi-class-logistic-regression-with-optimization-methods
• Owner: AlessioChen
• Created: 2024-04-14T16:28:53.000Z (9 months ago)
• Default Branch: main
• Last Pushed: 2024-05-15T18:46:41.000Z (8 months ago)
• Last Synced: 2024-05-16T17:56:44.120Z (8 months ago)
• Topics: armijo, armijo-condition, cauchy-point, line-search, mlr, multiclass-logistic-regression, optimization, optimization-algorithms, trust-region, trust-region-optimization
• Language: Jupyter Notebook
• Homepage:
• Size: 3.31 MB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
#  Multi-Class Logstic Regression with Optmization Methods

This repository contains Python code for implementing multi-class logistic regression using gradient descent with Armijo line search and trust region optimization methods with cauchy step. 

## Introduction 

Multi-class logistic regression is a popular technique for classification problems where the target variable can take multiple classes. This repository provides an implementation of multi-class logistic regression along with two optimization methods:

- **Gradient Descent with Armijo Line Search**: This method optimizes the model parameters by iteratively updating them based on the gradient of the loss function, with step size determined using the Armijo line search algorithm.

- **Trust Region Algorithm:**  This method iteratively minimizes the loss function while considering a trust region around the current solution. It utilizes Taylor's second-order model and the Cauchy step to update the model parameters within the trust region.

## Implementation 

The implementation is provided in Python using NumPy for numerical computations. The main components of the implementation include:

- MultiClassLogisticRegression class: Implements multi-class logistic regression with methods for training the model using gradient descent and trust region optimization.

- Functions for computing loss, gradient, Hessian matrix, Armijo line search, and Cauchy step.

## Experimantation 

This repository includes experimentation on a benchmark of multi-class classification datasets. It measures the training loss decrease and runtime for both optimization methods. Additionally, it reports the accuracy of the final model solutions.

## Datasets

Small set of real-world dataset for multi-class classification taska are used. These dataset are directy imported from sklearn datasets or they are download from [LIBSVM](https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/) and imported in the datasets directory.