{"id":23658665,"url":"https://github.com/zakircodearchitect/intent-classifiers-on-regression-models","last_synced_at":"2026-05-02T17:37:33.586Z","repository":{"id":270130387,"uuid":"909420121","full_name":"ZakirCodeArchitect/Intent-classifiers-on-regression-models","owner":"ZakirCodeArchitect","description":"This project compares five optimization algorithms (GD, SGD, Momentum, RMSProp, and Adam) on Univariate Linear Regression and a neural network for Intent Classification with the ATIS dataset. It evaluates convergence speed, stability, and final loss, showing that Adam delivers the best performance across both models.","archived":false,"fork":false,"pushed_at":"2024-12-28T16:49:06.000Z","size":0,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-12-28T17:29:11.212Z","etag":null,"topics":["atis-dataset","intent-classification","jupyter-notebook","keras","keras-neural-networks","matplotlib","numpy","python","regression-models","tenserflow"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ZakirCodeArchitect.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-28T16:45:18.000Z","updated_at":"2024-12-28T16:52:12.000Z","dependencies_parsed_at":"2024-12-28T18:16:07.352Z","dependency_job_id":null,"html_url":"https://github.com/ZakirCodeArchitect/Intent-classifiers-on-regression-models","commit_stats":null,"previous_names":["zakircodearchitect/intent-classifiers-on-regression-models"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZakirCodeArchitect%2FIntent-classifiers-on-regression-models","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZakirCodeArchitect%2FIntent-classifiers-on-regression-models/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZakirCodeArchitect%2FIntent-classifiers-on-regression-models/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZakirCodeArchitect%2FIntent-classifiers-on-regression-models/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ZakirCodeArchitect","download_url":"https://codeload.github.com/ZakirCodeArchitect/Intent-classifiers-on-regression-models/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239644128,"owners_count":19673577,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["atis-dataset","intent-classification","jupyter-notebook","keras","keras-neural-networks","matplotlib","numpy","python","regression-models","tenserflow"],"created_at":"2024-12-29T00:48:50.482Z","updated_at":"2025-10-30T09:32:37.218Z","avatar_url":"https://github.com/ZakirCodeArchitect.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Optimization Algorithms Comparison\n## 📖 Project Overview\n\nIn this project, we compare and analyze the performance of five different optimization algorithms applied to two distinct models:\n\n1. **Univariate Linear Regression** with varying polynomial degrees and training set sizes.\n2. **Intent Classification (Neural Network)** using the ATIS dataset for a classification task.\n\nThe optimization algorithms evaluated in this study include:\n- **Gradient Descent (GD)**\n- **Stochastic Gradient Descent (SGD)**\n- **Gradient Descent with Momentum**\n- **RMSProp**\n- **Adam Optimizer**\n\n## 🔍 Objectives\n\nThe key objectives of this project were:\n- To explore how different optimization algorithms perform under various conditions, such as polynomial degrees and training set sizes in linear regression.\n- To evaluate how the optimizers behave when applied to a neural network for a classification task on the ATIS dataset.\n- To compare convergence speed, stability, and the final loss to determine the best optimizer for both linear regression and neural network tasks.\n\n## 🧑‍💻 Key Features\n\n### 1. **Model 1: Univariate Linear Regression**\n- **Polynomial Degrees:** 1, 2, 3, 4\n- **Training Set Sizes:** Varied\n- **Objective:** To evaluate the performance of optimization algorithms under different complexities and dataset sizes.\n- **Key Findings:**  \n    - **Gradient Descent (GD):** Slow convergence, often getting stuck in local minima, especially with higher polynomial degrees.\n    - **Stochastic Gradient Descent (SGD):** Faster convergence but introduces oscillations, particularly with larger learning rates.\n    - **Momentum:** Enhanced convergence by reducing oscillations, outperforming GD and SGD.\n    - **RMSProp:** Superior convergence, adaptive to learning rates, outperforms other algorithms in terms of speed.\n    - **Adam Optimizer:** Best overall performance, combining the strengths of both Momentum and RMSProp.\n\n### 2. **Model 2: Intent Classification with Neural Networks**\n- **Dataset:** ATIS (Airline Travel Information System)\n- **Objective:** To evaluate how optimization algorithms handle neural network training on a classification task.\n- **Key Findings:**  \n    - **SGD:** Slow convergence with higher final loss, demonstrating instability during training.\n    - **SGD with Momentum:** Faster convergence than SGD, but still lacked the stability of more advanced methods.\n    - **RMSProp:** Smooth convergence with lower loss, demonstrating its adaptive learning rate as a key advantage.\n    - **Adam:** Fastest convergence and lowest final loss, showing clear superiority in terms of performance and efficiency.\n\n## 🚀 Optimizers Performance\n\nThe following summarizes the performance of each optimizer:\n\n| Optimizer                   | Key Characteristics                                           | Performance Summary                 |\n|-----------------------------|---------------------------------------------------------------|-------------------------------------|\n| **Gradient Descent (GD)**    | Simple, consistent updates.                                   | Slow convergence; prone to local minima. |\n| **Stochastic Gradient Descent (SGD)** | Faster updates with noisy gradients.                         | Fast convergence, but prone to instability and oscillations. |\n| **Momentum**                 | Smoothing updates by adding momentum to previous gradients.   | Improved convergence over GD and SGD, reducing oscillations. |\n| **RMSProp**                  | Adaptive learning rate per parameter.                         | Fast convergence with adaptive learning rate, avoiding overshooting. |\n| **Adam Optimizer**           | Combines Momentum and RMSProp techniques.                     | Best performance overall, fast convergence, and stable final loss. |\n\n## 🧠 Analysis and Findings\n\n- **Gradient Descent vs. SGD:**  \n    - **GD** performs well on small datasets but struggles with more complex models due to its slow convergence, especially in high-dimensional spaces.\n    - **SGD**, while faster, can exhibit instability, particularly when the learning rate is too high. However, it performs better on large datasets where individual updates are more beneficial.\n\n- **Momentum and Its Benefits:**  \n    - **Momentum** helps smooth out the oscillations of SGD and can drastically improve convergence speed, especially when dealing with complex or noisy datasets.\n\n- **RMSProp and Adaptive Learning Rates:**  \n    - **RMSProp** adjusts the learning rate dynamically, providing a significant advantage over SGD and basic Momentum in terms of stability and speed. This makes it particularly suited for complex models and large datasets.\n\n- **Adam Optimizer's Superiority:**  \n    - **Adam** optimizer combines the benefits of **Momentum** and **RMSProp**, delivering superior results in terms of both convergence speed and final accuracy. It is the preferred choice for both regression and neural network tasks due to its robustness and efficiency.\n\n## 🔧 How to Use\n\nTo run the comparison and see the optimizer performances yourself, follow these steps:\n\n1. **Clone the repository:**\n    ```bash\n    git clone https://github.com/your-username/optimization-algorithms-comparison.git\n    ```\n\n2. **Install dependencies:**\n    - Make sure to have Python installed.\n    - Install required libraries:\n    ```bash\n    pip install -r requirements.txt\n    ```\n\n3. **Run the models:**\n    - Navigate to the directory and run the models:\n    ```bash\n    python linear_regression.py\n    python neural_network.py\n    ```\n\n## 📊 Results\n\nThe detailed results, including training loss graphs, comparison plots, and algorithm-specific performance data, can be found in the **`results`** folder.\n\n## 📝 Conclusion\n\nThe results from this project demonstrate that advanced optimizers like **Adam** significantly outperform traditional methods like **Gradient Descent** and **Stochastic Gradient Descent**. **Momentum** and **RMSProp** also show considerable improvements over basic SGD, especially in terms of stability and convergence speed. **Adam**'s combination of the benefits from both Momentum and RMSProp makes it the top choice for both **univariate linear regression** and **neural network classification** tasks.\n\n### 🛠️ Future Work:\n- Expand dataset size for further testing.\n- Explore additional neural network architectures and optimizers.\n- Experiment with hyperparameter tuning for even better results.\n\n## 👨‍💻 Contribution\n\nFeel free to fork this project, raise issues, or make contributions by submitting pull requests. Contributions are welcome to enhance the model performance and explore other optimization techniques.\n\n---\n\n**License**: This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzakircodearchitect%2Fintent-classifiers-on-regression-models","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzakircodearchitect%2Fintent-classifiers-on-regression-models","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzakircodearchitect%2Fintent-classifiers-on-regression-models/lists"}