{"id":30850101,"url":"https://github.com/farukalpay/psd","last_synced_at":"2026-01-20T17:52:56.372Z","repository":{"id":311166840,"uuid":"1042727505","full_name":"farukalpay/PSD","owner":"farukalpay","description":"Perturbed Saddle-escape Descent (PSD): a first-order optimizer that escapes strict saddle points in nonconvex problems.","archived":false,"fork":false,"pushed_at":"2025-08-22T15:28:00.000Z","size":26,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-22T15:37:53.678Z","etag":null,"topics":["first-order-methods","gradient-descent","machine-learning","nonconvex-optimization","numerical-methods","optimization","python","saddle-points"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/farukalpay.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-08-22T13:30:43.000Z","updated_at":"2025-08-22T15:28:04.000Z","dependencies_parsed_at":"2025-08-22T15:49:01.857Z","dependency_job_id":null,"html_url":"https://github.com/farukalpay/PSD","commit_stats":null,"previous_names":["farukalpay/psd"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/farukalpay/PSD","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/farukalpay%2FPSD","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/farukalpay%2FPSD/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/farukalpay%2FPSD/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/farukalpay%2FPSD/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/farukalpay","download_url":"https://codeload.github.com/farukalpay/PSD/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/farukalpay%2FPSD/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273997157,"owners_count":25204479,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-07T02:00:09.463Z","response_time":67,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["first-order-methods","gradient-descent","machine-learning","nonconvex-optimization","numerical-methods","optimization","python","saddle-points"],"created_at":"2025-09-07T05:05:31.068Z","updated_at":"2025-10-03T15:26:19.780Z","avatar_url":"https://github.com/farukalpay.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Perturbed Saddle-escape Descent (PSD)\n\n[![CI](https://github.com/farukalpay/PSD/actions/workflows/ci.yml/badge.svg)](https://github.com/farukalpay/PSD/actions/workflows/ci.yml)\n[![Coverage](https://img.shields.io/badge/coverage-90%25-brightgreen)](./)\n[![Docs](https://img.shields.io/badge/docs-latest-blue)](./docs/index.md)\n[![Python](https://img.shields.io/badge/python-3.8%2B-blue)](https://www.python.org/)\n\n## Project Summary\n\nThis repository implements the **Perturbed Saddle-escape Descent (PSD)**\nalgorithm for escaping saddle points in non-convex optimisation problems, as described in [Alpay and Alakkad (2025)](https://arxiv.org/abs/2508.16540).\nIt contains reference NumPy implementations, framework specific optimisers\nfor PyTorch and TensorFlow, and utilities for reproducing the synthetic\nexperiments reported in the accompanying manuscript.\n\n## Features\n\n* Reference implementations of PSD, PSD-Probe and baseline gradient descent\n  variants in pure NumPy.\n* Suite of analytic test functions with gradients and Hessians.\n* Synthetic data generator producing the tables and figures used in the\n  paper (`experiments.py`).\n* Framework specific optimisers: `PSDTorch`, `PSDTensorFlow` and a\n  `PSDOptimizer`/`PerturbedAdam` package for PyTorch.\n* Example training scripts for MNIST and CIFAR-10.\n\n## Technology Stack\n\nThe core project depends on the following libraries:\n\n| Library | Purpose |\n| ------- | ------- |\n| `numpy` | numerical routines for reference implementations |\n| `torch`, `torchvision` | deep-learning framework and datasets |\n| `optuna` | hyper-parameter search utilities |\n| `matplotlib` | visualisation in notebooks |\n\nPython 3.8 or later is required.\n\n## Installation\n\nInstall the published optimiser package:\n\n```bash\npip install psd-optimizer\n```\n\nOr install the repository in editable mode for development:\n\n```bash\ngit clone https://github.com/farukalpay/PSD.git\ncd PSD\npip install -e \".[dev]\"\n```\n\n## Quick Start\n\n```python\nimport numpy as np\nfrom psd import algorithms, functions\n\nx0 = np.array([1.0, -1.0])\nx_star, _ = algorithms.gradient_descent(x0, functions.SEPARABLE_QUARTIC.grad)\n```\n\nFurther examples are available in the [`examples/`](./examples) directory and the\n[documentation](./docs/index.md).\n\n## Usage\n\n### Using the Reference Algorithms\n\nThe core PSD routines and test functions can be imported from the\n``psd`` package:\n\n```python\nimport numpy as np\nfrom psd import algorithms, functions\n\nx0 = np.array([1.0, -1.0])\nx_star, _ = algorithms.gradient_descent(x0, functions.SEPARABLE_QUARTIC.grad)\n```\n\nThis structure allows you to experiment with the reference NumPy\nimplementations directly in your projects.\n\nThe PyTorch optimisers ``PSDOptimizer`` and ``PerturbedAdam`` are also\navailable directly via ``from psd import ...``.\n\n### All-in-One \"Monster\" Interface\n\nFor rapid experimentation without navigating submodules, import the aggregated\n``psd.monster`` module.  It re-exports the core algorithms, analytic test\nfunctions and framework-specific optimisers in a single namespace:\n\n```python\nimport numpy as np\nfrom psd import monster\n\nx0 = np.array([1.0, -1.0])\nx_star, _ = monster.gradient_descent(x0, monster.SEPARABLE_QUARTIC.grad)\n```\n\nThis unified view aims to be approachable for both humans and language models\nexploring the project.\n\n### Generating Synthetic Data\n\n```bash\npython experiments.py\n```\n\nThe command writes CSV summaries to `results/` and training curves to\n`data/`.\n\n## Performance\n\nProfiling identified `rosenbrock_hess` as a hot path when computing the\nRosenbrock Hessian.  Vectorising the computation removed explicit\nPython loops and yielded the following improvements (dimension 1000):\n\n| Version | Mean time (ms) | Peak memory (MB) |\n|---------|----------------|-----------------|\n| Before  | 3.52           | 8.00            |\n| After   | 1.01           | 8.04            |\n\nBenchmarking is automated via `pytest-benchmark` using a fixed NumPy seed.\nHard time and memory thresholds guard against major regressions.\n\n### Training with the PyTorch Optimiser\n\n```python\nfrom psd_optimizer import PSDOptimizer\n\nmodel = ...\nopt = PSDOptimizer(model.parameters(), lr=1e-3)\n\ndef closure():\n    opt.zero_grad()\n    output = model(x)\n    loss = criterion(output, y)\n    loss.backward()\n    return loss\n\nopt.step(closure)\n```\n\nExample scripts using this API are available in the `notebooks/`\ndirectory.\n\n### Training a Small Language Model\n\nAn illustrative example for fine-tuning a compact transformer with\n``PSDOptimizer`` is provided in ``scripts/train_small_language_model.py``.\nThe script downloads a tiny GPT-style model from the Hugging Face Hub and\noptimizes it on a short dummy corpus.\n\nRun the example with default settings:\n\n```bash\npython scripts/train_small_language_model.py\n```\n\nSpecify a different pretrained model and number of epochs:\n\n```bash\npython scripts/train_small_language_model.py --model distilgpt2 --epochs 5\n```\n\n## Documentation\n\nFull API documentation and guides are available in the\n[``docs/`` directory](./docs/index.md).\nAdditional materials include:\n\n* `notebooks/10_minute_start.ipynb` – an interactive notebook showcasing the optimiser.\n* `docs/section_1_5_extension.md` – theoretical notes on extending PSD to stochastic settings.\n* `notebooks/navigation.ipynb` – links to all example notebooks including `advanced_usage.ipynb`.\n\n## Testing\n\nAfter installing the repository in editable mode, run the test suite to\nverify that everything works:\n\n```bash\npytest\n```\n\nThe current suite is small but helps prevent regressions.\n\n## Repository Structure\n\n```\npsd/              # Reference implementations and framework-specific optimisers\n    algorithms.py # PSD and baseline algorithms\n    functions.py  # Analytic test functions and registry\npsd_optimizer/    # PyTorch optimiser package\nexperiments.py    # Synthetic data generation\n```\n\n## Contributing\n\nContributions are welcome!  Please open an issue or pull request on GitHub\nand see `CONTRIBUTING.md` for guidelines.  By participating you agree to\nabide by the `CODE_OF_CONDUCT.md`.\n\n## Citation\n\nIf you use PSD in your research, please cite the following:\n\n```bibtex\n@misc{alpay2025escapingsaddlepointscurvaturecalibrated,\n      title={Escaping Saddle Points via Curvature-Calibrated Perturbations: A Complete Analysis with Explicit Constants and Empirical Validation},\n      author={Faruk Alpay and Hamdi Alakkad},\n      year={2025},\n      eprint={2508.16540},\n      archivePrefix={arXiv},\n      primaryClass={cs.LG},\n      url={https://arxiv.org/abs/2508.16540},\n}\n```\n\n## License\n\nThis project is released under the MIT License.  See `LICENSE` for details.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffarukalpay%2Fpsd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffarukalpay%2Fpsd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffarukalpay%2Fpsd/lists"}