{"id":13477167,"url":"https://github.com/amirgholami/PyHessian","last_synced_at":"2025-03-27T04:32:38.474Z","repository":{"id":52085769,"uuid":"222982416","full_name":"amirgholami/PyHessian","owner":"amirgholami","description":"PyHessian is a Pytorch library for second-order based analysis and training of Neural Networks","archived":false,"fork":false,"pushed_at":"2024-04-16T13:48:55.000Z","size":2148,"stargazers_count":720,"open_issues_count":17,"forks_count":121,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-03-02T21:05:13.268Z","etag":null,"topics":["hessian","hessian-free","pytorch-library","second-order","second-order-optimization"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/amirgholami.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-11-20T16:37:49.000Z","updated_at":"2025-02-22T08:14:41.000Z","dependencies_parsed_at":"2024-06-21T16:52:16.078Z","dependency_job_id":null,"html_url":"https://github.com/amirgholami/PyHessian","commit_stats":{"total_commits":31,"total_committers":8,"mean_commits":3.875,"dds":0.5806451612903225,"last_synced_commit":"c76c666fadd5d8a411b341247d5c7a15ec6fced1"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amirgholami%2FPyHessian","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amirgholami%2FPyHessian/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amirgholami%2FPyHessian/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amirgholami%2FPyHessian/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/amirgholami","download_url":"https://codeload.github.com/amirgholami/PyHessian/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245785591,"owners_count":20671628,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hessian","hessian-free","pytorch-library","second-order","second-order-optimization"],"created_at":"2024-07-31T16:01:38.966Z","updated_at":"2025-03-27T04:32:38.465Z","avatar_url":"https://github.com/amirgholami.png","language":"Jupyter Notebook","funding_links":[],"categories":["Jupyter Notebook"],"sub_categories":[],"readme":"![Block](misc/pyhessian-01.png)\n\n![Block](misc/hessian.png)\n\n## Introduction\nPyHessian is a pytorch library for Hessian based analysis of neural network models. The library enables computing the following metrics:\n\n- Top Hessian eigenvalues\n- The trace of the Hessian matrix\n- The full Hessian Eigenvalues Spectral Density (ESD)\n\n\nFor more details please see:\n\n- [The Hessian tutorial notebook](./Hessian_Tutorial.ipynb)\n- [Video explanation of tutorial](https://www.youtube.com/watch?v=S87ancnZ0MM\u0026feature=youtu.be\u0026t=43m20s)\n- [The PyHessian paper](https://arxiv.org/pdf/1912.07145.pdf).\n\n\n## Publication List\nThis project was supported through NSF funding and we are interested in documenting related publications\nwritten on or with the help of PyHessian. This will allows us to continue developing the library,\nand will also be a good summary for related and on going work on second order methods. [You can see the current list here](publication_list.md).\nPlease contact us if you have a related paper and we would be glad to add it to the list.\n\n\n## Usage\n### Install from Pip\nYou can install the library from pip\n```\npip install pyhessian\n```\n\n### Install from source\nYou can also compile the library from source\n```\ngit clone https://github.com/amirgholami/PyHessian.git\npython setup.py install\n```\n\nBefore running the Hessian code, we need a (pre-trained) NN model. Here, we provide a training file to train ResNet20 model on Cifar-10 dataset:\n```\nexport CUDA_VISIBLE_DEVICES=0; python training.py [--batch-size] [--test-batch-size] [--epochs] [--lr] [--lr-decay] [--lr-decay-epoch] [--seed] [--weight-decay] [--batch-norm] [--residual] [--cuda] [--saving-folder]\n\noptional arguments:\n--batch-size                training batch size (default: 128)\n--test-batch-size           testing batch size (default:256)\n--epochs                    total number of training epochs (default: 180)\n--lr                        initial learning rate (default: 0.1)\n--lr-decay                  learning rate decay ratio (default: 0.1)\n--lr-decay-epoch            epoch for the learning rate decaying (default: 80, 120)\n--seed                      used to reproduce the results (default: 1)\n--weight-decay              weight decay value (default: 5e-4)\n--batch-norm                do we need batch norm in ResNet or not (default: True)\n--residual                  do we need residual connection or not (default: True)\n--cuda                      do we use gpu or not (default: True)\n--saving-folder             saving path of the final checkpoint (default: checkpoints/)\n```\n\nAfter the model checkpoint is saved, we can run the following code to get the top eigenvalue, trace, and the Eigenvalue Spectral Density of Hessian:\n```\nexport CUDA_VISIBLE_DEVICES=0; python example_pyhessian_analysis.py [--mini-hessian-batch-size] [--hessian-batch-size] [--seed] [--batch-norm] [--residual] [--cuda] [--resume]\n\noptional arguments:\n--mini-hessian-batch-size   mini hessian batch size (default: 200)\n--hessian-batch-size        hessian batch size (default:200)\n--seed                      used to reproduce the results (default: 1)\n--batch-norm                do we need batch norm in ResNet or not (default: True)\n--residual                  do we need residual connection or not (default: True)\n--cuda                      do we use gpu or not (default: True)\n--resume                    resume path of the checkpoint (default: none, must be filled by user)\n```\n\nThe output density plot is saved as example.pdf \n\n\n## Citation\nPyHessian has been developed as part of the following paper. We appreciate it if you would please cite the following paper if you found the library useful for your work:\n\n* Z. Yao, A. Gholami, K Keutzer, M. Mahoney. PyHessian:  Neural Networks Through the Lens of the Hessian, Spotlight at ICML workshop on Beyond First-Order Optimization Methods in Machine Learning, 2020 (also in proceedings of IEEE Conferenceon big data), [PDF](https://arxiv.org/pdf/1912.07145.pdf).\n\n## Copyright\n\nTHIS SOFTWARE AND/OR DATA WAS DEPOSITED IN THE BAIR OPEN RESEARCH COMMONS REPOSITORY ON 02/27/23.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famirgholami%2FPyHessian","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famirgholami%2FPyHessian","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famirgholami%2FPyHessian/lists"}