{"id":16137658,"url":"https://github.com/rockerboo/lora-inspector","last_synced_at":"2025-04-09T18:08:51.774Z","repository":{"id":158260582,"uuid":"622356228","full_name":"rockerBOO/lora-inspector","owner":"rockerBOO","description":"LoRA (Low-Rank Adaptation) inspector for Stable Diffusion","archived":false,"fork":false,"pushed_at":"2024-09-20T21:17:17.000Z","size":318,"stargazers_count":95,"open_issues_count":5,"forks_count":7,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-03-23T20:08:19.241Z","etag":null,"topics":["analysis","lora","low-rank-approximation","machine-learning","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rockerBOO.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"rockerBOO","patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":null}},"created_at":"2023-04-01T21:36:18.000Z","updated_at":"2025-03-14T13:46:20.000Z","dependencies_parsed_at":null,"dependency_job_id":"d1fb394d-5491-432c-9ece-b4334d20920e","html_url":"https://github.com/rockerBOO/lora-inspector","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rockerBOO%2Flora-inspector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rockerBOO%2Flora-inspector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rockerBOO%2Flora-inspector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rockerBOO%2Flora-inspector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rockerBOO","download_url":"https://codeload.github.com/rockerBOO/lora-inspector/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248085157,"owners_count":21045135,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analysis","lora","low-rank-approximation","machine-learning","pytorch"],"created_at":"2024-10-09T23:28:52.710Z","updated_at":"2025-04-09T18:08:51.755Z","avatar_url":"https://github.com/rockerBOO.png","language":"Python","funding_links":["https://github.com/sponsors/rockerBOO"],"categories":[],"sub_categories":[],"readme":"# LoRA inspector\n\n\u003c!--toc:start--\u003e\n\n- [LoRA inspector](#lora-inspector)\n  - [Install](#install)\n  - [Usage](#usage)\n    - [Inspect](#inspect)\n    - [Save meta](#save-meta)\n    - [Average weights](#average-weights)\n    - [Tag frequency](#tag-frequency)\n    - [Dataset](#dataset)\n    - [Definition](#definition)\n  - [Update metadata](#update-metadata)\n    - [Usage](#usage)\n  - [Changelog](#changelog)\n  - [Development](#development)\n  - [Future](#future)\n  - [Reference](#reference)\n  \u003c!--toc:end--\u003e\n\n![lora-inspector](https://user-images.githubusercontent.com/15027/230981999-1af9ec4e-4c05-40bc-a10a-b825c73b1013.png)\n\nInspect LoRA files for meta info and  quantitative analysis of the\nLoRA weights.\n\n- view training parameters\n- extract metadata to be stored (we can store it in JSON currently)\n- only `safetensors` are supported (want to support all LoRA files)\n- only metadata from kohya-ss LoRA (want to parse all metadata in LoRA files)\n\n---\n\n_NOTE_ this is a work in progress and not meant for production use. _NOTE_\n\nConsider using the new web interface [LoRA Inspector](https://lora-inspector.rocker.boo) for a GUI representation.\n\n---\n\n## Install\n\nClone this repo or download the python script file.\n\nRequires dependencies:\n\n```\ntorch\nsafetensors\ntqdm\n```\n\nCan install them one of the following:\n\n- Add this script to your training directory and use the virtual environment\n  (`venv`). **RECOMMENDED**\n- Make/use with a venv/conda\n- `pip install safetensors tqdm` (See\n  [Get started](https://pytorch.org/get-started/locally/) for instructions on\n  how to install PyTorch)\n\n## Usage\n\n### Inspect\n\n```bash\n$ python lora-inspector.py --help\nusage: lora-inspector.py [-h] [-s] [-w] [-t] [-d] lora_file_or_dir\n\npositional arguments:\n  lora_file_or_dir  Directory containing the lora files\n\noptions:\n  -h, --help        show this help message and exit\n  -s, --save_meta   Should we save the metadata to a file?\n  -w, --weights     Show the average magnitude and strength of the weights\n  -t, --tags        Show the most common tags in the training set\n  -d, --dataset     Show the dataset metadata including directory names and number of images\n```\n\nYou can add a directory or file:\n\n```bash\n$ python lora-inspector.py /mnt/900/training/sets/landscape-2023-11-06-200718-e4d7120b -w\n/mnt/900/training/sets/landscape-2023-11-06-200718-e4d7120b/landscape-2023-11-06-200718-e4d7120b-000015.safetensors\nDate: 2023-11-06T20:16:34 Title: landscape\nLicense: CreativeML Open RAIL-M Author: rockerBOO\nDescription: High quality landscape photos\nResolution: 512x512 Architecture: stable-diffusion-v1/lora\nNetwork Dim/Rank: 16.0 Alpha: 8.0 Dropout: 0.3 dtype: torch.float32\nModule: networks.lora : {'block_dims': '4,4,4,4,4,4,4,4,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8', 'block_alphas': '16,16,16,16,16,16,16,16,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32', 'block_dropout': '0.01, 0.010620912260804992, 0.01248099020159499, 0.015572268683063176, 0.01988151037617019, 0.02539026244641935, 0.032074935571726845, 0.03990690495552037, 0.04885263290251277, 0.058873812432261884, 0.0699275313155418, 0.08196645583109653, 0.09493903345590124, 0.10878971362098, 0.12345918558747097, 0.13888463242431537, 0.155, 0.17173627983648962, 0.18902180461412393, 0.20678255506208312, 0.22494247692026895, 0.2434238066153228, 0.26214740425618505, 0.2810330925232585', 'dropout': 0.3}\nLearning Rate (LR): 2e-06 UNet LR: 1.0 TE LR: 1.0\nOptimizer: prodigyopt.prodigy.Prodigy(weight_decay=0.1,betas=(0.9, 0.9999),d_coef=1.5,use_bias_correction=True)\nScheduler: cosine  Warmup steps: 0\nEpoch: 15 Batches per epoch: 57 Gradient accumulation steps: 24\nTrain images: 57 Regularization images: 0\nNoise offset: 0.05 Adaptive noise scale: 0.01 IP noise gamma: 0.1  Multires noise discount: 0.3\nMin SNR gamma: 5.0 Zero terminal SNR: True Debiased Estimation: True\nUNet weight average magnitude: 0.7865518983141094\nUNet weight average strength: 0.00995593195090544\nNo Text Encoder found in this LoRA\n----------------------\n/mnt/900/training/sets/landscape-2023-11-06-200718-e4d7120b/landscape-2023-11-06-200718-e4d7120b.safetensors\nDate: 2023-11-06T20:27:12 Title: landscape\nLicense: CreativeML Open RAIL-M Author: rockerBOO\nDescription: High quality landscape photos\nResolution: 512x512 Architecture: stable-diffusion-v1/lora\nNetwork Dim/Rank: 16.0 Alpha: 8.0 Dropout: 0.3 dtype: torch.float32\nModule: networks.lora : {'block_dims': '4,4,4,4,4,4,4,4,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8', 'block_alphas': '16,16,16,16,16,16,16,16,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32', 'block_dropout': '0.01, 0.010620912260804992, 0.01248099020159499, 0.015572268683063176, 0.01988151037617019, 0.02539026244641935, 0.032074935571726845, 0.03990690495552037, 0.04885263290251277, 0.058873812432261884, 0.0699275313155418, 0.08196645583109653, 0.09493903345590124, 0.10878971362098, 0.12345918558747097, 0.13888463242431537, 0.155, 0.17173627983648962, 0.18902180461412393, 0.20678255506208312, 0.22494247692026895, 0.2434238066153228, 0.26214740425618505, 0.2810330925232585', 'dropout': 0.3}\nLearning Rate (LR): 2e-06 UNet LR: 1.0 TE LR: 1.0\nOptimizer: prodigyopt.prodigy.Prodigy(weight_decay=0.1,betas=(0.9, 0.9999),d_coef=1.5,use_bias_correction=True)\nScheduler: cosine  Warmup steps: 0\nEpoch: 30 Batches per epoch: 57 Gradient accumulation steps: 24\nTrain images: 57 Regularization images: 0\nNoise offset: 0.05 Adaptive noise scale: 0.01 IP noise gamma: 0.1  Multires noise discount: 0.3\nMin SNR gamma: 5.0 Zero terminal SNR: True Debiased Estimation: True\nUNet weight average magnitude: 0.8033398082829257\nUNet weight average strength: 0.010114916750103732\nNo Text Encoder found in this LoRA\n----------------------\n```\n\n```bash\n$ python lora-inspector.py /mnt/900/lora/testing/landscape-2023-11-06-200718-e4d7120b.safetensors\n/mnt/900/lora/testing/landscape-2023-11-06-200718-e4d7120b.safetensors\nDate: 2023-11-06T20:27:12 Title: landscape\nLicense: CreativeML Open RAIL-M Author: rockerBOO\nDescription: High quality landscape photos\nResolution: 512x512 Architecture: stable-diffusion-v1/lora\nNetwork Dim/Rank: 16.0 Alpha: 8.0 Dropout: 0.3 dtype: torch.float32\nModule: networks.lora : {'block_dims': '4,4,4,4,4,4,4,4,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8', 'block_alphas': '16,16,16,16,16,16,16,16,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32', 'block_dropout': '0.01, 0.010620912260804992, 0.01248099020159499, 0.015572268683063176, 0.01988151037617019, 0.02539026244641935, 0.032074935571726845, 0.03990690495552037, 0.04885263290251277, 0.058873812432261884, 0.0699275313155418, 0.08196645583109653, 0.09493903345590124, 0.10878971362098, 0.12345918558747097, 0.13888463242431537, 0.155, 0.17173627983648962, 0.18902180461412393, 0.20678255506208312, 0.22494247692026895, 0.2434238066153228, 0.26214740425618505, 0.2810330925232585', 'dropout': 0.3}\nLearning Rate (LR): 2e-06 UNet LR: 1.0 TE LR: 1.0\nOptimizer: prodigyopt.prodigy.Prodigy(weight_decay=0.1,betas=(0.9, 0.9999),d_coef=1.5,use_bias_correction=True)\nScheduler: cosine  Warmup steps: 0\nEpoch: 30 Batches per epoch: 57 Gradient accumulation steps: 24\nTrain images: 57 Regularization images: 0\nNoise offset: 0.05 Adaptive noise scale: 0.01 IP noise gamma: 0.1  Multires noise discount: 0.3\nMin SNR gamma: 5.0 Zero terminal SNR: True Debiased Estimation: True\nUNet weight average magnitude: 0.8033398082829257\nUNet weight average strength: 0.010114916750103732\nNo Text Encoder found in this LoRA\n----------------------\n```\n\n### Save meta\n\nWe also have support for saving the meta that is extracted and converted from\nstrings. We can then save those to a JSON file. These will save the metadata\ninto `meta/alorafile.safetensors-{session_id}.json` in the current working\ndirectory.\n\n```bash\n$ python lora-inspector.py ~/loras/alorafile.safetensors --save_meta\n```\n\n```bash\n$ python lora-inspector.py /mnt/900/training/cyberpunk-anime-21-min-snr/unet-1.15-te-1.15-noise-0.1-steps--linear-DAdaptation-networks.lora/last.safetensors --save_meta\n/mnt/900/training/cyberpunk-anime-21-min-snr/unet-1.15-te-1.15-noise-0.1-steps--linear-DAdaptation-networks.lora/last.safetensors\ntrain images: 1005 regularization images: 32000\nlearning rate: 1.15 unet: 1.15 text encoder: 1.15\nepoch: 1 batches: 2025\noptimizer: dadaptation.dadapt_adam.DAdaptAdam lr scheduler: linear\nnetwork dim/rank: 8.0 alpha: 4.0 module: networks.lora\n----------------------\n```\n\n### Average weights\n\nFind the average magnitude and average strength of your weights. Compare these\nwith other LoRAs to see how powerful or not so powerful your weights are. _NOTE_\nWeights shown are not conclusive to a good value. They are an initial example.\n\n```bash\n$ python lora-inspector.py /mnt/900/lora/studioGhibliStyle_offset.safetensors -w\nUNet weight average magnitude: 4.299801171795097\nUNet weight average strength: 0.01127891692482733\nText Encoder weight average magnitude: 3.128134997225176\nText Encoder weight average strength: 0.00769676965767913\n```\n\n### Tag frequency\n\nShows the frequency of a tag (words separated by commas). Trigger words are\ngenerally the most frequent, as they would use that word across the whole\ntraining dataset.\n\n```\n$ python lora-inspector.py -t /mnt/900/lora/booscapes.safetensors\n...\n-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=\nTags\n-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=\n4k photo”                         23\nspectacular mountains             17\naward winning nature photo        16\nryan dyar                         14\nimage credit nasa nat geo         11\nsunset in a valley                11\ngarden                            10\nbritish columbia                  10\ndramatic autumn landscape         10\nautumn mountains                  10\nan amazing landscape image        10\naustria                           9\nnature scenery                    9\npristine water                    9\nboreal forest                     9\nscenic view of river              9\nalpes                             9\nmythical floral hills             8\nmisty environment                 8\na photo of a lake on a sunny day  8\nmajestic beautiful world          8\nbreathtaking stars                8\nlush valley                       7\ndramatic scenery                  7\nsolar storm                       7\nsiberia                           7\ncosmic skies                      7\ndolomites                         7\noregon                            6\nlandscape photography 4k          6\nvery long spires                  6\nbeautiful forests and trees       6\nwildscapes                        6\nmountain behind meadow            6\ncolorful wildflowers              6\nphoto of green river              6\nbeautiful night sky               6\nswitzerland                       6\nnatural dynamic range color       6\nmiddle earth                      6\njessica rossier color scheme      6\narizona                           6\nenchanting and otherworldly       6\n```\n\n### Dataset\n\nA pretty basic view of the dataset with the directories and number of images.\n\n```\n$ python lora-inspector.py -d /mnt/900/lora/booscapes.safetensors\nDataset dirs: 2\n    [source] 50 images\n    [p7] 4 images\n```\n\n### Definition\n\n- epoch: an epoch is seeing the entire dataset once\n- Batches per epoch: how many batches per each epoch (does not include gradient\n  accumulation steps)\n- Gradient accumulation steps: gradient accumulation steps\n- Train images: number of training images you have\n- Regularization images: number of regularization images\n- Scheduler: the learning rate scheduler (cosine, cosine_with_restart, linear,\n  constant, …)\n- Optimizer: the optimizer (Adam, Prodigy, DAdaptation, Lion, …)\n- Network dim/rank: the rank of the LoRA network\n- Alpha: the alpha to the rank of the LoRA network\n- Module: the python module that created the network\n- Noise offset: noise offset option\n- Adaptive noise scale: adaptive noise scale\n- IP noise gamma: Input Perturbation noise gamma\n  [Input Perturbation Reduces Exposure Bias in Diffusion Models](https://arxiv.org/abs/2301.11706)\n\n  - \u003e …we propose a very simple but effective training regularization,\n    \u003e consisting in perturbing the ground truth samples to simulate the\n    \u003e inference time prediction errors.\n\n- multires noise discount: multires noise discount (See\n  [Multi-Resolution Noise for Diffusion Model Training](https://wandb.ai/johnowhitaker/multires_noise/reports/Multi-Resolution-Noise-for-Diffusion-Model-Training--VmlldzozNjYyOTU2))\n- multires noise scale: multires noise scale\n\n- average magnitude: square each weight, add them up, get the square root\n- average strength: abs each weight, add them up, get average\n- debiased estimation loss:\n  [Debias the Training of Diffusion Models](https://arxiv.org/abs/2310.08442)\n\n## Update metadata\n\nSimple script to update your metadata values. Helpful for changing\n`ss_output_name` for applications that use this value to set a good name for it.\n\nTo see your current metadata values, save the metadata using\n`lora-inspector.py --save_meta ...` and inspect the JSON file.\n\n```\n$ python update_metadata.py --help\nusage: update_metadata.py [-h] [--key KEY] [--value VALUE] safetensors_file\n\npositional arguments:\n  safetensors_file\n\noptions:\n  -h, --help        show this help message and exit\n  --key KEY         Key to change in the metadata\n  --value VALUE     Value to set to the metadata\n```\n\n### Usage\n\n```\n$ python update_metadata.py /mnt/900/lora/testing/armored-core-2023-08-02-173642-ddb4785e.safetensors --key ss_output_name --value mechBOO_v2\nUpdated ss_output_name with mechBOO_v2\nSaved to /mnt/900/lora/testing/armored-core-2023-08-02-173642-ddb4785e.safetensors\n```\n\n## Changelog\n\n- 2023-11-11 — Add debiased estimation loss, dtype (precision)\n- 2023-10-27 — Add IP noise gamma\n- 2023-08-27 — Add max_grad_norm, scale weight norms, gradient accumulation\n  steps, dropout, and datasets\n- 2023-08-08 — Add simple metadata updater script\n- 2023-07-31 — Add SDXL support\n- 2023-07-17 — Add network dropout, scale weight norms, adaptive noise scale,\n  and steps\n- 2023-07-06 — Add Tag Frequency\n- 2023-04-12 — Add gradient norm, gradient checkpoint metadata\n- 2023-04-03 — Add clip_skip, segment off LoCon/conv layers in average weights\n- 2023-04-03 — Add noise_offset, min_snr_gamma (when added to kohya-ss), and\n  network_args (for LoCon values)\n- 2023-04-02 — Add `--weights` which allows you to see the average magnitude and\n  strength of your LoRA UNet and Text Encoder weights.\n\n## Development\n\nFormatted using [`black`](https://github.com/psf/black).\n\n## Future\n\nWhat else do you want to see? Make an issue or a PR.\n\nUse cases/ideas that this can expand into:\n\n- Extract metadata from LoRA files to be used elsewhere\n- Put the metadata into a database or search engine to find specific trainings\n- Find possible issues with the training due to the metadata\n- Compare LoRA files together\n\n## Reference\n\n- https://github.com/Zyin055/Inspect-Embedding-Training\n- https://github.com/kohya-ss/sd-scripts\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frockerboo%2Flora-inspector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frockerboo%2Flora-inspector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frockerboo%2Flora-inspector/lists"}