{"id":23205489,"url":"https://github.com/sbartlett97/model-understanding","last_synced_at":"2025-04-05T10:41:20.563Z","repository":{"id":267359681,"uuid":"901004631","full_name":"sbartlett97/model-understanding","owner":"sbartlett97","description":"Playing around with understanding how different tokens affect the generation in LLMs","archived":false,"fork":false,"pushed_at":"2025-03-22T21:53:59.000Z","size":44,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-22T22:35:53.849Z","etag":null,"topics":["deep-learning","interpretable-ai","interpretable-machine-learning","llm","llm-inference","llms","machine-learning","model-understanding"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sbartlett97.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-09T21:39:54.000Z","updated_at":"2025-03-22T21:54:02.000Z","dependencies_parsed_at":"2024-12-14T00:15:55.021Z","dependency_job_id":null,"html_url":"https://github.com/sbartlett97/model-understanding","commit_stats":null,"previous_names":["sbartlett97/model-understanding"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sbartlett97%2Fmodel-understanding","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sbartlett97%2Fmodel-understanding/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sbartlett97%2Fmodel-understanding/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sbartlett97%2Fmodel-understanding/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sbartlett97","download_url":"https://codeload.github.com/sbartlett97/model-understanding/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247325649,"owners_count":20920713,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","interpretable-ai","interpretable-machine-learning","llm","llm-inference","llms","machine-learning","model-understanding"],"created_at":"2024-12-18T16:37:41.457Z","updated_at":"2025-04-05T10:41:20.558Z","avatar_url":"https://github.com/sbartlett97.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# [WIP] Model Understanding with Saliency and Attention Mapping\n\nThis repository provides tools for exploring and understanding transformer-based models using saliency maps and attention visualizations. By analyzing saliency and attention, users can gain insights into how models make predictions or generate text, which helps in debugging and interpreting model behavior.\n\n## Features\n\n- **Saliency Maps**: Visualize token importance by computing gradients of the input with respect to model outputs.\n- **Attention Visualizations**: Plot attention scores to understand token-to-token interactions.\n- **Support for Classification Models**: Works with `AutoModelForSequenceClassification` from the Hugging Face Transformers library.\n- **Support for Generative Models**: Compatible with `AutoModelForCausalLM` for analyzing autoregressive text generation.\n- **Custom Visualizations**: Generate bar plots for saliency scores and heatmaps for attention weights.\n\n## Installation\n\nClone the repository and install the required dependencies:\n\n```bash\ngit clone https://github.com/sbartlet97/model-visualization.git\ncd model-visualization\npip install -r requirements.txt\n```\n\n## Usage\n\n### For Classification Models\n\n```python\nfrom model_understanding import ModelVisualizer\n\n# Initialize the visualizer with a classification model\nvisualizer = ModelVisualizer(\"bert-base-uncased\")\n\n# Analyze a sample input text\ninput_text = \"The movie was absolutely fantastic!\"\nvisualizer.analyze(input_text)\n```\n\n### For Generative Models\n\n```python\nfrom model_understanding import GenerativeModelVisualizer\n\n# Initialize the visualizer with a generative model\ngen_visualizer = GenerativeModelVisualizer(\"gpt2\")\n\n# Analyze a sample prompt\nprompt = \"Once upon a time in a land far away,\"\ngen_visualizer.analyze(prompt)\n```\n\n### Outputs\n\n1. **Saliency Map**: A bar plot showing the relative importance of each token in the input.\n2. **Attention Weights**: A heatmap displaying token-to-token attention scores.\n\n## Implementation Details\n\n### ModelVisualizer\n\nThis class handles models for classification tasks. It computes saliency maps by backpropagating gradients through the model and extracts attention weights from the last layer.\n\n- **Methods**:\n  - `compute_saliency`: Computes normalized gradients for input tokens.\n  - `compute_attention`: Averages attention scores across heads.\n  - `plot_saliency`: Visualizes saliency as a bar plot.\n  - `plot_attention`: Visualizes attention scores as a heatmap.\n\n### GenerativeModelVisualizer\n\nThis class supports autoregressive models like GPT-2. It computes token saliency for the prompt and generates attention heatmaps for decoding steps.\n\n- **Methods**:\n  - `compute_saliency`: Uses the loss to compute input token gradients.\n  - `compute_attention`: Extracts and averages attention scores across heads.\n  - `plot_saliency`: Visualizes saliency as a bar plot.\n  - `plot_attention`: Visualizes attention scores as a heatmap.\n\n## Example Visualizations\n\n### Saliency Map Example\n\n![Saliency Map](./classification_saliency_map.png)\n\n### Attention Heatmap Example\n\n![Attention Heatmap](./classification_attention_fig.png)\n\n## Dependencies\n\n- `transformers`\n- `torch`\n- `matplotlib`\n- `seaborn`\n\n## License\n\nThis repository is licensed under the MIT License. See the `LICENSE` file for details.\n\n## Contributing\n\nContributions are welcome! Please feel free to submit issues or pull requests.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsbartlett97%2Fmodel-understanding","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsbartlett97%2Fmodel-understanding","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsbartlett97%2Fmodel-understanding/lists"}