{"id":50260589,"url":"https://github.com/joe0731/hf_vram_calc","last_synced_at":"2026-05-27T09:34:47.209Z","repository":{"id":310083671,"uuid":"1038474723","full_name":"joe0731/hf_vram_calc","owner":"joe0731","description":"A CLI tool for estimating GPU VRAM requirements for Hugging Face models, supporting various data types, parallelization strategies, and fine-tuning scenarios like LoRA.","archived":false,"fork":false,"pushed_at":"2025-10-20T15:36:04.000Z","size":238,"stargazers_count":1,"open_issues_count":6,"forks_count":2,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-20T17:14:42.905Z","etag":null,"topics":["gpu-memory","hugging-face-transformers","huggingface","huggingface-datasets","huggingface-models","memory-estimation","pipeline-parallelism","vram","vram-calculator","vram-memory-estimation","vram-monitoring"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/joe0731.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-15T09:09:08.000Z","updated_at":"2025-10-20T15:04:05.000Z","dependencies_parsed_at":"2025-09-03T11:21:47.737Z","dependency_job_id":"a1baa403-c4c1-4890-a514-944a244ed3d0","html_url":"https://github.com/joe0731/hf_vram_calc","commit_stats":null,"previous_names":["joe0731/model_vram_calc","joe0731/hf_vram_calc"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/joe0731/hf_vram_calc","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joe0731%2Fhf_vram_calc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joe0731%2Fhf_vram_calc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joe0731%2Fhf_vram_calc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joe0731%2Fhf_vram_calc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/joe0731","download_url":"https://codeload.github.com/joe0731/hf_vram_calc/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joe0731%2Fhf_vram_calc/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33560727,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-27T02:00:06.184Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gpu-memory","hugging-face-transformers","huggingface","huggingface-datasets","huggingface-models","memory-estimation","pipeline-parallelism","vram","vram-calculator","vram-memory-estimation","vram-monitoring"],"created_at":"2026-05-27T09:34:46.869Z","updated_at":"2026-05-27T09:34:47.203Z","avatar_url":"https://github.com/joe0731.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HF VRAM Calculator\n\n[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![uv](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json)](https://github.com/astral-sh/uv)\n\nA professional Python CLI tool for estimating GPU memory requirements for Hugging Face models with different data types and parallelization strategies.\n\n\u003e **⚡ Latest Features**: Smart dtype detection, MHA/MQA/GQA-aware KV cache, 12 quantization formats, 20+ GPU models, professional Rich UI\n\n## Quick Demo\n\n```bash\n# Install and run\npip install hf-vram-calc\n\n# Set up authentication (required for most models)\nhf auth login --token yourtoken --add-to-git-credential\n\n# Calculate memory requirements\nhf-vram-calc microsoft/DialoGPT-medium\n\n# Output: Beautiful tables showing 0.9GB inference, GPU compatibility, parallelization strategies\n```\n\n## Features\n\n- 🔍 **Automatic Model Analysis**: Fetch configurations from Hugging Face Hub automatically\n- 🧠 **Smart Data Type Detection**: Intelligent dtype recommendation from model names, config, or defaults\n- 📊 **Comprehensive Data Type Support**: fp32, fp16, bf16, fp8, int8, int4, mxfp4, nvfp4, awq_int4, gptq_int4, nf4, fp4\n- 🎯 **Multi-Scenario Memory Estimation**:\n  - **Inference**: Model weights + KV cache overhead (MHA/MQA/GQA-aware, ×1.2 factor)\n  - **Training**: Full Adam optimizer states (×4×1.3 factors)\n  - **LoRA Fine-tuning**: Low-rank adaptation with trainable parameter overhead\n- ⚡ **Advanced Parallelization Analysis**:\n  - Tensor Parallelism (TP): 1, 2, 4, 8\n  - Pipeline Parallelism (PP): 1, 2, 4, 8  \n  - Expert Parallelism (EP) for MoE models\n  - Data Parallelism (DP): 2, 4, 8\n  - Combined strategies (TP + PP combinations)\n- 🎮 **GPU Compatibility Matrix**:\n  - 20+ GPU models (RTX 4090, A100, H100, L40S, etc.)\n  - Automatic compatibility checking for inference/training/LoRA\n  - Minimum GPU memory requirement calculations\n- 📈 **Professional Rich UI**:\n  - 🎨 Beautiful color-coded tables and panels\n  - 📊 Real-time progress indicators\n  - 🚀 Modern CLI interface with emoji icons\n  - 💡 Smart recommendations and warnings\n- 🔧 **Flexible Configuration**:\n  - Customizable LoRA rank, batch size, sequence length\n  - External JSON configuration files\n  - User-defined GPU models and data types\n- 📋 **Parameter Display**: Raw count + human-readable format (e.g., \"405,016,576 (405.0M)\")\n\n## Installation\n\n### Quick Install (from PyPI)\n\n```bash\npip install hf-vram-calc\n```\n\n### Build from Source\n\n```bash\n# Clone the repository\ngit clone \u003crepository-url\u003e\ncd hf-vram-calc\n\n# Build with uv (recommended)\nuv build\nuv pip install dist/hf_vram_calc-1.0.0-py3-none-any.whl\n\n# Or install directly\nuv pip install .\n```\n\n\u003e **Dependencies**: `requests` (HTTP), `rich` (beautiful CLI), Python ≥3.8\n\nFor detailed build instructions, see: [BUILD.md](BUILD.md)\n\n## Authentication Setup\n\nMany models require a Hugging Face token. Get yours at https://huggingface.co/settings/tokens, then:\n\n```bash\nhf auth login --token yourtoken --add-to-git-credential\n```\n\n## Usage\n\n### Basic Usage - Smart Dtype Detection\n\n```bash\n# Automatic dtype recommendation from model config/name\nhf-vram-calc --model mistralai/Mistral-7B-v0.1\n```\n\n### Specify Data Type Override\n\n```bash\n# Override with specific data type\nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --dtype bf16\nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --dtype bf16,fp8\n```\n\n### Advanced Configuration\n\n```bash\n# Custom batch size and sequence length\nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --max_batch_size 4 --max_seq_len 4096\n\n# Custom LoRA rank for fine-tuning estimation  \nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --lora_rank 128\n\n# Detailed analysis (disabled by default)\nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --log_level verbose\n```\n\n### YAML Configuration\n\n```bash\n# Use YAML configuration file (trtllm-bench compatible)\nhf-vram-calc --extra_llm_api_options example_config.yaml\n\n# Override YAML with command line arguments\nhf-vram-calc --extra_llm_api_options  example_config.yaml --max_batch_size 128\n```\n\n### JSON Output\n\n```bash\n# Save results to JSON file\nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --dtype bf16,fp8 --output_json results.json\n```\n\n### System Information\n\n```bash\n# List all available data types and GPU models\nhf-vram-calc --list_types\n\n# Use custom configuration directory\nhf-vram-calc --config_dir ./my_config --model mistralai/Mistral-7B-v0.1\n\n# Show help\nhf-vram-calc --help\n```\n\n## Command Line Arguments\n\n### Required\n- `--model MODEL`: Hugging Face model name (e.g., `mistralai/Mistral-7B-v0.1`)\n\n### Data Type Control  \n- `--dtype {fp32,fp16,bf16,fp8,int8,int4,mxfp4,nvfp4,awq_int4,fp4,nf4,gptq_int4}`: Override automatic dtype detection\n- `--list_types`: List all available data types and GPU models\n\n### Memory Estimation Parameters\n- `--max_batch_size BATCH_SIZE`: Batch size for activation estimation (default: 1)\n- `--max_seq_len SEQUENCE_LENGTH`: Sequence length for memory calculation (default: 2048)  \n- `--lora_rank LORA_RANK`: LoRA rank for fine-tuning estimation (default: 64)\n\n### Parallelization Settings\n- `--tp TP`: Tensor parallelism size (default: 1)\n- `--pp PP`: Pipeline parallelism size (default: 1)\n- `--ep EP`: Expert parallelism size (default: 1)\n\n### Configuration \u0026 Output\n- `--model_path MODEL_PATH`: Path to local model directory containing config.json\n- `--extra_llm_api_options YAML_FILE`: Path to YAML configuration file (trtllm-bench compatible)\n- `--output_json JSON_FILE`: Path to save results as JSON file\n- `--log_level {info,verbose}`: Log level for output (default: info)\n- `--config_dir CONFIG_DIR`: Custom configuration directory path\n- `--help`: Show complete help message with examples\n\n### Smart Behavior\n- **No `--dtype`**: Uses intelligent priority (model name → config → fp16 default)\n- **With `--dtype`**: Overrides automatic detection with specified type\n- **YAML + CLI**: Command line arguments override YAML configuration\n- **Invalid model**: Graceful error handling with helpful suggestions\n\n## Quick Start Examples\n\n```bash\n# Set up authentication first time\nhf auth login --token yourtoken --add-to-git-credential\n\n# Estimate memory for different models\nhf-vram-calc --model mistralai/Mistral-7B-v0.1              # → ~14GB inference (BF16)\nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --dtype fp16 # → ~14GB inference (FP16)\nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --dtype fp8  # → ~7GB inference (FP8)\n\n# estimate size for specified quantization versions\nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --dtype fp16     # → ~14GB\nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --dtype int4     # → ~3.5GB  \nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --dtype awq_int4 # → ~3.5GB\n\n# for private access models, it is recommended to use --model_path\nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --model_path /llm_data/llm-models/Mistral-7B-v0.1\n\n# Find optimal parallelization strategy\nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --log_level verbose  # → TP/PP recommendations\n\n# Save results to JSON\nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --output_json results.json\n\n# Use YAML configuration (trtllm-bench compatible)\nhf-vram-calc --extra_llm_api_options config.yaml\n\n# Check what's available\nhf-vram-calc --list_types                               # → All types \u0026 GPUs\n```\n## Data Type Priority \u0026 Detection\n\n### Automatic Data Type Recommendation\n\nThe tool uses intelligent priority-based dtype selection:\n\n1. **Model Name Detection** (Highest Priority)\n   - `model-fp16`, `model-bf16` → Extracts from model name  \n   - `model-4bit`, `model-gptq`, `model-awq` → Detects quantization\n   \n2. **Config torch_dtype** (Medium Priority)\n   - Reads `torch_dtype` from model's `config.json`\n   - Maps `torch.float16` → `fp16`, `torch.bfloat16` → `bf16`, etc.\n\n3. **Default Fallback** (Lowest Priority)\n   - Defaults to `fp16` when no dtype detected\n\n### Supported Data Types\n\n| Data Type | Bytes/Param | Description | Detection Patterns |\n|-----------|-------------|-------------|--------------------|\n| **fp32**  | 4.0 | 32-bit floating point | `fp32`, `float32` |\n| **fp16**  | 2.0 | 16-bit floating point | `fp16`, `float16`, `half` |\n| **bf16**  | 2.0 | Brain Float 16 | `bf16`, `bfloat16` |\n| **fp8**   | 1.0 | 8-bit floating point | `fp8`, `float8` |\n| **int8**  | 1.0 | 8-bit integer | `int8`, `8bit` |\n| **int4**  | 0.5 | 4-bit integer | `int4`, `4bit` |\n| **mxfp4** | 0.5 | Microsoft FP4 | `mxfp4` |\n| **nvfp4** | 0.5 | NVIDIA FP4 | `nvfp4` |\n| **awq_int4** | 0.5 | AWQ 4-bit quantization | `awq`, `awq-int4` |\n| **gptq_int4** | 0.5 | GPTQ 4-bit quantization | `gptq`, `gptq-int4` |\n| **nf4**   | 0.5 | 4-bit NormalFloat | `nf4`, `bnb-4bit` |\n| **fp4**   | 0.5 | 4-bit floating point | `fp4` |\n\n## YAML Configuration (trtllm-bench Compatible)\n\nThe `--extra_llm_api_options` argument allows you to use YAML configuration files with the same hierarchical structure as trtllm-bench:\n\n```yaml\n# config.yaml\nmodel: \"mistralai/Mistral-7B-v0.1\"\nkv_cache_config:\n  dtype: \"fp8\"\n  mamba_ssm_cache_dtype: \"fp16\"\nenable_chunked_prefill: true\nbuild_config:\n  max_batch_size: 64\n  max_num_tokens: 8192\n  max_seq_len: 4096\nquant_config:\n  quant_algo: \"fp8\"\n  kv_cache_quant_algo: \"fp8\"\nlora_config:\n  lora_dir: \"/path/to/lora/weights\"\n  max_lora_rank: 16\nperformance_options:\n  cuda_graphs: true\n  multi_block_mode: true\nlog_level: \"verbose\"\n```\n\n### YAML Section Mappings\n\n- `build_config.max_batch_size` → `--max_batch_size`\n- `build_config.max_seq_len` → `--max_seq_len`\n- `lora_config.max_lora_rank` → `--lora_rank`\n- `kv_cache_config.dtype` → `--dtype`\n- `quant_config.quant_algo` → `--dtype` (with algorithm-to-dtype mapping)\n\n## JSON Output\n\nThe `--output_json` argument saves calculation results in a simplified JSON format:\n\n```json\n{\n  \"model\": {\n    \"name\": \"mistralai/Mistral-7B-v0.1\",\n    \"architecture\": \"mistral\",\n    \"parameters\": 7241732096,\n    \"parameters_formatted\": \"7.24B\",\n    \"original_torch_dtype\": \"torch.bfloat16\",\n    \"user_specified_dtype\": \"FP8,BF16\"\n  },\n  \"memory_requirements\": [\n    {\n      \"dtype\": \"FP8\",\n      \"batch_size\": 1,\n      \"sequence_length\": 2048,\n      \"lora_rank\": 64,\n      \"model_size_gib\": 6.75,\n      \"kv_cache_size_gib\": 0.13,\n      \"inference_total_gib\": 8.10,\n      \"training_gib\": 35.07,\n      \"lora_size_gib\": 8.37\n    },\n    {\n      \"dtype\": \"BF16\",\n      \"batch_size\": 1,\n      \"sequence_length\": 2048,\n      \"lora_rank\": 64,\n      \"model_size_gib\": 13.49,\n      \"kv_cache_size_gib\": 0.25,\n      \"inference_total_gib\": 16.19,\n      \"training_gib\": 70.14,\n      \"lora_size_gib\": 16.73\n    }\n  ]\n}\n```\n\n## Parallelization Strategies\n\n### Tensor Parallelism (TP)\nSplits model weights by tensor dimensions across multiple GPUs.\n\n### Pipeline Parallelism (PP)\nDistributes different model layers to different GPUs.\n\n### Expert Parallelism (EP)\nFor MoE (Mixture of Experts) models, distributes expert networks to different GPUs.\n\n### Data Parallelism (DP)\nEach GPU holds a complete model copy, only splitting data.\n\n## Example Output\n\n### Smart Dtype Detection Example\n\n```bash\n$ hf-vram-calc --model mistralai/Mistral-7B-v0.1 --log_level verbose\n```\n\n```\nUsing recommended data type: FP16\nUse --dtype to specify different type, or see --list_types for all options\n  🔍 Fetching configuration for mistralai/Mistral-7B-v0.1...\nUsing recommended data type: FP16\nUse --dtype to specify different type, or see --list_types for all options\n  📋 Parsing model configuration...                         \n  🧮 Calculating model parameters...                        \n  💾 Computing memory requirements...                       \n\n                          ╭─────── 🤖 Model Information ───────╮\n                          │                                    │\n                          │  Model: mistralai/Mistral-7B-v0.1  │\n                          │  Architecture: mistral             │\n                          │  Parameters: 7,241,732,096 (7.24B) │\n                          │  Recommended dtype: FP16           │\n                          │                                    │\n                          ╰────────────────────────────────────╯\n\n        💾 Memory Requirements by Data Type and Scenario                \n╭──────────────┬──────────────┬─────────────────┬─────────────────┬─────────────────┬──────────────╮\n│              │   Model Size │        KV Cache │       Inference │        Training │         LoRA │\n│  Data Type   │         (GB) │            (GB) │      Total (GB) │     (Adam) (GB) │         (GB) │\n├──────────────┼──────────────┼─────────────────┼─────────────────┼─────────────────┼──────────────┤\n│     FP16     │         0.76 │            0.19 │            0.91 │            3.94 │         0.94 │\n╰──────────────┴──────────────┴─────────────────┴─────────────────┴─────────────────┴──────────────╯\n================================================================================\n          ⚡ Parallelization Strategies (FP16 Inference)                 \n╔════════════════════╤══════╤══════╤══════╤══════╤══════════════╤══════════════╗\n║                    │      │      │      │      │   Memory/GPU │   Min GPU    ║\n║ Strategy           │  TP  │  PP  │  EP  │  DP  │         (GB) │   Required   ║\n╟────────────────────┼──────┼──────┼──────┼──────┼──────────────┼──────────────╢\n║ Single GPU         │  1   │  1   │  1   │  1   │         0.91 │     4GB+     ║\n║ Tensor Parallel    │  2   │  1   │  1   │  1   │         0.45 │     4GB+     ║\n║ TP + PP            │  4   │  4   │  1   │  1   │         0.06 │     4GB+     ║\n╚════════════════════╧══════╧══════╧══════╧══════╧══════════════╧══════════════╝\n\n                  🎮 GPU Compatibility Matrix                         \n┏━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━┯━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━┓\n┃ GPU Type        │   Memory   │  Inference   │   Training   │     LoRA     ┃\n┠─────────────────┼────────────┼──────────────┼──────────────┼──────────────┨\n┃ RTX 4090        │    24GB    │      ✓       │      ✓       │      ✓       ┃\n┃ A100 80GB       │    80GB    │      ✓       │      ✓       │      ✓       ┃\n┃ H100 80GB       │    80GB    │      ✓       │      ✓       │      ✓       ┃\n┗━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━┷━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━┛\n\n╭─── 📋 Minimum GPU Requirements ────╮\n│                                   │\n│  Single GPU Inference: 0.9GB      │\n│  Single GPU Training: 3.9GB       │  \n│  Single GPU LoRA: 0.9GB           │\n│                                   │\n╰───────────────────────────────────╯\n```\n\n### Large Model with User Override\n\n```bash\n$ hf-vram-calc nvidia/DeepSeek-R1-0528-FP4 --dtype nvfp4\n\n$ hf-vram-calc Qwen/Qwen-72B-Chat \n\n```\n\n```\n                          ╭──────── 🤖 Model Information ────────╮\n                          │                                      │\n                          │  Model: nvidia/DeepSeek-R1-0528-FP4  │\n                          │  Architecture: deepseek_v3           │\n                          │  Parameters: 30,510,606,336 (30.5B)  │\n                          │  Original torch_dtype: bfloat16      │\n                          │  User specified dtype: NVFP4         │\n                          │                                      │\n                          ╰──────────────────────────────────────╯\n\n        💾 Memory Requirements by Data Type and Scenario                \n╭──────────────┬──────────────┬──────────────┬─────────────────┬──────────────╮\n│              │   Total Size │    Inference │        Training │         LoRA │\n│  Data Type   │         (GB) │         (GB) │     (Adam) (GB) │         (GB) │\n├──────────────┼──────────────┼──────────────┼─────────────────┼──────────────┤\n│    NVFP4     │        14.21 │        17.05 │           73.88 │        19.34 │\n╰──────────────┴──────────────┴──────────────┴─────────────────┴──────────────╯\n```\n\n### List Available Types\n\n```bash\n$ hf-vram-calc --list_types\n```\n\n```\nAvailable Data Types:\n╭───────────┬─────────────┬────────────────────────╮\n│ Data Type │ Bytes/Param │ Description            │\n├───────────┼─────────────┼────────────────────────┤\n│ FP32      │           4 │ 32-bit floating point  │\n│ FP16      │           2 │ 16-bit floating point  │\n│ BF16      │           2 │ Brain Float 16         │\n│ NVFP4     │         0.5 │ NVIDIA FP4             │\n│ AWQ_INT4  │         0.5 │ AWQ 4-bit quantization │\n│ GPTQ_INT4 │         0.5 │ GPTQ 4-bit quantization│\n╰───────────┴─────────────┴────────────────────────╯\n\nAvailable GPU Types:\n╭───────────────────┬─────────────┬────────────┬──────────────╮\n│ GPU Name          │ Memory (GB) │ Category   │ Architecture │\n├───────────────────┼─────────────┼────────────┼──────────────┤\n│ RTX 4090          │          24 │ consumer   │ Ada Lovelace │\n│ A100 80GB         │          80 │ datacenter │ Ampere       │\n│ H100 80GB         │          80 │ datacenter │ Hopper       │\n╰───────────────────┴─────────────┴────────────┴──────────────╯\n```\n\n## Calculation Formulas\n\n### Inference Memory\n```\nInference Memory = Model Weights × 1.2\n```\nIncludes model weights and KV cache overhead.\n\n### KV Cache Memory\n```\nKV Cache (GB) = 2 × Batch_Size × Sequence_Length × Head_Dim × Num_KV_Heads × Num_Layers × Precision ÷ 1,073,741,824\n```\n- Head_Dim = hidden_size ÷ num_attention_heads\n- Num_KV_Heads = config.num_key_value_heads (if present) else num_attention_heads\n- Automatically supports MHA, MQA, and GQA via model config; KV cache uses FP16/BF16 for quantized models\n\n### Training Memory (with Adam)\n```\nTraining Memory = Model Weights × 4 × 1.3\n```\n- 4x factor: Model weights (1x) + Gradients (1x) + Adam optimizer states (2x)\n- 1.3x factor: 30% additional overhead (activation caching, etc.)\n\n### LoRA Fine-tuning Memory\n```\nLoRA Memory = (Model Weights + LoRA Parameter Overhead) × 1.2\n```\nLoRA parameter overhead calculated based on rank and target module ratio.\n\n## Advanced Features\n\n### Configuration System\n\nExternal JSON configuration files for maximum flexibility:\n\n- **`data_types.json`** - Add custom quantization formats\n- **`gpu_types.json`** - Define new GPU models and specifications  \n- **`display_settings.json`** - Customize UI appearance and limits\n\n```bash\n# Use custom config directory\nhf-vram-calc --config-dir ./custom_config model_name\n\n# Add custom data type example (data_types.json)\n{\n  \"my_custom_int2\": {\n    \"bytes_per_param\": 0.25,\n    \"description\": \"Custom 2-bit quantization\"\n  }\n}\n```\n\n### Memory Calculation Details\n\n| Scenario | Formula | Explanation |\n|----------|---------|-------------|\n| **Inference** | Model × 1.2 | Includes KV cache and activation overhead |\n| **Training** | Model × 4 × 1.3 | Weights(1x) + Gradients(1x) + Adam(2x) + 30% overhead |\n| **LoRA** | (Model + LoRA_params×4) × 1.2 | Base model + trainable parameters with optimizer |\n\n### Parallelization Efficiency\n\n- **TP (Tensor Parallel)**: Near-linear scaling, slight communication overhead\n- **PP (Pipeline Parallel)**: Good efficiency, pipeline bubble ~10-15%  \n- **EP (Expert Parallel)**: MoE-specific, depends on expert routing efficiency\n- **DP (Data Parallel)**: No memory reduction per GPU, full model replica\n\n## Supported Architectures\n\n### Fully Supported ✅\n- **GPT Family**: GPT-2, GPT-3, GPT-4, GPT-NeoX, etc.\n- **LLaMA Family**: LLaMA, LLaMA-2, Code Llama, Vicuna, etc.\n- **Mistral Family**: Mistral 7B, Mixtral 8x7B (MoE), etc.\n- **Other Transformers**: BERT, RoBERTa, T5, FLAN-T5, etc.\n- **New Architectures**: DeepSeek, Qwen, ChatGLM, Baichuan, etc.\n\n### Architecture Detection\n- **Automatic field mapping** for different config.json formats\n- **Fallback support** for uncommon architectures\n- **MoE handling** for Mixture-of-Experts models\n\n## Accuracy \u0026 Limitations\n\n### ✅ Highly Accurate For:\n- **Parameter counting** (exact calculation)\n- **Memory estimation** (within 5-10% of actual)\n- **Parallelization ratios** (theoretical maximum)\n\n### ⚠️ Considerations:\n- **Activation memory** varies with sequence length and optimization\n- **Real-world efficiency** may differ due to framework overhead  \n- **Quantization accuracy** depends on specific implementation\n- **MoE models** require expert routing consideration\n\n## Build \u0026 Development\n\nBuilt with modern Python tooling:\n- **uv**: Fast Python package management and building\n- **Rich**: Professional terminal interface\n- **Requests**: HTTP client for model config fetching\n- **JSON configuration**: Flexible external configuration system\n\nFor development setup, see: [BUILD.md](BUILD.md)\n\n## Contributing\n\nWe welcome contributions! Areas for improvement:\n\n- 🔧 **New quantization formats** (add to `data_types.json`)\n- 🎮 **GPU models** (update `gpu_types.json`)  \n- 📊 **Architecture support** (enhance config parsing)\n- 🚀 **Performance optimizations**\n- 📚 **Documentation improvements**\n- 🧪 **Test coverage expansion**\n\n## See Also\n\n- 📚 **[BUILD.md](BUILD.md)** - Complete build and installation guide\n- ⚙️ **[CONFIG_GUIDE.md](CONFIG_GUIDE.md)** - Configuration customization details\n- 📝 **Examples in help**: `hf-vram-calc --help` for usage examples\n\n## Version History\n\n- **v1.0.0**: Complete rewrite with uv build, smart dtype detection, professional UI\n- **v0.x**: Legacy single-file version (deprecated)\n\n## License\n\nMIT License - see LICENSE file for details.\n\n---\n\n**Made with ❤️ for the ML community** | Built with [uv](https://github.com/astral-sh/uv) and [Rich](https://github.com/Textualize/rich)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoe0731%2Fhf_vram_calc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoe0731%2Fhf_vram_calc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoe0731%2Fhf_vram_calc/lists"}