{"id":13407474,"url":"https://github.com/oobabooga/text-generation-webui","last_synced_at":"2026-03-07T16:07:03.319Z","repository":{"id":65283926,"uuid":"580642043","full_name":"oobabooga/text-generation-webui","owner":"oobabooga","description":"A Gradio web UI for Large Language Models with support for multiple inference backends.","archived":false,"fork":false,"pushed_at":"2025-03-03T20:09:55.000Z","size":29785,"stargazers_count":42801,"open_issues_count":213,"forks_count":5520,"subscribers_count":343,"default_branch":"main","last_synced_at":"2025-03-09T03:30:52.995Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://oobabooga.gumroad.com/l/deep_reason","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oobabooga.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"patreon":"oobabooga"}},"created_at":"2022-12-21T04:17:37.000Z","updated_at":"2025-03-09T02:46:26.000Z","dependencies_parsed_at":"2023-10-15T17:13:34.516Z","dependency_job_id":"4cfd49dc-fff0-45fe-b9e1-2f183a563eae","html_url":"https://github.com/oobabooga/text-generation-webui","commit_stats":{"total_commits":3664,"total_committers":354,"mean_commits":"10.350282485875706","dds":"0.24890829694323147","last_synced_commit":"cc8c7ed2093cbc747e7032420eae14b5b3c30311"},"previous_names":[],"tags_count":57,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oobabooga%2Ftext-generation-webui","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oobabooga%2Ftext-generation-webui/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oobabooga%2Ftext-generation-webui/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oobabooga%2Ftext-generation-webui/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oobabooga","download_url":"https://codeload.github.com/oobabooga/text-generation-webui/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243521128,"owners_count":20304183,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-30T20:00:40.940Z","updated_at":"2026-01-18T04:34:19.462Z","avatar_url":"https://github.com/oobabooga.png","language":"Python","funding_links":["https://patreon.com/oobabooga"],"categories":["Toolkits","Python","Others","Recursos","AI Сервисы для разработки ПО","Tools for Self-Hosting","others","精选开源项目合集","推理 Inference","Langchain","Chat Bots ([home](#awesome-llm))","A01_文本生成_文本对话","Project List","Large Language Models (LLMs)","HarmonyOS","Applications","[:robot: machine-learning]([robot-machine-learning)](\u003chttps://github.com/stars/ketsapiwiq/lists/robot-machine-learning\u003e))","网页用户界面工具","🆕 New and Emerging AI Tools","Writing","Open-Source Local LLM Projects","Repos","Application Recommendation","📚 Contents","User Interfaces","Table of Contents","Inference UI","App","12. User Interfaces \u0026 Self-hosted Platforms","One-Click Runners \u0026 Installers (15)","LLMs Backend","Notable Open-Source Projects and Repositories","Applied AI","8. Inference Engines","🛠️ Developer Infrastructure","Browse The Shelves","Large Language Models | 大语言模型"],"sub_categories":["Others","Web и Desktop клиенты для чатинга с LLM через API","LLMs","GPT开源平替机器人🔥🔥🔥","大语言对话模型及数据","\u003cspan id=\"tool\"\u003eLLM (LLM \u0026 Tool)\u003c/span\u003e","Contribute to our Repository","Windows Manager","提示语（魔法）","Other Cloud Provider Credits","Prompt Generator","GPT开源平替机器人","🧠 AI Applications","Web Interfaces","LLM GUI","Chat Apps","Desktop / Local","Model Deployment \u0026 Local Inference","Local LLM developer tools","Local Models | 本地模型"],"readme":"\u003cdiv align=\"center\" markdown=\"1\"\u003e\n   \u003csup\u003eSpecial thanks to:\u003c/sup\u003e\n   \u003cbr\u003e\n   \u003cbr\u003e\n   \u003ca href=\"https://go.warp.dev/text-generation-webui\"\u003e\n      \u003cimg alt=\"Warp sponsorship\" width=\"400\" src=\"https://raw.githubusercontent.com/warpdotdev/brand-assets/refs/heads/main/Github/Sponsor/Warp-Github-LG-02.png\"\u003e\n   \u003c/a\u003e\n\n### [Warp, built for coding with multiple AI agents](https://go.warp.dev/text-generation-webui)\n[Available for macOS, Linux, \u0026 Windows](https://go.warp.dev/text-generation-webui)\u003cbr\u003e\n\u003c/div\u003e\n\u003chr\u003e\n\n# Text Generation Web UI\n\nA Gradio web UI for Large Language Models.\n\n[Try the Deep Reason extension](https://oobabooga.gumroad.com/l/deep_reason)\n\n|![Image1](https://github.com/oobabooga/screenshots/raw/main/INSTRUCT-3.5.png) | ![Image2](https://github.com/oobabooga/screenshots/raw/main/CHAT-3.5.png) |\n|:---:|:---:|\n|![Image1](https://github.com/oobabooga/screenshots/raw/main/DEFAULT-3.5.png) | ![Image2](https://github.com/oobabooga/screenshots/raw/main/PARAMETERS-3.5.png) |\n\n## 🔥 News\n\n- The project now supports **image generation**! Including Z-Image-Turbo, 4bit/8bit quantization, `torch.compile`, and LLM-generated prompt variations ([tutorial](https://github.com/oobabooga/text-generation-webui/wiki/Image-Generation-Tutorial)).\n\n## Features\n\n- Supports multiple local text generation backends, including [llama.cpp](https://github.com/ggerganov/llama.cpp), [Transformers](https://github.com/huggingface/transformers), [ExLlamaV3](https://github.com/turboderp-org/exllamav3), [ExLlamaV2](https://github.com/turboderp-org/exllamav2), and [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) (the latter via its own [Dockerfile](https://github.com/oobabooga/text-generation-webui/blob/main/docker/TensorRT-LLM/Dockerfile)).\n- Easy setup: Choose between **portable builds** (zero setup, just unzip and run) for GGUF models on Windows/Linux/macOS, or the one-click installer that creates a self-contained `installer_files` directory.\n- 100% offline and private, with zero telemetry, external resources, or remote update requests.\n- **File attachments**: Upload text files, PDF documents, and .docx documents to talk about their contents.\n- **Vision (multimodal models)**: Attach images to messages for visual understanding ([tutorial](https://github.com/oobabooga/text-generation-webui/wiki/Multimodal-Tutorial)).\n- **Image generation**: A dedicated tab for `diffusers` models like **Z-Image-Turbo**. Features 4-bit/8-bit quantization and a persistent gallery with metadata ([tutorial](https://github.com/oobabooga/text-generation-webui/wiki/Image-Generation-Tutorial)).\n- **Web search**: Optionally search the internet with LLM-generated queries to add context to the conversation.\n- Aesthetic UI with dark and light themes.\n- Syntax highlighting for code blocks and LaTeX rendering for mathematical expressions.\n- `instruct` mode for instruction-following (like ChatGPT), and `chat-instruct`/`chat` modes for talking to custom characters.\n- Automatic prompt formatting using Jinja2 templates. You don't need to ever worry about prompt formats.\n- Edit messages, navigate between message versions, and branch conversations at any point.\n- Multiple sampling parameters and generation options for sophisticated text generation control.\n- Switch between different models in the UI without restarting.\n- Automatic GPU layers for GGUF models (on NVIDIA GPUs).\n- Free-form text generation in the Notebook tab without being limited to chat turns.\n- OpenAI-compatible API with Chat and Completions endpoints, including tool-calling support – see [examples](https://github.com/oobabooga/text-generation-webui/wiki/12-%E2%80%90-OpenAI-API#examples).\n- Extension support, with numerous built-in and user-contributed extensions available. See the [wiki](https://github.com/oobabooga/text-generation-webui/wiki/07-%E2%80%90-Extensions) and [extensions directory](https://github.com/oobabooga/text-generation-webui-extensions) for details.\n\n## How to install\n\n#### ✅ Option 1: Portable builds (get started in 1 minute)\n\nNo installation needed – just download, unzip and run. All dependencies included.\n\nCompatible with GGUF (llama.cpp) models on Windows, Linux, and macOS.\n\nDownload from here: **https://github.com/oobabooga/text-generation-webui/releases**\n\n#### Option 2: Manual portable install with venv\n\nVery fast setup that should work on any Python 3.9+:\n\n```bash\n# Clone repository\ngit clone https://github.com/oobabooga/text-generation-webui\ncd text-generation-webui\n\n# Create virtual environment\npython -m venv venv\n\n# Activate virtual environment\n# On Windows:\nvenv\\Scripts\\activate\n# On macOS/Linux:\nsource venv/bin/activate\n\n# Install dependencies (choose appropriate file under requirements/portable for your hardware)\npip install -r requirements/portable/requirements.txt --upgrade\n\n# Launch server (basic command)\npython server.py --portable --api --auto-launch\n\n# When done working, deactivate\ndeactivate\n```\n\n#### Option 3: One-click installer\n\nFor users who need additional backends (ExLlamaV3, Transformers) or extensions (TTS, voice input, translation, etc). Requires ~10GB disk space and downloads PyTorch.\n\n1. Clone the repository, or [download its source code](https://github.com/oobabooga/text-generation-webui/archive/refs/heads/main.zip) and extract it.\n2. Run the startup script for your OS: `start_windows.bat`, `start_linux.sh`, or `start_macos.sh`.\n3. When prompted, select your GPU vendor.\n4. After installation, open `http://127.0.0.1:7860` in your browser.\n\nTo restart the web UI later, run the same `start_` script.\n\nYou can pass command-line flags directly (e.g., `./start_linux.sh --help`), or add them to `user_data/CMD_FLAGS.txt` (e.g., `--api` to enable the API).\n\nTo update, run the update script for your OS: `update_wizard_windows.bat`, `update_wizard_linux.sh`, or `update_wizard_macos.sh`.\n\nTo reinstall with a fresh Python environment, delete the `installer_files` folder and run the `start_` script again.\n\n\u003cdetails\u003e\n\u003csummary\u003e\nOne-click installer details\n\u003c/summary\u003e\n\n### One-click-installer\n\nThe script uses Miniforge to set up a Conda environment in the `installer_files` folder.\n\nIf you ever need to install something manually in the `installer_files` environment, you can launch an interactive shell using the cmd script: `cmd_linux.sh`, `cmd_windows.bat`, or `cmd_macos.sh`.\n\n* There is no need to run any of those scripts (`start_`, `update_wizard_`, or `cmd_`) as admin/root.\n* To install requirements for extensions, it is recommended to use the update wizard script with the \"Install/update extensions requirements\" option. At the end, this script will install the main requirements for the project to make sure that they take precedence in case of version conflicts.\n* For automated installation, you can use the `GPU_CHOICE`, `LAUNCH_AFTER_INSTALL`, and `INSTALL_EXTENSIONS` environment variables. For instance: `GPU_CHOICE=A LAUNCH_AFTER_INSTALL=FALSE INSTALL_EXTENSIONS=TRUE ./start_linux.sh`.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\nManual full installation with conda or docker\n\u003c/summary\u003e\n\n### Full installation with Conda\n\n#### 0. Install Conda\n\nhttps://github.com/conda-forge/miniforge\n\nOn Linux or WSL, Miniforge can be automatically installed with these two commands:\n\n```\ncurl -sL \"https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh\" \u003e \"Miniforge3.sh\"\nbash Miniforge3.sh\n```\n\nFor other platforms, download from: https://github.com/conda-forge/miniforge/releases/latest\n\n#### 1. Create a new conda environment\n\n```\nconda create -n textgen python=3.11\nconda activate textgen\n```\n\n#### 2. Install Pytorch\n\n| System | GPU | Command |\n|--------|---------|---------|\n| Linux/WSL | NVIDIA | `pip3 install torch==2.7.1 --index-url https://download.pytorch.org/whl/cu128` |\n| Linux/WSL | CPU only | `pip3 install torch==2.7.1 --index-url https://download.pytorch.org/whl/cpu` |\n| Linux | AMD | `pip3 install torch==2.7.1 --index-url https://download.pytorch.org/whl/rocm6.2.4` |\n| MacOS + MPS | Any | `pip3 install torch==2.7.1` |\n| Windows | NVIDIA | `pip3 install torch==2.7.1 --index-url https://download.pytorch.org/whl/cu128` |\n| Windows | CPU only | `pip3 install torch==2.7.1` |\n\nThe up-to-date commands can be found here: https://pytorch.org/get-started/locally/.\n\nIf you need `nvcc` to compile some library manually, you will additionally need to install this:\n\n```\nconda install -y -c \"nvidia/label/cuda-12.8.1\" cuda\n```\n\n#### 3. Install the web UI\n\n```\ngit clone https://github.com/oobabooga/text-generation-webui\ncd text-generation-webui\npip install -r requirements/full/\u003crequirements file according to table below\u003e\n```\n\nRequirements file to use:\n\n| GPU | CPU | requirements file to use |\n|--------|---------|---------|\n| NVIDIA | has AVX2 | `requirements.txt` |\n| NVIDIA | no AVX2 | `requirements_noavx2.txt` |\n| AMD | has AVX2 | `requirements_amd.txt` |\n| AMD | no AVX2 | `requirements_amd_noavx2.txt` |\n| CPU only | has AVX2 | `requirements_cpu_only.txt` |\n| CPU only | no AVX2 | `requirements_cpu_only_noavx2.txt` |\n| Apple | Intel | `requirements_apple_intel.txt` |\n| Apple | Apple Silicon | `requirements_apple_silicon.txt` |\n\n### Start the web UI\n\n```\nconda activate textgen\ncd text-generation-webui\npython server.py\n```\n\nThen browse to\n\n`http://127.0.0.1:7860`\n\n#### Manual install\n\nThe `requirements*.txt` above contain various wheels precompiled through GitHub Actions. If you wish to compile things manually, or if you need to because no suitable wheels are available for your hardware, you can use `requirements_nowheels.txt` and then install your desired loaders manually.\n\n### Alternative: Docker\n\n```\nFor NVIDIA GPU:\nln -s docker/{nvidia/Dockerfile,nvidia/docker-compose.yml,.dockerignore} .\nFor AMD GPU:\nln -s docker/{amd/Dockerfile,amd/docker-compose.yml,.dockerignore} .\nFor Intel GPU:\nln -s docker/{intel/Dockerfile,amd/docker-compose.yml,.dockerignore} .\nFor CPU only\nln -s docker/{cpu/Dockerfile,cpu/docker-compose.yml,.dockerignore} .\ncp docker/.env.example .env\n#Create logs/cache dir :\nmkdir -p user_data/logs user_data/cache\n# Edit .env and set:\n#   TORCH_CUDA_ARCH_LIST based on your GPU model\n#   APP_RUNTIME_GID      your host user's group id (run `id -g` in a terminal)\n#   BUILD_EXTENIONS      optionally add comma separated list of extensions to build\n# Edit user_data/CMD_FLAGS.txt and add in it the options you want to execute (like --listen --cpu)\n#\ndocker compose up --build\n```\n\n* You need to have Docker Compose v2.17 or higher installed. See [this guide](https://github.com/oobabooga/text-generation-webui/wiki/09-%E2%80%90-Docker) for instructions.\n* For additional docker files, check out [this repository](https://github.com/Atinoda/text-generation-webui-docker).\n\n### Updating the requirements\n\nFrom time to time, the `requirements*.txt` change. To update, use these commands:\n\n```\nconda activate textgen\ncd text-generation-webui\npip install -r \u003crequirements file that you have used\u003e --upgrade\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\nList of command-line flags\n\u003c/summary\u003e\n\n```txt\nusage: server.py [-h] [--multi-user] [--model MODEL] [--lora LORA [LORA ...]] [--model-dir MODEL_DIR] [--lora-dir LORA_DIR] [--model-menu] [--settings SETTINGS]\n                 [--extensions EXTENSIONS [EXTENSIONS ...]] [--verbose] [--idle-timeout IDLE_TIMEOUT] [--loader LOADER] [--ctx-size N] [--cache-type N] [--model-draft MODEL_DRAFT]\n                 [--draft-max DRAFT_MAX] [--gpu-layers-draft GPU_LAYERS_DRAFT] [--device-draft DEVICE_DRAFT] [--ctx-size-draft CTX_SIZE_DRAFT] [--gpu-layers N] [--mmproj MMPROJ] [--streaming-llm]\n                 [--tensor-split TENSOR_SPLIT] [--row-split] [--no-mmap] [--mlock] [--no-kv-offload] [--batch-size BATCH_SIZE] [--threads THREADS] [--threads-batch THREADS_BATCH] [--numa]\n                 [--extra-flags EXTRA_FLAGS] [--cpu] [--cpu-memory CPU_MEMORY] [--disk] [--disk-cache-dir DISK_CACHE_DIR] [--load-in-8bit] [--bf16] [--no-cache] [--trust-remote-code]\n                 [--force-safetensors] [--no_use_fast] [--attn-implementation IMPLEMENTATION] [--load-in-4bit] [--use_double_quant] [--compute_dtype COMPUTE_DTYPE] [--quant_type QUANT_TYPE]\n                 [--enable-tp] [--tp-backend TP_BACKEND] [--gpu-split GPU_SPLIT] [--autosplit] [--cfg-cache] [--no_flash_attn] [--no_xformers] [--no_sdpa] [--num_experts_per_token N] [--cpp-runner]\n                 [--deepspeed] [--nvme-offload-dir NVME_OFFLOAD_DIR] [--local_rank LOCAL_RANK] [--alpha_value ALPHA_VALUE] [--rope_freq_base ROPE_FREQ_BASE] [--compress_pos_emb COMPRESS_POS_EMB]\n                 [--listen] [--listen-port LISTEN_PORT] [--listen-host LISTEN_HOST] [--share] [--auto-launch] [--gradio-auth GRADIO_AUTH] [--gradio-auth-path GRADIO_AUTH_PATH]\n                 [--ssl-keyfile SSL_KEYFILE] [--ssl-certfile SSL_CERTFILE] [--subpath SUBPATH] [--old-colors] [--portable] [--api] [--public-api] [--public-api-id PUBLIC_API_ID] [--api-port API_PORT]\n                 [--api-key API_KEY] [--admin-key ADMIN_KEY] [--api-enable-ipv6] [--api-disable-ipv4] [--nowebui]\n\nText Generation Web UI\n\noptions:\n  -h, --help                                show this help message and exit\n\nBasic settings:\n  --multi-user                              Multi-user mode. Chat histories are not saved or automatically loaded. Warning: this is likely not safe for sharing publicly.\n  --model MODEL                             Name of the model to load by default.\n  --lora LORA [LORA ...]                    The list of LoRAs to load. If you want to load more than one LoRA, write the names separated by spaces.\n  --model-dir MODEL_DIR                     Path to directory with all the models.\n  --lora-dir LORA_DIR                       Path to directory with all the loras.\n  --model-menu                              Show a model menu in the terminal when the web UI is first launched.\n  --settings SETTINGS                       Load the default interface settings from this yaml file. See user_data/settings-template.yaml for an example. If you create a file called\n                                            user_data/settings.yaml, this file will be loaded by default without the need to use the --settings flag.\n  --extensions EXTENSIONS [EXTENSIONS ...]  The list of extensions to load. If you want to load more than one extension, write the names separated by spaces.\n  --verbose                                 Print the prompts to the terminal.\n  --idle-timeout IDLE_TIMEOUT               Unload model after this many minutes of inactivity. It will be automatically reloaded when you try to use it again.\n\nModel loader:\n  --loader LOADER                           Choose the model loader manually, otherwise, it will get autodetected. Valid options: Transformers, llama.cpp, ExLlamav3_HF, ExLlamav2_HF, ExLlamav2,\n                                            TensorRT-LLM.\n\nContext and cache:\n  --ctx-size N, --n_ctx N, --max_seq_len N  Context size in tokens.\n  --cache-type N, --cache_type N            KV cache type; valid options: llama.cpp - fp16, q8_0, q4_0; ExLlamaV2 - fp16, fp8, q8, q6, q4; ExLlamaV3 - fp16, q2 to q8 (can specify k_bits and v_bits\n                                            separately, e.g. q4_q8).\n\nSpeculative decoding:\n  --model-draft MODEL_DRAFT                 Path to the draft model for speculative decoding.\n  --draft-max DRAFT_MAX                     Number of tokens to draft for speculative decoding.\n  --gpu-layers-draft GPU_LAYERS_DRAFT       Number of layers to offload to the GPU for the draft model.\n  --device-draft DEVICE_DRAFT               Comma-separated list of devices to use for offloading the draft model. Example: CUDA0,CUDA1\n  --ctx-size-draft CTX_SIZE_DRAFT           Size of the prompt context for the draft model. If 0, uses the same as the main model.\n\nllama.cpp:\n  --gpu-layers N, --n-gpu-layers N          Number of layers to offload to the GPU.\n  --mmproj MMPROJ                           Path to the mmproj file for vision models.\n  --streaming-llm                           Activate StreamingLLM to avoid re-evaluating the entire prompt when old messages are removed.\n  --tensor-split TENSOR_SPLIT               Split the model across multiple GPUs. Comma-separated list of proportions. Example: 60,40.\n  --row-split                               Split the model by rows across GPUs. This may improve multi-gpu performance.\n  --no-mmap                                 Prevent mmap from being used.\n  --mlock                                   Force the system to keep the model in RAM.\n  --no-kv-offload                           Do not offload the K, Q, V to the GPU. This saves VRAM but reduces the performance.\n  --batch-size BATCH_SIZE                   Maximum number of prompt tokens to batch together when calling llama_eval.\n  --threads THREADS                         Number of threads to use.\n  --threads-batch THREADS_BATCH             Number of threads to use for batches/prompt processing.\n  --numa                                    Activate NUMA task allocation for llama.cpp.\n  --extra-flags EXTRA_FLAGS                 Extra flags to pass to llama-server. Format: \"flag1=value1,flag2,flag3=value3\". Example: \"override-tensor=exps=CPU\"\n\nTransformers/Accelerate:\n  --cpu                                     Use the CPU to generate text. Warning: Training on CPU is extremely slow.\n  --cpu-memory CPU_MEMORY                   Maximum CPU memory in GiB. Use this for CPU offloading.\n  --disk                                    If the model is too large for your GPU(s) and CPU combined, send the remaining layers to the disk.\n  --disk-cache-dir DISK_CACHE_DIR           Directory to save the disk cache to. Defaults to \"user_data/cache\".\n  --load-in-8bit                            Load the model with 8-bit precision (using bitsandbytes).\n  --bf16                                    Load the model with bfloat16 precision. Requires NVIDIA Ampere GPU.\n  --no-cache                                Set use_cache to False while generating text. This reduces VRAM usage slightly, but it comes at a performance cost.\n  --trust-remote-code                       Set trust_remote_code=True while loading the model. Necessary for some models.\n  --force-safetensors                       Set use_safetensors=True while loading the model. This prevents arbitrary code execution.\n  --no_use_fast                             Set use_fast=False while loading the tokenizer (it's True by default). Use this if you have any problems related to use_fast.\n  --attn-implementation IMPLEMENTATION      Attention implementation. Valid options: sdpa, eager, flash_attention_2.\n\nbitsandbytes 4-bit:\n  --load-in-4bit                            Load the model with 4-bit precision (using bitsandbytes).\n  --use_double_quant                        use_double_quant for 4-bit.\n  --compute_dtype COMPUTE_DTYPE             compute dtype for 4-bit. Valid options: bfloat16, float16, float32.\n  --quant_type QUANT_TYPE                   quant_type for 4-bit. Valid options: nf4, fp4.\n\nExLlamaV3:\n  --enable-tp, --enable_tp                  Enable Tensor Parallelism (TP) to split the model across GPUs.\n  --tp-backend TP_BACKEND                   The backend for tensor parallelism. Valid options: native, nccl. Default: native.\n\nExLlamaV2:\n  --gpu-split GPU_SPLIT                     Comma-separated list of VRAM (in GB) to use per GPU device for model layers. Example: 20,7,7.\n  --autosplit                               Autosplit the model tensors across the available GPUs. This causes --gpu-split to be ignored.\n  --cfg-cache                               ExLlamav2_HF: Create an additional cache for CFG negative prompts. Necessary to use CFG with that loader.\n  --no_flash_attn                           Force flash-attention to not be used.\n  --no_xformers                             Force xformers to not be used.\n  --no_sdpa                                 Force Torch SDPA to not be used.\n  --num_experts_per_token N                 Number of experts to use for generation. Applies to MoE models like Mixtral.\n\nTensorRT-LLM:\n  --cpp-runner                              Use the ModelRunnerCpp runner, which is faster than the default ModelRunner but doesn't support streaming yet.\n\nDeepSpeed:\n  --deepspeed                               Enable the use of DeepSpeed ZeRO-3 for inference via the Transformers integration.\n  --nvme-offload-dir NVME_OFFLOAD_DIR       DeepSpeed: Directory to use for ZeRO-3 NVME offloading.\n  --local_rank LOCAL_RANK                   DeepSpeed: Optional argument for distributed setups.\n\nRoPE:\n  --alpha_value ALPHA_VALUE                 Positional embeddings alpha factor for NTK RoPE scaling. Use either this or compress_pos_emb, not both.\n  --rope_freq_base ROPE_FREQ_BASE           If greater than 0, will be used instead of alpha_value. Those two are related by rope_freq_base = 10000 * alpha_value ^ (64 / 63).\n  --compress_pos_emb COMPRESS_POS_EMB       Positional embeddings compression factor. Should be set to (context length) / (model's original context length). Equal to 1/rope_freq_scale.\n\nGradio:\n  --listen                                  Make the web UI reachable from your local network.\n  --listen-port LISTEN_PORT                 The listening port that the server will use.\n  --listen-host LISTEN_HOST                 The hostname that the server will use.\n  --share                                   Create a public URL. This is useful for running the web UI on Google Colab or similar.\n  --auto-launch                             Open the web UI in the default browser upon launch.\n  --gradio-auth GRADIO_AUTH                 Set Gradio authentication password in the format \"username:password\". Multiple credentials can also be supplied with \"u1:p1,u2:p2,u3:p3\".\n  --gradio-auth-path GRADIO_AUTH_PATH       Set the Gradio authentication file path. The file should contain one or more user:password pairs in the same format as above.\n  --ssl-keyfile SSL_KEYFILE                 The path to the SSL certificate key file.\n  --ssl-certfile SSL_CERTFILE               The path to the SSL certificate cert file.\n  --subpath SUBPATH                         Customize the subpath for gradio, use with reverse proxy\n  --old-colors                              Use the legacy Gradio colors, before the December/2024 update.\n  --portable                                Hide features not available in portable mode like training.\n\nAPI:\n  --api                                     Enable the API extension.\n  --public-api                              Create a public URL for the API using Cloudfare.\n  --public-api-id PUBLIC_API_ID             Tunnel ID for named Cloudflare Tunnel. Use together with public-api option.\n  --api-port API_PORT                       The listening port for the API.\n  --api-key API_KEY                         API authentication key.\n  --admin-key ADMIN_KEY                     API authentication key for admin tasks like loading and unloading models. If not set, will be the same as --api-key.\n  --api-enable-ipv6                         Enable IPv6 for the API\n  --api-disable-ipv4                        Disable IPv4 for the API\n  --nowebui                                 Do not launch the Gradio UI. Useful for launching the API in standalone mode.\n```\n\n\u003c/details\u003e\n\n## Downloading models\n\nModels should be placed in the folder `text-generation-webui/user_data/models`. They are usually downloaded from [Hugging Face](https://huggingface.co/models?pipeline_tag=text-generation\u0026sort=downloads\u0026search=gguf).\n\nTo check if a GGUF model will fit in your hardware before downloading it, you can use this tool I created:\n\n[Accurate GGUF VRAM Calculator](https://huggingface.co/spaces/oobabooga/accurate-gguf-vram-calculator)\n\n* GGUF models are a single file and should be placed directly into `user_data/models`. Example:\n\n```\ntext-generation-webui\n└── user_data\n    └── models\n        └── llama-2-13b-chat.Q4_K_M.gguf\n```\n\n* The remaining model types (like 16-bit Transformers models and EXL3 models) are made of several files and must be placed in a subfolder. Example:\n\n```\ntext-generation-webui\n└── user_data\n    └── models\n        └── lmsys_vicuna-33b-v1.3\n            ├── config.json\n            ├── generation_config.json\n            ├── pytorch_model-00001-of-00007.bin\n            ├── pytorch_model-00002-of-00007.bin\n            ├── pytorch_model-00003-of-00007.bin\n            ├── pytorch_model-00004-of-00007.bin\n            ├── pytorch_model-00005-of-00007.bin\n            ├── pytorch_model-00006-of-00007.bin\n            ├── pytorch_model-00007-of-00007.bin\n            ├── pytorch_model.bin.index.json\n            ├── special_tokens_map.json\n            ├── tokenizer_config.json\n            └── tokenizer.model\n```\n\nIn both cases, you can use the \"Model\" tab of the UI to download the model from Hugging Face automatically. It is also possible to download it via the command-line with:\n\n```\npython download-model.py organization/model\n```\n\nRun `python download-model.py --help` to see all the options.\n\n## Documentation\n\nhttps://github.com/oobabooga/text-generation-webui/wiki\n\n## Google Colab notebook\n\nhttps://colab.research.google.com/github/oobabooga/text-generation-webui/blob/main/Colab-TextGen-GPU.ipynb\n\n## Community\n\nhttps://www.reddit.com/r/Oobabooga/\n\n## Acknowledgments\n\n- In August 2023, [Andreessen Horowitz](https://a16z.com/) (a16z) provided a generous grant to encourage and support my independent work on this project. I am **extremely** grateful for their trust and recognition.\n- This project was inspired by [AUTOMATIC1111/stable-diffusion-webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui) and wouldn't exist without it.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foobabooga%2Ftext-generation-webui","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foobabooga%2Ftext-generation-webui","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foobabooga%2Ftext-generation-webui/lists"}