{"id":28384444,"url":"https://github.com/runpod-workers/worker-sglang","last_synced_at":"2025-06-25T23:30:50.813Z","repository":{"id":250656488,"uuid":"835035853","full_name":"runpod-workers/worker-sglang","owner":"runpod-workers","description":"SGLang is fast serving framework for large language models and vision language models.","archived":false,"fork":false,"pushed_at":"2025-02-10T03:03:45.000Z","size":68,"stargazers_count":23,"open_issues_count":6,"forks_count":9,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-30T09:27:31.286Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/runpod-workers.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-29T03:05:59.000Z","updated_at":"2025-05-08T06:26:22.000Z","dependencies_parsed_at":"2024-08-10T00:20:47.827Z","dependency_job_id":"9d3657bf-bfa7-4124-afda-181d6dee7f3e","html_url":"https://github.com/runpod-workers/worker-sglang","commit_stats":null,"previous_names":["pandyamarut/worker-sglang"],"tags_count":2,"template":false,"template_full_name":"runpod-workers/worker-template","purl":"pkg:github/runpod-workers/worker-sglang","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/runpod-workers%2Fworker-sglang","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/runpod-workers%2Fworker-sglang/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/runpod-workers%2Fworker-sglang/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/runpod-workers%2Fworker-sglang/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/runpod-workers","download_url":"https://codeload.github.com/runpod-workers/worker-sglang/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/runpod-workers%2Fworker-sglang/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261972489,"owners_count":23238536,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-30T08:38:40.946Z","updated_at":"2025-06-25T23:30:50.800Z","avatar_url":"https://github.com/runpod-workers.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n\u003ch1\u003e SgLang Worker\u003c/h1\u003e\n\n🚀 | SGLang is fast serving framework for large language models and vision language models.\n\u003c/div\u003e\n\n## RunPod Worker Images\n\nBelow is a summary of the available RunPod Worker images, categorized by image stability\n\n| Stable Image Tag                  | Development Image Tag             | \n-----------------------------------|-----------------------------------|\n `runpod/worker-sglang:v0.4.1stable` | `runpod/worker-sglang:v0.4.1dev` | \n\n\n## 📖 | Getting Started\n\n1. Clone this repository.\n2. Build a docker image - ```docker build -t \u003cyour_username\u003e:worker-sglang:v1 .```\n3. ```docker push \u003cyour_username\u003e:worker-sglang:v1```\n\n\n***Once you have built the Docker image and deployed the endpoint, you can use the code below to interact with the endpoint***: \n\n```\nimport runpod\n\nrunpod.api_key = \"your_runpod_api_key_found_under_settings\"\n\n# Initialize the endpoint\nendpoint = runpod.Endpoint(\"ENDPOINT_ID\")\n\n# Run the endpoint with input data\nrun_request = endpoint.run({\"your_model_input_key\": \"your_model_input_value\"})\n\n# Check the status of the endpoint run request\nprint(run_request.status())\n\n# Get the output of the endpoint run request, blocking until the run is complete\nprint(run_request.output()) \n```\n\n### OpenAI compatible API\n```python\nfrom openai import OpenAI\nimport os\n\n# Initialize the OpenAI Client with your RunPod API Key and Endpoint URL\nclient = OpenAI(\n    api_key=os.getenv(\"RUNPOD_API_KEY\"),\n    base_url=f\"https://api.runpod.ai/v2/\u003cendpoint_id\u003e/openai/v1\",\n)\n```\n\n`Chat Completions (Non-Streaming)`\n```python\nresponse = client.chat.completions.create(\n    model=\"meta-llama/Meta-Llama-3-8B-Instruct\",\n    messages=[{\"role\": \"user\", \"content\": \"Give a two lines on Planet Earth ?\"}],\n    temperature=0,\n    max_tokens=100,\n    \n)\nprint(f\"Response: {response}\")\n```\n\n`Chat Completions (Streaming)`\n```python\nresponse_stream = client.chat.completions.create(\n    model=\"meta-llama/Meta-Llama-3-8B-Instruct\",\n    messages=[{\"role\": \"user\", \"content\": \"Give a two lines on Planet Earth ?\"}],\n    temperature=0,\n    max_tokens=100,\n    stream=True\n    \n)\nfor response in response_stream:\n    print(response.choices[0].delta.content or \"\", end=\"\", flush=True)\n```\n\n\n\n## SGLang Server Configuration\nWhen launching an endpoint, you can configure the SGLang server using environment variables. These variables allow you to customize various aspects of the server's behavior without modifying the code.\n\n### How to Use\n\nDefine these variables in your endpoint template.\nThe SGLang server will read these variables at startup and configure itself accordingly.\nIf a variable is not set, the server will use its default value.\n\n### Available Environment Variables\nThe following table lists all available environment variables for configuring the SGLang server:\n\n\n| Environment Variable | Description | Default | Options |\n|----------------------|-------------|---------|---------|\n| `MODEL_PATH` | Path of the model weights | \"meta-llama/Meta-Llama-3-8B-Instruct\" | Local folder or Hugging Face repo ID |\n| `HOST` | Host of the server | \"0.0.0.0\" | |\n| `PORT` | Port of the server | 30000 | |\n| `TOKENIZER_PATH` | Path of the tokenizer | | |\n| `ADDITIONAL_PORTS` | Additional ports for the server | | |\n| `TOKENIZER_MODE` | Tokenizer mode | \"auto\" | \"auto\", \"slow\" |\n| `LOAD_FORMAT` | Format of model weights to load | \"auto\" | \"auto\", \"pt\", \"safetensors\", \"npcache\", \"dummy\" |\n| `DTYPE` | Data type for weights and activations | \"auto\" | \"auto\", \"half\", \"float16\", \"bfloat16\", \"float\", \"float32\" |\n| `CONTEXT_LENGTH` | Model's maximum context length | | |\n| `QUANTIZATION` | Quantization method | | \"awq\", \"fp8\", \"gptq\", \"marlin\", \"gptq_marlin\", \"awq_marlin\", \"squeezellm\", \"bitsandbytes\" |\n| `SERVED_MODEL_NAME` | Override model name in API | | |\n| `CHAT_TEMPLATE` | Chat template name or path | | |\n| `MEM_FRACTION_STATIC` | Fraction of memory for static allocation | | |\n| `MAX_RUNNING_REQUESTS` | Maximum number of running requests | | |\n| `MAX_NUM_REQS` | Maximum requests in memory pool | | |\n| `MAX_TOTAL_TOKENS` | Maximum tokens in memory pool | | |\n| `CHUNKED_PREFILL_SIZE` | Max tokens in chunk for chunked prefill | | |\n| `MAX_PREFILL_TOKENS` | Max tokens in prefill batch | | |\n| `SCHEDULE_POLICY` | Request scheduling policy | | \"lpm\", \"random\", \"fcfs\", \"dfs-weight\" |\n| `SCHEDULE_CONSERVATIVENESS` | Conservativeness of schedule policy | | |\n| `TENSOR_PARALLEL_SIZE` | Tensor parallelism size | | |\n| `STREAM_INTERVAL` | Streaming interval in token length | | |\n| `RANDOM_SEED` | Random seed | | |\n| `LOG_LEVEL` | Logging level for all loggers | | |\n| `LOG_LEVEL_HTTP` | Logging level for HTTP server | | |\n| `API_KEY` | API key for the server | | |\n| `FILE_STORAGE_PTH` | Path of file storage in backend | | |\n| `DATA_PARALLEL_SIZE` | Data parallelism size | | |\n| `LOAD_BALANCE_METHOD` | Load balancing strategy | | \"round_robin\", \"shortest_queue\" |\n| `NCCL_INIT_ADDR` | NCCL init address for multi-node | | |\n| `NNODES` | Number of nodes | | |\n| `NODE_RANK` | Node rank | | |\n\n**Boolean Flags** (set to \"true\", \"1\", or \"yes\" to enable):\n\n| Flag | Description |\n|------|-------------|\n| `SKIP_TOKENIZER_INIT` | Skip tokenizer init |\n| `TRUST_REMOTE_CODE` | Allow custom models from Hub |\n| `LOG_REQUESTS` | Log inputs and outputs of requests |\n| `SHOW_TIME_COST` | Show time cost of custom marks |\n| `DISABLE_FLASHINFER` | Disable flashinfer attention kernels |\n| `DISABLE_FLASHINFER_SAMPLING` | Disable flashinfer sampling kernels |\n| `DISABLE_RADIX_CACHE` | Disable RadixAttention for prefix caching |\n| `DISABLE_REGEX_JUMP_FORWARD` | Disable regex jump-forward |\n| `DISABLE_CUDA_GRAPH` | Disable cuda graph |\n| `DISABLE_DISK_CACHE` | Disable disk cache |\n| `ENABLE_TORCH_COMPILE` | Optimize model with torch.compile |\n| `ENABLE_P2P_CHECK` | Enable P2P check for GPU access |\n| `ENABLE_MLA` | Enable Multi-head Latent Attention |\n| `ATTENTION_REDUCE_IN_FP32` | Cast attention results to fp32 |\n| `EFFICIENT_WEIGHT_LOAD` | Enable memory efficient weight loading |\n\n## 💡 | Note: \nThis is an initial and preview phase of the worker's development. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frunpod-workers%2Fworker-sglang","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frunpod-workers%2Fworker-sglang","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frunpod-workers%2Fworker-sglang/lists"}