{"id":20790790,"url":"https://github.com/guidance-ai/llgtrt","last_synced_at":"2025-10-16T12:36:18.037Z","repository":{"id":259294003,"uuid":"867848139","full_name":"guidance-ai/llgtrt","owner":"guidance-ai","description":"TensorRT-LLM server with Structured Outputs (JSON) built with Rust","archived":false,"fork":false,"pushed_at":"2025-04-25T21:49:55.000Z","size":185,"stargazers_count":52,"open_issues_count":12,"forks_count":10,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-05-05T20:54:58.500Z","etag":null,"topics":["cfg","guidance","json","openai-api","regex","structured-generation","tensorrt-llm"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/guidance-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-10-04T21:01:54.000Z","updated_at":"2025-05-03T21:57:31.000Z","dependencies_parsed_at":"2024-10-24T06:10:44.399Z","dependency_job_id":"25b33c05-a3d4-4eed-9937-cbf5ef780660","html_url":"https://github.com/guidance-ai/llgtrt","commit_stats":null,"previous_names":["guidance-ai/llgtrt"],"tags_count":14,"template":false,"template_full_name":null,"purl":"pkg:github/guidance-ai/llgtrt","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guidance-ai%2Fllgtrt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guidance-ai%2Fllgtrt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guidance-ai%2Fllgtrt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guidance-ai%2Fllgtrt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/guidance-ai","download_url":"https://codeload.github.com/guidance-ai/llgtrt/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guidance-ai%2Fllgtrt/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279192221,"owners_count":26123265,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-16T02:00:06.019Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cfg","guidance","json","openai-api","regex","structured-generation","tensorrt-llm"],"created_at":"2024-11-17T15:37:42.975Z","updated_at":"2025-10-16T12:36:17.987Z","avatar_url":"https://github.com/guidance-ai.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# llgtrt (llguidance + TensorRT-LLM)\n\nThis project implements a REST HTTP server with \n[OpenAI-compatible API](https://platform.openai.com/docs/api-reference/introduction),\nbased on [NVIDIA TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM)\nand [llguidance library](https://github.com/microsoft/llguidance) for constrained output.\n\nThe server supports regular completions and chat endpoints with JSON schema enforcement (\"Structured Output\"), as well as full context-free grammars using the [Guidance library](https://github.com/guidance-ai/guidance).\n\nThis server is similar in spirit to the [TensorRT-LLM OpenAI server example](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/apps/openai_server.py), but it is Python-free (implemented in Rust) and includes support for constrained output. Like the example above, it **does not** use the NVIDIA Triton Inference Server.\n\n## Structured Output\n\nThe sampling can be constrained by the [Low-Level Guidance library](https://github.com/microsoft/llguidance), part of the [Guidance project](https://github.com/guidance-ai/guidance). While TensorRT computes logits (token probabilities) for the next token, the llguidance library computes a set of tokens allowed by the grammar (whether JSON schema, regular expression, or a full context-free grammar (CFG)) in the form of a bitmask. When both the logits and bitmask are ready, a custom CUDA kernel applies the mask to the logits, and the result is used for sampling inside of TensorRT-LLM.\n\nThere is no significant startup cost for all realistic sizes of grammars (no measurable impact on time to first token (TTFT)). The overhead on generation speed (median time between tokens (TBT)) is typically 1-3% (and comes mostly from apply masking kernels on the GPU). The mask computation takes on the order of 100 us of single-core CPU time per token per sequence in the batch. Thus, with 16 cores and a TBT of around 10 ms, batch sizes of up to 1600 are not CPU-bound. Typically, the unconstrained TBT is higher at such batch sizes, and more cores are available, so batch size is not a problem in production.\n\nThis approach differs from [Outlines](https://github.com/dottxt-ai/outlines) and [XGrammar](https://github.com/mlc-ai/xgrammar) (which both pre-compute masks, resulting in a startup cost and limits on schema complexity) and is more similar in spirit to [llama.cpp grammars](https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md), though it is much faster due to the use of a custom lexer with [derivative-based regexes](https://github.com/microsoft/derivre), an Earley parser, and a [highly optimized](https://github.com/guidance-ai/llguidance/blob/main/docs/optimizations.md) token prefix tree.\n\n### Lark grammars\n\nLlgtrt follows OpenAI API when `response_format` is set to `json_object` or `json_schema` \n(including handling of `strict` field in the schema).\n\nFor more flexible constraints, `response_format` can be set to `lark_grammar`.\nFor example, you can `POST` to `/v1/chat/completions`:\n\n```json\n{ \"model\": \"model\", \n  \"messages\": [\n    { \"role\": \"user\",\n      \"content\": \"Please tell me a one line joke.\"\n    } ],\n  \"response_format\": {\n    \"type\": \"lark_grammar\",\n    \"lark_grammar\": \"start: /[A-Z ]+/\"\n  },\n  \"max_tokens\": 100\n}\n```\n\nThis results in a (bad) joke in uppercase.\n\nAnother example involves reasoning models distilled from Deepseek-R1\n(the chat format in these models seems to already include `\u003cthink\u003e\\n`,\nso it should not be part of the grammar):\n\n```json\n{ \"model\": \"model\", \"messages\": [\n    { \"role\": \"user\",\n      \"content\": \"How many 'r' in strawberry?\"\n    } ],\n  \"response_format\": {\n    \"type\": \"lark_grammar\",\n    \"lark_grammar\": \"start: /(.|\\\\n){1000,2000}/ \u003c/think\u003e \\\"\\\\\\\\boxed{\\\" /[0-9]+/ \\\"}\\\"\"\n  },\n  \"max_tokens\": 1000\n}\n```\n\nThe `\"lark_grammar\"` is JSON-encoded version of \n`start: /(.|\\n){1000,2000}/ \u003c/think\u003e \"\\\\boxed{\" /[0-9]+/ \"}\"`.\nOf course you can also use `{0,2000}` to only place upper bound on thinking,\n`{1000,}` to place lower bound, or `*` to avoid any bounds.\n\nYou can [convert GBNF](https://github.com/guidance-ai/llguidance/blob/main/python/llguidance/gbnf_to_lark.py) grammars to Lark syntax, as it's strictly more expressive.\nLearn more in [llguidance docs](https://github.com/guidance-ai/llguidance/blob/main/docs/syntax.md).\n\n\n## Requirements\n\nYou will need a Linux machine with an NVIDIA GPU and Docker set up to use the `nvidia-docker` runtime.\n\nSo far, we have only tested it on 4xA100 (and single A100).\n\n## Running\n\nOverview of steps:\n\n- Build or pull the `llgtrt/llgtrt` Docker container.\n- Build a trtllm engine (likely using the container).\n- Create configuration files.\n- Use the container to run the engine.\n\n### Building or Pulling Docker Container\n\nTo use a pre-built container, run:\n\n```bash\ndocker pull llgtrt/llgtrt\n```\n\nTo build a container, use:\n\n```bash\n./docker/build.sh\n```\n\nThe build script will initialize submodules if they are missing. It takes about 15 minutes on a GitHub runner and should typically be faster on a local machine.\n\n#### Optional: Building TensorRT-LLM from source\n\nThe build process above uses prebuilt binaries from a release version of TensorRT-LLM.  It is also possible to build your own version of TensorRT-LLM from source and create a build of llgtrt based on that.  This can be used to build a version of llgtrt that will work with versions of TensorRT-LLM newer than the released versions in nVidia's repositories.\n\nTo do so, first build TensorRT-LLM from source following the instructions in https://nvidia.github.io/TensorRT-LLM/installation/build-from-source-linux.html\n\nNow, build llgtrt based on the Docker image you built above\n```bash\n./docker/build.sh --trtllm tensorrt_llm/release\n```\n\n### Building the TensorRT-LLM Engine\n\nThis is based on the [TensorRT-LLM Quick-start](https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html).\nFollow the steps here, and look into that guide if needed.\n\nFirst, use the `llgtrt/llgtrt` container to run bash.\n\n```bash\n./docker/bash.sh --volume /path/to/hf-models:/models\n```\n\nThe following steps are done inside the container:\n\n```bash\n# Convert the HF model to a checkpoint\npython3 /opt/TensorRT-LLM-examples/llama/convert_checkpoint.py \\\n    --dtype bfloat16 \\\n    --model_dir /models/Meta-Llama-3.1-8B-Instruct \\\n    --output_dir /models/model-ckpt \\\n    --tp_size 1\n\n# Then, run trtllm build\ntrtllm-build --checkpoint_dir /models/model-ckpt \\\n    --gemm_plugin bfloat16 \\\n    --output_dir /models/model-engine \\\n    --use_paged_context_fmha enable\n\n# Clean up checkpoint (optional)\nrm -rf /models/model-ckpt\n\n# Finally, copy tokenizer and preprocessor files to engine folder\ncp /models/Meta-Llama-3.1-8B-Instruct/tokenizer*.json /models/model-engine\ncp /models/Meta-Llama-3.1-8B-Instruct/preprocessor*.json /models/model-engine # this may be missing\n\n# Exit the container\nexit\n```\n\nMake sure to modify the path to the input model (it needs to contain the HF Transformers `config.json` as well as the `.safetensors` files and `tokenizer.json`). If you're running on more than one GPU, modify the `--tp_size` argument.\n\n### Running the Engine\n\n```bash\nPORT=3001 ./docker/run.sh /path/to/hf-models/model-engine\n```\n\nThe command will print out the actual `docker run` invocation on the first line if you want to invoke it directly later. `PORT` defaults to 3000.\n\n### Update Configuration (Optional)\n\nThe defaults should be mostly reasonable, but you can modify them. First, generate a template configuration file:\n\n```bash\n./docker/run.sh /path/to/hf-models/model-engine --print-config \u003e llgtrt.json5\n```\n\nThe file will contain commented-out defaults for all supported options (JSON5 is a superset of JSON, so you can use comments). Edit it and move it to the engine folder.\n\nTo modify the chat template, you can either use `--print-complete-config`, which will include the chat template from `tokenizer_config.json`, or preferably create a separate `chat_template.j2` file in the engine folder:\n\n```bash\n./docker/run.sh /path/to/hf-models/model-engine --print-chat-template \u003e chat_template.j2\nmv chat_template.j2 /path/to/hf-models/model-engine\n```\n\nThe paths to `llgtrt.json5` and `chat_template.j2` are controlled by command line arguments. See `--help` for more info:\n\n```bash\n./docker/run.sh /path/to/hf-models/model-engine --help\n```\n\nYou can specify multiple JSON5 config files, and they will be merged in the order specified (with later ones overriding earlier ones). This way, you can separate configuration for the tokenizer, runtime, and guidance parser.\n\n### Running phi-3\n\nThe phi-3 tokenizer, while based on llama2 is slightly different.\nDrop the following `llgtrt.json5` file in engine folder:\n\n```json5\n{\n  \"tokenizer\": {\n    \"bos_token\": null,\n    \"n_vocab_override\": 32064\n  }\n}\n```\n\n## LoRA and Multi-LoRA\n\nSupport for finetuned models via LoRA is provided but requires a few extra steps.\nDetailed instructions are in [LoRA.md](LoRA.md)\n\n## Development\n\nFirst, build the Docker container to be used in the dev container. If you have already followed the steps above, you can skip this. Otherwise, run `./docker/build.sh`.\n\nNext, in VSCode, reopen the folder in the container.\n\n## Credits\n\nThe basic structure of the server borrows inspiration from [npuichigo/openai_trtllm](https://github.com/npuichigo/openai_trtllm), which has similar aims but uses NVIDIA Triton Server wrapping TensorRT-LLM.\n\n## TODO\n\n- [X] multi-LoRA\n- [x] test phi-3.5\n- [ ] multi-modal input\n- [ ] when streaming, and stop is set, we need to buffer the output so as not to return the stop sequence itself\n- [ ] logprobs (currently only work with TP\u003e1; TRTLLM bug?)\n- [ ] logprobs with argmax sampling and constraints\n- [ ] expose the 'surprise' measure somehow\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fguidance-ai%2Fllgtrt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fguidance-ai%2Fllgtrt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fguidance-ai%2Fllgtrt/lists"}