{"id":19815934,"url":"https://github.com/replicate/cog-triton","last_synced_at":"2025-05-01T10:31:52.680Z","repository":{"id":223954582,"uuid":"745066552","full_name":"replicate/cog-triton","owner":"replicate","description":"A cog implementation of Nvidia's Triton server","archived":false,"fork":false,"pushed_at":"2024-08-08T14:04:04.000Z","size":785,"stargazers_count":11,"open_issues_count":4,"forks_count":0,"subscribers_count":13,"default_branch":"main","last_synced_at":"2024-08-08T15:59:53.305Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/replicate.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-18T15:22:36.000Z","updated_at":"2024-07-19T01:48:41.000Z","dependencies_parsed_at":"2024-07-10T01:55:03.980Z","dependency_job_id":"cb932590-28dd-4404-808a-5f21220149f3","html_url":"https://github.com/replicate/cog-triton","commit_stats":null,"previous_names":["replicate/cog-triton"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/replicate%2Fcog-triton","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/replicate%2Fcog-triton/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/replicate%2Fcog-triton/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/replicate%2Fcog-triton/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/replicate","download_url":"https://codeload.github.com/replicate/cog-triton/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224253515,"owners_count":17280932,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T10:07:47.778Z","updated_at":"2024-11-12T10:07:48.424Z","avatar_url":"https://github.com/replicate.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# cog-triton\nA cog implementation of Nvidia's Triton server\n\n## Error codes\n\nWe are using \"E[Category][Subcategory][Sequence] [Short Error Name]: [Description]\n\n### Universal user errors:\n\nCategory 1 (user error), subcategory 0 (framework-agnostic user errors).\n\n* E1000 GenericError: Generic user error (reserved)\n* E1001 PromptRequired: A prompt is required, but your formatted prompt is blank\n* E1002 PromptTooLong: Prompt length exceeds maximum input length.\n* E1003 BadPromptTemplate: You have submitted both a prompt and a prompt template that doesn't include '{prompt}'.\n* E1004 PromptTemplateError: Prompt template must be a valid python format spec\n\n### Triton user errors:\n\nCategory 1 (user error), subcategory 1 (triton-specific user errors)\n\n* E1101 InvalidArgumentMinTokens: Can't set both min_tokens ({min_tokens}) and min_new_tokens ({min_new_tokens})\n* E1102 InvalidArgumentMaxTokens: Can't set both max_tokens ({max_tokens}) and max_new_tokens ({max_new_tokens})\n\n### Triton errors:\n\nCategory 2 (framework error), subcategory 1 (triton system error)\n\n* E2100 TritonUnknownError: Unknown error\n* E2101 TritonTimeout: Triton timed out after {TRITON_TIMEOUT}s: httpx.ReadTimeout.\n* E2102 TritonTokenizerError: Tokenizer error: ... the first token of the stop sequence IDs was not '!', which suggests there is a problem with the tokenizer that you are using.\n* E2103 TritonMalformedJSON: Triton returned malformed JSON\n* E2104 TritonMalformedEvent: Triton returned malformed event (no output_ids or error key)\n\nOther frameworks like vLLM might start their error numbering from E2200. \n\n## Create a Replicate Model with cog-triton\n\nCurrently, we use [yolo](https://github.com/replicate/yolo), a CLI tool we've built to help with non-standard Replicate workflows. To get started, install yolo:\n\n```\nsudo curl -o /usr/local/bin/yolo -L \"https://github.com/replicate/yolo/releases/latest/download/yolo_$(uname -s)_$(uname -m)\"\nsudo chmod +x /usr/local/bin/yolo\n```\n\nOnce you have yolo installed, follow these steps:\n\n1. **Compile a TensorRT engine with cog-triton**\n\n2. **If it doesn't exist already, you'll need to create the Replicate Model to which you'll push your cog-triton model**\n\nYou can create a new Replicate Model via web or our API. To keep things simple, we'll use the latter method.\n\nFirst, set a Replicate API token.\n\n```\nexport REPLICATE_API_TOKEN=\u003cyour-api-token\u003e\n```\n\n```\ncurl -s -X POST -H \"Authorization: Token $REPLICATE_API_TOKEN\" \\\n    -d '{\"owner\": \"my-username\", \"name\": \"my-new-model\", \"visibility\": \"private\", \"hardware\": \"gpu-a40-large\"}' \\\n    https://api.replicate.com/v1/models \n```\n\n\nWe'll call our model `staging-gpt2-triton-trt-llm`\n\n2. **Instantiate a cog-triton model with your TRT-LLM engine**\n\nstaging-gpt2-triton-trt-llm\n\n```\nyolo push \\\n--base r8.im/replicate-internal/cog-triton@sha256:5d784bf5f449a0578ceb903265bb756dae146a267fc075b4c77021babedc6637 \\\n--dest r8.im/replicate-internal/staging-gpt2-triton-trt-llm \\\n-e COG_WEIGHTS=https://replicate.delivery/pbxt/CUDp32x5hO6GMBWprN8o24vWOLZbnYm7AAoRTxLfe0CUfglkA/engine.tar\n```\n\n\n# Run cog-triton locally\n\nTo run cog-triton locally, you must either pull the cog-triton Replicate image or build your own image.\n\n\n## Preparation to run cog-triton locally with Replicate image\n\n### Pull and tag the cog-triton image\n\nGo [here](https://replicate.com/replicate-internal/cog-triton/versions) and pick the version you want to run locally. For our purposes, we'll set the version ID as an environment variable so that the code chunks below won't get stale.\n\n```\nexport COG_TRITON_VERSION=\u003cversion-id-here\u003e\n```\n\n\nThen, click the version hash. We need to set our Replicate API Token and you can do that manually, or navigate to the HTTP tab in your browser and copy the export command.\n\n```\nexport REPLICATE_API_TOKEN=\u003ctoken-here\u003e\n```\n\nNext, navigate to the `Docker` tab under Input. This will display a code chunk like with a `Docker run` command like:\n\n```\ndocker run -d -p 5000:5000 --gpus=all r8.im/replicate-internal/cog-triton@sha256:2db2b5c2e199975fef07ed9045608ed7adc7796744041fa54d3ae9d13db6c3cf\n```\n\nWe'll use the image reference to write a pull command:\n\n```\ndocker pull r8.im/replicate-internal/cog-triton@sha256:${COG_TRITON_VERSION}\n```\n\nAfter the image has been pulled, you should tag it so that all the docker commands in this README will work. First, find the `IMAGE ID` for the image you just pulled, e.g. via `docker images`. Then run the command below after replacing `\u003cimage id\u003e` with your image id.\n\n```\ndocker tag \u003cimage id\u003e cog-triton:latest\n```\n\n### Run an engine built with cog-trt-llm\n\nCopy all model artifacts from `cog-trt-llm/engine_outputs/` to `triton_model_repo/tensorrt_llm/1/`:\n\n```\ncp -r ../cog-trt-llm/engine_outputs/* triton_model_repo/tensorrt_llm/1/\n```\n\nRun the cog-triton image:\n\n```\ndocker run --rm -it -p 5000:5000 --gpus=all --workdir /src  --net=host --volume $(pwd)/.:/src/. --ulimit memlock=-1 --shm-size=20g cog-triton /bin/bash\npython -m cog.server.http\n```\n\n\nMake a request:\n\n\n```\ncurl -s -X POST \\\n  -H \"Content-Type: application/json\" \\\n  -d $'{\n    \"input\": {\n        \"prompt\": \"What is machine learning?\"\n    }\n  }' \\\n  http://localhost:5000/predictions\n```\n\n# Performance tests with test_perf.py\n\n```\ntime python3 scripts/test_perf.py --target cog-triton --rate 8 --unit rps --duration 30 --n_input_tokens 100 --n_output_tokens 100\n```\n\n# Development\n\n\nThis repository builds 4 different images:\n\n- `cog-triton-builder`, which builds TRT-LLM engines.\n- `cog-triton-runner-80`, suitable to run engines built on, and for, [nvidia A100's](https://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/#ampere-cuda-11-1-and-later.)\n- `cog-triton-runner-86`, suitable for A40\n- `cog-triton-runner-90`, suitable for H100 and H200.\n\n[Here's a full GPU compatibility list](https://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/#ampere-cuda-11-1-and-later.).\n\n\n## End-to-end build process \n\nCog-triton is pre-release and not stable. This build process currently requires `nix` to be installed (with the config setting `experimental-features = nix-command flakes`). We recommend the [DeterminateSystems Nix installer](https://github.com/DeterminateSystems/nix-installer), which will set this setting for you.\n\n1. Install nix:\n```console\n$ curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install --extra-conf \"trusted-users = $USER\"\n```\n\n2. Clone the cog-triton repo:\n\n```console\n$ git clone https://github.com/replicate/cog-triton\n$ cd cog-triton\n```\n\n3. Build cog-triton-builder (builder)\n\n\n```console\n$ nix build .#packages.x86_64-linux.cog-triton-builder \u0026\u0026 ./result load\n[...]\nLoaded image: cog-triton-builder:1hz2v478b382h6qqwdgxivqqb2bm1kad\n```\nThis command will eventually output the image id loaded into the local docker daemon.\n\n4. Build cog-triton-runner-86 (runner)\n\n```console\n$ nix build .#packages.x86_64-linux.cog-triton-runner-86 \u0026\u0026 ./result load\n[..]\nLoaded image: cog-triton-runner-86:zknc2pj8kx9kmmicmjmswd2yj343lpd1\n```\n\n## Build a TRT-LLM Model\n\nThe cog-triton-builder image takes in a [cog-trt-llm](https://github.com/replicate/cog-trt-llm) build configs and outputs an engine.tar suitable to run on the same hardware it's running on. In this example, we grab one from official-language-models. Open [mistral-7b-instruct-v0.2/build_config.yaml raw](https://github.com/replicate/official-language-models/raw/main/models/mistral-7b-instruct-v0.2/build_config.yaml) in your browser and copy-paste the URL with the token.\n\n```console\n$ docker run -d -p 5000:5000 --gpus=all cog-triton-builder:1hz2v478b382h6qqwdgxivqqb2bm1kad\n$ curl -s -X POST \\\n  -H \"Content-Type: application/json\" \\\n  -d $'{\n    \"input\": { \n        \"config\":\"https://raw.githubusercontent.com/replicate/official-language-models/main/models/mistral-7b-instruct-v0.2/build_config.yaml?token=\u003cexample-token-see-instructions\u003e\"\n    }\n  }' http://localhost:5000/predictions\n$ docker container cp \u003ccontainer name\u003e:/src/engine.tar ./engine.tar\n```\n\n## Run a TRT-LLM Model\n\nWhen you have this engine, you can use it with the cog-triton-runner images.\n\n1. Extract the `engine.tar`\n```console\n$ rm -rf triton_model_repo/tensorrt_llm/1/\n$ mkdir -p triton_model_repo/tensorrt_llm/1/\n$ tar xvf ./engine.tar -C triton_model_repo/tensorrt_llm/1/\n```\n\n2. Run the image\nThis runs the cog-triton-runner-86 image that's been built in the previous steps. Adjust accordingly for your GPU and image tag.\n\n```console\n$ docker run --rm -it -p 5000:5000 -p 8000:8000 --gpus=all --ulimit memlock=-1 --shm-size=20g --volume $(pwd)/triton_model_repo/tensorrt_llm/1/:/src/triton_model_repo/tensorrt_llm/1/ cog-triton-runner-86:zknc2pj8kx9kmmicmjmswd2yj343lpd1\n$ \n```\n7. Curl a request\n\nYou can curl directly to the Triton server:\n```\ncurl -s -X POST \\\n  -H \"Content-Type: application/json\" \\\n  -d $'{\n    \"input\": {\n        \"prompt\": \"What is machine learning?\"\n    }\n  }' \\\n  http://localhost:5000/predictions\n```\n\n```\ncurl -X POST localhost:8000/v2/models/ensemble/generate -d '{\"text_input\": \"What is machine learning?\", \"max_tokens\": 20, \"bad_words\": \"\", \"stop_words\": \"\"}'\n```\n\n```\ncurl -X POST localhost:8000/v2/models/ensemble/generate -d '{\"text_input\": \"Water + Fire = Steam\\nEarth + Water = Plant\\nHuman + Robe = Judge\\nCow + Fire = Steak\\nKing + Ocean = Poseidon\\nComputer + Spy =\", \"max_tokens\": 20, \"bad_words\": \"\", \"stop_words\": \"\"}'\n```\n\n```\ncurl -s -X POST \\\n  -H \"Content-Type: application/json\" \\\n  -d $'{\n    \"input\": {\n        \"prompt\": \"Water + Fire = Steam\\nEarth + Water = Plant\\nHuman + Robe = Judge\\nCow + Fire = Steak\\nKing + Ocean = Poseidon\\nComputer + Spy =\"\n    }\n  }' \\\n  http://localhost:5000/predictions\n```\n\n```\n  curl -s -X POST \\\n  -H \"Content-Type: application/json\" \\\n  -d $'{\n    \"input\": {\n        \"prompt\": \"Water + Fire = Steam\\nEarth + Water = Plant\\nHuman + Robe = Judge\\nCow + Fire = Steak\\nKing + Ocean = Poseidon\\nComputer + Spy =\"\n    }\n  }' \\\n  http://localhost:5000/predictions\n```\n\n# Local Testing\n\n## Test cog performance\n\nThe `mock_cog_triton` directory provides a mocked `cog-triton` server that emits tokens at a fixed rate. It also includes a performance test script that reports client side and server-side performance metrics.\n\nThis should eventually be integrated into a test suite, but to maintain some visibility into performance continuity, we can run it manually and eyeball.\n\nTo do this, start the `mock_cog_triton` cog server:\n```\ndocker run --rm -it -p 5000:5000  --workdir /src  --net=host --volume $(pwd)/.:/src/. cog-triton bash -c \"cd mock_cog_triton \u0026\u0026 python -m cog.server.http\"\n```\n\nand execute the performance script:\n\n```\npython3 mock_cog_triton/test_perf.py  --unit batch --duration 10 --tps 100 --n_output_tokens 128 --output_method yield --rate 24\n```\n\nExpected output with these input parameters is shown below. Note that Single-stream TPS metrics are quite close to the server-side metrics.\n\n```\n------------------------------\nTest Configuration:\n------------------------------\nOutput Method: yield\nMode: batch\nRate: 24.0 batch\nDuration: 60 seconds\nOutput tokens: 128\n------------------------------\nConcurrency levels:\nMode concurrency: 24\nMean concurrency: 13.8055\nMedian concurrency: 24.0\nMax concurrency: 24\nMin concurrency: 0\n------------------------------\nStatistics for completed predictions:\n------------------------------\nSingle-stream TPS:\nSSTPS - Std: 1.498\nSSTPS - Median: 93.177\nSSTPS - Mean: 93.082\nSSTPS - Max: 96.346\nSSTPS - Min: 87.296\n------------------------------\nLatency - Std: 0.023 seconds\nMedian response latency: 1.374 seconds\nMean response latency: 1.375 seconds\nMax response latency: 1.466 seconds\nMin response latency: 1.329 seconds\n------------------------------\nServer-side metrics:\n------------------------------\nServer-side TPS\n--Expected mean: 100.000, Actual mean: 95.557\n--Expected std: 0.000, Actual std: 1.198\n--Expected median: 100.000, Actual median: 95.535\n--Expected min: 100.000, Actual min: 90.650\n--Expected max: 100.000, Actual max: 97.710\nResponse Latency\n--Expected mean: 1.280, Actual mean: 1.340\n--Expected std: 0.000, Actual std: 0.018\n--Expected median: 1.280, Actual median: 1.340\n--Expected min: 1.280, Actual min: 1.310\n--Expected max: 1.280, Actual max: 1.410\nTime to First Token\n--Expected mean: 0.010, Actual mean: 0.010\n--Expected std: 0.000, Actual std: 0.000\n--Expected median: 0.010, Actual median: 0.010\n--Expected min: 0.010, Actual min: 0.010\n--Expected max: 0.010, Actual max: 0.010\n------------------------------\nTotal requests made: 600\nTotal requests started: 600\nTotal requests completed: 600\nFailure rate: 0.000, Total failures: 0\nCog already running prediction: 0\nE2E throughput: 9.988 rps\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freplicate%2Fcog-triton","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Freplicate%2Fcog-triton","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freplicate%2Fcog-triton/lists"}