{"id":13607564,"url":"https://github.com/aws-samples/amazon-eks-machine-learning-with-terraform-and-kubeflow","last_synced_at":"2025-04-12T11:32:57.409Z","repository":{"id":137021014,"uuid":"163337511","full_name":"aws-samples/amazon-eks-machine-learning-with-terraform-and-kubeflow","owner":"aws-samples","description":"Distributed training using Kubeflow on Amazon EKS","archived":false,"fork":false,"pushed_at":"2025-04-01T23:14:40.000Z","size":1187,"stargazers_count":86,"open_issues_count":4,"forks_count":44,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-04-02T00:20:22.132Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aws-samples.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-12-27T21:28:41.000Z","updated_at":"2025-04-01T23:14:43.000Z","dependencies_parsed_at":"2024-01-20T20:24:31.760Z","dependency_job_id":"5831ddb6-5b0f-40cc-9830-e906b5ab089e","html_url":"https://github.com/aws-samples/amazon-eks-machine-learning-with-terraform-and-kubeflow","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws-samples%2Famazon-eks-machine-learning-with-terraform-and-kubeflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws-samples%2Famazon-eks-machine-learning-with-terraform-and-kubeflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws-samples%2Famazon-eks-machine-learning-with-terraform-and-kubeflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws-samples%2Famazon-eks-machine-learning-with-terraform-and-kubeflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aws-samples","download_url":"https://codeload.github.com/aws-samples/amazon-eks-machine-learning-with-terraform-and-kubeflow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248560316,"owners_count":21124630,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T19:01:19.723Z","updated_at":"2025-04-12T11:32:57.403Z","avatar_url":"https://github.com/aws-samples.png","language":"HCL","funding_links":[],"categories":["Containers"],"sub_categories":["Workshops"],"readme":"# Serve Mistral 7B Instruct v0.1 and Llama3 8B concurrently using Triton Inference Server with TensorRT-LLM\n\nThis example shows how to serve [mistral-7b-instruct-v01](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1) and [Meta Llama 3 8B Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) models concurrently using [Triton Inference Server](https://github.com/triton-inference-server) with [TensorRT-LLM backend](https://github.com/triton-inference-server/tensorrtllm_backend/tree/main).  \n\n## Prerequisites\n\nBefore proceeding, complete the [Prerequisites](../../../../../README.md#prerequisites) and [Getting started](../../../../../README.md#getting-started). \n\nSee [What is in the YAML file](../../../../../README.md#what-is-in-the-yaml-file) to understand the common fields in the Helm values files. There are some fields that are specific to a machine learning chart.\n\n## Build and Push Docker Container\n\nThis example uses a custom Docker container for Tritonserver TensorRT-LLM. Build and push this container using following command (replace `aws-region` with your AWS Region name):\n\n     cd ~/amazon-eks-machine-learning-with-terraform-and-kubeflow\n     ./containers/tritonserver-trtllm/build_tools/build_and_push.sh aws-region\n      \n## Hugging Face Mistral 7B Instruct v0.1 pre-trained model weights\n\nTo download Hugging Face Mistral 7B Instruct v0.1 pre-trained model weights, replace `YourHuggingFaceToken` with your Hugging Face token below, and execute:\n\n    cd ~/amazon-eks-machine-learning-with-terraform-and-kubeflow\n    helm install --debug triton-mistral-7b-llama3-8b-trtllm     \\\n        charts/machine-learning/model-prep/hf-snapshot    \\\n        --set-json='env=[{\"name\":\"HF_MODEL_ID\",\"value\":\"mistralai/Mistral-7B-Instruct-v0.1\"},{\"name\":\"HF_TOKEN\",\"value\":\"YourHuggingFaceToken\"}]' \\\n        -n kubeflow-user-example-com\n\nUninstall the Helm chart at completion:\n\n    helm uninstall triton-mistral-7b-llama3-8b-trtllm  -n kubeflow-user-example-com\n\n## Hugging Face Meta Llama 3 8B Instruct  pre-trained model weights\n\nTo download Hugging Face Meta Llama 3 8B Instruct  pre-trained model weights, replace `YourHuggingFaceToken` with your Hugging Face token below, and execute:\n\n    cd ~/amazon-eks-machine-learning-with-terraform-and-kubeflow\n    helm install --debug triton-mistral-7b-llama3-8b-trtllm    \\\n        charts/machine-learning/model-prep/hf-snapshot    \\\n        --set-json='env=[{\"name\":\"HF_MODEL_ID\",\"value\":\"meta-llama/Meta-Llama-3-8B-Instruct\"},{\"name\":\"HF_TOKEN\",\"value\":\"YourHuggingFaceToken\"}]' \\\n        -n kubeflow-user-example-com\n\nUninstall the Helm chart at completion:\n\n    helm uninstall triton-mistral-7b-llama3-8b-trtllm -n kubeflow-user-example-com\n\n## Convert Mistral 7B Instruct HuggingFace Checkpoint to TensorRT-LLM Checkpoint\n\nTo convert checkpoint:\n\n    cd ~/amazon-eks-machine-learning-with-terraform-and-kubeflow\n    helm install --debug triton-mistral-7b-llama3-8b-trtllm  \\\n        charts/machine-learning/data-prep/data-process \\\n        -f examples/inference/triton-inference-server/tensorrtllm_backend/mistral-7b-llama3-8b/mistral_7b_hf_to_trtllm.yaml -n kubeflow-user-example-com\n\nTo monitor the logs, execute:\n\n    kubectl logs -f data-process-triton-mistral-7b-llama3-8b-trtllm  -n kubeflow-user-example-com\n\nUninstall the Helm chart at completion:\n\n    helm uninstall triton-mistral-7b-llama3-8b-trtllm  -n kubeflow-user-example-com\n\n## Convert LLama3 8B Instruct HuggingFace Checkpoint to TensorRT-LLM Checkpoint\n\nTo convert checkpoint:\n\n    cd ~/amazon-eks-machine-learning-with-terraform-and-kubeflow\n    helm install --debug triton-mistral-7b-llama3-8b-trtllm \\\n        charts/machine-learning/data-prep/data-process \\\n        -f examples/inference/triton-inference-server/tensorrtllm_backend/mistral-7b-llama3-8b/llama3_8b_hf_to_trtllm.yaml -n kubeflow-user-example-com\n\nTo monitor the logs, execute:\n\n    kubectl logs -f data-process-triton-mistral-7b-llama3-8b-trtllm  -n kubeflow-user-example-com\n\nUninstall the Helm chart at completion:\n\n    helm uninstall triton-mistral-7b-llama3-8b-trtllm  -n kubeflow-user-example-com\n\n\n## Build Mistral 7B TensorRT-LLM Engine\n\nTo build TensorRT-LLM engine:\n\n    cd ~/amazon-eks-machine-learning-with-terraform-and-kubeflow\n    helm install --debug triton-mistral-7b-llama3-8b-trtllm  \\\n        charts/machine-learning/data-prep/data-process \\\n        -f examples/inference/triton-inference-server/tensorrtllm_backend/mistral-7b-llama3-8b/trtllm_mistral_7b_engine.yaml -n kubeflow-user-example-com\n\nTo monitor the logs, execute:\n\n    kubectl logs -f data-process-triton-mistral-7b-llama3-8b-trtllm  -n kubeflow-user-example-com\n\nUninstall the Helm chart at completion:\n\n    helm uninstall triton-mistral-7b-llama3-8b-trtllm  -n kubeflow-user-example-com\n\n## Build Llama3 8B TensorRT-LLM Engine\n\nTo build TensorRT-LLM engine:\n\n    cd ~/amazon-eks-machine-learning-with-terraform-and-kubeflow\n    helm install --debug triton-mistral-7b-llama3-8b-trtllm  \\\n        charts/machine-learning/data-prep/data-process \\\n        -f examples/inference/triton-inference-server/tensorrtllm_backend/mistral-7b-llama3-8b/trtllm_llama3_8b_engine.yaml -n kubeflow-user-example-com\n\nTo monitor the logs, execute:\n\n    kubectl logs -f data-process-triton-mistral-7b-llama3-8b-trtllm  -n kubeflow-user-example-com\n\nUninstall the Helm chart at completion:\n\n    helm uninstall triton-mistral-7b-llama3-8b-trtllm  -n kubeflow-user-example-com\n\n## Build Mistral 7B Triton Model\n\nTo build Triton model:\n\n    cd ~/amazon-eks-machine-learning-with-terraform-and-kubeflow\n    helm install --debug triton-mistral-7b-llama3-8b-trtllm  \\\n        charts/machine-learning/data-prep/data-process \\\n        -f examples/inference/triton-inference-server/tensorrtllm_backend/mistral-7b-llama3-8b/triton_mistral_7b_model.yaml -n kubeflow-user-example-com\n\nTo monitor the logs, execute:\n\n    kubectl logs -f data-process-triton-mistral-7b-llama3-8b-trtllm  -n kubeflow-user-example-com\n\nUninstall the Helm chart at completion:\n\n    helm uninstall triton-mistral-7b-llama3-8b-trtllm  -n kubeflow-user-example-com\n\n## Build Llama3 8B Triton Model\n\nTo build Triton model:\n\n    cd ~/amazon-eks-machine-learning-with-terraform-and-kubeflow\n    helm install --debug triton-mistral-7b-llama3-8b-trtllm  \\\n        charts/machine-learning/data-prep/data-process \\\n        -f examples/inference/triton-inference-server/tensorrtllm_backend/mistral-7b-llama3-8b/triton_llama3_8b_model.yaml -n kubeflow-user-example-com\n\nTo monitor the logs, execute:\n\n    kubectl logs -f data-process-triton-mistral-7b-llama3-8b-trtllm  -n kubeflow-user-example-com\n\nUninstall the Helm chart at completion:\n\n    helm uninstall triton-mistral-7b-llama3-8b-trtllm  -n kubeflow-user-example-com\n\n## Launch Triton Server\n\nTo launch Triton server:\n\n    cd ~/amazon-eks-machine-learning-with-terraform-and-kubeflow\n    helm install --debug triton-mistral-7b-llama3-8b-trtllm  \\\n        charts/machine-learning/serving/triton-inference-server \\\n        -f examples/inference/triton-inference-server/tensorrtllm_backend/mistral-7b-llama3-8b/triton_server.yaml -n kubeflow-user-example-com\n\n\n## Stop Service\n\nTo stop the service:\n\n    cd ~/amazon-eks-machine-learning-with-terraform-and-kubeflow\n    helm uninstall triton-mistral-7b-llama3-8b-trtllm  -n kubeflow-user-example-com\n\n### Logs\n\nTriton server logs are available in `/efs/home/triton-mistral-7b-llama3-8b-trtllm /logs` folder. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faws-samples%2Famazon-eks-machine-learning-with-terraform-and-kubeflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faws-samples%2Famazon-eks-machine-learning-with-terraform-and-kubeflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faws-samples%2Famazon-eks-machine-learning-with-terraform-and-kubeflow/lists"}