{"id":23564174,"url":"https://github.com/selectel/mks-gpu-examples","last_synced_at":"2025-09-09T08:46:47.010Z","repository":{"id":255353924,"uuid":"839780739","full_name":"selectel/mks-gpu-examples","owner":"selectel","description":"This repo includes examples for deploying, configuring and using selectel managed kubernetes with GPU nodes form ML issues","archived":false,"fork":false,"pushed_at":"2024-08-26T15:20:04.000Z","size":18,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-12-26T17:13:06.768Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/selectel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-08T10:04:19.000Z","updated_at":"2024-10-09T11:20:03.000Z","dependencies_parsed_at":"2024-08-29T13:39:49.180Z","dependency_job_id":"39f2db7b-de8d-40bf-a6c8-8b3f866097a3","html_url":"https://github.com/selectel/mks-gpu-examples","commit_stats":null,"previous_names":["selectel/mks-gpu-examples"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/selectel%2Fmks-gpu-examples","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/selectel%2Fmks-gpu-examples/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/selectel%2Fmks-gpu-examples/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/selectel%2Fmks-gpu-examples/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/selectel","download_url":"https://codeload.github.com/selectel/mks-gpu-examples/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239323631,"owners_count":19620052,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-26T17:13:13.461Z","updated_at":"2025-02-17T16:26:18.561Z","avatar_url":"https://github.com/selectel.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mks-gpu-examples\nДанный репозиторий включает в себя примеры использования terraform для деплоя Managed Kubernetes с GPU нодами и с автоскейлингом.\nА также примеры настройки GPU нод с помощью GPU-operator\n\n# Подготовка инфраструктуры\n## Добавление зеркала terraform\nесли вы хотите использовать зеркало, создайте отдельный конфигурационный файл ~/.terraformrc и добавьте в него блок:\n\n```\nprovider_installation {\n  network_mirror {\n    url = \"https://mirror.selectel.ru/3rd-party/terraform-registry/\"\n    include = [\"registry.terraform.io/*/*\"]\n  }\n  direct {\n    exclude = [\"registry.terraform.io/*/*\"]\n  }\n}\n```\nПодробнее о настройках зеркал в инструкции [CLI Configuration File](https://developer.hashicorp.com/terraform/cli/config/config-file) документации HashiCorp.\n\n\n## Создание кластера с помощью terraform\n\nЕсли вы хотите хранить terraform state в selectel s3, то нужно создать `backend.tfvars` и указать там\n```toml\naccess_key = \u003cs3_access_key\u003e\nsecret_key = \u003csecret_key\u003e\n```\n\nТакже создайте `terraform.tfvars` файл, где укажите\n```toml\n# Данные для провайдера Selectel\n# Задаются в личном кабинете my.selectel.ru\nselectel_domain_name = \n\n# Данные для провайдера Openstack\nproject_id             = \nselectel_user_name     = \nselectel_user_password = \n\n```\n\n```bash\ncd terraform\nterraform init -reconfigure -backend-config=backend.tfvars\nterraform plan\nterraform apply\n```\n\n## Подготовка ноды для работы с GPU\nУстановите prometheus stack и gpu operator с нужными values\n```\ncd kubernetes\nhelm upgrade --install prometheus-stack prometheus-community/kube-prometheus-stack -f prometheus-stack/values.yaml\nhelm upgrade --install gpu-operator -n gpu-operator --create-namespace nvidia/gpu-operator -f gpu-operator/values.yaml\n```\n\n### Включение MIG\nЕсли вы используете ноды с GPU архитектурой Ampere и выше, после установки GPU оператор вы можете включить поддержку MIG\nВ values gpu operator добавим следующие поля\n```yaml\nmig:\n  strategy: single\n\nmigManager:\n  enabled: true\n  config:\n    name: \"default-mig-parted-config\"\n    default: \"all-disabled\"\n```\nДалее после установки нового релиза вы можете залейблить ноду, где требуется конфигурация MIG\n```bash\nkubectl label node \u003cnode-name\u003e nvidia.com/mig.config=all-2g.10gb --overwrite\n```\n\n# Запуск инференса ChatGPT2 в кластере MKS\n## Запуск ChatGPT2 в кластере MKS\nУстановим инференс с chatpgt2\n```bash\nkubectl apply -f common\n```\nПробросим порт и перейдем в браузере по адресу localhost:8000/docs\n```\nkubectl port-forward svc/vllm-openai-svc 8000:8000 --address='0.0.0.0'\n```\n## Как настроить автоскейлинг инференса по кастомным метрикам\nПоставим prometheus-adapter с нужной кастомной метрикой\n```\nhelm upgrade --install prometheus-adapter prometheus-community/prometheus-adapter -f vllm/prometheus-adapter.yaml\n```\nустановим vllm с loadbalancer и публичным ip адресом\n```bash\nkubectl apply -f ha\n```\nПолезная информация про мониторинг vllm [здесь](https://github.com/vllm-project/vllm/tree/main/examples/production_monitoring)\nЗаимпортим в grafana [дашборд](kubernetes/vllm/grafana.json)\nИ подадим нагрузку с помощью [genai perf](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client/src/c%2B%2B/perf_analyzer/genai-perf/README.html)\n```\ndocker run --net host -it -v /tmp:/workspace nvcr.io/nvidia/tritonserver:24.05-py3-sdk\ngenai-perf   -m gpt2   --service-kind openai   --endpoint v1/completions   --concurrency 50 --url \u003cloadbalancer_ip\u003e:8000 --endpoint-type completions --num-prompts 100 --random-seed 123 --synthetic-input-tokens-mean 20 --synthetic-input-tokens-stddev 0 --tokenizer hf-internal-testing/llama-tokenizer --measurement-interval 1000 -p 100000\n```\nЕсли менять --concurrency от 50 до 100, то при этом average задержка будет варьироваться от 200мс до 400\n\nВ файле artifacts/gpt2-openai-completions-concurrency50/llm_inputs.json хранятся сгенерированные промты\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fselectel%2Fmks-gpu-examples","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fselectel%2Fmks-gpu-examples","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fselectel%2Fmks-gpu-examples/lists"}