{"id":18028499,"url":"https://github.com/interactivetech/kserve-pytorch","last_synced_at":"2025-08-21T22:18:48.103Z","repository":{"id":155679305,"uuid":"630209503","full_name":"interactivetech/kserve-pytorch","owner":"interactivetech","description":"Installation steps kserve with pytorch","archived":false,"fork":false,"pushed_at":"2023-04-27T05:50:36.000Z","size":69104,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-04T20:46:43.757Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/interactivetech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-19T22:43:28.000Z","updated_at":"2023-04-27T05:50:40.000Z","dependencies_parsed_at":null,"dependency_job_id":"10917264-05ee-4b07-87e8-dc748ba06795","html_url":"https://github.com/interactivetech/kserve-pytorch","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/interactivetech/kserve-pytorch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/interactivetech%2Fkserve-pytorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/interactivetech%2Fkserve-pytorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/interactivetech%2Fkserve-pytorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/interactivetech%2Fkserve-pytorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/interactivetech","download_url":"https://codeload.github.com/interactivetech/kserve-pytorch/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/interactivetech%2Fkserve-pytorch/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271553232,"owners_count":24779820,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-21T02:00:08.990Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-30T08:42:21.834Z","updated_at":"2025-08-21T22:18:48.086Z","avatar_url":"https://github.com/interactivetech.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"End to end kserve installation\n\nDet-environments-24586f0\n\n\nssh -i \"andrew-determined-env.pem\" ubuntu@ec2-54-82-63-88.compute-1.amazonaws.com -L 8008:localhost:8008\n\nInstall Kind\n\nsudo apt-get update \u0026\u0026 sudo apt-get install nano screen\n\ncurl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.18.0/kind-linux-amd64\nchmod +x ./kind\nsudo mv ./kind /usr/local/bin/kind\n\nkind create cluster\n\nInstall kubectl \n\nInstall kubectl\ncurl -LO \"https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl\"\n\ncurl -LO \"https://dl.k8s.io/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256\"\n\nsudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl\n\nCopy and Paste kserve_install.sh\nnano kserve_install.sh\nbash kserve_install.sh\n\n\nPatch domain for local connection\n\nkubectl patch cm config-domain --patch '{\"data\":{\"example.com\":\"\"}}' -n knative-serving\n\nCreate namespace\nkubectl create ns kserve-test\n\n\nCreate InferenceService\n\nkubectl apply -n kserve-test -f - \u003c\u003cEOF\napiVersion: \"serving.kserve.io/v1beta1\"\nkind: \"InferenceService\"\nmetadata:\n  name: \"sklearn-iris\"\nspec:\n  predictor:\n    model:\n      modelFormat:\n        name: sklearn\n      storageUri: \"gs://kfserving-examples/models/sklearn/1.0/model\"\nEOF\n\nCreate Test input\n\ncat \u003c\u003cEOF \u003e \"./iris-input.json\"\n{ \n  \"instances\": [\n    [6.8,  2.8,  4.8,  1.4],\n    [6.0,  3.4,  4.5,  1.6]\n  ]\n}\nEOF\n\n\nJupyterlab\n\nsudo apt-get update \u0026\u0026 sudo apt-get  install python3.8-venv\npython3 -m venv kserve_env python=3.8\n\nsource kserve_env/bin/activate\nPip install kserve\n\njupyter lab --ip=0.0.0.0 --port=8008 --NotebookApp.token='' --NotebookApp.password=''\n\nCreate two terminals\n\nTerminal one\nINGRESS_GATEWAY_SERVICE=$(kubectl get svc --namespace istio-system --selector=\"app=istio-ingressgateway\" --output jsonpath='{.items[0].metadata.name}')\n\necho $INGRESS_GATEWAY\n\nTerminal 2\nTest kserve \n\nMODEL_NAME=sklearn-iris\nINPUT_PATH=@./iris-input.json\nCLUSTER_IP=localhost:8080\n SERVICE_HOSTNAME=$(kubectl get -n kserve-test inferenceservice ${MODEL_NAME} -o jsonpath='{.status.url}' | cut -d \"/\" -f 3)\ncurl -v -H \"Host: ${SERVICE_HOSTNAME}\" -H \"Cookie: authservice_session=${SESSION}\" http://${CLUSTER_IP}/v1/models/${MODEL_NAME}:predict -d ${INPUT_PATH}\n\n\nSet up Persistent Volume Claim to run models locally\nhttps://kserve.github.io/website/0.10/modelserving/storage/pvc/pvc/#copy-model-to-pv \nhttps://kserve.github.io/website/0.10/modelserving/storage/pvc/pvc/#create-pv-and-pvc\n\nCreate folder called k8s_pvc_files/ and create file pv-and-pvc.yaml\n\nPaste content:\u2028\napiVersion: v1\nkind: PersistentVolume\nmetadata:\n  name: task-pv-volume\n  labels:\n    type: local\nspec:\n  storageClassName: manual\n  capacity:\n    storage: 2Gi\n  accessModes:\n    - ReadWriteOnce\n  hostPath:\n    path: \"/home/ubuntu/mnt/data\"\n---\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: task-pv-claim\nspec:\n  storageClassName: manual\n  accessModes:\n    - ReadWriteOnce\n  resources:\n    requests:\n      storage: 1Gi\n\nRun command:\nkubectl apply -f k8s_pvc_files/pv-and-pvc.yaml \n\nCreate Pytorch files for local model loading\nhttps://kserve.github.io/website/0.10/modelserving/v1beta1/torchserve/ \nGit clone https://github.com/pytorch/serve.git \n\npip install torch-model-archiver\n\ncd serve/examples/image_classifier/\nexport ENABLE_TORCH_PROFILER=true\n \nmkdir mnist_torchserve\ncd mnist_torchserve\nmkdir config\nmkdir model-store\ncd ..\n\n\ntorch-model-archiver --model-name mnist --version 1.0 --model-file /home/ubuntu/serve/examples/image_classifier/mnist/mnist.py --serialized-file /home/ubuntu/serve/examples/image_classifier/mnist/mnist_cnn.pt --handler  /home/ubuntu/serve/examples/image_classifier/mnist/mnist_handler.py\u2028\u2028\nmv mnist.mar mnist_torchserve/model-store/\n\nCreate properties.txt in model-store\n\n[\n  {\n    \"model-name\": \"mnist\",\n    \"version\": \"1.0\",\n    \"model-file\": \"\",\n    \"serialized-file\": \"mnist_cnn.pt\",\n    \"extra-files\": \"\",\n    \"handler\": \"mnist_handler.py\",\n    \"min-workers\" : 1,\n    \"max-workers\": 3,\n    \"batch-size\": 1,\n    \"max-batch-delay\": 100,\n    \"response-timeout\": 120,\n    \"requirements\": \"\"\n  },\n  {\n    \"model-name\": \"densenet_161\",\n    \"version\": \"1.0\",\n    \"model-file\": \"\",\n    \"serialized-file\": \"densenet161-8d451a50.pth\",\n    \"extra-files\": \"index_to_name.json\",\n    \"handler\": \"image_classifier\",\n    \"min-workers\" : 1,\n    \"max-workers\": 3,\n    \"batch-size\": 1,\n    \"max-batch-delay\": 100,\n    \"response-timeout\": 120,\n    \"requirements\": \"\"\n  }\n]\n\nRename to properties.json\n\nCreate config.properties in mnist_torchserve/config\n\nUpload artifacts to PVC\n\n\nkubectl cp  mnist_torchserve/config model-store-pod:/pv/config\nkubectl cp  mnist_torchserve/model-store model-store-pod:/pv/model-store\nkubectl exec -it model-store-pod -- ls /pv\n\nCreate Torch Inference Service\nhttps://github.com/kserve/kserve/issues/1810 \n\nKubectl get pods\nGet name: torchserve-predictor-default-00001-deployment-5464467b4-kchxz\n\nWait until initialized\n\nCheck files mounted correctly\n\nkubectl logs -n default torchserve-predictor-default-00001-deployment-5464467b4-kchxz -c storage-initializer\n\nCheck if server is running\nkubectl logs -f torchserve-predictor-default-00001-deployment-5464467b4-kchxz\n\nPrepare Data\n\nhttps://github.com/kserve/kserve/tree/master/docs/samples/v1beta1/torchserve/v1/imgconv#convert-image-to-bytearray\n\ngit clone https://github.com/kserve/kserve.git\n\ncd ~/kserve/docs/samples/v1beta1/torchserve/v1/imgconv/\n\n\nRun Inference\nMODEL_NAME=mnist\nSERVICE_HOSTNAME=$(kubectl get inferenceservice torchserve -o jsonpath='{.status.url}' | cut -d \"/\" -f 3)\nCLUSTER_IP=localhost:8080\nINPUT_PATH=@./input.json\nINPUT_PATH_1=@./input_1.json\ncurl -v -H \"Host: ${SERVICE_HOSTNAME}\"  http://${CLUSTER_IP}/v1/models/${MODEL_NAME}:predict -d ${INPUT_PATH}\n\n\ncurl -v -H \"Host: ${SERVICE_HOSTNAME}\"  http://${CLUSTER_IP}/v1/models/${MODEL_NAME}:predict -d ${INPUT_PATH_1}\n\n\nDensenet Test\n\nwget https://download.pytorch.org/models/densenet161-8d451a50.pth\nhttps://kserve.github.io/website/0.10/modelserving/v1beta1/torchserve/model-archiver/#221-create-propertiesjson-file \ntorch-model-archiver --model-name densenet161 --version 1.0 --model-file /home/ubuntu/serve/examples/image_classifier/densenet_161/model.py --serialized-file densenet161-8d451a50.pth --handler image_classifier --extra-files /home/ubuntu/serve/examples/image_classifier/index_to_name.json\n\nUpdate properties.json\n  {\n    \"model-name\": \"densenet_161\",\n    \"version\": \"1.0\",\n    \"model-file\": \"\",\n    \"serialized-file\": \"densenet161-8d451a50.pth\",\n    \"extra-files\": \"index_to_name.json\",\n    \"handler\": \"image_classifier\",\n    \"min-workers\" : 1,\n    \"max-workers\": 3,\n    \"batch-size\": 1,\n    \"max-batch-delay\": 100,\n    \"response-timeout\": 120,\n    \"requirements\": \"\"\n  }\nLook at test.txt to see how to update config.properties\n\nInspiration: https://github.com/kserve/kserve/blob/07e4d5d3ea54164026bbf7d87de0b06c6d34c014/docs/samples/v1beta1/torchserve/model-archiver/model-archiver-image/dockerd-entrypoint.sh \n\nkubectl exec --tty model-store-pod -- cat /pv/config/config.properties\n\nkubectl cp mnist_torchserve/model-store/densenet161.mar model-store-pod:/pv/model-store/densenet161.mar\n\nkubectl exec --tty model-store-pod -- cat /pv/config/config.properties\n\nkubectl cp  mnist_torchserve/model-store model-store-pod:/pv/model-store/\n\nkubectl cp  mnist_torchserve/model-store model-store-pod:/pv/model-store\n\n\nMODEL_NAME_d=densenet161\n\nexamples/image_classifier/kitten.jpg\npython img2bytearray.py /home/ubuntu/serve/examples/image_classifier/kitten.jpg\n\nmv input.json ~/cat.json\nINPUT_PATH_cat=@./cat.json\n\ncurl -v -H \"Host: ${SERVICE_HOSTNAME}\"  http://${CLUSTER_IP}/v1/models/${MODEL_NAME_d}:predict -d ${INPUT_PATH}\n\n\n# Object Detection FasterRCNN Kserve\n\nwget https://download.pytorch.org/models/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth\n\ntorch-model-archiver --model-name fasterrcnn --version 1.0 --model-file /home/ubuntu/kserve-pytorch/serve/examples/object_detector/fast-rcnn/model.py --serialized-file fasterrcnn_resnet50_fpn_coco-258fb6c6.pth --handler object_detector --extra-files /home/ubuntu/kserve-pytorch/serve/examples/object_detector/index_to_name.json\n\npip install torchserve nvgpu\n\nMkdir model-store-test\nMv fasterrcnn.mar model-store-test/\ntorchserve --start --model-store model-store-test/ --models fastrcnn=fasterrcnn.mar\ntorchserve --stop\n\n\nMODEL_NAME_f=fasterrcnn\nSERVICE_HOSTNAME=$(kubectl get inferenceservice torchserve -o jsonpath='{.status.url}' | cut -d \"/\" -f 3)\nCLUSTER_IP=localhost:8080\n\n\nmv input.json ~/cat.json\nINPUT_PATH_cat=@./cat.json\n\ncurl -v -H \"Host: ${SERVICE_HOSTNAME}\"  http://${CLUSTER_IP}/v1/models/${MODEL_NAME_f}:predict -d ${INPUT_PATH_cat}\n\nscp -i \"andrew-determined-env.pem\" /Users/mendeza/Downloads/faster_rcnn_e2e_checkpoints/model_8.pth ubuntu@ec2-54-90-185-179.compute-1.amazonaws.com:/home/ubuntu/kserve-pytorch\n\n\n—\n\n# Xview FasterRCNN Test\n\nscp -i \"andrew-determined-env.pem\" /Users/mendeza/Downloads/faster_rcnn_e2e_checkpoints/model_8.pth ubuntu@ec2-54-90-185-179.compute-1.amazonaws.com:/home/ubuntu/kserve-pytorch\n\nStrip model pth: notebook \n\ntorch-model-archiver --model-name xview-fasterrcnn --version 1.0 --model-file /home/ubuntu/kserve-pytorch/serve/examples/object_detector/fast-rcnn/model-xview.py --serialized-file model_8_stripped.pth --handler /home/ubuntu/kserve-pytorch/serve/examples/object_detector/fasterrcnn_handler.py --extra-files /home/ubuntu/kserve-pytorch/serve/examples/object_detector/fast-rcnn/xview-labels/index_to_name.json\n\nUpdate test.txt\nUpdate config.properties\nAdd .mar to model-store\nUpdate properties.json\n\nkubectl cp mnist_torchserve/config/config.properties model-store-pod:/pv/config/config.properties\n\nkubectl exec --tty model-store-pod -- cat /pv/config/config.properties\n\nkubectl cp mnist_torchserve/model-store/xview-fasterrcnn.mar model-store-pod:/pv/model-store\n\nkubectl exec --tty model-store-pod -- ls /pv/model-store\n\nkubectl cp mnist_torchserve/model-store/properties.json model-store-pod:/pv/model-store\n\nExec into container\n\nkubectl exec -it  torchserve-predictor-default-00001-deployment-69ff797b68-zxh7v -c kserve-container sh\n\nISSUE INSTALL the version that is in the container!!!\n\n pip install torch==1.11.0 torchvision==0.12.0\n\nMODEL_NAME_x=xview-fasterrcnn\nSERVICE_HOSTNAME=$(kubectl get inferenceservice torchserve -o jsonpath='{.status.url}' | cut -d \"/\" -f 3)\nCLUSTER_IP=localhost:8080\n\n\nmv input.json ~/cat.json\nINPUT_PATH_cat=@./cat.json\n\ncurl -v -H \"Host: ${SERVICE_HOSTNAME}\"  http://${CLUSTER_IP}/v1/models/${MODEL_NAME_x}:predict -d ${INPUT_PATH_cat}\n\nkserve-pytorch/serve/examples/object_detector/fast-rcnn/xview-labels/index_to_name.json","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finteractivetech%2Fkserve-pytorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finteractivetech%2Fkserve-pytorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finteractivetech%2Fkserve-pytorch/lists"}