{"id":16827001,"url":"https://github.com/retocode/knative-kserve","last_synced_at":"2026-03-18T22:29:08.056Z","repository":{"id":154107286,"uuid":"626953841","full_name":"ReToCode/knative-kserve","owner":"ReToCode","description":null,"archived":false,"fork":false,"pushed_at":"2023-08-21T09:27:16.000Z","size":385,"stargazers_count":0,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-24T07:08:28.475Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ReToCode.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-12T13:45:04.000Z","updated_at":"2024-11-29T16:56:01.000Z","dependencies_parsed_at":null,"dependency_job_id":"7264c4d7-bc02-4c2f-a1b0-fa973f658763","html_url":"https://github.com/ReToCode/knative-kserve","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ReToCode%2Fknative-kserve","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ReToCode%2Fknative-kserve/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ReToCode%2Fknative-kserve/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ReToCode%2Fknative-kserve/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ReToCode","download_url":"https://codeload.github.com/ReToCode/knative-kserve/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244102039,"owners_count":20398379,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T11:19:21.754Z","updated_at":"2026-01-04T13:02:45.697Z","avatar_url":"https://github.com/ReToCode.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# OpenShift + KServe + Knative with Service Mesh\n\n## Prerequisites\n* An OpenShift cluster, configured with a reachable domain for OCP routes\n* OC client pointing to that cluster\n\n## Installation of Knative with Service Mesh\n```bash\n# Install Service Mesh operators\noc apply -f service-mesh/operators.yaml\nsleep 30\noc wait --for=condition=ready pod -l name=istio-operator -n openshift-operators --timeout=300s\noc wait --for=condition=ready pod -l name=jaeger-operator -A --timeout=300s\noc wait --for=condition=ready pod -l name=kiali-operator -n openshift-operators --timeout=300s\n\n# Create an istio instance\noc apply -f service-mesh/namespace.yaml\noc apply -f service-mesh/smcp.yaml\nsleep 15\noc wait --for=condition=ready pod -l app=istiod -n istio-system --timeout=300s\noc wait --for=condition=ready pod -l app=istio-ingressgateway -n istio-system --timeout=300s\noc wait --for=condition=ready pod -l app=istio-egressgateway -n istio-system --timeout=300s\n\noc create ns kserve-demo\noc create ns knative-serving\noc apply -f service-mesh/smmr.yaml\n\n# Install OpenShift Serverless operator\noc apply -f serverless/operator.yaml\nsleep 30\noc wait --for=condition=ready pod -l name=knative-openshift -n openshift-serverless --timeout=300s\noc wait --for=condition=ready pod -l name=knative-openshift-ingress -n openshift-serverless --timeout=300s\noc wait --for=condition=ready pod -l name=knative-operator -n openshift-serverless --timeout=300s\n\n# Create a Knative Serving installation\noc apply -f serverless/knativeserving-istio.yaml\nsleep 15\noc wait --for=condition=ready pod -l app=controller -n knative-serving --timeout=300s\noc wait --for=condition=ready pod -l app=net-istio-controller -n knative-serving --timeout=300s\noc wait --for=condition=ready pod -l app=net-istio-webhook -n knative-serving --timeout=300s\noc wait --for=condition=ready pod -l app=autoscaler-hpa -n knative-serving --timeout=300s\noc wait --for=condition=ready pod -l app=domain-mapping -n knative-serving --timeout=300s\noc wait --for=condition=ready pod -l app=webhook -n knative-serving --timeout=300s\noc wait --for=condition=ready pod -l app=activator -n knative-serving --timeout=300s\noc wait --for=condition=ready pod -l app=autoscaler -n knative-serving --timeout=300s\n\n# Create the Knative gateways\noc apply -f serverless/gateways.yaml\n\n# Install cert-manager operator\noc apply -f cert-manager/operator.yaml\nsleep 60\noc wait --for=condition=ready pod -l app=webhook -n cert-manager --timeout=300s\noc wait --for=condition=ready pod -l app=cainjector -n cert-manager --timeout=300s\noc wait --for=condition=ready pod -l app=cert-manager -n cert-manager --timeout=300s\n```\n\n## Installation of KServe using RHODS\n\n```bash\n# Install RHODS (or ODH) operator\noc apply -f rhods/operator.yaml\nsleep 15\noc wait --for=condition=ready pod -l name=rhods-operator -n redhat-ods-operator --timeout=300s\n\n# Install KServe using KfDef\noc apply -f rhods/kserve-kfdef.yaml\n```\n\n## Creating a serving runtime\n\n\u003e πŸ“ Note: Make sure that your serving runtime image does not need to run as a root or a specific uid.\n\u003e Otherwise `anyuid` is required on the target namespace.\n\n```bash\n# Install a runtime that does work without `anyuid`\noc apply -f rhods/serving-runtime.yaml\n```\n\n## Testing RHODS installation\n\n```bash\n# Deploy an InferenceService\ncat \u003c\u003c-EOF | oc apply -f -\napiVersion: \"serving.kserve.io/v1beta1\"\nkind: \"InferenceService\"\nmetadata:\n name: \"sklearn-iris\"\n namespace: kserve-demo\n annotations:\n sidecar.istio.io/inject: \"true\"\n sidecar.istio.io/rewriteAppHTTPProbers: \"true\"\n serving.knative.openshift.io/enablePassthrough: \"true\"\nspec:\n predictor:\n model:\n runtime: kserve-sklearnserver\n modelFormat:\n name: sklearn\n storageUri: \"gs://kfserving-examples/models/sklearn/1.0/model\"\nEOF\n```\n\n```bash\n# Testing it\noc get isvc -n kserve-demo\nNAME URL READY PREV LATEST PREVROLLEDOUTREVISION LATESTREADYREVISION AGE\nsklearn-iris http://sklearn-iris-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com True 100 sklearn-iris-predictor-00001 2m14s\n\ncurl -k https://sklearn-iris-predictor-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com/v1/models/sklearn-iris:predict -d @./kserve/samples/input-iris.json\n{\"predictions\":[1,1]}\n```\n\n\n## Testing vanilla KServe installation\n\n### From google bucket\nFrom https://kserve.github.io/website/0.10/get_started/first_isvc/#2-create-an-inferenceservice\n```bash\n# Istio\noc apply -f kserve/samples/istio/sklearn-iris.yaml\n# Kourier\noc apply -f kserve/samples/kourier/sklearn-iris.yaml\n\ncurl -k https://sklearn-iris-predictor-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com/v1/models/sklearn-iris:predict -d @./kserve/samples/input-iris.json\n{\"predictions\":[1,1]}% \n```\n\n### From PVC\n```bash\n# Preparation\noc apply -f kserve/samples/pvc.yaml\noc apply -f kserve/samples/pvc-pod.yaml\noc cp kserve/samples/model.joblib model-store-pod:/pv/model.joblib -c model-store -n kserve-demo\noc delete pod --force model-store-pod -n kserve-demo\n\n# Istio\noc apply -f kserve/samples/istio/sklearn-pvc.yaml\n# Kourier\noc apply -f kserve/samples/kourier/sklearn-pvc.yaml\n\ncurl -k https://sklearn-pvc-predictor-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com/v1/models/sklearn-pvc:predict -d @./kserve/samples/input-iris.json \n{\"predictions\":[1,1]}% \n```\n\n### Using GRPC\n* From https://kserve.github.io/website/0.10/modelserving/v1beta1/triton/torchscript/#run-a-prediction-with-grpcurl\n \n```bash\n# Istio\noc apply -f kserve/samples/istio/torchscript-grpc.yaml\n# Kourier\noc apply -f kserve/samples/kourier/torchscript-grpc.yaml\n\nexport PROTO_FILE=kserve/samples/grpc_predict_v2.proto\ngrpcurl -insecure -proto $PROTO_FILE torchscript-grpc-predictor-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com:443 inference.GRPCInferenceService.ServerReady\n{\n \"ready\": true\n}\n\ngrpcurl -insecure -proto $PROTO_FILE -d @ torchscript-grpc-predictor-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com:443 inference.GRPCInferenceService.ModelInfer \u003c\u003c\u003c $(cat kserve/samples/input-grpc.json)\n\n{\n \"modelName\": \"cifar10\",\n \"modelVersion\": \"1\",\n \"outputs\": [\n {\n \"name\": \"OUTPUT__0\",\n \"datatype\": \"FP32\",\n \"shape\": [\n \"1\",\n \"10\"\n ]\n }\n ],\n \"rawOutputContents\": [\n \"viwGwNpLDL7icgK/dusyQAaAD79/KP8/IX2QP4fAs7+HuRk/1+oHwA==\"\n ]\n}\n```\n\n### Canary Deployment\n```bash\n# Istio\noc apply -f kserve/samples/istio/sklearn-iris.yaml\n# Kourier\noc apply -f kserve/samples/kourier/sklearn-iris.yaml\n\noc get isvc sklearn-iris -n kserve-demo\n\nNAME URL READY PREV LATEST PREVROLLEDOUTREVISION LATESTREADYREVISION AGE\nsklearn-iris http://sklearn-iris-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com True 100 sklearn-iris-predictor-00001 109s\n\n# Canary rollout\n## Istio\noc apply -f kserve/samples/istio/sklearn-iris-v2.yaml\n## Kourier\noc apply -f kserve/samples/kourier/sklearn-iris-v2.yaml\n\noc get isvc sklearn-iris -n kserve-demo\n\nNAME URL READY PREV LATEST PREVROLLEDOUTREVISION LATESTREADYREVISION AGE\nsklearn-iris http://sklearn-iris-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com False 90 10 sklearn-iris-predictor-00001 sklearn-iris-predictor-00001 3m40s\n\ncurl -k https://sklearn-iris-predictor-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com/v1/models/sklearn-iris:predict -d @./kserve/samples/input-iris.json\n{\"predictions\":[1,1]}\n```\n\n```bash\n# Previous revision\noc get isvc sklearn-iris -n kserve-demo -ojsonpath=\"{.status.components.predictor}\" | jq\n\n{\n \"address\": {\n \"url\": \"http://sklearn-iris-predictor.kserve-demo.svc.cluster.local\"\n },\n \"latestCreatedRevision\": \"sklearn-iris-predictor-00002\",\n \"latestReadyRevision\": \"sklearn-iris-predictor-00002\",\n \"latestRolledoutRevision\": \"sklearn-iris-predictor-00001\",\n \"traffic\": [\n {\n \"latestRevision\": true,\n \"percent\": 10,\n \"revisionName\": \"sklearn-iris-predictor-00002\"\n },\n {\n \"latestRevision\": false,\n \"percent\": 90,\n \"revisionName\": \"sklearn-iris-predictor-00001\",\n \"tag\": \"prev\",\n \"url\": \"https://prev-sklearn-iris-predictor-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com\"\n }\n ],\n \"url\": \"http://sklearn-iris-predictor-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com\"\n}\n\n# Curl the previous revision\ncurl -k https://prev-sklearn-iris-predictor-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com/v1/models/sklearn-iris:predict -d @./kserve/samples/input-iris.json\n{\"predictions\":[1,1]}\n```\n\n```bash\n# Tag based routing\n## Istio\noc apply -f kserve/samples/istio/sklearn-iris-tag-based.yaml\n## Kourier\noc apply -f kserve/samples/kourier/sklearn-iris-tag-based.yaml\n\noc get isvc sklearn-iris -n kserve-demo -ojsonpath=\"{.status.components.predictor}\" | jq\n\n{\n \"address\": {\n \"url\": \"http://sklearn-iris-predictor.kserve-demo.svc.cluster.local\"\n },\n \"latestCreatedRevision\": \"sklearn-iris-predictor-00003\",\n \"latestReadyRevision\": \"sklearn-iris-predictor-00003\",\n \"latestRolledoutRevision\": \"sklearn-iris-predictor-00001\",\n \"traffic\": [\n {\n \"latestRevision\": true,\n \"percent\": 10,\n \"revisionName\": \"sklearn-iris-predictor-00003\",\n \"tag\": \"latest\",\n \"url\": \"https://latest-sklearn-iris-predictor-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com\"\n },\n {\n \"latestRevision\": false,\n \"percent\": 90,\n \"revisionName\": \"sklearn-iris-predictor-00001\",\n \"tag\": \"prev\",\n \"url\": \"https://prev-sklearn-iris-predictor-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com\"\n }\n ],\n \"url\": \"http://sklearn-iris-predictor-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com\"\n}\n\n# Curl the latest revision\ncurl -k https://latest-sklearn-iris-predictor-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com/v1/models/sklearn-iris:predict -d @./kserve/samples/input-iris.json\n{\"predictions\":[1,1]}\n\n# Curl the previous revision\ncurl -k https://prev-sklearn-iris-predictor-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com/v1/models/sklearn-iris:predict -d @./kserve/samples/input-iris.json\n{\"predictions\":[1,1]}%\n```\n\n### Batching\n* From https://kserve.github.io/website/0.10/modelserving/batcher/batcher/\n\n```bash\n# Istio\noc apply -f kserve/samples/istio/torchserve-batch.yaml\n# Kourier\noc apply -f kserve/samples/kourier/torchserve-batch.yaml\n\n# Basic request\ncurl -k https://torchserve-predictor-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com/v1/models/mnist:predict -d @./kserve/samples/input-mnist.json\n\n# Use hey to create some parallel requests\nhey -z 10s -c 5 -m POST -D ./kserve/samples/input-mnist.json https://torchserve-predictor-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com/v1/models/mnist:predict\n\nSummary:\n Total: 10.3765 secs\n Slowest: 0.7197 secs\n Fastest: 0.5339 secs\n Average: 0.5461 secs\n Requests/sec: 9.1553\n\n Total data: 7695 bytes\n Size/request: 81 bytes\n\nResponse time histogram:\n 0.534 [1] |\n 0.552 [89] |β– β– β– β– β– β– β– β– β– β– β– β– β– β– β– β– β– β– β– β– β– β– β– β– β– β– β– β– β– β– β– β– β– β– β– β– β– β– β– β– \n 0.571 [0] |\n 0.590 [0] |\n 0.608 [0] |\n 0.627 [0] |\n 0.645 [0] |\n 0.664 [0] |\n 0.683 [0] |\n 0.701 [0] |\n 0.720 [5] |β– β– \n\n\nLatency distribution:\n 10% in 0.5353 secs\n 25% in 0.5357 secs\n 50% in 0.5366 secs\n 75% in 0.5374 secs\n 90% in 0.5379 secs\n 95% in 0.7197 secs\n 0% in 0.0000 secs\n\nDetails (average, fastest, slowest):\n DNS+dialup: 0.0096 secs, 0.5339 secs, 0.7197 secs\n DNS-lookup: 0.0073 secs, 0.0000 secs, 0.1380 secs\n req write: 0.0000 secs, 0.0000 secs, 0.0001 secs\n resp wait: 0.5364 secs, 0.5338 secs, 0.5385 secs\n resp read: 0.0000 secs, 0.0000 secs, 0.0001 secs\n\nStatus code distribution:\n [200] 95 responses\n```\n\n\n### Custom transformer (has service to service communication)\n* From https://kserve.github.io/website/0.10/modelserving/v1beta1/transformer/torchserve_image_transformer/\n\n```bash\n# Istio\noc apply -f kserve/samples/istio/torch-transformer.yaml\n# Kourier\noc apply -f kserve/samples/kourier/torch-transformer.yaml\n\n# Two Knative Services are created\noc get ksvc -n kserve-demo\nNAME URL LATESTCREATED LATESTREADY READY REASON\ntorch-transformer-predictor https://torch-transformer-predictor-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com torch-transformer-predictor-00002 torch-transformer-predictor-00002 True\ntorch-transformer-transformer https://torch-transformer-transformer-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com torch-transformer-transformer-00001 torch-transformer-transformer-00001 True\n\n# Run prediction\ncurl -k https://torch-transformer-transformer-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com/v1/models/mnist:predict -d @./kserve/samples/input-image.json\n\n{\"predictions\":[[2]]}%\n\n# In this example the transformer communicates with the predictor:\noc get po -n kserve-demo torch-transformer-transformer-00001-deployment-56db8449cb-wlcbp -o yaml | grep args -A 6\n- args:\n- --model_name\n- mnist\n- --predictor_host\n- torch-transformer-predictor.kserve-demo\n- --http_port\n- \"8080\"\n\n# So it will actually communicate with the DNS `torch-transformer-predictor.kserve-demo` and reach\n# 1) with kourier: the cluster-local gateway and go through Kourier\noc get svc -n kserve-demo torch-transformer-predictor\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\ntorch-transformer-predictor ExternalName \u003cnone\u003e kourier-internal.knative-serving-ingress.svc.cluster.local 80/TCP 13m\n\n# 2) with istio: bypass the cluster-local gateway and go directly through the mesh\noc get svc -n kserve-demo torch-transformer-predictor\n\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\ntorch-transformer-predictor ExternalName \u003cnone\u003e knative-local-gateway.istio-system.svc.cluster.local 80/TCP 29s\n\n# bypassed because of the mesh virtual-services:\noc get virtualservice -n kserve-demo | grep mesh\nNAME GATEWAYS HOSTS AGE\ntorch-transformer-predictor-mesh [\"mesh\"] [\"torch-transformer-predictor.kserve-demo\",\"torch-transformer-predictor.kserve-demo.svc\",\"torch-transformer-predictor.kserve-demo.svc.cluster.local\"] 89s\ntorch-transformer-transformer-mesh [\"mesh\"] [\"torch-transformer-transformer.kserve-demo\",\"torch-transformer-transformer.kserve-demo.svc\",\"torch-transformer-transformer.kserve-demo.svc.cluster.local\"] 2m\n```\n\n### Inference Graph (has service to service communication)\n* From https://kserve.github.io/website/0.10/modelserving/inference_graph/image_pipeline/\n* This is currently broken in KServe. Wait for https://github.com/kserve/kserve/pull/2830 to be merged to re-test.\n* An additional PR was necessary: https://github.com/kserve/kserve/pull/2839\n\n```bash\n# Patch the kserve cluster role to allow setting finalizers on `InferenceGraphs`. Otherwise we end up with\n# \"error\":\"services.serving.knative.dev \\\"dog-breed-pipeline\\\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , \u003cnil\u003e\"\n# PR to KServe to fix this: https://github.com/kserve/kserve/pull/2839\noc apply -f kserve/kserve-cluster-role-patch.yaml\n\n# Istio\noc apply -f kserve/samples/istio/cat-dog-breed.yaml\noc apply -f kserve/samples/istio/inference-graph.yaml\n# Kourier\noc apply -f kserve/samples/kourier/cat-dog-breed.yaml\noc apply -f kserve/samples/kourier/inference-graph.yaml\n\n# Get the URL of the inference graph\noc get ig dog-breed-pipeline -n kserve-demo\nNAME URL READY AGE\ndog-breed-pipeline https://dog-breed-pipeline-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com True 24s\n\n# Calling the service\ncurl -k https://dog-breed-pipeline-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com -d @./kserve/samples/input-cat.json\n\n{\"predictions\": [\"cat\"]}% \n\n# Kourier\nTODO: This is currently broken in KServe. Wait for https://github.com/kserve/kserve/pull/2830 to be merged to re-test.\nIssue created: https://github.com/kserve/kserve/issues/2856\n```\n\nAdditional VS that KServe generates for the wrapper service:\n```yaml\napiVersion: networking.istio.io/v1beta1\nkind: VirtualService\nmetadata:\n annotations:\n serving.knative.openshift.io/enablePassthrough: \"true\"\n sidecar.istio.io/inject: \"true\"\n sidecar.istio.io/rewriteAppHTTPProbers: \"true\"\n labels:\n serviceEnvelope: kserve\n name: cat-dog-classifier\n namespace: kserve-demo\n ownerReferences:\n - apiVersion: serving.kserve.io/v1beta1\n blockOwnerDeletion: true\n controller: true\n kind: InferenceService\n name: cat-dog-classifier\n uid: 61182a3a-376a-4db3-a8b0-a9e403b82903\n uid: 91204ee2-c62f-4eed-be71-a1563502266d\nspec:\n gateways:\n - knative-serving/knative-local-gateway\n - knative-serving/knative-ingress-gateway\n hosts:\n - cat-dog-classifier.kserve-demo.svc.cluster.local\n - cat-dog-classifier-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com\n http:\n - headers:\n request:\n set:\n Host: cat-dog-classifier-predictor.kserve-demo.svc.cluster.local\n match:\n - authority:\n regex: ^cat-dog-classifier\\.kserve-demo(\\.svc(\\.cluster\\.local)?)?(?::\\d{1,5})?$\n gateways:\n - knative-serving/knative-local-gateway\n - authority:\n regex: ^cat-dog-classifier-kserve-demo\\.apps\\.rlehmann-ocp-4-12\\.serverless\\.devcluster\\.openshift\\.com(?::\\d{1,5})?$\n gateways:\n - knative-serving/knative-ingress-gateway\n route:\n - destination:\n host: knative-local-gateway.istio-system.svc.cluster.local\n port:\n number: 80 -\u003e 8081 (http2)\n weight: 100\n```\n\n### Model Explainability\n* From https://kserve.github.io/website/0.10/modelserving/explainer/alibi/moviesentiment/\n* Has upstream issues: https://github.com/kserve/kserve/issues/2843\n* Does only work with a patch on the kserve images: \n * https://github.com/kserve/kserve/issues/2844\n * https://github.com/kserve/kserve/pull/2845\n\n\n```bash\n# Istio\noc apply -f kserve/samples/istio/moviesentiment.yaml\n\n# Kourier\noc apply -f kserve/samples/kourier/moviesentiment.yaml\n\n# Predict\ncurl -k https://moviesentiment-predictor-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com/v1/models/moviesentiment:predict -d '{\"instances\":[\"a visually flashy but narratively opaque and emotionally vapid exercise .\"]}'\n{\"predictions\":[0]}\n\ncurl -k https://moviesentiment-predictor-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com/v1/models/moviesentiment:predict -d '{\"instances\":[\"a touching , sophisticated film that almost seems like a documentary in the way it captures an italian immigrant family on the brink of major changes .\"]}'\n{\"predictions\":[1]}\n\n# Explain\ncurl -k https://moviesentiment-explainer-kserve-demo.apps.rlehmann-ocp-4-12.serverless.devcluster.openshift.com/v1/models/moviesentiment:explain -d '{\"instances\":[\"a visually flashy but narratively opaque and emotionally vapid exercise .\"]}'\n{\"meta\":{\"name\":\"AnchorText\",\"type\":[\"blackbox\"],\"explanations\":[\"local\"],\"params\":{\"seed\":0,\"sample_proba\":0.5},\"version\":\"0.6.5\"},\"data\":{\"anchor\":[\"exercise\"],\"precision\":1.0,\"coverage\":0.5005,\"raw\":{\"feature\":[9],\"mean\":[1.0],\"precision\":[1.0],\"coverage\":[0.5005],\"examples\":[{\"covered_true\":[\"UNK UNK flashy but narratively UNK UNK emotionally vapid exercise UNK\",\"UNK visually flashy but UNK UNK UNK UNK vapid exercise .\",\"UNK visually flashy UNK UNK opaque UNK UNK UNK exercise .\",\"UNK UNK UNK but narratively opaque and emotionally vapid exercise UNK\",\"UNK UNK flashy but narratively UNK and emotionally vapid exercise .\",\"a visually UNK but UNK opaque UNK UNK UNK exercise UNK\",\"a visually UNK UNK narratively UNK UNK UNK UNK exercise .\",\"UNK UNK flashy UNK narratively UNK UNK UNK vapid exercise UNK\",\"a UNK flashy UNK UNK UNK and UNK UNK exercise UNK\",\"a visually UNK but UNK UNK and emotionally vapid exercise .\"],\"covered_false\":[],\"uncovered_true\":[],\"uncovered_false\":[]}],\"all_precision\":0,\"num_preds\":1000000,\"success\":true,\"names\":[\"exercise\"],\"positions\":[63],\"instance\":\"a visually flashy but narratively opaque and emotionally vapid exercise .\",\"instances\":[\"a visually flashy but narratively opaque and emotionally vapid exercise .\"],\"prediction\":[0]}}}\n```\n## Testing KServe with Knative Eventing\n\n### Prerequisites\n\nMake sure you followed the instructions in [Testing KServe installation - Prerequisites](#prerequisites-1) first.\n\nThen follow these:\n```bash\n\n# Create a Knative Eventing installation\noc apply -f serverless/knativeeventing.yaml\n\n# Create resources for inference logging: broker, trigger, message dumper ksvc\noc apply -f serverless/inference-logger-resources.yaml\n```\n\n### Inference logging\n\n```\n β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”\n β”‚ β”‚\n β”‚ InferenceService β”‚\n β”‚ β”‚\nβ”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚\nβ”‚ β”œβ”€β”€β”€β”Όβ”€β”€β–Ίβ”‚ Predictor β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”\nβ”‚ Client β”‚ β”‚ β”‚ Service β”‚ β”‚ β”‚ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚\nβ”‚ │◄──┼──── β”‚ β”‚ β”Œβ”€β”€β”€β–Ίβ”‚ Broker β”‚ Triggerβ”œβ”€β”€β”€β–Ίβ”‚ Event Display β”‚\nβ””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ Service β”‚\n β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚\n β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜\n β”‚ β”‚ β”‚ β”‚ β”‚\n β”‚ β”‚ Logger β”œβ”€β”€β”€β”Όβ”€β”€β”€β”˜\n β”‚ β”‚ β”‚ β”‚\n β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚\n β”‚ β”‚\n β”‚ β”‚\n β”‚ β”‚\n β”‚ β”‚\n β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜\n```\n\n```bash\n# Same as the previous tests, only this time we have inference log events sent to Knative Eventing Broker\n$ oc apply -f kserve/samples/istio/sklearn-iris-inference-logging.yaml\n\n$ curl -k https://sklearn-iris-predictor-kserve-demo.apps.aliok-c088.serverless.devcluster.openshift.com/v1/models/sklearn-iris:predict -d @./kserve/samples/input-iris.json\n{\"predictions\":[1,1]}% \n\n$ k logs -n kserve-demo message-dumper-00001-deployment-7f7874bd66-ssv89\n\n☁️ cloudevents.Event\nValidation: valid\nContext Attributes,\n specversion: 1.0\n type: org.kubeflow.serving.inference.response\n source: http://localhost:9081/\n id: 464d1291-1bbd-4c05-b592-d6ae24de6555\n time: 2023-05-02T14:05:37.874021224Z\n datacontenttype: application/json\nExtensions,\n component: predictor\n endpoint:\n inferenceservicename: sklearn-iris\n knativearrivaltime: 2023-05-02T14:05:37.877365133Z\n namespace: kserve-demo\n traceparent: 00-aa63c8db0dff507e246a7abf10a21a7d-d082068f6e5be1d3-00\nData,\n {\n \"predictions\": [\n 1,\n 1\n ]\n }\n☁️ cloudevents.Event\nValidation: valid\nContext Attributes,\n specversion: 1.0\n type: org.kubeflow.serving.inference.request\n source: http://localhost:9081/\n id: 464d1291-1bbd-4c05-b592-d6ae24de6555\n time: 2023-05-02T14:05:37.871883206Z\n datacontenttype: application/x-www-form-urlencoded\nExtensions,\n component: predictor\n endpoint:\n inferenceservicename: sklearn-iris\n knativearrivaltime: 2023-05-02T14:05:37.878989816Z\n namespace: kserve-demo\n traceparent: 00-f53727be09d26dfb9f7d75bb436a9280-d6b647fe8adc12ff-00\nData,\n { \"instances\": [ [6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6] ]}\n```\n\nPRs:\n- https://github.com/kserve/kserve/pull/2792\n\n### Drift detection\n\n```\n β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”\n β”‚ β”‚\n β”‚ InferenceService β”‚\n β”‚ β”‚\nβ”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚\nβ”‚ β”œβ”€β”€β”€β”Όβ”€β”€β–Ίβ”‚ Predictor β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”\nβ”‚ Client β”‚ β”‚ β”‚ Service β”‚ β”‚ β”‚ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚\nβ”‚ │◄──┼──── β”‚ β”‚ β”Œβ”€β”€β”€β–Ίβ”‚ Broker β”‚ Triggerβ”œβ”€β”€β”€β–Ίβ”‚ Drift Detector β”‚\nβ””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ Service β”‚\n β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚\n β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜\n β”‚ β”‚ β”‚ β”‚ β”‚\n β”‚ β”‚ Logger β”œβ”€β”€β”€β”Όβ”€β”€β”€β”˜\n β”‚ β”‚ β”‚ β”‚\n β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚\n β”‚ β”‚\n β”‚ β”‚\n β”‚ β”‚\n β”‚ β”‚\n β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜\n```\n\nSee https://github.com/aliok/yaml/tree/main/data-science#drift-detection-on-openshift\n\nPRs:\n- https://github.com/kserve/kserve/pull/2787\n- https://github.com/kserve/kserve/pull/2888\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fretocode%2Fknative-kserve","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fretocode%2Fknative-kserve","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fretocode%2Fknative-kserve/lists"}