{"id":16826961,"url":"https://github.com/retocode/ocp-serverless-perfromance-testing","last_synced_at":"2025-10-19T15:51:57.596Z","repository":{"id":191136972,"uuid":"683997474","full_name":"ReToCode/ocp-serverless-perfromance-testing","owner":"ReToCode","description":null,"archived":false,"fork":false,"pushed_at":"2023-09-11T08:44:50.000Z","size":233,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-17T19:49:11.313Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ReToCode.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-28T08:27:16.000Z","updated_at":"2023-09-01T07:45:03.000Z","dependencies_parsed_at":"2024-11-24T02:52:01.333Z","dependency_job_id":null,"html_url":"https://github.com/ReToCode/ocp-serverless-perfromance-testing","commit_stats":null,"previous_names":["retocode/ocp-serverless-perfromance-testing"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ReToCode/ocp-serverless-perfromance-testing","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ReToCode%2Focp-serverless-perfromance-testing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ReToCode%2Focp-serverless-perfromance-testing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ReToCode%2Focp-serverless-perfromance-testing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ReToCode%2Focp-serverless-perfromance-testing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ReToCode","download_url":"https://codeload.github.com/ReToCode/ocp-serverless-perfromance-testing/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ReToCode%2Focp-serverless-perfromance-testing/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266578709,"owners_count":23951150,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T11:19:14.186Z","updated_at":"2025-10-19T15:51:52.546Z","avatar_url":"https://github.com/ReToCode.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OpenShift Serverless Performance Testing\n\n## Setup\n\n### Environment\n\nThe scripts expect the following environment to be set\n```bash\nexport KO_DOCKER_REPO=quay.io/rlehmann\nexport SYSTEM_NAMESPACE=knative-serving\nexport KO_DEFAULTPLATFORMS=linux/amd64\nexport SERVING=/Users/rlehmann/code/knative/serving\n```\n\n### Influx DB\n```bash\nkubectl create ns influx\noc apply -f ./yaml/ocp-hack.yaml\nhelm upgrade --install -n influx local-influx --set persistence.enabled=false,ingress.enabled=true,ingress.hostname=influx.apps.rlehmann-ocp.serverless.devcluster.openshift.com influxdata/influxdb2\nexport INFLUX_URL=http://influx.apps.rlehmann-ocp.serverless.devcluster.openshift.com\necho $(kubectl get secret local-influx-influxdb2-auth -o \"jsonpath={.data['admin-token']}\" --namespace influx | base64 --decode)\necho $(kubectl get secret local-influx-influxdb2-auth -o \"jsonpath={.data['admin-password']}\" --namespace influx | base64 --decode)\nexport INFLUX_TOKEN=$(kubectl get secret local-influx-influxdb2-auth -o \"jsonpath={.data['admin-token']}\" --namespace influx | base64 --decode)\n$SERVING/test/performance/visualization/setup-influx-db.sh\n```\n\n### Backing up InfluxDB\nOnly relevant on demand (if you want to save the data):\n```bash\nexport INFLUX_URL=http://influx.apps.rlehmann-ocp.serverless.devcluster.openshift.com\ninflux backup --host=$INFLUX_URL backup\n```\n\n### Grafana\n```bash\n# Setup grafana\noc apply -f yaml/grafana-subscription.yaml\noc apply -f yaml/grafana.yaml\necho $(kubectl get secret grafana-admin-credentials -o \"jsonpath={.data['GF_SECURITY_ADMIN_PASSWORD']}\" --namespace influx | base64 --decode)\n```\n\nUsing influxDB as a datasource for Grafana\n* Navigate to Grafana UI and log in using the user from above (e.g. https://grafana-route-influx.apps.rlehmann-ocp.serverless.devcluster.openshift.com/)\n* Create a new datasource for InfluxDB\n* Select the flux query language\n* Server-URL: http://local-influx-influxdb2.influx:80\n* Organization: Knativetest\n* Bucket: knative-serving\n* Token: \u003cyour influx-db token\u003e\n\n\n## Manually creating Knative Services\n\nThis is only relevant for manual testing\n```bash\nkubectl create ns knative-performance\nfor i in {1..1000}\ndo\n   kn service create \"hello-go-$i\" --image=gcr.io/knative-samples/helloworld-go -n knative-performance \u0026\ndone\n```\n\n## Running the tests in the scenarios\n\n### Cluster setup: scenario small\n\n```bash\n# Scale machines\nfor name in $(oc get machineset -n openshift-machine-api -o name); do oc scale $name -n openshift-machine-api --replicas=2; done\n\n# Set resource scenario\noc apply -f scenarios/knative-serving-minimal.yaml\n\n# hack to stop HPA from interfering - this is optional (depending on tests)\noc scale deploy -n openshift-serverless knative-operator-webhook --replicas=0\noc -n knative-serving patch hpa activator --patch '{\"spec\":{\"minReplicas\":1, \"maxReplicas\": 1}}'\noc -n knative-serving patch hpa webhook --patch '{\"spec\":{\"minReplicas\":1, \"maxReplicas\": 1}}'\n```\n\n### Cluster setup: scenario limits\nThis scenario is just to find the limits of the system.\n\n```bash\n# Scale machines\nfor name in $(oc get machineset -n openshift-machine-api -o name); do oc scale $name -n openshift-machine-api --replicas=4; done\noc wait --for=jsonpath={.status.availableReplicas}=4 machineset --all -n openshift-machine-api --timeout=-1s\n\n# Set resource scenario\noc apply -f scenarios/knative-serving-limits.yaml\n\n# hack to stop HPA from interfering - this is optional (depending on tests)\noc scale deploy -n openshift-serverless knative-operator-webhook --replicas=0\noc -n knative-serving patch hpa activator --patch '{\"spec\":{\"minReplicas\":10, \"maxReplicas\": 10}}'\noc -n knative-serving patch hpa webhook --patch '{\"spec\":{\"minReplicas\":2, \"maxReplicas\": 2}}'\n\n# optional patch for reconciliation-delay\noc patch cm config-autoscaler -n knative-serving -p '{\"data\": {\"allow-zero-initial-scale\": \"true\"}}'\n\n# optional if we have issues here\noc scale deploy -n knative-serving-ingress 3scale-kourier-gateway --replicas=20\n```\n\n**Running the tests: limits**\nRunning the tests needs some tweaks, because one calling pod is not enough:\n\n```bash\n# for local debugging only\nexport ARTIFACTS=$PWD/logs\nname= xxx; stern -n default --selector=job-name=\"$name\" | tee \"$ARTIFACTS/stern-$name.log\"\n```\n\n```bash\n# Environment\nexport KO_DOCKER_REPO=quay.io/rlehmann\nexport SYSTEM_NAMESPACE=knative-serving\nexport KO_DEFAULTPLATFORMS=linux/amd64\nexport SERVING=/Users/rlehmann/code/knative/serving\nexport INFLUX_URL=http://local-influx-influxdb2.influx:80\nexport ARTIFACTS=$PWD/logs\n\n# You can run the tests individually\n./scripts/run-reconciliation-delay.sh \"1000ms\"  # frequency (default=5s)\n\n./scripts/run-load-test.sh 5 # parallelism (default=1)\n\n./scripts/run-rollout-probe.sh 200 250 # min-replicas, max-replicas (defaults: 100, 150)\n\n./scripts/run-dataplane-probe.sh 15 # parallelism (default=1)\n\n./scripts/run-real-traffic-test.sh\n```\n\n### Cluster setup: Kourier with KnativeServing defaults\n\n```bash\n# Cluster setup\n# Scale machines\nfor name in $(oc get machineset -n openshift-machine-api -o name); do oc scale $name -n openshift-machine-api --replicas=4; done\noc wait --for=jsonpath={.status.availableReplicas}=4 machineset --all -n openshift-machine-api --timeout=-1s\n\n# Set resource scenario\noc apply -f scenarios/knative-serving-kourier-defaults.yaml\n\n# Environment\nexport KO_DOCKER_REPO=quay.io/rlehmann\nexport SYSTEM_NAMESPACE=knative-serving\nexport KO_DEFAULTPLATFORMS=linux/amd64\nexport SERVING=/Users/rlehmann/code/knative/serving\nexport INFLUX_URL=http://local-influx-influxdb2.influx:80\nexport ARTIFACTS=$PWD/logs\n\n./scripts/run-all-performance-tests.sh\n```\n\n### Cluster setup: Istio with KnativeServing defaults\n\n```bash\n# Cluster setup\n# \u003e Install Istio + Serverless, SMCP, SMMR and gateways as per S-O script \n# \u003e Scale machines\nfor name in $(oc get machineset -n openshift-machine-api -o name); do oc scale $name -n openshift-machine-api --replicas=4; done\noc wait --for=jsonpath={.status.availableReplicas}=4 machineset --all -n openshift-machine-api --timeout=-1s\n\n# Set resource scenario\noc apply -f scenarios/knative-serving-istio-defaults.yaml\n\n# Deploy the testing webhook, that injects istio annotations to ksvc\nexport SERVERLESS_OPERATOR=/Users/rlehmann/code/openshift-knative/serverless-operator\noc apply -f $SERVERLESS_OPERATOR/serving/metadata-webhook/config\nsed \"s|registry.ci.openshift.org/knative/openshift-serverless-nightly:metadata-webhook|quay.io/openshift-knative/metadata-webhook:latest|g\" $SERVERLESS_OPERATOR/serving/metadata-webhook/config/webhook.yaml | oc apply -f - \n\n# Environment\nexport KO_DOCKER_REPO=quay.io/rlehmann\nexport SYSTEM_NAMESPACE=knative-serving\nexport KO_DEFAULTPLATFORMS=linux/amd64\nexport SERVING=/Users/rlehmann/code/knative/serving\nexport INFLUX_URL=http://local-influx-influxdb2.influx:80\nexport ARTIFACTS=$PWD/logs\n\n# Manually (for now) make sure to add istio-inject annotations to all jobs in $SERVING and the deployment in dataplane-probe-setup.yaml\n#   metadata:\n#     annotations:\n#       sidecar.istio.io/inject: \"true\"\n#       proxy.istio.io/config: { \"holdApplicationUntilProxyStarts\": true }\n        \n./scripts/run-all-performance-tests.sh\n```\n\n⛔️ Note: The tests do not run with our default settings. See below for a changed setup.\n\n\n### Cluster setup: Istio with KnativeServing increased resources\n\n```bash\n# Cluster setup\n# \u003e Install Istio + Serverless, SMCP, SMMR and gateways as per S-O script\n\n# Patch the SMCP + gateway with increased defaults\noc apply -f scenarios/customizations-istio\n \n# \u003e Scale machines\nfor name in $(oc get machineset -n openshift-machine-api -o name); do oc scale $name -n openshift-machine-api --replicas=4; done\noc wait --for=jsonpath={.status.availableReplicas}=4 machineset --all -n openshift-machine-api --timeout=-1s\n\n# Set KnativeServing with increased resources\noc apply -f scenarios/knative-serving-istio-increased-limits.yaml\n\n# Deploy the testing webhook, that injects istio annotations to ksvc\nexport SERVERLESS_OPERATOR=/Users/rlehmann/code/openshift-knative/serverless-operator\noc apply -f $SERVERLESS_OPERATOR/serving/metadata-webhook/config\nsed \"s|registry.ci.openshift.org/knative/openshift-serverless-nightly:metadata-webhook|quay.io/openshift-knative/metadata-webhook:latest|g\" $SERVERLESS_OPERATOR/serving/metadata-webhook/config/webhook.yaml | oc apply -f - \n\n# Environment\nexport KO_DOCKER_REPO=quay.io/rlehmann\nexport SYSTEM_NAMESPACE=knative-serving\nexport KO_DEFAULTPLATFORMS=linux/amd64\nexport SERVING=/Users/rlehmann/code/knative/serving\nexport INFLUX_URL=http://local-influx-influxdb2.influx:80\nexport ARTIFACTS=$PWD/logs\n\n# Manually (for now) make sure to add istio-inject annotations to all jobs in $SERVING and the deployment in dataplane-probe-setup.yaml\n#   metadata:\n#     annotations:\n#       sidecar.istio.io/inject: \"true\"\n#       proxy.istio.io/config: { \"holdApplicationUntilProxyStarts\": true }\n        \n./scripts/run-all-performance-tests.sh\n```\n\n## Testing throughput and resources of data-path components\n\n### Setup\n```bash\n# Setup OSS with Kourier\n# Make sure we only have one 3scale-kourier-gateway and one activator\noc apply -f scenarios/pod-throughput-limits/knative-serving-kourier.yaml\n```\n\n### 3scale-kourier-gateway\n\n```bash\noc apply -f scenarios/pod-throughput-limits/ksvc-no-activator.yaml\n\n# No payload\noc apply -f scenarios/pod-throughput-limits/vegeta-3scale-kourier-gateway.yaml\n\n# With payload\noc apply -f scenarios/pod-throughput-limits/vegeta-3scale-kourier-gateway-body.yaml\n\n# Dump the logs\noc logs -f -n default \"job.batch/load-test\" \u003e\"$ARTIFACTS/load-test-kourier.log\"\n```\n\n### Activator\n\n```bash\n# Stop HPA from interfering\noc scale deploy -n openshift-serverless knative-operator-webhook --replicas=0\noc -n knative-serving patch hpa activator --patch '{\"spec\":{\"minReplicas\":1, \"maxReplicas\": 1}}'\n\n# Deploy backing pod\noc apply -f scenarios/pod-throughput-limits/ksvc-always-activator.yaml\n\n# No payload\noc apply -f scenarios/pod-throughput-limits/vegeta-activator.yaml\n\n# With payload\noc apply -f scenarios/pod-throughput-limits/vegeta-activator-boyd.yaml\n\n# Dump the logs\noc logs -f -n default \"job.batch/load-test\" \u003e\"$ARTIFACTS/load-test-activator.log\"\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fretocode%2Focp-serverless-perfromance-testing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fretocode%2Focp-serverless-perfromance-testing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fretocode%2Focp-serverless-perfromance-testing/lists"}