{"id":16827038,"url":"https://github.com/retocode/knative-scaling-scenarios","last_synced_at":"2025-03-17T19:45:46.059Z","repository":{"id":154107295,"uuid":"622982943","full_name":"ReToCode/knative-scaling-scenarios","owner":"ReToCode","description":null,"archived":false,"fork":false,"pushed_at":"2023-04-06T13:48:56.000Z","size":1170,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-24T07:08:32.808Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ReToCode.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-03T13:10:17.000Z","updated_at":"2023-04-04T13:18:58.000Z","dependencies_parsed_at":null,"dependency_job_id":"754d4e11-6e85-4407-bbd8-f671cbb5d80c","html_url":"https://github.com/ReToCode/knative-scaling-scenarios","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ReToCode%2Fknative-scaling-scenarios","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ReToCode%2Fknative-scaling-scenarios/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ReToCode%2Fknative-scaling-scenarios/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ReToCode%2Fknative-scaling-scenarios/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ReToCode","download_url":"https://codeload.github.com/ReToCode/knative-scaling-scenarios/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244102040,"owners_count":20398379,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T11:19:27.244Z","updated_at":"2025-03-17T19:45:46.030Z","avatar_url":"https://github.com/ReToCode.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Knative Scaling Scenarios\n\n## References\n* [Upstream performance testing](https://github.com/knative/serving/issues/1625#issuecomment-511930023)\n* [Upstream autoscale-go](https://github.com/knative/docs/tree/main/docs/serving/autoscaling/autoscale-go)\n* [xk6-output-prometheus-remote](https://github.com/grafana/xk6-output-prometheus-remote)\n\n## Prerequisites\n* A kubernetes cluster\n* Configured `kubectl` pointing to the cluster\n* `Knative Serving` installed (HEAD: `ko apply -Rf config/core`)\n* `net-kourier` installed (HEAD: `ko apply -Rf config`)\n\n## Configuration\n```bash\n# Create random payload\nhead -c $( echo 10K | numfmt --from=iec ) \u003c/dev/urandom \u003e payload/10K\nhead -c $( echo 100K | numfmt --from=iec ) \u003c/dev/urandom \u003e payload/100K\nhead -c $( echo 1000K | numfmt --from=iec ) \u003c/dev/urandom \u003e payload/1000K\n```\n\n### Kind\n```bash\n# Knative config\nkubectl patch configmap/config-network \\\n  --namespace knative-serving \\\n  --type merge \\\n  --patch '{\"data\":{\"ingress-class\":\"kourier.ingress.networking.knative.dev\"}}'\nkubectl patch cm/config-features -n knative-serving -p '{\"data\":{\"'kubernetes.podspec-init-containers'\":\"'Enabled'\"}}'\nkubectl patch cm/config-autoscaler -n knative-serving -p '{\"data\":{\"'allow-zero-initial-scale'\":\"'true'\"}}'\nkubectl patch cm/config-observability -n knative-serving -p '{\"data\":{\"'profiling.enable'\":\"'true'\"}}'\n\n# Metrics for kind\nkubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.6.3/components.yaml\nkubectl patch -n kube-system deployment metrics-server --type=json \\\n  -p '[{\"op\":\"add\",\"path\":\"/spec/template/spec/containers/0/args/-\",\"value\":\"--kubelet-insecure-tls\"}]'\n\n# Make sure kourier is scaled enough to not be the bottleneck\nkubectl -n kourier-system patch hpa 3scale-kourier-gateway --patch '{\"spec\":{\"minReplicas\":10}}'\n```\n\n### OpenShift\n```bash\n# Install Operator and Knative Serving\noc apply -f openshift/serverless-operator.yaml\noc apply -f openshift/knative-serving.yaml\n```\n\n## Running test scenarios\n### Environment set up\n```bash\nkubectl apply -f visualization/prometheus\nkubectl apply -f visualization/grafana\n```\n\n// TODO: HERE\n\nAll the test scenarios share the following configuration\n```bash\nexport DOMAIN=10.89.0.200.sslip.io\n```\n\n### Results\nThe results will be written to `results` and `results/heap`. The files are prefixed with the run-timestamp:\n```bash\n./run_tests.sh\nStarting a new run for timestamp: 1680616481\nStarting background process to monitor activator stats and profile memory\nRunning performance scenarios\nTrying to stop the background jobs, if this fails please manually check [1]-  Running                 kubectl port-forward deployment/activator -n knative-serving 8008:8008 \u003e /dev/null 2\u003e\u00261 \u0026\n[2]+  Running                 while true; do\n    kubectl top -n knative-serving pod --containers=true | grep activator \u003e\u003e results_local/\"${date}\"-activator-stats.log; curl http://localhost:8008/debug/pprof/heap \u003e results_local/heap/\"${date}\"-$(date +%s)-activator-heap.out \u003e /dev/null 2\u003e\u00261; sleep 1;\ndone \u0026 and stop them\nThe results_local were written to the results_local folder\n\ntree results_local            \nresults_local\n├── 1680616481-activator-stats.log\n├── 1680616481-config\n├── 1680616481-k6s-stats.log\n└── heap\n    ├── 1680616481-1680616481-activator-heap.out\n    ├── 1680616481-1680616482-activator-heap.out\n    ├── 1680616481-1680616483-activator-heap.out\n```\n\n### Checking heap dumps\n```bash\ngo tool pprof -http=:8080 results_local/heap/xxx.out\n\n# Running GC and compare\ncurl \"http://localhost:8008/debug/pprof/heap\" \u003e out1\ncurl \"http://localhost:8008/debug/pprof/heap?gc=1\"\n\n# wait a while and dump again\ncurl \"http://localhost:8008/debug/pprof/heap\" \u003e out2\n\n# Compare the dumps\ngo tool pprof -http=:8080 -diff_base out1 out2\n```\n\n\nCheck the results at [http://localhost:3000](http://localhost:3000)\n\n## Scenarios\n### 1) Activator always in path + scale to the limit\n**Description**\nThis scenario tests the scaling limit of one activator and checks resource usage of that pod.\n* We have 20 already running KServices without any delays or sleeps\n* We have only one activator (HPA is set to 1-1)\n* The activator always stays in the path\n* We start with $BASE_REQUEST_TARGET and double it every 30 seconds\n\n**Preparation**\n```bash\n# Kind: Patch Activator HPA\nkubectl -n knative-serving patch hpa activator --patch '{\"spec\":{\"minReplicas\":1, \"maxReplicas\": 1}}'\n\n# OCP: Patch Activator HPA\noc apply -f openshift/knative-serving-one-activator.yaml\nkubectl -n knative-serving patch hpa activator --patch '{\"spec\":{\"minReplicas\":1, \"maxReplicas\": 1}}'\n\n# Create the KService\nkubectl apply -f scenarios/activator-limit/services\n\n# Wait for all services to be ready\nkubectl wait kservice --for=condition=Ready -n default --all\n```\n\n**Running**\n```bash\n# Starting RPS, this will be doubled 10 times every 30 seconds \nexport SERVICE_COUNT=1\nexport BASE_REQUEST_TARGET=10\n./run_tests.sh scenarios/activator-limit/tests.js\n```\n\n\n### 2) Scaling with delays\n**Description**\nWe have three types of KServices:\n* No startup delay\n* 5s startup delay\n* 15s startup delay\n* We have only one activator (HPA is set to 1-1) \n* All have target of 10 RPS, so if we send 50 requests, activator will scale to 5 instances\n \nThis scenario consists of batch requests per VU with the following matrix\n* Amount of target KServices (e.g. 1-15)\n* Startup delay: (no, 5s, 15s) --\u003e random value\n* Slow responses: (none, 10ms, 100ms, 1000ms, 5000ms) --\u003e random value\n* Payload: (none, 10K, 100K, 500K) --\u003e random value\n\n**Preparation**\n```bash\n# Kind: Patch Activator HPA\nkubectl -n knative-serving patch hpa activator --patch '{\"spec\":{\"minReplicas\":1, \"maxReplicas\": 1}}'\n\n# OCP: Patch Activator HPA\n// TODO\n\n# Create the requested amount of KServices\nexport SERVICE_COUNT=15\nfor (( i = 0; i \u003c SERVICE_COUNT; i++ )); do\n    cat scenarios/scaling/services/* | sed -e \"s/COUNT/${i}/g\" | kubectl apply -f -\ndone\n\n# Wait for all services to be ready\nkubectl wait kservice --for=condition=Ready -n default --all\n```\n\n**Running**\n```bash\n# This targets k6's Virtual Users (https://k6.io/docs/get-started/running-k6/)\n# each VU does multiple requests, depending on the scenario. VUs are essentially parallel while(true) loops.\n# Tests will start with this base VU value and increase the load over time\nexport BASE_REQUEST_TARGET=30\nexport SERVICE_COUNT=15 # same as above\n\n./run_tests.sh scenarios/scaling/tests.js\n```\n\n\n## Cleanup\n```bash\n# Kind\nkubectl -n knative-serving patch hpa activator --patch '{\"spec\":{\"minReplicas\":1, \"maxReplicas\": 20}}'\n\n# OCP\noc apply -f openshift/knative-serving-default.yaml\n\n# Cleanup KServices\nkubectl delete ksvc --all -A\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fretocode%2Fknative-scaling-scenarios","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fretocode%2Fknative-scaling-scenarios","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fretocode%2Fknative-scaling-scenarios/lists"}