{"id":18589372,"url":"https://github.com/pmorie/middleware-keynote-demo-blog","last_synced_at":"2026-02-14T21:31:15.041Z","repository":{"id":138121658,"uuid":"38634283","full_name":"pmorie/middleware-keynote-demo-blog","owner":"pmorie","description":null,"archived":false,"fork":false,"pushed_at":"2015-07-22T03:00:07.000Z","size":150,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-10T06:14:59.248Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pmorie.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-07-06T17:23:53.000Z","updated_at":"2015-07-06T17:23:53.000Z","dependencies_parsed_at":"2023-03-23T00:17:30.596Z","dependency_job_id":null,"html_url":"https://github.com/pmorie/middleware-keynote-demo-blog","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/pmorie/middleware-keynote-demo-blog","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pmorie%2Fmiddleware-keynote-demo-blog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pmorie%2Fmiddleware-keynote-demo-blog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pmorie%2Fmiddleware-keynote-demo-blog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pmorie%2Fmiddleware-keynote-demo-blog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pmorie","download_url":"https://codeload.github.com/pmorie/middleware-keynote-demo-blog/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pmorie%2Fmiddleware-keynote-demo-blog/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29456216,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-14T21:29:27.764Z","status":"ssl_error","status_checked_at":"2026-02-14T21:28:11.111Z","response_time":53,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T00:51:52.209Z","updated_at":"2026-02-14T21:31:15.019Z","avatar_url":"https://github.com/pmorie.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Middleware Keynote Demo at Summit 2015\n\nIn this blog post, we'll deconstruct the OpenShift portions of the Middleware Keynote Demo that was\nshow at Red Hat Summit 2015.  If you haven't seen the keynote and demo, you can watch them\n[here](https://www.redhat.com/en/about/videos/craig-muzilla-middleware-keynote-2015).  We got a lot\nof positive feedback about this demo and so we thought it might be interesting if we pulled the\ncovers back a little and examined some of the work that went into making it happen.\n\n## The requirements\n\nWe had a few fairly challenging requirements for OpenShift during this demo.  The main requirements\nthis blog post will discuss are:\n\n1.  We had to be able to do a build of the 'sketchpod' application in under 30 seconds\n2.  We had to be able to scale from 1 replicas of the built pod to 10 replicas in under 20 seconds\n3.  We had to be able to scale from 10 to 1026 replicas in under 3 minutes\n4.  Since OpenShift v3 GA wasn't finished, we had to make the beta4 release meet all of the above\n\nThe timing requirements were dictated by the storyboard for the presentation.  I had never worked\non demo this public before.  It was a lot more like what I imagine working on a television show\nwould be like than working on a typical demo.\n\n## The demo architecture\n\nTODO: \n\nLet's take a look at the high-level architecture for this portion of the demo to make sure we have\na good shared understanding to base our discussion on:\n\n\nTODO: block diagram here and flesh out explanation\nOpenShift arch diagram:\n```\n                                   ------------------\n                                  | OpenShift Master |\n                                   ------------------\n                                           |\n                     ----------------------------------------------\n                    /               /            \\                 \\\n                   |               |              |                 |\n                   v               v              v                 v\n             ------------   -------------   -------------       -------------\n            | Infra Node | | App Node 1  | | App Node 2  | ... | App Node 20 |\n             ------------   -------------   -------------       -------------\n                   |             \\                |                   /\n            -----------           ------------------------------------\n           /           \\                  |                          |\n          |             |                 |                          |\n          v             v                 v                          v\n -----------------   --------         -----------               ------------\n| Custom Registry | | Router |       | Proxy App |             | Sketch App |\n -----------------   --------         -----------               ------------\n                                          |                          |\n                                    -------------               -----------------\n                                   /             \\             /                 \\\n                                  |               |           |                   |\n                              ---------       ---------    ----------       -------------\n                             | Proxy 1 | ... | Proxy 8 |  | Sketch 1 | ... | Sketch 1026 |\n                              ---------       ---------    ----------       -------------\n```\nTODO: Overall demo arch diagram, since the logical flow would be difficult to show in the context of the OpenShift arch diagram.\n\nMobile App -\u003e Feed Henry -\u003e Hex UI -\u003e Sketch Proxy -\u003e Sketch Pod\n\nWe'll probably also want to discuss the arch of the Hex UI a bit and how it was doing the api watch to get the pod status as they came up, and also map to the pod IPs.\n\n\n## Initial state: a semi-primed cluster\n\nDuring this post, we're going to be walking through the requirements in the order that we\ninvestigated and tuned them while preparing for the demo.  That being the case, it makes sense\nto articulate the state that the cluster was in once we really started locking down the performance\nrequirements we needed to hit.  The state the cluster was in once we in earnest began tuning for\nperformance was, at a very high level:\n\n1.  We had functional completeness for the demo -- ie, we could run through the demo fully, and\n    were basically optimizing for time\n2.  We were using a cluster with 5 nodes to host the application pods, an infrastructure node, and\n    a separate AWS vm for the master and datastore\n3.  The cluster was created from an ansible playbook that incorporated a step of pulling the base\n    image for the app onto all of the nodes in the cluster\n\nA quick note about the third bullet.  The step that pulls the nodejs builder image onto all the\nnodes in the cluster is what we'll refer to in this blog post as 'priming'.  The idea of priming\na system is pretty simple: many software systems need to do things like establish data caches or\nconnect to third party systems when they start up.  This means that the performance of the system\nin the steady state is often different from the performance when a system is starting up.  Priming\nis the activity you do when you act on a 'clean' system to make it perform like it would under\nsteady state.  You'll find that many of the optimizations we did took the form of some kind of\npriming.\n\nLet's take a closer look at the actual specifics of the cluster:\n\n1.  The cluster was comprised of size AWS m3.large instances: a master/datastore host, an\n    infrastructure node (router and registry), and four nodes to host the application pods\n2.  The cluster was created from an ansible playbook that does the following workflow:\n    1.  Spin up the ec2 instances\n    2.  Create and assign the security groups\n    3.  Create route53 entries for each of the hosts\n    4.  Create route53 entry for the wildcard dns domain used by applications\n    5.  Call the official openshift-ansible playbooks for installation and configuration\n    6.  Create the demo user and project\n    7.  Create the openshift-router\n    8.  Create the openshift-registry\n    9.  Pre-pull all needed images on the hosts\n        1.  pod, builder, deployer, router, registry images for infra node\n        2.  pod, builder, deployer, nodejs images for the app nodes\n\n## Building the 'sketchpod' application in under 30 seconds\n\nThe first requirement we attacked was the one to reliably build the sketchpod application and push\nit to the internal registry in under 30 seconds.\n\nWhen I am given a requirement like this one, the first thing I like to do is get what's called a\n\"p90\" value around how the system performs without any optimizations.  A p90 value is a measure of\nthe time that 90% of measurements fall _under_.  In other words, if I perform some experiment X\nnumber of times and my results indicate a p90 value of one minute, it means that 90% of the time\nwhen I perform the experiment, it completes in _under one minute_. You can read more about p90\nvalues [here](https://en.wikipedia.org/wiki/Cumulative_frequency_analysis).\n\nWhile performing time trials to determine the p90 value for this requirement, we noticed that the\nresults 'clustered' into 2 groups -- builds that took approximately 25-30 seconds and builds that\ntook approximately two minutes.  We expected that the timing requirement here would be easy to hit,\nand so it was very surprising to see the builds taking two minutes.  \n\nWhen we dug into the builds in the 'approximately 2 minutes' cluster, we found that most of the\nduration of the build was spent doing a docker push to the OpenShift internal registry.  The root\ncause, it turned out, was that the docker engine's checksum calculation algorithm can yield\ndifferent checksums for the same image layer bytes depending on attributes like ctime and mtime for\nthe layer's files on disk.  During the two-minute builds, a different checksum was being calculated\nfor the layers in the nodejs base image, and so the entire bytes for all the layers were being\npushed to the daemon, instead of only the bytes for the top-most layer containing the sketchpod\napplication code.\n\nLuckily, we have a docker registry expert on the OpenShift team,\n[Andy Goldstein](https://github.com/ncdc).  After helping us diagnose the problem, Andy built a\nboutique version of the OpenShift registry that accepted new checksums for layers it already\ncontained and aliased them.  This allowed us to push only the topmost layer of the image we built.\nSince it is important to us to fix this issues for everyone, Andy created\n[an issue](https://github.com/docker/docker/issues/14018)\nin the upstream docker project to share this knowledge and help us drive a fix in the upstream\nregistry.  \n\nIn conjunction with the modified registry, we added a priming step to the Ansible playbook that:\n\n1.  Creates a project namespace for priming the registry\n2.  Tags the nodejs builder image on each of the nodes\n3.  Push the tagged nodejs builder image to the registry from each node\n\nAfter adding the priming steps to the playbook, we found that our 5 node cluster satisfied the\nthe build time requirements, and the requirements for scaling up from a single replica to ten\nreplicas.\n\n## Requirement 3: Scale from 10 to 1026 replicas in under 3 minutes\n\nThe next major requirment we had to hit was resizing from ten replicas to the total number of\nreplicas required for the demo (1026).  \n\nTODO: what was our scale up time on the smaller cluster size?\n\nThe Kubernetes kubelet is the node-level agent that ensures that each node stays in the desired\nstate.  To do this, the kubelet compares the pods scheduled onto a node with the pods that are\nactually running, and takes action to make these two lists match.  In the context of optimizing\nscale-up of a single replication controller throughout the cluster, the relevant behavior to know\nabout is that the kubelet basically loops through the scheduled pods and starts ones that aren't\nrunning, one at a time.\n\nTherefore, the total number of pods that can be being started at any one time is linear in the\nnumber of nodes in the cluster.  We increased the cluster size from five nodes to ten nodes to\ntwenty nodes and found that a twenty-node cluster was able to acheive our requirement for the\nresize.\n\nThe mood was jubilant once we had acheived the time requirement for the large resize, but our work\nwasn't done yet.\n\n## Running the demo at scale\n\nTODO: discuss soak testing and proxy work\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpmorie%2Fmiddleware-keynote-demo-blog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpmorie%2Fmiddleware-keynote-demo-blog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpmorie%2Fmiddleware-keynote-demo-blog/lists"}