{"id":15760618,"url":"https://github.com/flrs/sd_covid_dashboard","last_synced_at":"2026-04-09T17:32:43.070Z","repository":{"id":201981394,"uuid":"291371857","full_name":"flrs/sd_covid_dashboard","owner":"flrs","description":"San Diego COVID-19 Dashboard on Kubernetes in the Google Cloud","archived":false,"fork":false,"pushed_at":"2020-12-09T05:28:55.000Z","size":13746,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-24T00:48:22.563Z","etag":null,"topics":["california","continuous-deployment","covid-19","dashboard","docker","github-actions","gke","google-cloud","jupyter","kubernetes","python","san-diego","terraform","traefik","voila"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flrs.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-08-30T00:30:48.000Z","updated_at":"2021-01-31T04:44:29.000Z","dependencies_parsed_at":null,"dependency_job_id":"942d2fbd-595c-49ca-9dae-7aec2e7ba51d","html_url":"https://github.com/flrs/sd_covid_dashboard","commit_stats":null,"previous_names":["flrs/sd_covid_dashboard"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flrs%2Fsd_covid_dashboard","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flrs%2Fsd_covid_dashboard/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flrs%2Fsd_covid_dashboard/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flrs%2Fsd_covid_dashboard/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flrs","download_url":"https://codeload.github.com/flrs/sd_covid_dashboard/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246443534,"owners_count":20778252,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["california","continuous-deployment","covid-19","dashboard","docker","github-actions","gke","google-cloud","jupyter","kubernetes","python","san-diego","terraform","traefik","voila"],"created_at":"2024-10-04T10:59:16.792Z","updated_at":"2025-12-30T21:06:39.251Z","avatar_url":"https://github.com/flrs.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# San Diego COVID-19 Dashboard\n*A self-updating data application on Kubernetes in the Cloud*\n\n\u003e “How are COVID-19 cases evolving in my San Diego neighborhood?”\n\n![Screen recording of dashboard](docs/screencast.gif)\n\nThis project primarily serves as an example of how to deploy a self-updating data application on Kubernetes in the \ncloud, using automated continuous integration workflows. Secondarily, the project offers a refined view on San Diego \nCounty COVID-19 data through a dashboard that focuses on breaking down case data by ZIP code.\n\n**Dashboard URL:** [http://35.225.51.112](http://35.225.51.112) \n\n\n## Table of Contents\n\n  * [Pipeline Status](#pipeline-status)\n  * [Infrastructure](#infrastructure)\n    + [Continuous Deployment Pipelines](#continuous-deployment-pipelines)\n    + [Cloud Foundation Layer: Google Cloud + Terraform](#cloud-foundation-layer--google-cloud---terraform)\n    + [Kubernetes Foundation Layer](#kubernetes-foundation-layer)\n      - [Local Development](#local-development)\n    + [App Layer](#app-layer)\n      - [Crawler](#crawler)\n      - [Dashboard](#dashboard)\n  * [Contributing](#contributing)\n  * [Credits](#credits)\n  * [License](#license)\n\n## Pipeline Status\n\nThis section tracks the status of the various production continuous integration pipelines.\n\n**Cloud Foundation Layer** \n\n![Deploy Infrastructure on GCP](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/flrs/78f880bf0781dd1ed6d5d676400d1ebe/raw/sha_infrastructure.json)\n\n**Kubernetes Foundation Layer**\n\n![Deploy Infrastructure on GKE](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/flrs/78f880bf0781dd1ed6d5d676400d1ebe/raw/sha_app.json)\n\n**App Layer**\n\n|  | Docker Build | Built Image | K8s Deployment |\n|-|-|-|-|\n| Crawler | ![Build Docker Crawler](https://github.com/flrs/sd_covid_dashboard/workflows/Build%20Docker%20Crawler/badge.svg) | [![Built Docker Crawler Image](https://images.microbadger.com/badges/version/flrs/sd_covid_dashboard_crawler.svg)](https://microbadger.com/images/flrs/sd_covid_dashboard_crawler \"Get your own version badge on microbadger.com\") | ![Deploy Infrastructure on GKE](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/flrs/78f880bf0781dd1ed6d5d676400d1ebe/raw/sha_crawler.json) |\n| Dashboard | ![Build Docker Dashboard](https://github.com/flrs/sd_covid_dashboard/workflows/Build%20Docker%20Dashboard/badge.svg) | [![Built Docker Dashboard Image](https://images.microbadger.com/badges/version/flrs/sd_covid_dashboard_dashboard.svg)](https://microbadger.com/images/flrs/sd_covid_dashboard_dashboard \"Get your own version badge on microbadger.com\") | ![Deploy Infrastructure on GKE](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/flrs/78f880bf0781dd1ed6d5d676400d1ebe/raw/sha_dashboard.json) |\n\n## Infrastructure\n\nThis section describes the project's infrastructure. The infrastructure is built with the following requirements in \nmind:\n\n- Have a module to automatically download new data (*crawler*)\n- Have a module to store new data (*database*)\n- Have a module to display analyses of those data (*dashboard*)\n- Be small-scale, since very little traffic is expected\n- Be relatively affordable to run\n- Be easy to update and maintain, so experiments can be run at low cost\n\nAs a direct result of these requirements, the project features a great amount of automation. Since the prime use case \nfor cloud computing is big-scale, and not of small-scale deployments (and may come with a bill to match), the project \nuses a custom cloud infrastructure that minimizes cost.\n\nThe below chart gives an overview of the repository's components:\n\n```\n.github/ \u003c-- Continuous integration pipelines\ncrawler/ \u003c-- Component: Crawler for getting COVID-19 case data\ndashboard/ \u003c-- Component: Dashboard for displaying case data \ndocs/ \u003c-- Documentation\nk8s/ \u003c-- Kubernetes manifests\n  base/ \u003c-- Common manifests for dev and production stages\n  overlays/\n    dev/ \u003c-- Manifests for development stage (local)\n    prod/ \u003c-- Manifests for production stage (in cloud)\nnotebooks/ \u003c-- Jupyter notebooks for exprimenting with data\nterraform/ \u003c-- Terraform configurations for Kubernetes and \n               database deployment on Google Cloud\n```\n\n### Continuous Deployment Pipelines\n\nA good starting point for understanding how this repository works are the continuous deployment pipelines. They \nconnect development to deployment. All continuous deployment pipelines are implemented as \n[GitHub Actions workflows](https://github.com/features/actions). Here are the typical use cases the pipelines cover:\n\n- **Deploy Infrastructure on GCP** *(GCP: Google Cloud Platform)* At the beginning, your Google cloud account includes no \n  resources related to this project whatsoever. The \n  [deploy_infrastructure_gcp](.github/workflows/deploy_infrastructure_gcp.yml) workflow uses \n  [Terraform](https://terraform.io) to provision a Kubernetes cluster with a pre-defined set of nodes, a Postgres \n  database, and related roles and permissions on your Google cloud account. These components are the foundation this \n  app needs to run in the cloud.\n  \n- **Deploy App on GKE** *(GKE: Google Kubernetes Engine)* The [deploy_app_gke](.github/workflows/deploy_app_gke.yml) \n  workflow deploys all components of the app into the Kubernetes cluster created on GCP. \n  \n  The workflow also takes care of updating the Kubernetes deployment when any of the components change. For example, \n  the workflow will rollout dashboard updates on Kubernetes when a new Docker image for the dashboard is available.\n  \n- **Build Docker Crawler/Build Docker Dashboard** The [build_docker_crawler](.github/workflows/build_docker_crawler.yml)\n  and [build_docker_dashboard](.github/workflows/build_docker_dashboard.yml) workflows build the crawler and dashboard \n  Docker images and push them onto [DockerHub](https://hub.docker.com/u/flrs). The workflows are triggered whenever\n  resources belonging to one of these app components, located either in the [crawler](./crawler) or \n  [dashboard](dashboard) directories, are changed.\n  \n- **Destroy Infrastructure on GCP** The manually-triggered \n  [destroy_infrastructure_gcp](.github/workflows/destroy_infrastructure_gcp.yml) action uses Terraform to remove all\n  resources created on Google cloud. This is useful during development or when the project has reached its end of life.\n  \n### Cloud Foundation Layer: Google Cloud + Terraform\n\nTerraform creates several resources on Google Cloud to make this project work. The [terraform templates](terraform) \nare based on [Niko Kosonen's tutorial](https://dev.to/verkkokauppacom/how-to-kubernetes-for-cheap-on-google-cloud-1aei)\nand the associated [gke-tutorial GitHub repository](https://github.com/nkoson/gke-tutorial/tree/b2279e72dc65a4a62b9fec779135d48473a66e91).\n\nNiko's terraform templates help to build a small-scale Kubernetes cluster on Google Cloud that circumvents Google's \ndefault load balancer. For very small clusters, the default load balancer often adds significant cost. Niko's solution \n is much less costly and its performance aligns better with the low performance demands of a small cluster. The next \n section describes the Kubernetes infrastructure in more detail.\n\nFor this project, Niko's terraform templates have been extended with a \n[Google Cloud SQL Postgres database](terraform/gke/database.tf) and some \n[output variables](terraform/cluster.tf#L107) to ease continuous deployment. The \n[deploy_infrastructure_gcp](.github/workflows/deploy_infrastructure_gcp.yml#L96) workflow uses the Terraform outputs to \nwrite, encrypt and commit secrets to this repo, so that they can be picked up in the \n[deploy_app_gke](.github/workflows/deploy_app_gke.yml#L60) workflow.\n\n### Kubernetes Foundation Layer\n\nAfter the cloud foundation has been laid by terraform, the Kubernetes infrastructure can now be built on top of it. \nKubernetes manifests for production deployment are stored in the [k8s/overlays/prod](k8s/overlays/prod) direct.\n\nOnce again, a good part of the Kubernetes infrastructure comes from \n[Niko Kosonen's tutorial](https://dev.to/verkkokauppacom/how-to-kubernetes-for-cheap-on-google-cloud-1aei). The \nKubernetes cluster has two nodes. One node serves as an ingress to the cluster and has a static IP that can be \naccessed by the public. Tying a static IP\nto the ingress node is the task of [KubeIP](https://kubeip.com/). The other node is ephemeral and hosts all pods\nneeded for running the apps. [Traefik](https://docs.traefik.io/) is deployed as an internal load balancer in the \ncluster and routes incoming traffic to the apps.\n\nThe app's crawler is triggered by a CronJob. Native Kubernetes CronJobs are not timezone-aware and run in the timezone\nof the hosting node, which is typically UTC. New COVID-19 data from San Diego County becomes available at a point in \ntime relative to Pacific Time. Pacific Time observes Daylight Savings Time. As a result, the offset to UTC changes\nthroughout the year. The Kubernetes-native CronJob does not support changing offsets. As an alternative, this project\nuses Hidde Beydals' timezone-aware [CronJobber](https://github.com/hiddeco/cronjobber) and deploys it \n[alongside](.github/workflows/deploy_app_gke.yml#L114) the other manifests. \n\nThe resulting Kubernetes deployment has 4 namespaces, `default`, `kube-system`, `traefik`, and `dashboard`.\n\n| `default` namespace | `traefik` namespace |\n| -| -|\n| \u003cimg src=\"https://github.com/flrs/sd_covid_dashboard/blob/main/docs/default.png\" alt=\"Kubernetes default namespace\" height=\"800\"\u003e | \u003cimg src=\"https://github.com/flrs/sd_covid_dashboard/blob/main/docs/traefik.png\" alt=\"Kubernetes traefik namespace\" height=\"800\"\u003e |\n\n#### Local Development\n\nThis repository also contains a [Skaffold](https://skaffold.dev/) [pipeline](skaffold.yaml) that helps to develop and \ndeploy the project on a local machine. The [k8s/overlays/dev](k8s/overlays/dev) directory contains the associated \nKubernetes manifests.\n\n### App Layer\n\nThe app layer consists of 2 components: The [crawler](crawler) and the [dashboard](dashboard).\n\nBoth components use Kubernetes' `dashboard` namespace:\n\n\u003cimg src=\"https://github.com/flrs/sd_covid_dashboard/blob/main/docs/dashboard.png\" alt=\"Kubernetes dashboard namespace\" height=\"800\"\u003e\n\n#### Crawler\n\nThe [crawler](crawler)'s job is to acquire San Diego County's COVID-19 data and store it in the project's database, so that it can\nbe received by the dashboard. The crawler is a implemented as [Python script](crawler/main.py).\n\nIt is triggered by a timezone-aware CronJob as described above. In addition to this CronJob, the Kubernetes deployment \nalso includes a [job](k8s/overlays/prod/crawler-seedjob.yaml) to seed the database with previously crawled data, \nsaving traffic. This job is run once, at initialization of the app layer cluster. \n\n#### Dashboard\n\nThe [dashboard](dashboard) is the user-facing part of the application. It is implemented as a \n[voila application](https://github.com/voila-dashboards/voila) and uses \n[voila-vuetify](https://github.com/voila-dashboards/voila-vuetify) to render a material UI-style user interface.\n\nIn comparison to other dashboarding tools, Voila is relatively slow. However, great user-facing performance is not a \nrequirement for this project. Voila was chosen because it makes it easy to create working data applications fast, based\non [Jupyter Notebooks](dashboard/app.ipynb).\n\n## Contributing\n\nFor questions or feedback, please file an issue here on GitHub. You may also use the \n[feedback form](https://survey.survicate.com/a8285d0a4afe7aa3/) from within the app.\n\nFeel free to open pull requests for any changes you think the project could benefit from.\n\n## Credits\n\n- Niko Kosonen's tutorial [How to: Kubernetes for Cheap on Google Cloud](https://dev.to/verkkokauppacom/how-to-kubernetes-for-cheap-on-google-cloud-1aei)\n  and the associated [GitHub repository](https://github.com/nkoson/gke-tutorial)\n\n- Hidde Beydals' [Cronjobber](https://github.com/hiddeco/cronjobber) for Kubernetes\n\n## License\n\nThis project is published under the [MIT License](LICENSE.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflrs%2Fsd_covid_dashboard","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflrs%2Fsd_covid_dashboard","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflrs%2Fsd_covid_dashboard/lists"}