{"id":22353525,"url":"https://github.com/dp6/site-speed-dashboard","last_synced_at":"2025-07-30T08:34:17.304Z","repository":{"id":40574896,"uuid":"390759235","full_name":"DP6/site-speed-dashboard","owner":"DP6","description":"Tool for extracting and visualizing performance data (mobile and desktop) from a given list of urls","archived":false,"fork":false,"pushed_at":"2023-03-06T07:37:36.000Z","size":816,"stargazers_count":12,"open_issues_count":12,"forks_count":2,"subscribers_count":23,"default_branch":"master","last_synced_at":"2024-11-28T23:32:27.610Z","etag":null,"topics":["crux","dp6","hacktoberfest","monitoring-tool","psi","sitespeed"],"latest_commit_sha":null,"homepage":"https://dp6.github.io/site-speed-dashboard/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DP6.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-07-29T14:55:42.000Z","updated_at":"2024-07-01T14:41:59.000Z","dependencies_parsed_at":"2023-02-06T20:46:16.668Z","dependency_job_id":null,"html_url":"https://github.com/DP6/site-speed-dashboard","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":"DP6/template-js-cloudfunction-with-terraform","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DP6%2Fsite-speed-dashboard","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DP6%2Fsite-speed-dashboard/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DP6%2Fsite-speed-dashboard/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DP6%2Fsite-speed-dashboard/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DP6","download_url":"https://codeload.github.com/DP6/site-speed-dashboard/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228114910,"owners_count":17871742,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crux","dp6","hacktoberfest","monitoring-tool","psi","sitespeed"],"created_at":"2024-12-04T13:08:53.519Z","updated_at":"2024-12-04T13:08:54.069Z","avatar_url":"https://github.com/DP6.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Site Speed Dashboard\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"https://raw.githubusercontent.com/DP6/templates-centro-de-inovacoes/main/public/images/centro_de_inovacao_dp6.png\" height=\"100px\" /\u003e\n\u003c/div\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#badge\"\u003e\n    \u003cimg alt=\"semantic-release\" src=\"https://img.shields.io/badge/%20%20%F0%9F%93%A6%F0%9F%9A%80-semantic--release-e10079.svg\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://www.codacy.com/gh/DP6/3ef6b0f421634cb881539914e7fa59f1/dashboard?utm_source=github.com\u0026amp;utm_medium=referral\u0026amp;utm_content=DP6/{{repo}}\u0026amp;utm_campaign=Badge_Coverage\"\u003e\u003cimg alt=\"Code coverage\" src=\"https://app.codacy.com/project/badge/Coverage/3ef6b0f421634cb881539914e7fa59f1\"/\u003e\u003c/a\u003e\n  \u003ca href=\"#badge\"\u003e\n    \u003cimg alt=\"Test\" src=\"https://github.com/dp6/site-speed-dashboard/actions/workflows/test.yml/badge.svg\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://www.codacy.com/gh/DP6/site-speed-dashboard/dashboard?utm_source=github.com\u0026amp;utm_medium=referral\u0026amp;utm_content=DP6/site-speed-dashboard\u0026amp;utm_campaign=Badge_Grade\"\u003e\n    \u003cimg alt=\"Code quality\" src=\"https://app.codacy.com/project/badge/Grade/3ef6b0f421634cb881539914e7fa59f1\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\nSite Speed Dashboard is an open source tool developed by DP6 to help you measure performance in your websites. The project is structured in [Google Cloud Plataform](https://console.cloud.google.com) (GCP) and composed of a JS script that will extract simulated data from Google's [PageSpeed Insights](https://developers.google.com/speed/docs/insights/v5/about) API, and a Python script that will extract real data from Google's [Chrome User Experience Report](https://developers.google.com/web/tools/chrome-user-experience-report) public dataset in Google BigQuery.\n\nAll this data is then stored in BigQuery's tables and connected to a Google Data Studio Dashboard in order to help you visualize the data and get insights in how to improve performance!\n\n# Content\n- [Disclaimer Crux](#disclaimer-crux)\n- [GCP Requirements](#gcp-requirements)\n- [Local Environment Requirements](#local-environment-requirements)\n- [Installation](#installation)\n- [Configuration](#configuration)\n- [Dashboard](#dashboard)\n- [How to Contribute](#how-to-contribute)\n- [Support](#support)\n\n# Disclaimer Crux\nThis project uses CrUX's public BigQuery table and, because of that, the project is subject to CrUX's rules, which are:\n- Data update frequency is M-1. That means that the table is refreshed once per month (usually the first tuesday of the month) with data from the previous month. So if you're querying after the first tuesday of April, you'll get March's data.\n- Data is extracted only from the origin URL / home page of sites. For example, at CrUX's table we can find \"https://www.dp6.com.br/\" but not \"https://www.dp6.com.br/servicos/\".\n\nSo when you're replicating this project, take this in note so you're not suprised if there's urls missing or your data isn't being refreshed daily.\n\n# GCP Requirements \n\nThis project creates resources in Google Cloud Plataform to extract and store data. Because of that, the first requirement is to have a **GCP's project with active billing**.\n## **Resources**\nDuring terraform's execution, these resources will be created:\n- Cloud Storage\n- BigQuery\n- Cloud Function\n- Cloud Scheduler\n\nAnd in order for the creation to succeed, you'll need to active the following:\n- Cloud Storage\n- BigQuery API\n- Cloud Functions API\n- Cloud Build API\n- Cloud Resource Manager API\n- BigQuery Data Transfer API\n- Cloud Scheduler API\n\nIf you are not familiar with activating resouces in GCP, follow [this](https://cloud.google.com/endpoints/docs/openapi/enable-api) guide and search for the sources listed above. \n\n## **Service Account**\nAfter the activation, you'll need to create a [Service Account](https://cloud.google.com/iam/docs/creating-managing-service-accounts) with the following permissions:\n- Storage Admin\n- Cloud Functions Admin\n- BigQuery Admin\n- Service Account User\n\n## **Next Steps**\n\nIn the next step, you'll need to run a terraform script. You can use a local environment in order to do so or you can run directly into GCP, using [Google Cloud Shell](https://cloud.google.com/shell/docs). \n\nWe personally recommend using Google Cloud Shell because of its facility, as you don't need to install anything. If you're going to use your local environment, check [Local Environment Requirements](#local-environment-requirements). Otherwise, skip this section.\n\n# Local Environment Requirements\n\nIn order for the code to run locally, you'll need to install the following:\n\n- Package [zip](https://www.tecmint.com/install-zip-and-unzip-in-linux/)\n- Package [unzip](https://www.tecmint.com/install-zip-and-unzip-in-linux/)\n- Package [curl](https://www.tecmint.com/install-curl-in-linux/)\n- [Google Cloud SDK](https://cloud.google.com/sdk/docs/install?hl=pt-br)\n- [Terraform](https://www.terraform.io/downloads.html)\n\nAfter that, you'll need to set GOOGLE_APPLICATION_CREDENTIALS environment variable with the service account key. You can download this key from GCP, in the same place where the service account was created. If you're not familiar with setting environment variables, follow [this](https://cloud.google.com/docs/authentication/getting-started#setting_the_environment_variable) guide.\n\n  \n# Installation\n\n## Create an API Key\n\nTo be able to get PSI data, you'll need an API key, which you can get in [this](https://developers.google.com/speed/docs/insights/v5/get-started) link, which will automatically create a key and return its value. You can also create an API key directly in the [credentials page](https://console.developers.google.com/apis/credentials) in GCP.\n\n## Running Terraform Script\n\nClone this project into you local computer or to Cloud Shell.\n\n```console\ngit clone https://github.com/DP6/site-speed-dashboard.git\n```\n\nAfter that, run the following:\n\n```console\ncd site-speed-dashboard/\nsh terraform_deploy.sh\n```\n\nWait for the code to run and after finishing, check GCP in order to confirm if the resources were created successfully.\n\n# Configuration\n\n### URLs\nThe next step is to set the configuration file that was created in Cloud Storage. To do that, access [this](https://console.cloud.google.com/storage/browser) link and search for a bucket with the name [project-prefix]-sitespeed, with project-prefix beeing the variable of the same name that was set during terraform's execution. \n\nWithin the bucket, open the \"config\" folder, find the \"config.json\" file and download it. Open the file and edit the URLS array, adding the URLs that you want to analyze in the format shown below.\n\n\u003cimg src=\"https://raw.githubusercontent.com/DP6/templates-centro-de-inovacoes/main/public/images/site_speed_dashboard_config_file.png\"/\u003e\n\n### CrUX Countries (Optional)\nIn case you want to change in which countries crux data will be extracted, you can also alter the COUNTRIES array, adding or removing countries using the [ISO 3166-1 alpha-2](https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2) format.\n\n```\n\"COUNTRIES\": [\"br\", \"fr\"]   // In this case, Brazil's and France's data will be extracted  \n```\n\n### Deployment\nAfter finishing the setup, upload the file into the same GCP bucket, overwriting the previous file. After that, the setup is finished and you'll start collecting performance data on your BQ tables!\n\n# Dashboard\nAfter installing and seting up the urls and countries, you'll have the \"back-end\" done and ready to work, but you still won't be able to visualize the data you are collecting. In order to do that, you'll have to follow through the steps below.\n\n## Duplicating Data Sources\nIn the dataset created in your GCP project, you can find three tables:\n\n- crux_table\n- psi_metrics_results\n- psi_suggestion_results\n\nEach table will compose a different Data Studio data source and all these datasources will compose the dashboard, that, in turn, will create the data visualization. You'll have to duplicate all the data sources below:\n\n- [CrUX Table - Template [Data Source - Bigquery]](https://datastudio.google.com/datasources/521b4106-d002-45d4-b428-f359ea9d4b33)\n- [PSI Table Results - Template [Data Source - Bigquery]](https://datastudio.google.com/datasources/4184e227-e6bf-4323-b224-f684ad58ee9b)\n- [PSI Table Suggestions - Template [Data Source - Bigquery]](https://datastudio.google.com/datasources/1313adbe-412a-406e-beef-e7313238c43b)\n\nTo duplicate, follow the steps below:\n1. Click the\n\u003cimg src=\"https://lh3.googleusercontent.com/V6jKS63Ya_QVoDo7PfKcheXZ-j9Zcih4H8VxaBOMSGesBIT2m97cFIsHazykztgXwOrS=w36\"  style=\"height: 20px; width:20px;\"/\u003e icon \non the top menu of each data source and then click \"Copy Data Source\".\n\n2. Select the GCP project in which you created your site speed dashboard BigQuery tables\n\n3. Select the BigQuery data source created when executing the terraform script\n\n4. Select the corresponding table that matches the Data Studio data source selected\n\nAt the end, you'll have 3 new data sources connected to your tables.\n\n## Duplicating the dashboard template\n\nAfter duplicating the data sources, you'll need to duplicate the dashboard itself. To do that, follow this steps:\n\n1. Open the [Site Speed Dashboard - Template](https://datastudio.google.com/reporting/19dd4850-55a1-42ef-9de1-43f776b9ddbb/page/p_3exd7jginc) dashboard.\n\n2. Duplicate the dashboard by clicking at the \n\u003cimg src=\"https://lh3.googleusercontent.com/Kz1M6nAGoh2SvZ44eaZ_c1scIotCBl_qJzei-gV6Q3RwuLm21XVaqfQMUK2bgzxmoLU=w36-h36\" style=\"height: 20px; width:20px;\"/\u003e icon on the top menu and then at \u003cimg src=\"https://lh3.googleusercontent.com/V6jKS63Ya_QVoDo7PfKcheXZ-j9Zcih4H8VxaBOMSGesBIT2m97cFIsHazykztgXwOrS=w36\"  style=\"height: 20px; width:20px;\"/\u003e *Make a Copy* option.\n\n3. Link each Original Data Source to your newly created data sources\n\n4. Click *Copy Report*.\n\nNow you have a beautiful dashboard ready to be used!\n\n# How to contribute\n\nPull requests are welcome! We will love help to evolve this module. Feel free to browse open issues looking for something you can do. If you have a new feature or bug, please open a new issue to be followed up by our team.\n\n## Mandatory requirements\n\nOnly contributions that meet the following requirements will be accepted:\n\n- [Commit pattern](https://www.conventionalcommits.org/en/v1.0.0/)\n\n## Api Docs\n\n- [PSI](https://github.com/dp6/site-speed-dashboard/blob/master/functions/psi)\n- [CrUX](https://github.com/dp6/site-speed-dashboard/blob/master/functions/crux)\n\n# Support:\n\n**DP6 Koopa-troopa Team**\n\n_e-mail: \u003ckoopas@dp6.com.br\u003e_\n\n\u003cimg src=\"https://raw.githubusercontent.com/DP6/templates-centro-de-inovacoes/main/public/images/koopa.png\" height=\"100px\" width=50px/\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdp6%2Fsite-speed-dashboard","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdp6%2Fsite-speed-dashboard","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdp6%2Fsite-speed-dashboard/lists"}