{"id":22382060,"url":"https://github.com/googleinterns/cluster-resource-forecast","last_synced_at":"2025-07-31T03:30:53.594Z","repository":{"id":46593962,"uuid":"261868936","full_name":"googleinterns/cluster-resource-forecast","owner":"googleinterns","description":"This repository contains experimental tools we developed to forecast a clusters' resource (CPU or memory) usage.","archived":false,"fork":false,"pushed_at":"2021-04-28T19:45:33.000Z","size":63,"stargazers_count":28,"open_issues_count":0,"forks_count":15,"subscribers_count":3,"default_branch":"master","last_synced_at":"2023-03-11T22:22:51.914Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/googleinterns.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/contributing.md","funding":null,"license":"LICENSE","code_of_conduct":"docs/code-of-conduct.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-05-06T20:16:25.000Z","updated_at":"2022-10-20T04:16:07.000Z","dependencies_parsed_at":"2022-09-14T05:20:28.329Z","dependency_job_id":null,"html_url":"https://github.com/googleinterns/cluster-resource-forecast","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/googleinterns%2Fcluster-resource-forecast","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/googleinterns%2Fcluster-resource-forecast/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/googleinterns%2Fcluster-resource-forecast/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/googleinterns%2Fcluster-resource-forecast/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/googleinterns","download_url":"https://codeload.github.com/googleinterns/cluster-resource-forecast/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228209948,"owners_count":17885595,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-05T00:11:41.008Z","updated_at":"2024-12-05T00:11:41.804Z","avatar_url":"https://github.com/googleinterns.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Overcommit Simulator\n\nOur  simulator  is  written  in  Python  using  [Apache  Beam](https://beam.apache.org/). The simulator uses the Google's publicly-available [cluster usage trace ](https://github.com/google/cluster-data), and mimics Google’s production environment in which Borg [Borg](https://research.google/pubs/pub49065/) operates. A Python script,named simulator-setup-walkthrough, develops a Beam pipeline to convert the Google workload trace to the desired format outlined in `table_schema.json`. Another Python script,named `fortune-teller-setup-walkthrough`, builds a Beam   pipeline   to   configure   simulation   scenarios   and   run   various   predictors   (see   Section   A.5 from the paper  further  details).  The  heart  of  the  simulator  is  in `simulator/fortune_teller.py`,  which  implements  the logic  to  run peak-oracle and  other  practical  predictors.The  configurations  for  the  simulator  are  defined  using protobuf’s text format. The format for the configuration files is described in the `simulator/config.proto`.\n\n## Software Dependencies\n\nOur  simulator  defines data  processing  pipelines  using  Apache  Beam.   Beam pipelines can be run on selected distributed processing back-ends that include Apache Flink, Apache Spark, and GoogleCloud Dataflow. Our simulator should run on any cloud platform or compute cluster, with slight modifications, that supports the aforementioned processing back-ends. How-ever, we have only tested the simulator with Dataflow onGCP. When running on GCP, the simulator uses the Compute Engine, BigQuery (BQ), Google Cloud Storage, and Dataflow APIs. We plan to test the simulator on other cloud platforms and the university-owned compute clusters in the future.\n\n\n## Installation\nInstructions for setting up the environment are available in\n`/docs/installation/`. The  instructions  for setting  up  the  GCP project  and enabling the required APIs  are  available  under `/docs/installation/gcp-project-config`. The instructions for creating a Google Compute Engine virtual machine (VM) instance with required permissions are available under `/docs/installation/gcp-vm-config`. Finally, the instructions for setting up a Python virtual environment with required dependencies for the simulator are available under   `/docs/installation/simulator-config`.\n\n## Experiment Workflow\nThere are three major steps for evaluating the performance of a predictor following the data pre-processing, simulation, and data post-processing framework: Join the table, then run the simulator on the joined table, and data analysis. \nThe first two steps can be merged into a single beam pipeline, technically. But because the joining tables take lots of time and resources, and its results can be reused, we choose to put it into a separate pipeline so that users can run it once and not worry about it later. \n\n### Data pre-processing \n\nThe Google workload trace provides event and usage information for instances in two separate tables: `InstanceEvents` table and `InstanceUsage` table. Our simulator expects this information in a single table. The instructions for joining Google trace tables are available under  `/docs/experiments/joining-tables`. The output is saved to a BQ table and is used as an input for the next step. \n\n### Simulation \n\nOur simulator enables user to configure various simulation scenarios and run multiple predictors in parallel. The instructions for configuring the simulation scenarios are available under `/docs/experiments/simulation-scenarios`. \nThe instructions for configuring and running various peak predictors are available under `/docs/experiments/running-predictors`. Each predictor saves its output to a BQ table, which contain the per-machine usage, sum of limits for all tasks running on each machine, and predicted peak for each machine, along with machine-specific information.\n\n\n### Data post-processing\nThe output BQ tables from simulations are processed in a Colab notebook to compute evaluation metrics and visualize the results. The instructions to using the Colab notebook and notebook's link are available under `/docs/visualization/`. \n\n\n## Experiment Customization\nOne of the key goals for our simulator is to enable future work on designing overcommit policies. We enable users to customize the parameters for the existing overcommit policies as well as contribute entirely new predictors to the simulator. The instruction for customizing the existing predictors are available under `/docs/customize-predictors`. Our simulator facilitates integrating new predictors and standardizes their interface to ease their later conversion to a production environment. We implement supporting classes in `simulator/predictor.py` that define the interfaces to implement new peak predictors. Users can add any data-driven, machine learning-based predictors as long as they use the specified interfaces. The instructions for contributing new predictors are available under `/docs/contribute`. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogleinterns%2Fcluster-resource-forecast","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogleinterns%2Fcluster-resource-forecast","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogleinterns%2Fcluster-resource-forecast/lists"}