{"id":18843449,"url":"https://github.com/ai-hypercomputer/ml-goodput-measurement","last_synced_at":"2025-04-14T07:32:27.718Z","repository":{"id":223568244,"uuid":"742121415","full_name":"AI-Hypercomputer/ml-goodput-measurement","owner":"AI-Hypercomputer","description":null,"archived":false,"fork":false,"pushed_at":"2025-03-25T04:28:00.000Z","size":181,"stargazers_count":12,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-27T21:38:39.818Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AI-Hypercomputer.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-11T20:01:26.000Z","updated_at":"2025-03-25T04:28:03.000Z","dependencies_parsed_at":"2024-05-28T23:13:23.421Z","dependency_job_id":"eaf3d1d0-cf90-4514-9e70-f2a2babc7f49","html_url":"https://github.com/AI-Hypercomputer/ml-goodput-measurement","commit_stats":null,"previous_names":["google/cloud_tpu_goodput"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AI-Hypercomputer%2Fml-goodput-measurement","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AI-Hypercomputer%2Fml-goodput-measurement/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AI-Hypercomputer%2Fml-goodput-measurement/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AI-Hypercomputer%2Fml-goodput-measurement/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AI-Hypercomputer","download_url":"https://codeload.github.com/AI-Hypercomputer/ml-goodput-measurement/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248839534,"owners_count":21169826,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T02:57:49.707Z","updated_at":"2025-04-14T07:32:27.681Z","avatar_url":"https://github.com/AI-Hypercomputer.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!--\n Copyright 2023 Google LLC\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n      https://www.apache.org/licenses/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n --\u003e\n# ML Goodput Measurement\n\n## Overview\n\n ML Goodput Measurement is a library intended to be used with Cloud accelerators\n to log necessary information and query a job's Goodput and Badput Breakdown. It\n can be pip installed to import its modules, and retrieve information about a\n training job's overall productive Goodput and sources of Badput. The package\n exposes API interfaces to log useful information from the user application and\n query Goodput for the job run, gain insight into the productivity of ML\n workloads and utilization of compute resources.\n\n The package also exposes Goodput Monitoring APIs which allow asynchronous query\n and export of the job's Goodput, Badput and Step Time Deviation to Tensorboard\n with configurable upload interval.\n\n## Components\n\n\n The ML Goodput Measurement library consists of the following main components: \n\n  - `GoodputRecorder`\n\n  - `GoodputCalculator`\n  - `GoodputMonitor`\n  - `GoodputCache`\n\n\n The `GoodputRecorder`\n exposes APIs to the client to export key timestamps while a training job makes\n progress, namely APIs that allow logging of productive step time and total job\n run time. The library will serialize and store this data in Google Cloud\n Logging.\n\n The `GoodputCalculator` exposes APIs to compute Goodput based on the\n recorded data. Cloud Logging handles its internal operations asynchronously.\n The recommended way to compute Goodput is to run an analysis program separate\n from the training application, either on a CPU instance or on the users'\n development machine.\n\n Under the hood, the `GoodputCalculator` uses a `GoodputCache` which is an\n internal component that locally caches pre-computations and useful logs such\n that repeated computations can be made inexpensive.\n\n The `GoodputMonitor` exposes APIs to query and upload goodput and step time\n deviation data to Tensorboard asynchronously. It does this by instantiating a\n `GoodputCaluclator` under the hood.\n\n## Installation\n\n To install the ML Goodput Measurement package, run the following command on the\n VM or machine you want to query or monitor your workload from:\n\n ```bash\n pip install ml-goodput-measurement\n ```\n\n## Usage\n\nThe usage of this package requires the setup of a Google Cloud project with\nbilling enabled to properly use Google Cloud Logging. If you don't have a Google\nCloud project, or if you don't have billing enabled for your Google Cloud\nproject, then do the following:\n\n1. In the Google Cloud console, on the project selector page,\n [select or create a Google Cloud project](https://cloud.google.com/resource-manager/docs/creating-managing-projects).\n\n2. Make sure that billing is enabled for your Google Cloud project. Instructions can be found [here](https://cloud.google.com/billing/docs/how-to/verify-billing-enabled#console)\n\n3. [Enable](https://console.cloud.google.com/flows/enableapi?apiid=logging.googleapis.com\u0026_ga=2.27841276.1571868865.1726250448-123998259.1726107009) the Cloud Logging API.\n\n  To run your training on Cloud accelerator, set up the environment by following\n  instructions [here](https://cloud.google.com/tpu/docs/setup-gcp-account).\n\n  To learn more about Google Cloud Logging, visit this [page](https://cloud.google.com/logging/docs).\n\n### Access Scopes\n\n You will need both read and write access scopes for cloud logging on both the\n GPU or TPU and CPU node pools. Full cloud logging access is granted by the\n following access scope during node pool creation:\n\n  - `https://www.googleapis.com/auth/cloud-platform`\n\n   XPK adds this access scope to the GPU, TPU and CPU node pools, so XPK is the recommended method to create clusters and node-pools in you intend to run your workloads on GKE.\n\n   Instructions on how to create clusters using XPK can be\n   found [here](https://github.com/AI-Hypercomputer/xpk/blob/main/README.md#cluster-create) and how to create workloads using XPK can be found\n   [here](https://github.com/AI-Hypercomputer/xpk/blob/main/README.md#workload-create).\n\n   \u003e **_NOTE:_** Access Scopes are immutable and workloads can only be migrated\n  to new node pools with required access scopes. Access scopes on already created clusters cannot be updated.\n\n### Import\n\n To use this package, import the `goodput` module:\n\n ```python\n from ml_goodput_measurement import goodput\n from ml_goodput_measurement import monitoring\n ```\n\n### Define the name of the Google Cloud Logging logger.\n\n Create a run-specific logger name where Cloud Logging entries can be written to and read from.\n\n For example:\n\n ```python\n goodput_logger_name = f'goodput_{config.run_name}'\n ```\n\n### Create a `GoodputRecorder` object\n\n Next, create a recorder object with the following parameters:\n\n 1. `job_name`: The full run name of the job.\n 2. `logger_name`: The name of the Cloud Logging logger object (created in the previous step).\n 3. `logging_enabled`: Whether or not this process has Cloud Logging enabled.\n\n \n \n  \u003e **_NOTE:_** For a multi-worker setup, please ensure that only one worker\n   writes the logs to avoid the duplication. In JAX, for example, the check \n   could be `if jax.process_index() == 0`\n\n\n  \u003e **_NOTE:_** `logging_enabled` defaults to `False` and Goodput computations cannot be completed if no logs are ever written.\n\n For example:\n\n\n ```python\n goodput_recorder = goodput.GoodputRecorder(job_name=config.run_name, logger_name=goodput_logger_name, logging_enabled=(jax.process_index() == 0))\n ```\n\n\n### Record Data with `GoodputRecorder`\n\n#### Record Job Start and End Time\n\n Use the recorder object to record the job's overall start and end time.\n \n For example:\n\n ```python\n def main(argv: Sequence[str]) -\u003e None:\n # Initialize configs…\n goodput_recorder.record_job_start_time(datetime.datetime.now())\n # Device Initialization and device scanning…\n # Set up other things for the main training loop…\n # Main training loop\n train_loop(config)\n goodput_recorder.record_job_end_time(datetime.datetime.now())\n ```\n\n\n#### Record Step Time\n\n Use the recorder object to record a step's start time using `record_step_start_time(step_count)`:\n\nFor example:\n\n ```python\n def train_loop(config, state=None):\n # Set up mesh, model, state, checkpoint manager…\n\n # Initialize functional train arguments and model parameters…\n\n # Define the compilation\n\n for step in np.arange(start_step, config.steps):\n   goodput_recorder.record_step_start_time(step)\n   # Training step…\n\n return state\n ```\n\n#### Record Device Initialization, Training Preparation and Data Loading Time\n\n  - Use the recorder object to record Device Initialization time using `record_tpu_init_start_time` and `record_tpu_init_end_time`.\n  - Use the recorder object to record Training Preparation time using `record_training_preparation_start_time` and `record_training_preparation_end_time`.\n  - Use the recorder object to record Data Loading time using `record_data_loading_start_time` and `record_data_loading_end_time`.\n\n\n  For example:\n\n  ```python\n  def train_loop(config, state=None):\n  goodput_recorder.record_tpu_init_start_time()\n  # Set up mesh, model, state, checkpoint manager…\n  goodput_recorder.record_tpu_init_end_time()\n  goodput_recorder.record_training_preparation_start_time()\n  # Set up training set, initialize functional train arguments and model parameters…\n  # Define the compilation\n  # Set up any metrics collectors\n  goodput_recorder.record_training_preparation_end_time()\n\n  for step in np.arange(start_step, config.steps):\n    goodput_recorder.record_data_loading_start_time()\n    example_batch = load_next_batch(data_iterator, example_batch, config)\n    goodput_recorder.record_data_loading_end_time()\n    goodput_recorder.record_step_start_time(step)\n    # Training step…\n\n  return state\n  ```\n\n### Retrieve Goodput with `GoodputCalculator`\n\nIn order to retrieve the Goodput of a job run, all you need to do is instantiate\na `GoodputCalculator` object with the job's run name and the Cloud Logging\nlogger name used to record data for that job run. Then call the `get_job_goodput`\nAPI to get the computed Goodput for the job run. \n\nIt is recommended to make the `get_job_goodput` calls for a job run from an\ninstance that runs elsewhere from your training machine.\n\n\n#### Create a `GoodputCalculator` object\n\nCreate the calculator object:\n\n```python\ngoodput_logger_name = f'goodput_{config.run_name}' # You can choose your own logger name.\ngoodput_calculator = goodput.GoodputCalculator(job_name=config.run_name, logger_name=goodput_logger_name)\n```\n\nIf you want to enable Pathways, turn on the `using_pathways` flag:\n\n```python\ngoodput_logger_name = f'goodput_{config.run_name}' # You can choose your own logger name.\ngoodput_calculator = goodput.GoodputCalculator(job_name=config.run_name, logger_name=goodput_logger_name, using_pathways=True)\n```\n\n#### Retrieve Goodput\n\nFinally, call the `get_job_goodput` API to retrieve Goodput for the entire job run. This API takes an optional parameter `include_badput_breakdown`. which defaults to `False`.\n\nThe returned result is a tuple of the job’s Goodput at query-time, a dictionary mapping various sources of Badput and their corresponding percentages and the last recorded step. If `include_badput_breakdown` is not set, an empty dictionary for Badput is returned.\n\n\nIf you are only interested in Goodput:\n\n```python\ntotal_goodput, _, _ = goodput_calculator.get_job_goodput()\nprint(f\"Total job goodput: {total_goodput:.2f}%\")\n```\n\n#### Retrieve Badput Breakdown\n\nBadput breakdown is dictionary representation of various sources of Badput mapped to its corresponding value. \nBadput is the percentage of time spent by the job doing work that is not training to the total lifetime of the job. This includes time spent doing Device initialization, training preparation, checkpoint loading, compilation or re-compilation, data loading, checkpoint saving and time lost due to disruptions.\n\nFollowing Badput Breakdown buckets are supported by the library at this time:\n\n```python\n# Supported Badput Types\nclass BadputType(enum.Enum):\n \"\"\"The type of Badput.\"\"\"\n  TPU_INITIALIZATION = 1\n  TRAINING_PREP = 2\n  PROGRAM_STARTUP = 3\n  DATA_LOADING_SYNC = 4\n  DATA_LOADING_ASYNC = 5\n  UNPRODUCTIVE_CHECKPOINT_SAVE_TIME = 6\n  UNPRODUCTIVE_CHECKPOINT_RESTORE_TIME = 7\n  WASTED_PROGRESS_FROM_DISRUPTION = 8\n  OTHER = 9\n```\n\n#### Badput Breakdown Details\n\n - Accelerator Initialization Time (TPU_INITIALIZATION)\n\n  This is the time spent on device discovery, slice initialization,\n  device driver re-initialization and reset, security setup, initialization of\n  pre-mapped buffers and more.\n\n - Training Preparation Time (TRAINING_PREP)\n\n  This is the time spent on the creation of checkpoint managers, checkpoint\n  loading, running mesh and model optimizers and more.\n\n - Program Startup Time (PROGRAM_STARTUP)\n\n  This is the time spent on framework specific function transformations\n  (such as JAX tracing), compilation tasks, runtime initialization etc.\n\n - Data Loading Time (DATA_LOADING_SYNC)\n\n  This is the time spent on loading each batch of data for the training at a\n  step to continue. This should be a small contribution to Badput if parallel\n  data loading is used.\n\n  Async data loading is accumulated overlapping with training steps and is\n  non-blocking, therefore is not unproductive time. The time spent on overlapped\n  data loading is stored in BadputType.DATA_LOADING_ASYNC, but does **not**\n  affect overall Goodput of the workload.\n\n - Checkpointing Time (UNPRODUCTIVE_CHECKPOINT_SAVE_TIME, UNPRODUCTIVE_CHECKPOINT_RESTORE_TIME)\n\n  This is the time spent on saving a checkpoint and restoring a checkpoint.\n\n  Depending on the type of checkpointing technology used by the program, there\n  could be unproductive time while saving a checkpoint. When checkpointing is\n  synchronous, the save operation will block training progress until it is complete.\n\n  During asynchronous checkpointing, the model parameters or weights have to be\n  transferred from the device memory to the host memory which is a blocking\n  operation on the device. After the transfer, the device can proceed with model\n  training while the CPU saves the checkpoint to storage in the background. The\n  first blocking operation contributes to unproductive checkpoint time.\n\n  If auto checkpointing is used, the checkpoint save operation is initiated upon\n  detection of a planned disruption signal. The save operation in type of\n  checkpointing is synchronous resulting in time lost to Badput.\n\n - Wasted Progress due to Disruption (WASTED_PROGRESS_FROM_DISRUPTION)\n\n Based on checkpointing frequency, a disruption may result in time lost in the\n form of wasted progress, i.e. time that was spent on productive training but\n lost after restart.\n\n  When there is a disruption, Badput is expected to accumulate in\n  each of the following buckets after restart:\n\n  - Accelerator Initialization\n  - Training Preparation\n  - Program Startup\n  - Wasted Progress due to Disruption\n\nIf you are interested in retrieving Badput Breakdown along with Goodput:\n\n```python\ngoodput, badput_breakdown, last_step = goodput_calculator.get_job_goodput(include_badput_breakdown=True)\nprint(f\"Last step recorded: {last_step}\")\nprint(f\"Goodput: {goodput:.2f}%\")\nprint(f\"Badput due to TPU initialization: {badput_breakdown[goodput.BadputType.TPU_INITIALIZATION]:.2f}%\")\nprint(f\"Badput due to training preparation: {badput_breakdown[goodput.BadputType.TRAINING_PREP]:.2f}%\")\nprint(f\"Badput due to program startup: {badput_breakdown[goodput.BadputType.PROGRAM_STARTUP]:.2f}%\")\nprint(f\"Badput due to data loading: {badput_breakdown[goodput.BadputType.DATA_LOADING_SYNC]:.2f}%\")\nprint(f\"Badput due to disruption and wasted progress: {badput_breakdown[goodput.BadputType.WASTED_PROGRESS_FROM_DISRUPTION]:.2f}%\")\nprint(f\"Badput due to checkpoint save: {badput_breakdown[goodput.BadputType.UNPRODUCTIVE_CHECKPOINT_SAVE_TIME]:.2f}%\")\nprint(f\"Badput due to checkpoint restore: {badput_breakdown[goodput.BadputType.UNPRODUCTIVE_CHECKPOINT_RESTORE_TIME]:.2f}%\")\nprint(f\"Badput from unknown source: {badput_breakdown[goodput.BadputType.OTHER]:.2f}%\")\n```\n\n#### Interval Query Goodput and Badput\n\nIf you are interested in retrieving Goodput and Badput of the workload within a\nspecific window of time, the `GoodputCalculator` exposes the\n`get_job_goodput_interval` API which computes metrics between the start and end\nof this window.\n\nThis API also returns the last step recorded for the job. the total job time in\nthis window and the number of disruptions within the interval window.\n\n\u003e **_IMPORTANT:_** **Use this API if** you know the exact window of time within the workload's total run time that you are interested in.\n\n\u003e **_IMPORTANT:_** **Do NOT use this API if** your workload has been manually disrupted. \n\n\u003e **_IMPORTANT:_** **Do NOT use this API if** you have accidentally re-used a previous `run_name`.\n\n```python\n# Example usage\nstart_time_str = \"2024-12-16 1:05:00\"\nstart_time_utc = convert_pst_to_utc(start_time_str)\nend_time_str = \"2024-12-17 2:00:00\"\nend_time_utc = convert_pst_to_utc(end_time_str)\ncurrent_goodput, badput_breakdown, last_step, total_time, disruptions = goodput_calculator.get_job_goodput_interval(start_time_utc, end_time_utc)\n```\n\n### Monitor Goodput with `GoodputMonitor`\n\nIn order to monitor the Goodput of a job run on Tensorboard, all you need to do\nis instantiate a `GoodputMonitor` object with the job's run name, cloud logger \nname and Goodput monitoring configurations (as described below). Then call the \n`start_goodput_uploader` API to asynchronously query and upload measured Goodput\nto the specified Tensorboard directory.\n\n\n#### Create a `GoodputMonitor` object\n\nCreate a `GoodputMonitor` object with the following parameters:\n\n 1. `job_name`: The full run name of the job.\n 2. `logger_name`: The name of the Cloud Logging logger object (created in the previous step).\n 3. `tensorboard_dir`: The directory to write TensorBoard data to.\n 4. `upload_interval`: The time interval at which to query and upload data to TensorBoard.\n 5. `monitoring_enabled`: Whether or not monitoring is enabled. If the application is\n      interested in monitoring Goodput, it should set this value to True. Only one worker \n      should enable monitoring.\n 6. `include_badput_breakdown`: Whether to query and upload badput breakdown data to Tensorboard.\n\n\u003e **_NOTE:_** Please ensure that only **one** worker enables monitoring of Goodput.\n   In JAX, for example, the check could be `if jax.process_index() == 0`\n\nFor example:\n\n```python\ngoodput_logger_name = f'goodput_{config.run_name}' # You can choose your own logger name.\ngoodput_monitoring_enabled = config.monitor_goodput and jax.process_index() == 0 # Check for configs whether or not the enable monitoring.\n\ngoodput_monitor = monitoring.GoodputMonitor(\n      job_name=config.run_name,\n      logger_name=logger_name,\n      tensorboard_dir=config.tensorboard_dir,\n      upload_interval=config.goodput_upload_interval_seconds,\n      monitoring_enabled=True,\n      include_badput_breakdown=True,\n    )\n```\n\nIf you want to enable Pathways, turn on the `pathway_enabled` flag:\n\n```python\ngoodput_logger_name = f'goodput_{config.run_name}' # You can choose your own logger name.\ngoodput_monitoring_enabled = config.monitor_goodput and jax.process_index() == 0 # Check for configs whether or not the enable monitoring.\n\ngoodput_monitor = monitoring.GoodputMonitor(\n      job_name=config.run_name,\n      logger_name=logger_name,\n      tensorboard_dir=config.tensorboard_dir,\n      upload_interval=config.goodput_upload_interval_seconds,\n      monitoring_enabled=True,\n      include_badput_breakdown=True,\n      pathway_enabled=True\n    )\n```\n\nIf you want to monitor Step Time Deviation, configure the `GoodputMonitor`\nas follows:\n\n```python\ngoodput_logger_name = f'goodput_{config.run_name}' # You can choose your own logger name.\ngoodput_monitoring_enabled = config.monitor_goodput and jax.process_index() == 0 # Check for configs whether or not the enable monitoring.\n\ngoodput_monitor = monitoring.GoodputMonitor(\n      job_name=config.run_name,\n      logger_name=logger_name,\n      tensorboard_dir=config.tensorboard_dir,\n      upload_interval=config.goodput_upload_interval_seconds,\n      monitoring_enabled=True,\n      include_badput_breakdown=True,\n      include_step_deviation=True,\n      configured_ideal_step_time=None # Optional, the library will compute ideal step time if it is not provided\n    )\n```\n\n#### Start asynchronous \"query and upload\" of Goodput\n\nCall the `start_goodput_uploader` API to spin off a thread which continuously\nqueries and uploads Goodput.\n\nNote: This will upload Goodput and Badput data to Google Cloud Monitoring\nby default.\n\n```python\ngoodput_monitor.start_goodput_uploader()\n```\n\n#### Start asynchronous \"query and upload\" of Step Time Deviation\n\nCall the `start_step_deviation_uploader` API to spin off a thread which\ncontinuously queries and uploads step time deviation.\n\nNote: This will upload Step Time Deviation data to Google Cloud Monitoring\nby default.\n\n```python\ngoodput_monitor.start_step_deviation_uploader()\n```\n\n#### Visualize on Tensorboard\n\n1. Make sure you have `tensorboard-plugin-profile`, `tensorflow` and `tensorboard` packages installed\n2. Follow instructions [here](https://cloud.google.com/tpu/docs/profile-tpu-vm#start_profiling_the_model_training) to start the Tensorboard server\n\n#### Access Goodput, Badput and Step Deviation on Google Cloud Monitoring\n\nBy default, performance data ([goodput](https://cloud.google.com/monitoring/api/metrics_gcp#:~:text=workload/goodput_time), [badput](https://cloud.google.com/monitoring/api/metrics_gcp#:~:text=workload/badput_time), and [step deviation](https://cloud.google.com/monitoring/api/metrics_gcp#:~:text=workload/performance)) is automatically sent to Google Cloud Monitoring, enabling visualization on dashboards.\n\nThis feature leverages Google VM metadata (project ID, location, accelerator type)\nand supports replica IDs for uniquely identifying workloads in multi-replica\ndeployments.\n\n```python\n\ngcp_options = goodput_utils.GCPOptions(\n      project_id=None, # If None, the library will automatically identify from GCE internal metadata\n      location=None, # If None, the library will automatically identify from GCE internal metadata\n      replica_id='0', # Default is '0'\n      acc_type=None, # If None, the library will automatically identify from GCE internal metadata\n      enable_gcp_goodput_metrics=True,\n      enable_gcp_step_deviation_metrics=True,\n    )\n\ngoodput_monitor = monitoring.GoodputMonitor(\n      job_name=config.run_name,\n      logger_name=logger_name,\n      tensorboard_dir=config.tensorboard_dir,\n      upload_interval=config.goodput_upload_interval_seconds,\n      monitoring_enabled=True,\n      include_badput_breakdown=True,\n      include_step_deviation=True,\n      configured_ideal_step_time=None, # Optional, the library will compute ideal step time if it is not provided\n      gcp_options=gcp_options\n    )\n```\n\nIf you do not wish to send metrics to Google Cloud Monitoring then please set\nthe flag `enable_gcp_goodput_metrics` to `False` for disabling goodput metrics\nand `enable_gcp_step_deviation_metrics` to `False` for disabling step deviation\nmetrics while creating the GCPOptions object.\n\nSetting `monitoring_enabled` to `False` will disable both tensorboard and GCM\nmonitoring.\n\n```python\n\ngcp_options = goodput_utils.GCPOptions(\n      project_id=None, # If None, the library will automatically identify from GCE internal metadata\n      location=None, # If None, the library will automatically identify from GCE internal metadata\n      replica_id='0', # Default is '0'\n      acc_type=None, # If None, the library will automatically identify from GCE internal metadata\n      enable_gcp_goodput_metrics=False,\n      enable_gcp_step_deviation_metrics=False,\n    )\n\n\ngoodput_monitor = monitoring.GoodputMonitor(\n      job_name=config.run_name,\n      logger_name=logger_name,\n      tensorboard_dir=config.tensorboard_dir,\n      upload_interval=config.goodput_upload_interval_seconds,\n      monitoring_enabled=True,\n      include_badput_breakdown=True,\n      include_step_deviation=True,\n      configured_ideal_step_time=None,\n      gcp_options=gcp_options,\n    )\n```\n\nIf you want to monitor Goodput and Badput metrics computed in a specific window\nof time, you can use the `start_goodput_interval_uploader` monitoring API.\n\n#### Create the `GoodputMonitor` with `enable_gcp_goodput_metrics` set to `True` in `GCPOptions`\n\n```python\n\ngcp_options = goodput_utils.GCPOptions(\n      project_id=None, # If None, the library will automatically identify from GCE internal metadata\n      location=None, # If None, the library will automatically identify from GCE internal metadata\n      replica_id='0', # Default is '0'\n      acc_type=None, # If None, the library will automatically identify from GCE internal metadata\n      enable_gcp_goodput_metrics=True,\n    )\n\ngoodput_monitor = monitoring.GoodputMonitor(\n      job_name=config.run_name,\n      logger_name=logger_name,\n      tensorboard_dir=config.tensorboard_dir,\n      upload_interval=config.goodput_upload_interval_seconds,\n      monitoring_enabled=True,\n      include_badput_breakdown=True,\n      gcp_options=gcp_options\n    )\n```\n\n#### Start asynchronous \"query and upload\" of Interval Goodput to GCM.\n\nCall the `start_goodput_interval_uploader` API and specify `window_size_seconds`\nto compute Goodput and Badput metrics only in the sliding time window.\nThe interval starts `window_size_seconds` prior to time of query, ends at time\nof query, and moves ahead by `upload_interval` seconds.\n\nThis call is asynchronous and will only upload Goodput and Badput data to\nGoogle Cloud Monitoring, and not to Tensorboard.\n\n```python\n# Set the window size to be 12h\ngoodput_monitor.start_goodput_interval_uploader(window_size_seconds = 43200)\n```\n\nNote: Google Cloud Monitoring will allow you to view all the metrics reported\nduring the entire workload. GCM will also allow you to filter by any time window\n(irrespective of `window_size_seconds`). Each data point that is reported by\nthis API will correspond to computation only within the sliding window of size\n`window_size_seconds`.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fai-hypercomputer%2Fml-goodput-measurement","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fai-hypercomputer%2Fml-goodput-measurement","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fai-hypercomputer%2Fml-goodput-measurement/lists"}