{"id":48356779,"url":"https://github.com/green-coding-solutions/cloud-energy","last_synced_at":"2026-04-05T11:33:37.317Z","repository":{"id":59269556,"uuid":"531897614","full_name":"green-coding-solutions/cloud-energy","owner":"green-coding-solutions","description":"Cloud Energy is an XGBoost \u0026 linear model based on the energy data from the SPECPower database for the cloud to estimate wattage consumption of server by just a few input variables","archived":false,"fork":false,"pushed_at":"2026-04-02T06:31:01.000Z","size":10528,"stargazers_count":31,"open_issues_count":4,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-04-02T20:13:02.097Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/green-coding-solutions.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-09-02T11:26:31.000Z","updated_at":"2026-04-02T06:30:58.000Z","dependencies_parsed_at":"2023-02-08T18:45:51.876Z","dependency_job_id":"39ac307c-d119-46f9-8bd3-38d4d9080497","html_url":"https://github.com/green-coding-solutions/cloud-energy","commit_stats":null,"previous_names":["green-coding-solutions/spec-power-model","green-coding-solutions/cloud-energy"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/green-coding-solutions/cloud-energy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/green-coding-solutions%2Fcloud-energy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/green-coding-solutions%2Fcloud-energy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/green-coding-solutions%2Fcloud-energy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/green-coding-solutions%2Fcloud-energy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/green-coding-solutions","download_url":"https://codeload.github.com/green-coding-solutions/cloud-energy/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/green-coding-solutions%2Fcloud-energy/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31434624,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-05T08:13:15.228Z","status":"ssl_error","status_checked_at":"2026-04-05T08:13:11.839Z","response_time":75,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-05T11:33:36.493Z","updated_at":"2026-04-05T11:33:37.295Z","avatar_url":"https://github.com/green-coding-solutions.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Cloud Energy is a locally trainable and runnable ML Model (based on XGBoost) to estimate\npower consumption [W] from CPU Utilization.\n\nThe model works everywhere where Python runs but can ingest live data best on *NIX Systems. \n\n# Table of Contents\n\n- [Installation](#installation)\n  + [Rebuild Training Data](#re-build-training-data)\n- [Use](#use)\n  + [Demo Reporter](#demo-reporter)\n- [Overview](#overview)\n- [Background](#background)\n- [Discovery of the parameters](#discovery-of-the-parameters)\n- [Model Details / EDA](#model-details-eda)\n- [Interpolation for output](#interpolation-for-output)\n- [Results](#results)\n- [Assumptions \u0026 Limitations](#assumptions-limitations)\n- [TODO](#todo)\n- [Credits](#credits)\n- [License](#license)\n\n## Installation\n\nNeeds at least python 3.11\n\n```\npython3 -m venv venv\nsource venv/bin/activate\npip3 install -r requirements.txt\n```\n\n### Re-build training data\nIf you want to rebuild the training data (`spec_data*.csv`) then you have to include \nthe git submodule with the raw data.\n\n```bash\ngit submodule update --init\n```\n\n## Use\nYou must call the python file `ols.py` or `xgb.py`. \nThis file is designed to accept streaming inputs.\n\nA typical call with a streaming binary that reports CPU Utilization could look like\nso: \n```\n$ ./static-binary | python3 xgb.py --tdp 240 \n191.939294374113\n169.99632303510703\n191.939294374113\n191.939294374113\n191.939294374113\n191.939294374113\n194.37740205685841\n191.939294374113\n169.99632303510703\n191.939294374113\n....\n```\n\nSince all possible outputs are infered directly into a dict the model is highly\nperformant to use in inline reporting scenarios.\n\n### Demo Reporter\n\nIf you want to use the demo reporter to read the CPU utilization there is a C reporter\nin the `demo-reporter` directory.\n\nCompile it with `gcc cpu-utilization.c`\n\nThen run it with `./a.out`\n\nOr feed it directly to the model with: `./a.out | python3 model.py --tdp ....`\n\n\n## Overview\nThis repository containes the needed data to train a Linear Model (OLS) / XGBoost for the [SPECPower\ndata set](https://www.spec.org/power_ssj2008/).\n\nThe models are built with dynamic variables designed to work \nin different cloud environments where some information may not be available.\n\nIts use is the estimation of the current power draw of the whole machine in \nWatts.\n\nCurrently the model supports following variables:\n- CPU Utilization `[float [0-100]]`\n    + The utilization of all your assigned threads cumulative and normalized to 0-100\n- CPU Chips `[integer [1,)]`\n    + The CPU chips installed on the mainboard. Most machines have either 1 or 2.\n    + If you do not know this value rather leave it off.\n- CPU Threads `[integer [1,)]`\n    + The total amount of CPU threads over all installed chips.\n    + Example: The CPU has 10 physical cores with two threads each and two chips installed you enter 10 * 2 * 2 = 40.\n    + Please note that if you are restricted to use only a subset of the threads, like it is typical \\\n      in virtualized or containerized environments you still enter the full capacity of the CPU. The ratio assigned \\\n      to you is handled by the parameter `vHost Ratio`\n- CPU Cores `[integer [1,)]`\n    + Threads and cores do not have to be equal. When Hyperthreading is active the amount of threads\n    + is typically greater than the amount of cores.\n    + If you do not know how many phyiscal cores you really have rather do not supply this argument\n- CPU Frequency `[integer [1,)]`\n    + The base frequency of the processor in MHz.\n    + This value is only used in the XGBoost variant of the model\n- Architecture `[str]`\n    * For example: \"haswell\"\n- CPU Make `[str]`\n    * either \"intel\" or \"amd\"\n- Release year `[int]`\n    + ex. 2011\n- RAM `[integer (0,]]`\n    * in Gigabytes\n- TDP `[integer (0,]]`\n    + In Watts\n    + The thermal design power of the CPU in your system. This value you typically find only on the data sheet online.\n- vHost Ratio `[float (0,1])`\n    + The vHost ratio on the system you are on. If you are on a bare metal machine this is 1\n    + If you are a guest and have e.g. 24 of the 96 Threads than the ratio would be 0.25\n    + Currently the model cannot account for non-balanced CPU and memory ratios.\n\nOnly the CPU Utilization parameter is mandatory. All other paramters are optional. \\\nvHost ratio is assumed to be 1 if not given.\n\nYou are free to supply only the utilization or as many additional parameters that\nthe model supports. The model will then be retrained on the new configuration on the spot.\n\nTypically the model gets more accurate the more parameters you can supply. Please see the *Assumptions \u0026 Limitations* part at the end to get an idea how accurate the model will be in different circumstances.\n\n## Background\n\nTypically in the cloud, especially when virtualized, it is not posssible to \naccess any energy metrics either from the [ILO](https://en.wikipedia.org/wiki/HP_Integrated_Lights-Out) / [IDRAC](https://en.wikipedia.org/wiki/Dell_DRAC) \ncontrollers or from [RAPL](https://en.wikipedia.org/wiki/Perf_(Linux)#RAPL).\n\nTherfore power draw must be estimated.\n\nMany approaches like this have been made so far:\n- https://www.cloudcarbonfootprint.org/\n- https://greenpixie.com/\n- https://medium.com/teads-engineering/evaluating-the-carbon-footprint-of-a-software-platform-hosted-in-the-cloud-e716e14e060c#3bf5\n\nCloud Carbon Footprint and Teads operate on Billing data and are too coarse \nfor a fast paced development that pushes changing code on a daily basis.\n\nTeads could theoretically solve this, but is strictily limited to AWS EC2. Also\nit provides no interface out of the box to inline monitor the emissions.\n\nTherefore we created a model out of the SPECPower dataset that also can be used\nin real-time.\n\n## Discovery of the parameters\n\nAt least utilization is needed as an input parameter.\n\nYou need some small script that streams the CPU utilization as pure float numbers\nline by line.\n\nThe solution we are using is a modified version of our [CPU Utilization reporter\nfrom the Green Metrics Tool](https://github.com/green-coding-solutions/green-metrics-tool/tree/main/metric_providers/cpu/utilization/procfs/system).\n\nThis one is tailored to read from the procfs. You might need something different in your case ...\n\n### Hyperthreading\n\nHT can be easily checked if the core-id is similar to the processor id.\n\nLast Core-ID should be processor_id+1\nIf Last core ID is \u003e processor_id+2  then HT is enabled\n\nAlternatively looking at `lscpu` might reveal some infos.\n\n### SVM / VT-X / VT-D / AMD-V ...\nThe presence of virtualization can be checked by looking at:\n\n`/dev/kvm`\n\nIf that directory is present this is a strong indicator, that virtualization is enabled.\n\nOne can also install cpu-checker and then run \n`sudo apt install kvm-ok -y \u0026\u0026 sudo kvm-ok`\n\nThis will tell with more checks if virtualization is on. even on AMD machines.\n\nHowever in a vHost this might not work at all, as the directory is generally hidden.\n\nHere it must be checked if a virtualization is already running through:\n`sudo apt install virt-what -y \u0026\u0026 sudo virt-what`\n\nAlso `lscpu` might provide some insights by having these lines:\n\n```\nVirtualization features:\n  Hypervisor vendor:     KVM\n  Virtualization type:   full\n```  \n\n### Hardware prefetchers\n\nThere are actually many to disable:\nThe above mentioned processors support 4 types of h/w prefetchers for prefetching data. There are 2 prefetchers associated with L1-data cache (also known as DCU DCU prefetcher, DCU IP prefetcher) and 2 prefetchers associated with L2 cache (L2 hardware prefetcher, L2 adjacent cache line prefetcher).\n\nThere is a Model Specific Register (MSR) on every core with address of 0x1A4 that can be used to control these 4 prefetchers. Bits 0-3 in this register can be used to either enable or disable these prefetchers. Other bits of this MSR are reserved.\n\nHowever it seems that for some processors this setting is only available in the BIOS\nas it is not necessary disclosed info by Intel how to disable it.\nFor servers it seems quite standard to do be an available feature apparently ...\n\nhttps://stackoverflow.com/questions/54753423/correctly-disable-hardware-prefetching-with-msr-in-skylake\nhttps://stackoverflow.com/questions/55967873/how-can-i-verify-that-my-hardware-prefetcher-is-disabled\nhttps://stackoverflow.com/questions/784041/how-do-i-programmatically-disable-hardware-prefetching\nhttps://stackoverflow.com/questions/19435788/unable-to-disable-hardware-prefetcher-in-core-i7\nhttps://stackoverflow.com/questions/784041/how-do-i-programmatically-disable-hardware-prefetching\n\n\n### Other variables\nOther variables to be discovered like CPU Make etc. can be found in these locations typically:\n\n- `/proc/stat`\n- `/proc/memory`\n- `/proc/cpuinfo`\n- `/sys/devices/virtual/dmi`\n- `dmidecode`\n- `lspci`\n- `lshw`\n- `/var/log/dmesg`\n\nInformations like the vHost-Ratio you can sometimes see in `/proc/stat`, but this\ninfo is usually given in the machine selector of your cloud provider.\n\nIf you cannot find out specific parameters the best thing is: Write an email to your cloud provider and ask :)\n\n## Model Details / EDA\n\n- Model uses SPECPower raw data\n    + Current copy is stored in `./data/raw`\n    + We only process the html data. It contains the same info as the text\n    + Look into `./scripts/create_data_csv.py`\n    + Unprocessed version is then in `./data/spec_data.csv`\n- CPU microarchitecture and TDP data is coming from\n    + David Mytton [Cloud carbon coefficients](https://github.com/cloud-carbon-footprint/cloud-carbon-coefficients) (only AMD Epyc info)\n    + Wikipedia (very! thorough source)\n- Data is cleaned. Look into `./scripts/data_cleaning.py`\n    + Cleaned and enriched version is then in `./data/spec_data_cleaned.csv`\n\nThe EDA is currently only on Kaggle, where you can see how we selected the subset of the \navailable variables and their interaction in our [Kaggle notebook](https://www.kaggle.com/code/arne3000/spec-power-eda-pass-2)\n\nIn order to create some columns we inspected the `SUT_BIOS` and `SUT_Notes` fields\nand created some feature columns dervied from them. Here is a quick summary:\n\n- *BIOS_P_States_Enabled*\n    + P-states are a power feature. P-State = 1 is the base frequency\n    + Setting P-states to off will set P-State to max non-turbo (aka 1) (https://www.thomas-krenn.com/en/wiki/Disable_CPU_Power_Saving_Management_in_BIOS)\n    + All P-States greater than 1 are power efficient states: https://www.thomas-krenn.com/en/wiki/Processor_P-states_and_C-states\n\n- *BIOS_Memory_Setting_Changed*\n    + When we found infos like \"DDR Frequency set to 1066 MHz\" we considered this memory tuning\n\n- *BIOS_HT_Enabled*\n    + We found Hyperthreading mostly not mentioned, but when than turned on. Which should be the default anyway.\n    \n- *BIOS_VT_Enabled*\n    + Virtualization was sometimes disabled, which is also very often the default\n    + However we believe it is almost always on in cloud environments, as it is for instance a prerequiste for KVM (EC2 hypervisor)\n    + Includes SVM from AMD\n   \n- *BIOS_Turbo_Boost_Enabled*\n    + Turbo Boost was very often turned off, which is a clear sign of tuning\n    + Turbo Boost is almost always on by default\n   \n- *BIOS_C_States_Enabled*\n    + C-States are a power saving feature. If they are fixed to a certain state this could well be considered tuning, as this is non default and very untypical for the cloud\n   \n- *BIOS_Prefetchers_Enabled*\n    + Prefetchers like DCU Prefetcher, Adjacent Cache Line Prefetch, MLC Spatial Prefetcher etc. are almost always on by default\n    + Most systems however have these disabled.\n    + We do not know the typical state in the cloud here.\n\n### Unclear data in SUT_BIOS / SUT_Notes\n\nSome info we thought might be related to energy, but we could not make sense of them.\nIf you can, please share and create and create a Pull Request:\n\n- The cores were mostly fixed to a JVM instance: *Each JVM instance was affinitized two logical processors on a single socket.*\n    + We do not know if this optimizing for the benchmark or a SPECPower requirement.\n    + Therefore not processed further\n\n- We found however settings with TurboBoost on and then the *Maximum Processor State: 100%.* was set. \n    + We are not exactly sure what that means, but it could indicate that TurboBoost although enabled could never be executed ...\n\n- We found settings like *SATA Controller = Disabled*\n    + This setting was mostly set cause the machines were running on PCIe / M2 disks\n\n- *Set \"Uncore Frequency Override = Power balanced\" in BIOS.* or *Power Option: Power Saver* or *\"Power Mode: Balanced\"*\n    + Unsure what does translates to really since \"power balanced\" has no defined meaning and changes for every vendor.\n    + Balanced might for instance include TurboBoost On for one vendor and Off for another\n\n- *DEMT -enabled.*\n    + Dynamic energy management\n    + Ignored cause we do not know how this really affects energy consumption\n\n- *Memory Data Scrambling: Disable* / *Set \"Memory Patrol Scrub = Disabled\"*\n    + Ignored cause we do not know how this really affects energy consumption\n\n- *EIST* is sometimes enabled and sometimes not. Although it can be a power saving feature it alone says nothing about power itself.\n    + We believe this column holds no information on its own\n\n- ASPM Support - Power saving for PCIe\n    + Ignored cause we do not know how this really affects energy consumption\n\n-  'USB Front Port Disabled.',\n    + Ignored cause we do not know how this really affects energy consumption\n    + Also we believe this is cloud standard\n\n- *CPU Power Management set to DAPC*\n    + Dell only feature for energy. Did not look into further\n\n- *EfficiencyModeEn = Enabled*\n    + Too few entries with feature\n\n- *SGX enabled / disabled* \n    + is also very curious ... unclear what the cloud setting is\n\n\n## Interpolation for output\n\nLike all tree based models our XGBoost model can only predict what it has seen so \nfar.\n\nSince the original data from SPECPower only has information for every 10% of \nutilization the model will by default for instance give the same value for 6%\nas well as for 7%.\n\nTo combat this behaviour we interpolate between the points where the model actually\nreports new data, which is:\n- 0-5\n- 5-15\n- 15-25\n- 25-35\n- 35-45\n- 45-55\n- 55-65\n- 65-75\n- 75-85\n- 85-95\n- 95-100\n\nThe data is just interpolated linearly. The interpolation is done directly \nwhen the `xgb.py` script is starting and thus all possible infered values for \nutilization (0.00 - 100.00) are stored in a dict.\nThis makes the model extremely performant at the cost of a minimal memory cost.\n\n## Results\n\nWe have first compared the model against a machine from SPECPower that we \ndid not include in the model training: [Hewlett Packard Enterprise Synergy 480 Gen10 Plus Compute Module](https://www.spec.org/power_ssj2008/results/res2022q1/power_ssj2008-20211207-01142.html)\n\nThis machine is comprised of 10 identical nodes, therefore the power values\nhave to be divided by 10 to get the approximate value that would have resulted \nif only one node was tested individually.\n\nAn individual node has the following characteristics as model parameters:\n- --cpu-freq 2300\n- --tdp 270\n- --ram 256\n- --cpu-threads 160\n- --cpu-chips 2\n\n![hp_synergy_480_Gen10_Plus.png](/img/hp_synergy_480_Gen10_Plus.png)\n\n\nThis is the comparison chart:\n\nSecondly we have bought a machine from the SPECPower dataset: [FUJITSU Server PRIMERGY RX1330 M3](https://www.spec.org/power_ssj2008/results/res2017q2/power_ssj2008-20170315-00744.html)\n\nThe machine has the following characteristics as model parameters:\n- --cpu-freq 3500\n- --tdp 24\n- --ram 16\n- --cpu-threads 8\n- --cpu-chips 1\n\nThis is the comparison chart for the SPEC data vs our modelling:\n![fujitsu_TX1330_SPEC.png](/img/fujitsu_TX1330_SPEC.png)\n\n\nThis is the comparison chart where we compare the standard BIOS setup against the *tuning* settings from SPECPower:\n![fujitsu_TX1330_measured.png](/img/fujitsu_TX1330_measured.png)\n\n### Summary\n- We can see that the SDIA model in its current form cannot account for the idle state of the machine and thus always underestimates here\n- The SDIA model underestimates 1-chip machines and greatly over-estimates 2-chip machines\n    + Taken into account that for 2-chip machines we only have SPECPower data at the moment and no real world data\n- The linear model is good for parameter exploration, but delivers badly fitted results\n- The XGBoost model is able to estimate a real world 1-chip machine and an out of sample 2-chip machine from SPECPower very nicely.\n    + However it tends to under-estimate\n- We see suprisingly no efficiency gain from applying the SPECPower BIOS settings but rather a smoothing of the curve. The reason to that is currently unknown.\n\n\n\n\n## Comparison with Interact DC variable selection\n\nRun the `interact_validation.py` to see a K-Folds comparison of our variable \nselection against the one from Interact DC.\n\nWithout Hyperparameter Tuning when comparing the available variables in the cloud\nthey are about the same.\n\n## Assumptions \u0026 Limitations\n- The model was trained on the SPECpower dataset which almost exclusively includes compute focussed machines. This means it will not be accurate for memory-heavy machines like database servers or ML machines that tend to use GPUs/TPUs or even ASICS\n- The main input variable for the model is CPU utilization. This metric is only reliable if the system frequencies do not change much. See our in depth article about [usefulness of CPU Utilization as a metric](https://www.green-coding.berlin/case-studies/cpu-utilization-usefulness/)\n- SPECPower machines tend to be rather tuned and do not necessarily represent the reality of current datacenter configurations. So you are likely to get a too small value than a too high value. This was also detailed in the analysis earlier in the README, where we talk about the turned off features.\n- If you are in a shared resource system like a Virtual Machine the model will assume a linear fraction of the load. This is debateable and might need improvement. See the discussion here: https://github.com/green-coding-solutions/spec-power-model/issues/4\n\n\n## TODO\n\n- vhost operating point\n- ~~validation of EC2 machines and the data from Teads.~~\n- ~~Performance optimizations for inline processing to get below 2% of utilization for 100ms intervals~~\n- Re-evaluating more machines from the SPECPower database in our lab and better understand what the BIOS settings really impact in regards to the server energy\n- Research what values in the cloud are typically set for the BIOS settings that SPECPower lists and if they can be configured in the cloud\n- Introspecting our models to understand which parameter in which setting will give the most energy gain when set on the machine so that developers can optimize these parameters\n\n## Credits\n\nA similar model has been developed in academia from [Interact DC](https://interactdc.com/) and the \npaper can be downloaded on [their official resources site](https://interactdc.com/static/images/documents/Elsevier_Journal.pdf).\n\nOur model was initially developed idependently but we have taken some inspiration \nfrom the paper to tune the model afterwards.\n\nA big thank you to [Rich Kenny](https://twitter.com/bigkatrich) from Interact DC to providing some insights to\nparameters and possible pitfalls during our model development.\n\n## License \n\nThe model is MIT Licensed. See License file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgreen-coding-solutions%2Fcloud-energy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgreen-coding-solutions%2Fcloud-energy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgreen-coding-solutions%2Fcloud-energy/lists"}