{"id":21447861,"url":"https://github.com/hsbc/mrpusa","last_synced_at":"2025-09-03T08:36:54.213Z","repository":{"id":243088826,"uuid":"811281483","full_name":"hsbc/mrpusa","owner":"hsbc","description":"Measuring Real Power Usage of Software Applications (mrpusa) is an application that serves as the basis for a software carbon emission measuring initiative.","archived":false,"fork":false,"pushed_at":"2024-08-11T23:54:08.000Z","size":174,"stargazers_count":5,"open_issues_count":2,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-08T14:44:35.703Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hsbc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-06T09:43:19.000Z","updated_at":"2024-11-22T19:31:28.000Z","dependencies_parsed_at":"2024-06-06T17:42:43.513Z","dependency_job_id":"5067022d-3a41-498f-8098-e5ed3ae36026","html_url":"https://github.com/hsbc/mrpusa","commit_stats":null,"previous_names":["hsbc/mrpusa"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/hsbc/mrpusa","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hsbc%2Fmrpusa","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hsbc%2Fmrpusa/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hsbc%2Fmrpusa/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hsbc%2Fmrpusa/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hsbc","download_url":"https://codeload.github.com/hsbc/mrpusa/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hsbc%2Fmrpusa/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273415429,"owners_count":25101633,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-03T02:00:09.631Z","response_time":76,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-23T03:12:27.086Z","updated_at":"2025-09-03T08:36:54.178Z","avatar_url":"https://github.com/hsbc.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Measuring Real Power Usage of Software Applications (mrpusa)\n\nMeasuring Real Power Usage of Software Applications: open source version \n\n# Methodology\n\n#### MRPUSA is an application that serves as the basis for a software carbon emission measuring initiative. This initiative, a system and method for measuring the real power draw and carbon emissions of software applications, allows businesses and consumers to accurately assess emissions at an individual software level providing a greater level of granularity in reporting.\n\nThe core proposition of this initiative is being able to connect a user device to a visualisation platform or service\nthat is informed by a series of modules that account for the intake of energy consumption data, mapping applications to\ndevices, attributing power to these applications from their mapped devices and converting the calculated energy consumed\nto emission figures.\n\u003cbr\u003e\n\n### Measuring Real Power Usage of Software Applications Overview:\n\nMRPUSA serves to configure and access data across different API endpoints and databases to\naggregate data. Then is then utilised according to the given methodology to arrive at an emissions figure for\na given device, cluster or service. Written primarily in Java, using the Spring framework and Maven build management, this\napplication can be broken down, at a high level into three components: configuration files, data collection jobs \u0026 write\nstreams for declaring collected data's destination. Currently, write streams and some configuration files are written\nand are dependent on Google's BigQuery and its Java libraries. However, for this open sourced version, this can be re-examined and should\nserve in its current state as a reference point for how the code for data collection could be pushed into one of the\nlarger Cloud provider's services for further analysis/manipulation \u0026 visualisation.\n\n\u003cbr\u003e\n\n#### Outlined below is a generalised overview of the methodology, of which is dependent on the data captured by this application:\n\n## Bottom-up: Mapping\n\n| Logical Step | Outline |\n|-------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| `Hardware \u003c-\u003e Physical Hosts \u003c-\u003e Virtual Hosts` | \u003cul\u003e\u003cli\u003eGet the full list of hardware from device inventory\u003c/li\u003e\u003cli\u003eMap the hardware to their physical hosts\u003c/li\u003e\u003cli\u003eMap these physical hosts to virtual hosts via hypervisor\u003c/li\u003e\u003cli\u003eMap Services and application instances to respective physical / virtual hosts\u003c/li\u003e |\n| `Utilisation Data` | \u003cul\u003e\u003cli\u003eBring in the utilisation data from the virtual / physical host keys\u003c/li\u003e\u003cli\u003eBring in the average CPU\u003c/li\u003e\u003cli\u003eWhere no host utilisation is found, an average estimate is applied\u003c/li\u003e |\n| `Filters` | Remove rows where:\u003cul\u003e\u003cli\u003eServer lifecycle is Demised\u003c/li\u003e\u003cli\u003eVirtual infrastructure services (VMware) where other services mapped to device\u003c/li\u003e |\n\n## Bottom-up: Energy Calculation\n\n| Logical Step | Outline |\n|----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| `Hardware Power Draw` | Actual Power Draw: \u003cul\u003e\u003cli\u003eBring in Actual Power draw for mapped physical servers via the power draw data API\u003c/li\u003e\u003cli\u003eActual power draw (AverageConsumedWatts) via power draw data API is taken every 20 minutes\u003c/li\u003e\u003cli\u003eIntervals aggregated up for a daily totals\u003c/li\u003e \u003cbr\u003eRated Power Draw - used where no actual power reading was found: \u003c/br\u003e\u003cbr\u003e \u003cli\u003eMaximum Rated Power is adjusted by the power factor at the mapped physical server utilisation range\u003c/li\u003e |\n| `Maximum Rated Power` | \u003cul\u003e\u003cli\u003eDevice maximum power (in Watts) is joined from Data Center Hardware inventory using the hardware key\u003c/li\u003e |\n| `Total Energy Consumption` | \u003cul\u003e\u003cli\u003eMultiply the power draw, Watts x duration, hours / 10^6 to get energy consumption (MWh) per device\u003c/li\u003e\u003cli\u003eSum the energy (MWh) to get the yearly energy consumption across the estate \u003c/li\u003e\u003cli\u003eApportion the data centre Actual – Estimated delta proportionally across apps \u003c/li\u003e |\n\n## Top-Down: Overheads\n\n| Logical Step | Outline |\n|---------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| `Overheads apportionment` | \u003cul\u003e\u003cli\u003eIT Energy estimated using the bottom up approach is lower than the IT Energy known to be consumed by the data centre\u003c/li\u003e\u003cli\u003eTo account for this, IT Energy delta (Compute, Storage, Network and Other) and Building Energy is apportioned back to the data centres\u003c/li\u003e\u003cli\u003eApportionment is performed on the application instance count basis\u003c/li\u003e |\n\n## Emissions\n\n| Logical Step | Outline |\n|----------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| `Energy → Emissions` | \u003cul\u003e\u003cli\u003eCarbon equivalent emissions are obtained by considering energy consumed and the emissions factor (location and market based) for a given location\u003c/li\u003e |\n\n## Bottom-Up IT Energy calculation Overview\n\nBottom-Up approach refers to the IT energy consumption which is derived by estimating the server energy consumption.\nThis is achieved by two main methods:\n\n### Actual Power method:\n\nRefers to energy consumption obtained directly from physical servers\n\n### Rated Power method (for physical servers with no actual energy consumption data available)\n\nRefers to energy consumption estimated, where Maximum Rated device power (Data Centre Device Data) is scaled down by the\nPower Factor. Power Factor is derived based on the utilisation % of a given server.\n\n### Device / physical server power draw is then apportioned to mapped downstream elements:\n\n- Directly to mapped applications\n- To applications via VMs, where servers are clustered\n\n### Actual Energy Consumption Calculation\n\nGetting a device's 'Actual' energy consumption is made possible only by power draw API, which may not be configured for\nall devices. Below is the logical steps taken in order to get this data for devices where this configuration is present.\n\n| Step | Method |\n|------|---------------------------------------------------------------------------------------------------------------------------------------|\n| 1 | Retrieve a Fully Qualified Domain Name of physical servers from device inventory |\n| 2 | Using the FQDN, retrieve a given physical server's power draw from power draw API |\n| 3 | Multiply the average consumed power draw (Watts) x Interval (mins) / 60 * 1.0E-6 to get energy consumption (MWh) per physical server |\n\n### Rated Energy Consumption Calculation\n\nWhere device power data is not configured for a device within an inventory, we have to draw from other data sources in\norder to gauge a device's power draw estimate.\n\n| Step | Method |\n|------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| 1 | Bring in hardware inventory list. Maximum power (in Watts) is joined from inventory using the hardware key |\n| 2 | Maximum Power is adjusted by the power factor at the utilisation range: \u003cbr\u003eOne Physical host per device: utilization range assumed to equal host utilization\u003c/br\u003e(Where no utilization data can be found for a given device, an average utilization figure based on existing data is applied) |\n| 3 | Multiply the average consumed power draw (Watts) x Interval (mins) / 60 * 1.0E-6 to get energy consumption (MWh) per physical server |\n| 4 | Sum the energy (MWh) to get the daily/weekly energy consumption |\n\n### Actual Energy Apportionment\n\nThe Actual energy is then apportioned out to give a picture of how this energy is consumed per application across\nvirtual machines and clusters.\n\nNote: downstream energy apportionment logic is the same regardless which method was used to estimate the device power\ndraw (actual / rated)\n\n| Step | Method |\n|------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| 1 | Physical servers power / Cluster power (Cluster power is summed up by its mapped physical servers power) is apportioned across mapped Virtual Machines based on Power Factor and Virtual Cores of the Virtual Machines, adjusted by the sum of Power Factor x Virtual Cores of the physical servers |\n| 2 | Partitioned by the number of mapped services / apps to avoid double counting |\n\n## Top-Down\n\n### Overview\n\nOur Top-Down approach contextualises our Bottom-Up calculations within their respective Data estate locations, as it\nprovides information on other energy usage outside of compute servers. The comparison of data between these two\napproaches also gives insight into the accuracy of our calculations and method as we are provided a ''Data Gap'' that\nrefers to parts of the estate our bottom-up approach has not accounted for.\n\n#### Definition of important key terms required for calculation:\n\nBuilding Energy Consumption: Total energy consumed by office buildings (in MWh) in each Data Centre\n\u003cbr\u003e\u003c/br\u003e\nIT resources Energy Consumption: Total energy consumed by IT Resources (in MWh) in each Data Centre\n\u003cbr\u003e\u003c/br\u003e\nBottom-Up Energy Estimate: Sum of all actual or estimated energy consumption (in MWh) calculated for the servers located\nin the Data Centre.\n\u003cbr\u003e\u003c/br\u003e\nttFractions of Overheads: Percentage of calculated energy consumed by IT Compute, IT Storage, IT Network and IT Others\nand Buildings, as provided by Data Centre.\n\u003cbr\u003e\u003c/br\u003e\nEnergy Consumption of Applications: Amount bottom-up calulated energy consumption of applications hosted on servers in\neach Data Centre, respectively apportioned to their compute categories.\n\u003cbr\u003e\u003c/br\u003e\nApportion data Centre overhead per Application: Portion of Energy consumed (in MWh) by each application mapped to Data\nCentre.\n\n### Overhead Apportionment\n\nBelow details the calculation methodology of Overhead Energy consumption at Data Centre and Application Level and how it\nis apportioned.\n\n| Step | Method |\n|------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| 1 | Get Energy Consumption Delta per Data Centre, calculated by IT Energy (actual) - Bottom-Up energy estimate |\n| 2 | Distribute the Energy Consumption Delta on IT resources (Fractions of Overheads, split as per Data Centre Data Groupings - e.g. Compute, Storage, Network, Others) |\n\n### Emission Factors\n\nAn emission factor gives the relationship between the amount of a pollutant produced, and the amount of raw material\nprocessed or burnt.\n\nThis methodology is designed for the reporting of Scope 2 emissions under the location-based and market-based methods of\nthe GHG Protocol.\n\n#### Location-Based\n\nA location-based method reflects the average emissions intensity of grids on which energy consumption occurs (using\nmostly grid-average emission factor data). For location based emission reporting; HSBC apply the following hierarchy of\nemission factors:\n\nRegional or subnational emission factors\nNational production emission factors provided by the International Energy Agency (IEA).\n\n#### Market-Based\n\nA market-based method reflects emissions from electricity that companies have purposefully chosen deriving emission\nfactors from contractual instruments. For market based emissions reporting, HSBC apply the following hierarchy of\nemission factors (this may differ in your organsation):\n\n- Factors provided by electricity attribute certificates or equivalent instruments\n- Factors provided by contracts for electricity, such as power purchase agreements (PPAs)\n- Factors provided by energy suppliers\n- Factors provided by the Association of Issuing Bodies (AIB) for the residual mixes in Europe - Version 1.0, 2021-05-31\n- Other grid-average emission factors (subnational or national)\n- Factors provided by the International Energy Agency (IEA) - 2020 edition of the Emission factors.\n\n#### Market / location based emission factors that is applied to the overall energy calculated:\n\nCarbon Emission in tCO2e = MWh * EF (in tCO2e/Mwh)\n\n## Visualization\n\nThe data collected via an application such as this should be easily connected to a visualisation platform, with the\nability to create and seperate views by splitting and filtering data accordingly.\n\nWithin our example, we have streamlined this process by working within Google's services, directly connecting our tables\nof collated data hosted in their BigQuery product to their Looker visualisation platform with no additional manipulation\nrequired.\n\nHowever, this should only be understood as an example and other means of visually presenting data collected via this\nmethodology can be achieved and distributed as per your/your users' needs.\n\n# Measuring Real Power Usage of Software Applications Project :\n\n# PART-1:Execute JAVA APPLICATION\n\n# PREREQUISITES:\n\n- Install JDK 11 and set up the JAVA_HOME environment variable correctly\n- Install Maven and set up the MAVEN_HOME environment variable and add the MAVEN bin directory to your system path.\n- Install git and configure it with your gitHub credentials.\n- clone the project repository in your local Machine.\n- Download and install an IDE like Eclipse or Intellij IDEA\n- after cloning the project from the GitHub navigate to the project directory\n- Download all dependencies required for the project, ref.pom.xml file to ensure all dependencies are specified\n correctly and downloaded.\n\n# Project directory Structure:\n\n| Folder | Description |\n|------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| config | The configuration files serve to define access to API endpoints and manages these connections, set schedules for accessing specified data sources.**_ScheduleConfig_**-Sets scheduled tasks for running the various data collector files,**_SwaggerConfig_**-Configures swagger API access within our project,**_WebConfig_**-Handles connections for web based data transfers. |\n| Controller | Contains structure of handling request and response starcure with there status code. |\n| Service | **_PhysicalServerCollector.java_**-Contain the code to fecth data from the redfish API for specific hostname and include functionality to filter data and add devices to hostname.**_PowerMetricCollector.java_**- fetch power metric data for the same hostname |\n| util | Contain the credentials used for the utilization of API and also the main spring boot class. |\n| resources | **_input_**-power-metric data in json format,**_output_**-contains conversion of json into csv format as output of PowerMetric data,**_Application.yml_**-contains various credetials and URLs required to fetch the data |\n| test | Contains small Junit tests required for json to csv code |\n| pom.xml | Our dependencies for this project are declared in the Maven wrapper's Pom.xml file located within the MRPUSA-main folder, along with declaring runtime version for the application. |\n\n# Running the Application:\n\n### RUNNING MRPUSA \n- Ensure JDK,Maven,GitHub,and MRPUSA project set up properly in IDE tool.\n- Run spring boot application(MRPUSA app).\n- Open postman or swagger and setup all credentials and URLs, Ref. controller class for structure.\n- In Postman or Swagger,trigger the endpoint \"OPENSOURCE_GENERIC\", \n It will convert the power-metric data response stored in JSON (src/main/resources/input/powermetricData.json) into CSV file as output (src/main/resources/output/physical_server_powermetric.csv).\n\n### RUNNING MRPUSA APP\n- Ensure JDK,Maven,GitHub,and MRPUSA project set up properly in IDE tool.\n- Replace all the credentials and URLs in the application.yml and in util file.\n- Add server_name/hostname of which you want to fetch power metric data into application-data.yaml file\n inside \u003cservar name\u003e tag.\n- Run spring boot application(MRPUSA app).\n- As per the scheduler,it triggers \"PHYSICAL_SERVER\" endpoint which start collecting server list and \n power-metric data of that servers for every 20 min.\n\n# PART-2:Implement SQL OPERATIONS\n\n- Create table which calculates the average power based on redfish output.\n- Create table which calculates the average power for servers in the inventory that don’t have an actual power\n measurement.\n- Create table which split clustered Bottom-up energy on VMs.\n- Create table which estimate energy split by type based on maximum rated power.\n- Create table which calculate breakdown of top-down energy to be apportioned.\n- Final stage is apportioning overheads and applying location and market-based factors.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhsbc%2Fmrpusa","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhsbc%2Fmrpusa","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhsbc%2Fmrpusa/lists"}