{"id":28606193,"url":"https://github.com/endava/awsimporter-impactframework-plugin","last_synced_at":"2026-04-28T01:33:06.205Z","repository":{"id":242694584,"uuid":"808678338","full_name":"Endava/awsimporter-impactframework-plugin","owner":"Endava","description":"Plugin for the Green Software Foundation's Impact Framework allowing retrieval of metadata and CloudWatch observations from EC2 instances and EBS volumes for use in a pipeline.","archived":false,"fork":false,"pushed_at":"2024-08-30T09:15:46.000Z","size":114,"stargazers_count":2,"open_issues_count":2,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-11T20:16:50.927Z","etag":null,"topics":["aws","aws-ebs","aws-ec2","aws-sdk","green-software-foundation","impact-framework"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Endava.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-31T15:19:22.000Z","updated_at":"2024-11-23T14:10:11.000Z","dependencies_parsed_at":"2024-06-04T13:52:34.979Z","dependency_job_id":"4e8881b9-1388-481b-a0e3-c20c4429dc0e","html_url":"https://github.com/Endava/awsimporter-impactframework-plugin","commit_stats":null,"previous_names":["endava/awsimporter-impactframework-plugin"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/Endava/awsimporter-impactframework-plugin","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Endava%2Fawsimporter-impactframework-plugin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Endava%2Fawsimporter-impactframework-plugin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Endava%2Fawsimporter-impactframework-plugin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Endava%2Fawsimporter-impactframework-plugin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Endava","download_url":"https://codeload.github.com/Endava/awsimporter-impactframework-plugin/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Endava%2Fawsimporter-impactframework-plugin/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32362781,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-27T20:07:02.737Z","status":"ssl_error","status_checked_at":"2026-04-27T20:07:00.910Z","response_time":128,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-ebs","aws-ec2","aws-sdk","green-software-foundation","impact-framework"],"created_at":"2025-06-11T19:39:34.366Z","updated_at":"2026-04-28T01:33:06.200Z","avatar_url":"https://github.com/Endava.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AwsImporter\n\nThe AwsImporter plugin allows users to include time-series observations from AWS resources within an Impact Framework pipeline, using Tags to identify resources and CloudWatch to provide metrics. This information is then used to populate usage params in your manifest's tree, providing disk, memory and CPU utilization data for further calculations of energy use, carbon emissions or intensity. The plugin currently supports EC2 VM and EBS attached storage types.\n\n## AWS Set-up\n\n### 1. Create an AWS VM instance\nYou can create one using [console.aws.amazon.com](https://console.aws.amazon.com/console/home).  \u003cbr/\u003e\n\n\u003cb\u003eNote:\u003c/b\u003e\nFor our porposes we used an Amazon Linux AMI. (amazon/al2023-ami-2023.3.20240312.0)\n\n### 2. Create readonly user\n\nCreate an IAM user without console access. This will allow read only access to the required metrics.\nAssign to the user only the following AWS Managed policies:\n1. CloudWatchReadOnlyAccess\n2. AmazonEC2ReadOnlyAccess\n\nExport the access keys and store them safely. Will be needed at step 6.\n\n### 3. Create a IAM role to allow EC2 to push data to CW\nCreate an IAM role and assign it the AWS Managed policy \"CloudWatchAgentServerPolicy\". Add a Trust Relationship as described below.\n\u003cdetails\u003e\n\u003csummary\u003eIAM Trust Relationship\u003c/summary\u003e\n\u003cbr\u003e\n\n```json\n{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        {\n            \"Sid\": \"\",\n            \"Effect\": \"Allow\",\n            \"Principal\": {\n                \"Service\": \"ec2.amazonaws.com\"\n            },\n            \"Action\": \"sts:AssumeRole\"\n        }\n    ]\n}\n```\n\n\u003c/details\u003e\n\n### 4. Configure CloudWatch agent\nInstall cloudwatch agent on the VM. Use the relevant command based on your distribution of Linux. You can use the [AWS Cloudwatch Agent Configuration wizard](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-cloudwatch-agent-configuration-file-wizard.html)\n\nThe cloudwatch agent config looked like this for us (by default is should be /opt/aws/amazon-cloudwatch-agent/bin/config. json if you used the wizard):\n\u003cdetails\u003e\n\u003csummary\u003eCloudwatch Agent Configuration\u003c/summary\u003e\n\u003cbr\u003e\n\n```json\n{\n    \"agent\": {\n        \"metrics_collection_interval\": 60,\n        \"run_as_user\": \"cwagent\"\n    },\n    \"metrics\": {\n        \"namespace\": \"CloudwatchAgentMetrics\",\n        \"append_dimensions\": {\n            \"ImageId\": \"${aws:ImageId}\",\n            \"InstanceId\": \"${aws:InstanceId}\",\n            \"InstanceType\": \"${aws:InstanceType}\"\n        },\n        \"metrics_collected\": {\n            \"cpu\": {\n                \"measurement\": [\n                    \"cpu_usage_user\",\n                    \"cpu_usage_system\"\n                ],\n                \"metrics_collection_interval\": 60,\n                \"totalcpu\": true,\n                \"resources\": [\n                    \"*\"\n                ]\n            },\n            \"disk\": {\n                \"measurement\": [\n                    \"used_percent\",\n                    \"free\",\n                    \"used\",\n                    \"total\"\n                ],\n                \"metrics_collection_interval\": 60,\n                \"resources\": [\n                    \"*\"\n                ]\n            },\n            \"diskio\": {\n                \"measurement\": [\n                    \"io_time\"\n                ],\n                \"metrics_collection_interval\": 60,\n                \"resources\": [\n                    \"*\"\n                ]\n            },\n            \"mem\": {\n                \"measurement\": [\n                    \"mem_used_percent\",\n                    \"mem_available_percent\"\n                    \"mem_total\"\n                ],\n                \"metrics_collection_interval\": 60\n            },\n            \"swap\": {\n                \"measurement\": [\n                    \"swap_used_percent\"\n                ],\n                \"metrics_collection_interval\": 60\n            }\n        }\n    }\n}\n```\n\n\u003c/details\u003e\n\n### 5. (Optional) Check the metrics in AWS Cloudwatch\n\nSee [here](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/PublishMetrics.html#ViewGraphs) for more details.\n\n### 6. Add credentials to `.env`\n\nTo manage your AWS authentication details securely, you can create a file called .env in the main directory of your project. This file serves as a convenient location to store sensitive information like access keys and secret keys. By storing this information in a dedicated file, you can easily access and manage your AWS authentication details across your project.\n\nIn the .env file, you'll define key-value pairs for your AWS credentials, following a specific format. For example:\n\n```txt\nAWS_ACCESS_KEY_ID=your_access_key_id\nAWS_SECRET_ACCESS_KEY=your_secret_access_key\n```\n\nReplace your_access_key_id and your_secret_access_key with your actual AWS access key ID and secret access key, respectively. Remember to keep this file secure and never expose it publicly, as it contains sensitive information that grants access to your AWS resources.\n\n\n## Node config\n\n- `aws-importer`: Specifies configurations related to AWS importing process\n  - `tag`: Used to identify the AWS resources you wish to observe. For example within an SCI score's 'application boundary'.\n  - `location`: Specifies the AWS region where your resources are located.\n  - `aws-observation`: Sets the observation period for AWS metrics in seconds. This defines how frequently metrics are collected and analyzed\n  - `aws-services`: Specifies which AWS resource types to output observations from, currently ec2 or eb2.\n- `metric`: Contains configurations related to metrics.\n  - `cloudwatch-namespace`: Defines the namespace for CloudWatch metrics. The namespace is used to categorize metrics, making it easier to organize and manage them.\n  - `client-namespace`: Specifies the namespace for the AWS EC2 client. This namespace is used to identify the AWS service to which the client belongs.\n\n## Inputs\n\nThe plugin config enables the identification of resources by service type and tagging. Input parameters allow you to define the time and duration of your observation period. This section requires the following mandatory params:\n\n- `timestamp`: Provide an ISO8601 `timestamp` representing the beginning of your observation period. By adding the `duration` to this initial `timestamp`, we determine the end time of your observation period.\n- `duration`: Specify the `duration` of your observation period in seconds. This `duration` value is added to the `timestamp` to calculate the end time of your observation period.\n\nAll of these details are given as `inputs`. Additionally, you'll need to set up an instance of the `aws-importer` plugin to manage the specific `input` data for AWS. Below is an example of a fully configured manifest:\n\n```yaml\nname: aws-importer\ndescription: example manifest utilizing the AWS plugin\ntags: null\ninitialize:\n  plugins:\n    aws-importer:\n      method: AwsImporter\n      path: 'aws-importer'\n      global-config:\n        aws-importer:\n          tag: 'GreenSoftware'\n          location: eu-central-1\n          aws-services: 'ec2, ebs'\n          aws-observation: 60\n        metric:\n          cloudwatch-namespace: 'CloudwatchAgentMetrics'\n          client-namespace: 'AWS/EC2'\n  outputs: ['yaml']\ntree:\n  children:\n    child:\n      pipeline:\n        - aws-importer\n      inputs:\n        - timestamp: '2024-03-26T14:08:00.000Z'\n          duration: 3600\n```\n\nThe input will grab AWS metrics for 3600 seconds (1 hour) period beginning at 14:08 UTC on 26th March 2024\n\nExecute the following command from the project root to run the unit tests\n```sh\nnpm run test\n```\n\n## Outputs\n\nThe AWS importer plugin will enrich the information in your `manifest`, resulting in the following output data:\n\n- `timestamp`: the per-input `timestamp`\n- `duration`: the per-input `duration` in seconds\n- `location`: The AWS region, initially defined with the global or node config\n- `geolocation`: LatLong of the resource's AWS region, based on Google geocoding for the region city\n- `cloud/vendor`: The cloud vendor. For this plugin it will always be aws\n- `cloud/service`: The AWS service associated with this resource, ie. ec2 or ebs\n- `cloud/instance-type`: The name/sku of the AWS resource\n\nFor each result, there is aditional data. It can contains utilization or storage details:\n\n- `memory/utilization`: percentage memory utilization\n- `cpu/utilization`: percentage CPU utilization\nOr:\n- `storage/type`: storage type\n- `storage/capacity`: storage capacity\n\nFor a one hour observation of a single EC2 VM and attached SSD storage, the output will look as follows:\n\n```yaml\noutputs:\n  - timestamp: '2024-03-26T14:08:00.000Z'\n    duration: 3600\n    location: eu-central-1\n    geolocation: 50.1213155,8.471759\n    cloud/vendor: aws\n    cloud/service: ec2\n    cloud/instance-type: t2.micro\n    memory/utilization: 0.6914698759172085\n    cpu/utilization: 0.9272994995117188\n  - timestamp: '2024-03-26T14:08:00.000Z'\n    duration: 3600\n    location: eu-central-1\n    geolocation: 50.1213155,8.471759\n    cloud/vendor: aws\n    cloud/service: ebs\n    storage/type: ssd\n    storage/capacity: 8\n```\n\n## Error Handling\n- The plugin conducts config validation using the zod library and will throw errors if required name/value pairs are missing.\n- The plugin conducts input validation using the zod library and will throw errors if required parameters are missing, or the provided parameters are invalid.\n- The plugin uses AWS credentials from .env vars. The AWS SDK will throw an error if the credentials are missing or invalid.\n\n\nExecuting the following command from the project root:\n\n```sh\nnpm install\n\nnpm run build\n\nnpm link\n```\n\nNavigate to the folder with your manifest file and run the following:\n```sh\nnpm link aws-importer\n```\n\nGo back to the root folder and run your manifest:\n```sh\nie --manifest ./examples/aws-importer.yml --output ./examples/aws-importer-computed.yml\n```\n\nThe results will be saved to a new `yaml` file in `./examples`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fendava%2Fawsimporter-impactframework-plugin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fendava%2Fawsimporter-impactframework-plugin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fendava%2Fawsimporter-impactframework-plugin/lists"}