{"id":13582245,"url":"https://github.com/banzaicloud/cloudinfo","last_synced_at":"2025-04-09T05:08:22.197Z","repository":{"id":39575889,"uuid":"139873273","full_name":"banzaicloud/cloudinfo","owner":"banzaicloud","description":"Cloud instance type and price information as a service","archived":false,"fork":false,"pushed_at":"2025-02-17T09:15:14.000Z","size":12373,"stargazers_count":168,"open_issues_count":85,"forks_count":37,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-04-02T13:50:08.708Z","etag":null,"topics":["cloud-billing-api","cloud-providers","price-information","spot-price"],"latest_commit_sha":null,"homepage":"https://banzaicloud.com/cloudinfo","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/banzaicloud.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-07-05T16:10:58.000Z","updated_at":"2025-01-05T14:08:04.000Z","dependencies_parsed_at":"2023-02-18T07:45:50.940Z","dependency_job_id":"e878e89f-e8ed-4187-818f-e6b5efd84bb1","html_url":"https://github.com/banzaicloud/cloudinfo","commit_stats":null,"previous_names":[],"tags_count":157,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/banzaicloud%2Fcloudinfo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/banzaicloud%2Fcloudinfo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/banzaicloud%2Fcloudinfo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/banzaicloud%2Fcloudinfo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/banzaicloud","download_url":"https://codeload.github.com/banzaicloud/cloudinfo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247980837,"owners_count":21027808,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud-billing-api","cloud-providers","price-information","spot-price"],"created_at":"2024-08-01T15:02:31.746Z","updated_at":"2025-04-09T05:08:22.180Z","avatar_url":"https://github.com/banzaicloud.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"\n[![CI](https://github.com/banzaicloud/cloudinfo/actions/workflows/ci.yaml/badge.svg)](https://github.com/banzaicloud/cloudinfo/actions/workflows/ci.yaml)\n[![Docker](https://github.com/banzaicloud/cloudinfo/actions/workflows/docker.yaml/badge.svg)](https://github.com/banzaicloud/cloudinfo/actions/workflows/docker.yaml)\n[![Helm chart](https://github.com/banzaicloud/cloudinfo/actions/workflows/helm_chart.yaml/badge.svg)](https://github.com/banzaicloud/cloudinfo/actions/workflows/helm_chart.yaml)\n[![Go Report Card](https://goreportcard.com/badge/github.com/banzaicloud/cloudinfo)](https://goreportcard.com/report/github.com/banzaicloud/cloudinfo)\n![license](http://img.shields.io/badge/license-Apache%20v2-orange.svg)\n[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/1651/badge)](https://bestpractices.coreinfrastructure.org/projects/1651)\n\n\n# Cloud price and service information\n\nThe Banzai Cloud Cloudinfo application is a standalone project in the [Pipeline](https://github.com/banzaicloud/pipeline) ecosystem.\nWhile AWS, Google Cloud, Azure, AliBaba or Oracle all provide some kind of APIs to query instance type attributes and product pricing information, these APIs are often responding with partly inconsistent data, or the responses are very cumbersome to parse.\nThe Cloudinfo service uses these cloud provider APIs to asynchronously fetch and parse instance type attributes and prices, while storing the results in an in memory cache and making it available as structured data through a REST API.\n\n## Quick start\n\nBuilding the project is as simple as running a go build command. The result is a statically linked executable binary.\n\n```\nmake build\n```\n\nThe following options can be configured when starting the exporter (with defaults):\n\n```\nbuild/cloudinfo --help\nUsage of Banzai Cloud Cloudinfo Service:\n      --config-vault string               enable config Vault\n      --config-vault-address string       config Vault address\n      --config-vault-token string         config Vault token\n      --config-vault-secret-path string   config Vault secret path\n      --log-level string                  log level (default \"info\")\n      --log-format string                 log format (default \"json\")\n      --metrics-enabled                   internal metrics are exposed if enabled\n      --metrics-address string            the address where internal metrics are exposed (default \":9090\")\n      --listen-address string             application listen address (default \":8000\")\n      --scrape                            enable cloud info scraping (default true)\n      --scrape-interval duration          duration (in go syntax) between renewing information (default 24h0m0s)\n      --provider-amazon                   enable amazon provider\n      --provider-google                   enable google provider\n      --provider-alibaba                  enable alibaba provider\n      --provider-oracle                   enable oracle provider\n      --provider-azure                    enable azure provider\n      --provider-digitalocean             enable digitalocean provider\n      --config string                     Configuration file\n      --version                           Show version information\n      --dump-config                       Dump configuration to the console (and exit)\n```\n\nCreate a permanent developer configuration:\n\n```bash\ncp config.toml.dist config.toml\n```\n\nRunning `cloudinfo` requires the `web/` project to be built (requires `Node.js` to be installed):\n\n```bash\ncd web/\nnpm run build-prod\ncd ..\nbuild/cloudinfo\n```\n\n## Cloud credentials\n\nThe cloudinfo service is querying the cloud provider APIs, so it needs credentials to access these.\n\n### AWS\n\nCloudinfo is using the AWS [Price List API](https://aws.amazon.com/blogs/aws/aws-price-list-api-update-new-query-and-metadata-functions/) that allows a user to query product pricing in a fine-grained way.\nAuthentication works through the standard [AWS SDK for Go](https://aws.amazon.com/sdk-for-go/), so credentials can be configured via\nenvironment variables, shared credential files and via AWS instance profiles. To learn more about that read the [Specifying Credentials](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html) section of the SDK docs.\n\nThe easiest way is through environment variables:\n\n```\nexport AWS_ACCESS_KEY_ID=\u003caccess-key-id\u003e\nexport AWS_SECRET_ACCESS_KEY=\u003csecret-access-key\u003e\ncloudinfo --provider-amazon\n```\n\nCreate AWS credentials with aws command-line tool:\n\n```\naws iam create-user --user-name cloudinfo\naws iam put-user-policy --user-name cloudinfo --policy-name cloudinfo_policy --policy-document file://credentials/amazon_cloudinfo_role.json\naws iam create-access-key --user-name cloudinfo\n```\n\n### Google Cloud\n\nOn Google Cloud the project is using two different APIs to collect the full product information: the Cloud Billing API and the Compute Engine API.\nAuthentication to the `Cloud Billing Catalog API` is done through an [API key](https://cloud.google.com/docs/authentication/api-keys) that can be generated on the [Google Cloud Console](https://console.cloud.google.com/apis/credentials).\nOnce you have an API key, billing is [enabled](https://cloud.google.com/billing/docs/how-to/modify-project) for the project and the Cloud Billing API is also [enabled](https://console.cloud.google.com/flows/enableapi?apiid=cloudbilling.googleapis.com) you can start using the API.\n\nThe `Compute Engine API` is doing authentication in the standard Google Cloud way with [service accounts](https://cloud.google.com/compute/docs/access/service-accounts) instead of API keys.\nOnce you have a service account, download the JSON credentials file from the Google Cloud Console, and set its account through an environment variable:\n\n```\nexport GOOGLE_CREDENTIALS_FILE=\u003cpath-to-my-service-account-file\u003e.json\nexport GOOGLE_PROJECT=\u003cgoogle-project-id\u003e\ncloudinfo --provider-google\n```\n\nCreate service account key with gcloud command-line tool:\n\n```\ngcloud services enable container.googleapis.com compute.googleapis.com cloudbilling.googleapis.com cloudresourcemanager.googleapis.com\ngcloud iam service-accounts create cloudinfoSA --display-name \"Service account used for managing Cloudinfo”\ngcloud iam roles create cloudinfo --project [PROJECT-ID] --title cloudinfo --description \"cloudinfo roles\" --permissions compute.machineTypes.list,compute.regions.list,compute.zones.list\ngcloud projects add-iam-policy-binding [PROJECT-ID] --member='serviceAccount:cloudinfoSA@[PROJECT-ID].iam.gserviceaccount.com' --role='projects/[PROJECT-ID]/roles/cloudinfo'\ngcloud iam service-accounts keys create cloudinfo.gcloud.json --iam-account=cloudinfoSA@[PROJECT-ID].iam.gserviceaccount.com\n```\n\n### Azure\n\nThere are two different APIs used for Azure that provide machine type information and SKUs respectively.\nPricing info can be queried through the [Rate Card API](https://msdn.microsoft.com/en-us/library/azure/mt219004).\nMachine types can be queried through the Compute API's [list virtual machine sizes](https://docs.microsoft.com/en-us/rest/api/compute/virtualmachinesizes/list) request.\nAuthentication is done via standard Azure service principals.\n\nFollow [this](https://docs.microsoft.com/en-us/go/azure/azure-sdk-go-qs-vm#create-a-service-principal) link to learn how to generate one with the Azure SDK\nand set an environment variable that points to the service account file:\n\n```\nexport AZURE_SUBSCRIPTION_ID=\u003csubscription-id\u003e\nexport AZURE_TENANT_ID=\u003ctenant-id\u003e\nexport AZURE_CLIENT_ID=\u003cclient-id\u003e\nexport AZURE_CLIENT_SECRET=\u003cclient-secret\u003e\ncloudinfo --provider-azure\n```\n\nCreate service principal with azure command-line tool:\n\n```\ncd credentials\naz provider register --namespace Microsoft.Compute\naz provider register --namespace Microsoft.Resources\naz provider register --namespace Microsoft.ContainerService\naz provider register --namespace Microsoft.Commerce\naz role definition create --verbose --role-definition @azure_cloudinfo_role.json\naz ad sp create-for-rbac --name \"CloudinfoSP\" --role \"Cloudinfo\" --sdk-auth true \u003e azure_cloudinfo.auth\n```\n\n### Oracle\n\nAuthentication is done via CLI configuration file. Follow [this](https://docs.cloud.oracle.com/iaas/Content/API/Concepts/sdkconfig.htm) link to learn how to create such a file and set an environment variable that points to that config file:\n\n```\nexport ORACLE_TENANCY_OCID=\u003ctenancy-ocid\u003e\nexport ORACLE_USER_OCID=\u003cuser-ocid\u003e\nexport ORACLE_REGION=\u003cregion\u003e\nexport ORACLE_FINGERPRINT=\u003cfingerprint\u003e\nexport ORACLE_PRIVATE_KEY=\u003cprivate-key\u003e\nexport ORACLE_PRIVATE_KEY_PASSPHRASE=\u003cprivate-key-passphrase\u003e\n# OR\nexport ORACLE_CONFIG_FILE_PATH=\u003cconfig-file-path\u003e\nexport ORACLE_PROFILE=\u003cprofile\u003e\n\ncloudinfo --provider-oracle\n```\n\n### Alibaba\n\nThe easiest way to authenticate is through environment variables:\n\n```\nexport ALIBABA_ACCESS_KEY_ID=\u003caccess-key-id\u003e\nexport ALIBABA_ACCESS_KEY_SECRET=\u003caccess-key-secret\u003e\nexport ALIBABA_REGION_ID=\u003cregion-id\u003e\ncloudinfo --provider-alibaba\n```\n\nCreate Alibaba credentials with Alibaba Cloud CLI:\n\n```\naliyun ram CreateUser --UserName CloudInfo --DisplayName CloudInfo\naliyun ram AttachPolicyToUser --UserName CloudInfo --PolicyName AliyunECSReadOnlyAccess --PolicyType System\naliyun ram AttachPolicyToUser --UserName CloudInfo --PolicyName AliyunBSSReadOnlyAccess --PolicyType System\naliyun ram CreateAccessKey --UserName CloudInfo\n```\n\n### DigitalOcean\n\n```\nexport DIGITALOCEAN_ACCESS_TOKEN=\u003caccess-token\u003e\ncloudinfo --provider-digitalocean\n```\n\nCreate a new API access token on [DigitalOcean Console](https://cloud.digitalocean.com/account/api/tokens).\n\n### Configuring multiple providers\n\nCloud providers can be configured one by one. To configure multiple providers simply list all of them and configure the credentials for all of them.\nHere's an example of how to configure three providers:\n```\nexport AWS_SECRET_ACCESS_KEY=\u003csecret-access-key\u003e\nexport AWS_ACCESS_KEY_ID=\u003caccess-key-id\u003e\nexport ALIBABA_ACCESS_KEY_ID=\u003caccess-key-id\u003e\nexport ALIBABA_ACCESS_KEY_SECRET=\u003caccess-key-secret\u003e\nexport ALIBABA_REGION_ID=\u003cregion-id\u003e\nexport DIGITALOCEAN_ACCESS_TOKEN=\u003caccess-token\u003e\n\ncloudinfo --provider-amazon --provider-alibaba --provider-digitalocean\n```\n\n## API calls\n\n*For a complete OpenAPI 3.0 documentation, check out this [URL](https://editor.swagger.io/?url=https://raw.githubusercontent.com/banzaicloud/cloudinfo/master/api/openapi-spec/cloudinfo.yaml).*\n\nHere's a few `cURL` examples to get started:\n\n```\ncurl  -ksL -X GET \"http://localhost:9090/api/v1/providers/azure/services/compute/regions/\" | jq .\n[\n  {\n    \"id\": \"centralindia\",\n    \"name\": \"Central India\"\n  },\n  {\n    \"id\": \"koreacentral\",\n    \"name\": \"Korea Central\"\n  },\n  {\n    \"id\": \"southindia\",\n    \"name\": \"South India\"\n  },\n  ...\n]\n```\n\n```\ncurl  -ksL -X GET \"http://localhost:9090/api/v1/providers/amazon/services/compute/regions/eu-west-1/products\" | jq .\n{\n  \"products\": [\n    {\n      \"type\": \"i3.8xlarge\",\n      \"onDemandPrice\": 2.752,\n      \"cpusPerVm\": 32,\n      \"memPerVm\": 244,\n      \"gpusPerVm\": 0,\n      \"ntwPerf\": \"10 Gigabit\",\n      \"ntwPerfCategory\": \"high\",\n      \"spotPrice\": [\n        {\n          \"zone\": \"eu-west-1c\",\n          \"price\": 1.6018\n        },\n        {\n          \"zone\": \"eu-west-1b\",\n          \"price\": 0.9563\n        },\n        {\n          \"zone\": \"eu-west-1a\",\n          \"price\": 2.752\n        }\n      ]\n    },\n    ...\n  ]\n}\n```\n\n## FAQ\n\n**1. The API responses with status code 500 after starting the `cloudinfo` app and making a `cURL` request**\n\nAfter the `cloudinfo` app is started, it takes a few minutes to cache all the product information from the providers.\nBefore the results are cached, responses may be unreliable. We're planning to solve it in the future. After a few minutes it should work fine.\n\n**2. Why is it needed to parse the product info asynchronously and periodically instead of relying on static data?**\n\nCloud providers are releasing new instance types and regions quite frequently and also changing on-demand pricing from time to time.\nSo it is necessary to keep this info up-to-date without needing to modify it manually every time something changes on the provider's side.\nAfter the initial query, the `cloudinfo` app will parse this info from the Cloud providers once per day.\nThe frequency of this querying and caching is configurable with the `--product-info-renewal-interval` switch and is set to `24h` by default.\n\n**3. What happens if the `cloudinfo` app cannot cache the AWS product info?**\n\nIf caching fails, the `cloudinfo` app will try to reach the AWS Pricing List API on the fly when a request is sent (and it will also cache the resulting information).\n\n**4. What kind of AWS permissions do I need to use the project?**\n\nThe `cloudinfo` app is querying the AWS [Pricing API](https://aws.amazon.com/blogs/aws/aws-price-list-api-update-new-query-and-metadata-functions/) to keep up-to-date info\nabout instance types, regions and on-demand pricing.\nYou'll need IAM access as described here in [example 11](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/billing-permissions-ref.html#example-policy-pe-api) of the AWS IAM docs.\n\nIf you don't use Prometheus to track spot instance pricing, you'll need to be able to access the [spot price history](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeSpotPriceHistory.html) from the AWS API as well with your IAM user.\nIt means giving permission to `ec2:DescribeSpotPriceHistory`.\n\n**5. What is the advantage of using Prometheus to determine spot prices?**\n\nPrometheus is becoming the de-facto monitoring solution in the cloud native world, and it includes a time series database as well.\nWhen using the Banzai Cloud [spot price exporter](https://github.com/banzaicloud/spot-price-exporter), spot price history will be collected as time series data and\ncan be queried for averages, maximums and predictions.\nIt gives a richer picture than relying on the current spot price that can be a spike, or on a downward or upward trend.\nYou can fine tune your query (with the `-prometheus-query` switch) if you want to change the way spot instance prices are scored.\nBy default the spot price averages of the last week are queried and instance types are sorted based on this score.\n\n**6. What happens if my Prometheus server cannot be reached or if it doesn't have the necessary spot price metrics?**\n\nIf the `cloudinfo` app fails to reach the Prometheus query API, or it couldn't find proper metrics, it will fall back to querying the current spot prices from the AWS API.\n\n### License\n\nCopyright (c) 2017-2019 [Banzai Cloud, Inc.](https://banzaicloud.com)\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbanzaicloud%2Fcloudinfo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbanzaicloud%2Fcloudinfo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbanzaicloud%2Fcloudinfo/lists"}