{"id":13557656,"url":"https://github.com/project-koku/koku","last_synced_at":"2026-02-20T04:03:22.570Z","repository":{"id":37470279,"uuid":"126496611","full_name":"project-koku/koku","owner":"project-koku","description":"An open source solution for cost management of cloud and hybrid cloud environments.","archived":false,"fork":false,"pushed_at":"2024-10-29T15:03:32.000Z","size":27998,"stargazers_count":269,"open_issues_count":10,"forks_count":92,"subscribers_count":23,"default_branch":"main","last_synced_at":"2024-10-29T17:14:23.044Z","etag":null,"topics":["cloud","cost","cost-control","cost-estimation","cost-optimization","koku"],"latest_commit_sha":null,"homepage":"https://github.com/project-koku/koku/tree/main/docs","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/project-koku.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"licenses/Apache-2.0.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-03-23T14:29:23.000Z","updated_at":"2024-10-29T13:13:31.000Z","dependencies_parsed_at":"2023-12-20T19:32:35.605Z","dependency_job_id":"1414d216-032b-44de-bf82-8f24c1547404","html_url":"https://github.com/project-koku/koku","commit_stats":null,"previous_names":[],"tags_count":351,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/project-koku%2Fkoku","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/project-koku%2Fkoku/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/project-koku%2Fkoku/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/project-koku%2Fkoku/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/project-koku","download_url":"https://codeload.github.com/project-koku/koku/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247002169,"owners_count":20867415,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud","cost","cost-control","cost-estimation","cost-optimization","koku"],"created_at":"2024-08-01T12:04:28.320Z","updated_at":"2025-04-03T12:30:54.886Z","avatar_url":"https://github.com/project-koku.png","language":"Python","funding_links":[],"categories":["Python","others","Cloud Cost Management Tools and Services","Repositories"],"sub_categories":["MultiCloud"],"readme":"# Koku README\n\n[![license](https://img.shields.io/github/license/project-koku/koku.svg)](https://github.com/project-koku/koku/blob/main/LICENSE)\n[![CI](https://github.com/project-koku/koku/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/project-koku/koku/actions/workflows/ci.yml)\n[![codecov](https://codecov.io/gh/project-koku/koku/branch/main/graph/badge.svg)](https://codecov.io/gh/project-koku/koku)\n\n## About\n\nKoku's goal is to provide an open source solution for cost management of cloud and hybrid cloud environments. This solution is offered via a web interface that exposes resource consumption and cost data in easily digestible and filterable views. The project also aims to provide insight into this data and ultimately provide suggested optimizations for reducing cost and eliminating unnecessary resource usage.\n\nFull documentation is available in [docs folder](docs).\n\nTo submit an issue please visit [https://issues.redhat.com/projects/COST/]().\n\n## Getting Started\n\nThis project is developed using Python 3.11. Make sure you have at least this version installed.\n\n### Prerequisites\n\n- Docker or Rancher Desktop\n- (macOS only) [Install Homebrew](https://brew.sh/)\n\n## Development\n\nTo get started developing against Koku you first need to clone a local\ncopy of the git repositories.:\n\n    git clone https://github.com/project-koku/koku\n    git clone https://github.com/project-koku/nise\n\nThis project is developed using the Django web framework. Many\nconfiguration settings can be read in from a `.env` file. To configure,\ndo the following:\n\n1.  Copy [`.env.example`](.env.example) into a `.env` and update the following in your `.env`:\n\n        AWS_RESOURCE_NAME=YOUR_COST_MANAGEMENT_AWS_ARN\n\n2.  Copy [`dev/credentials/aws.example`](dev/credentials/aws.example) into `dev/credentials/aws`, obtain AWS credentials, then update the credentials file:\n\n        [default]\n        aws_access_key_id=YOUR_AWS_ACCESS_KEY_ID\n        aws_secret_access_key=YOUR_AWS_SECRET_ACCESS_KEY\n\n3.  (macOS only) Install libraries for building wheels on ARM:\n\n        brew install openssl librdkafka postgresql@13\n\n3.  (Fedora only) Install libraries for building wheels on Linux:\n\n        dnf install openssl-devel libpq-devel postgresql golang-sigs-k8s-kustomize\n\n4.  (macOS only) Also add the following to your `.env` or shell profile:\n\n        LDFLAGS=\"-L$(brew --prefix openssl)/lib -L$(brew --prefix librdkafka)/lib -L$(brew --prefix postgresql@13)/lib\"\n        CPPFLAGS=\"-I$(brew --prefix openssl)/include -I$(brew --prefix librdkafka)/include -I$(brew --prefix postgresql@13)/include\"\n        PATH=\"$PATH:$(brew --prefix postgresql@13)/bin\"\n\n5.  Developing inside a virtual environment is recommended. A Pipfile is provided. Pipenv is recommended for combining virtual environment and dependency management. To install `pipenv`:\n\n        pip3 install pipenv\n\n6.  Then project dependencies and a virtual environment can be created using:\n\n        pipenv install --dev\n\n7.  To activate the virtual environment run:\n\n        pipenv shell\n\n8.  Install the pre-commit hooks for the repository:\n\n        pre-commit install\n\n### Developing with Docker Compose\n\nThis will explain how to start the server and its dependencies using Docker (or Rancher Desktop), create AWS/OCP sources, and view reports. This will not cover all API or scenarios but should give you an end-to-end flow.\n\n#### Starting Koku using Docker Compose\n\n\u003e **Note**\n\u003e\n\u003e In order for the `koku_base` image to build correctly, buildkit must be enabled by setting `DOCKER_BUILDKIT=1`. This is set in the `.env` file, but if you are having issues building the `koku_base` image, make sure buildkit is enabled.\n\n1.  Start the containers:\n\n        make docker-up-min\n\n2.  Display log output from the containers. It is recommended that logs be kept in a second terminal:\n\n        make docker-logs\n\nWith all containers running any source added will be processed by saving CSV files in MinIO and storing Parquet files in MinIO. The source's data will be summarized via Trino. Summarized data will land in the appropriate daily_summary table for the source type for consumption by the API.\n\nTo add test sources and data:\n\n    make create-test-customer\n    make load-test-customer-data # Optional parameters: start={start_date} end={end_date} test_source=AWS\n\nThe MinIO UI will be available at http://127.0.0.1:9090/minio/. Use the `S3_ACCESS_KEY` and `S3_SECRET` set in your `.env` file as login credentials.\n\nThe Trinio UI will be available at http://127.0.0.1:8080/ui/. Details can be found there on queries. This is particularly useful for troubleshooting failures.\n\nAccess the Trino CLI using the following command:\n\n    docker exec -it trino trino --server 127.0.0.1:8080 --catalog hive --schema org1234567 --user admin --debug\n\nExample usage:\n\n    SHOW tables;\n    SELECT * from aws_line_items WHERE source='{source}' AND year='2023' AND month='02' LIMIT 100;\n\n#### Run AWS Scenario\n\n1.  Create AWS Source:\n\n        make aws-source aws_name=AWS-SOURCE-001 bucket=cost-usage-bucket\n\n2.  Verify source exists by visiting\n\n    http://127.0.0.1:8000/api/cost-management/v1/sources/?name=AWS-SOURCE-001\n\n3.  Trigger MASU processing by visiting\n\n    http://127.0.0.1:5042/api/cost-management/v1/download/\n\n4.  Wait for processing to complete\n\n5.  Verify data existing using AWS API endpoints\n\n    -   http://127.0.0.1:8000/api/cost-management/v1/reports/aws/instance-types/\n    -   http://127.0.0.1:8000/api/cost-management/v1/reports/aws/costs/\n    -   http://127.0.0.1:8000/api/cost-management/v1/reports/aws/storage/\n\n#### Run OCP Scenario\n\n1.  Create OCP Source:\n\n        make ocp-source-from-yaml cluster_id=my_test_cluster srf_yaml=../nise/example_ocp_static_data.yml ocp_name=my_ocp_name\n\n2.  Verify provider exists by visiting\n\n    http://127.0.0.1:8000/api/cost-management/v1/sources/?name=my_ocp_name\n\n3.  Trigger MASU processing by visiting\n\n    http://127.0.0.1:5042/api/cost-management/v1/download/\n\n4.  Wait for processing to complete\n\n5.  Verify data exists using API endpoints\n\n    -   http://127.0.0.1:8000/api/cost-management/v1/reports/openshift/volumes/\n    -   http://127.0.0.1:8000/api/cost-management/v1/reports/openshift/memory/\n    -   http://127.0.0.1:8000/api/cost-management/v1/reports/openshift/compute/\n\n#### Run GCP Scenario\n\n1.  Set Environment variables:\n\n        GCP_DATASET - The name of the BigQuery dataset in your GCP setup.\n        GCP_TABLE_ID - The identifier for the table you are pulling for the billing information.\n        GCP_PROJECT_ID - The identifier for the GCP project.\n\n2.  Create GCP source:\n\n        make gcp-source gcp_name=my_gcp_source\n\n3.  Verify provider exists by visiting\n\n    http://127.0.0.1:8000/api/cost-management/v1/sources/?name=my_gcp_source\n\n#### Stopping Koku using Docker Compose\n\nTo bring down all the docker containers, run the following command:\n\n    make docker-down\n\n#### Database\n\nPostgreSQL is used as the database backend for Koku. A docker compose file is provided for creating a local database container. Assuming the default `.env` file values are used, to access the database directly using psql run:\n\n    PGPASSWORD=postgres psql postgres -U postgres -h 127.0.0.1 -p 15432\n\n\u003e **Note**\n\u003e\n\u003e There is a known limitation with Docker Compose and Linux environments with SELinux enabled. You may see the following error during the postgres container deployment:\n\u003e\n\u003e       \"mkdir: cannot create directory '/var/lib/pgsql/data/userdata': Permission denied\" can be resolved by granting ./pg_data ownership permissions to uid:26 (postgres user in centos/postgresql-96-centos7)\n\u003e\n\u003e If you see this error, run the following command (assuming you are at the project top level directory):\n\u003e\n\u003e       setfacl -m u:26:-wx ./pg_data\n\n\n#### Database Query Monitoring\n\nA basic level of query monitoring has been included leveraging a local grafana container which will be built with the docker-up make target.\n\nTo use the monitor, open a new web browser tab or window and enter the following URL: http://127.0.0.1:3001\n\nYou will be presented with the grafana login page. For this monitor, use the following credentials:\n\n    User: admin\n    Password: admin12\n\nOnce you have logged into the server, you will be taken straight to the main dashboard. It will have 5 panels.\n\n\u003ctable\u003e\n    \u003ctr\u003e\n        \u003ctd colspan=\"2\"\u003eQuery statistics\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eConnect States\u003c/td\u003e\u003ctd\u003eActive Queries\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eLock Types\u003c/td\u003e\u003ctd\u003eLock Detail\u003c/td\u003e\n    \u003c/tr\u003e\n\u003c/table\u003e\n\n\n-   Query Statistics - The max execution time, the mean execution time, number of calls and the query text\n-   Connect States - Shows the connection states (active, idle, idle in transaction, etc)\n-   Active Queries - Shows the approximate run time (based on the probe time) and the query text of queries detected\n-   Lock Types - Shows the discrete lock types detected during the probe\n-   Lock Detail - Shows any detail informaiton for the lock and the affected query.\n\nThe Query Statistics panel is cumulative. The remaining panels are ephemeral.\n\nInformation about PostgreSQL statistics can be found here: https://www.postgresql.org/docs/12/monitoring-stats.html\n\nInformation about Grafana dashboards can be found here: https://grafana.com/docs/grafana/latest/features/dashboard/dashboards/\n\n\n### Developing with OpenShift\n\nOur production deployment runs on OpenShift. At times you may need to run on OpenShift if you are working on deployment templates or would like to test in a production like environment. This is a more advanced scenario that many new developers will not need. To learn how to run OpenShift refer to [Working with OpenShift](docs/openshift.md).\n\n### Testing\n\nKoku uses tox to standardize the environment used when running tests. Essentially, tox manages its own virtual environment and a copy of required dependencies to run tests. To ensure a clean tox environment run:\n\n    tox -r\n\nThis will rebuild the tox virtual env and then run all tests.\n\nTo run unit tests specifically:\n\n    tox -e py39\n\nTo run a specific subset of unit tests, you can pass a particular module path to tox. To do this, use positional args using the -- separator. For example:\n\n    tox -e py39 -- masu.test.external.downloader.azure.test_azure_services.AzureServiceTest\n\nTo run IQE Smoke, Vortex or API tests, while on the Red Hat network and koku deployed via docker compose run:\n\n    make docker-iqe-smokes-tests\n    make docker-iqe-vortex-tests\n    make docker-iqe-api-tests\n\nIndividual IQE tests can be ran with run_test.sh:\n\n    \u003ckoku_topdir\u003e/testing/run_test.sh iqe tests plugin hccm -k test_api_cost_model_markup_calculation_ocp\n\n### Linting\n\nThis repository uses [pre-commit](https://pre-commit.com) to check and enforce code style. It uses [Black](https://github.com/psf/black) to reformat the Python code and [Flake8](http://flake8.pycqa.org) to check it afterwards. Other formats and text files are linted as well.\n\nTo run pre-commit checks:\n\n    pre-commit run --all-files\n\n### pgAdmin\n\nIf you want to interact with the Postgres database from a GUI:\n\n1.  Copy the [pgadmin_servers.json.example](pgadmin_servers.json.example) into a `pgadmin_servers.json` file and if necessary, change any variables to match your database.\n2.  `docker compose up -d pgadmin` causes pgAdmin to run on http://127.0.0.1:8432\n3.  In the login screen, the default login email:password is `postgres@local.dev:postgres`\n\n\u003e **Note**\n\u003e\n\u003e The pgadmin_servers.json file is written in [json format](https://www.pgadmin.org/docs/pgadmin4/development/import_export_servers.html#json-format).\n\n### Partitioned Tables\n\nThe koku project is now making use of partitioned tables with PostgreSQL 12. Currently, there are only a limited number of modules supporting table partitoning with Django. For this reason, the partitioned tables are being unmanaged by Django models.\n\nThere is a stored procedure that helps create table partitions on-the-fly during masu data processing. This procedure is called `create_date_partitons` and is in the `public` schema. It is designed to scan a table for partition range start values and compare them against a tracking table of table partitions. For any range start not present in the table, a table partition will be created. This allows for partition creation for any uncovered range, past, present, or future. See `db_functions/create_date_partitions.sql` for the code and parameter documentation.\n\n## Contributing\n\nPlease refer to [Contributing](https://github.com/project-koku/koku/blob/main/CONTRIBUTING.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproject-koku%2Fkoku","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fproject-koku%2Fkoku","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproject-koku%2Fkoku/lists"}