{"id":19748669,"url":"https://github.com/doitintl/bigquery-optimization-queries","last_synced_at":"2025-04-06T02:07:18.022Z","repository":{"id":86290446,"uuid":"569784283","full_name":"doitintl/bigquery-optimization-queries","owner":"doitintl","description":"Queries to assist with BigQuery cost and performance.","archived":false,"fork":false,"pushed_at":"2024-10-30T14:53:29.000Z","size":105,"stargazers_count":83,"open_issues_count":8,"forks_count":33,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-02T23:46:23.110Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/doitintl.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":"audit_log/billing_recommendations_per_query.sql","citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-23T15:53:02.000Z","updated_at":"2025-03-26T08:12:24.000Z","dependencies_parsed_at":"2023-12-09T01:35:24.170Z","dependency_job_id":"338f5c9a-0670-4be3-8af4-77d08bc65c00","html_url":"https://github.com/doitintl/bigquery-optimization-queries","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doitintl%2Fbigquery-optimization-queries","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doitintl%2Fbigquery-optimization-queries/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doitintl%2Fbigquery-optimization-queries/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doitintl%2Fbigquery-optimization-queries/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/doitintl","download_url":"https://codeload.github.com/doitintl/bigquery-optimization-queries/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247423512,"owners_count":20936626,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T02:22:52.654Z","updated_at":"2025-04-06T02:07:17.988Z","avatar_url":"https://github.com/doitintl.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Introduction\n\nNote that these queries go along with an [ebook](https://www.doit.com/resources/the-bigquery-optimization-handbook-preparing-to-save/) about optimizing BigQuery\ncosts written by Sayle Matthews of DoiT International. This also goes along with the DoiT Engineering Blog series located [here](https://engineering.doit.com/bigquery-optimizations-part-1-6d188689af1c)\n\nThese queries are to assist in optimizing BigQuery usage in projects and organizations.\nMost focus on costs, but there are a few that focus on concurrency and also some that\nrecommend whether a query will run better under an on-demand or flat-rate pricing scheme.\n\n## Usage\n\nIn each file is a variable called interval_in_days which is the number of days in the past\nthat the query will look at for doing its work. In order to increase or decrease the amount of\ndata processed by the query just change this value and it will be reflected throughout the \nrest of the query.\n\nNote that we have set this value to a default that reflects a good timeframe vs amount of\ndata processed by it.\n\nAdditionally each file should be named on what exactly it does and also has a comment at\nthe top of explaining what the query does.\n\n## Note on Costs\n\nSome of these queries can process through a LOT of data so it's HIGHLY recommended to\nverify the estimated cost of each query before running it. Depending upon how much\nusage your dataset sees over the specified timeframe then this could easily be upwards\nof tens of gigabytes if not more per query.\n\nBlindly run these queries at your own risk! It's very much recommended to reduce the\ninterval_in_days value when the query will be processing a very large amount of data.\n\n## Audit Log vs Information Schema\n\nThe queries are broken up into two different folders: audit_log and information_schema.\nThese correspond to the different schemas that may need to be queried. In general most\npeople will use the information_schema queries because they do not have a BigQuery audit\nlog sink setup.\n\nIf you have an audit log sink setup for BigQuery already (or are a DoiT customer with the\nBQ Lens feature enabled) read the blog entry for a detailed guide of how to discover the\nlocation of your sink.\n\nNote that if you are currently a DoiT International customer and have the BQ Lens feature enabled\nin your Cloud Management Platform (CMP) then you should use the audit log queries as you will\nalready have the tables created\n\n## Generating Queries for Your Project\n\nIf you look at the queries you will see some placeholders for \u003cproject-name\u003e\nand \u003cdataset-region\u003e in the SQL code. These need to be replaced with the\ncorrect values prior to running the queries.\n\nIn order to assist in doing this easier there is a file called generate_sql_files.py\ninside of this repository. This will perform a search and replace operation on all of\nthe .sql files with your specified values.\n\n## generate_sql_files.py Usage\n\n```bash\ngenerate_sql_files.py [--location \u003cdataset-location\u003e] \u003cproject\u003e \u003coutput-directory\u003e\n```\n\nNote that this will create an exact copy of the files in the directory you specify\nas the output-directory.\n\nlocation:  \nThis is the location of your BigQuery dataset. Note the format is the same as BigQuery's such\nas 'region-us' for the US multi-region or 'us-central1' for the US Central Zone 1 region. The\ndefault value is 'region-us' if you do not specify anything.\n\nproject:  \nThis is the name of the GCP project you are going to be querying against.\n\noutput-directory:  \nThe directory where you would like the generated files to be stored. The script will\nattempt to create this directory if it doesn't already exist. Note that this should not\nbe the same directory as the one where the script is located.\n\n## Contributing\nIf you see any bugs please feel free to reach out or perform a pull request on the code.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdoitintl%2Fbigquery-optimization-queries","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdoitintl%2Fbigquery-optimization-queries","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdoitintl%2Fbigquery-optimization-queries/lists"}