https://github.com/HariSekhon/SQL-scripts
100+ SQL Scripts - PostgreSQL, MySQL, Oracle, Google BigQuery, MariaDB, AWS Athena. DBA, Analytics, DevOps, performance engineering. Google BigQuery ML machine learning classification.
https://github.com/HariSekhon/SQL-scripts
athena aws aws-athena bigquery bigquery-ml dba devops gcp google-bigquery google-cloud-sql google-cloudsql-mysql machine-learning mariadb mysql oracle performance postgres postgresql rds sql
Last synced: 1 day ago
JSON representation
100+ SQL Scripts - PostgreSQL, MySQL, Oracle, Google BigQuery, MariaDB, AWS Athena. DBA, Analytics, DevOps, performance engineering. Google BigQuery ML machine learning classification.
- Host: GitHub
- URL: https://github.com/HariSekhon/SQL-scripts
- Owner: HariSekhon
- License: mit
- Created: 2020-08-05T20:14:49.000Z (over 4 years ago)
- Default Branch: master
- Last Pushed: 2025-02-28T17:06:11.000Z (about 2 months ago)
- Last Synced: 2025-04-08T08:12:25.538Z (6 days ago)
- Topics: athena, aws, aws-athena, bigquery, bigquery-ml, dba, devops, gcp, google-bigquery, google-cloud-sql, google-cloudsql-mysql, machine-learning, mariadb, mysql, oracle, performance, postgres, postgresql, rds, sql
- Language: Shell
- Homepage:
- Size: 620 KB
- Stars: 437
- Watchers: 19
- Forks: 120
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
- jimsghstars - HariSekhon/SQL-scripts - 100+ SQL Scripts - PostgreSQL, MySQL, Oracle, Google BigQuery, MariaDB, AWS Athena. DBA, Analytics, DevOps, performance engineering. Google BigQuery ML machine learning classification. (Shell)
README
# SQL Scripts
[](https://github.com/HariSekhon/SQL-scripts/stargazers)
[](https://github.com/HariSekhon/SQL-scripts/network)
[](https://github.com/HariSekhon/SQL-scripts#SQL-Scripts)
[](https://github.com/HariSekhon/SQL-scripts/blob/master/LICENSE)
[](https://www.linkedin.com/in/HariSekhon/)
[](https://github.com/HariSekhon/SQL-scripts/commits/master)
[](https://www.postgresql.org/)
[](https://www.mysql.com/)
[](https://mariadb.org/)
[](https://oracle.com/)
[](https://aws.amazon.com/athena/)
[](https://aws.amazon.com/aurora/)
[](https://cloud.google.com/bigquery)
[](https://bitbucket.org/HariSekhon/devops-bash-tools/src/master/STATUS.md)
[](https://github.com/HariSekhon/SQL-scripts/actions/workflows/shellcheck.yaml)
[](https://github.com/HariSekhon/SQL-scripts/actions/workflows/yaml.yaml)
[](https://github.com/HariSekhon/SQL-scripts/actions/workflows/markdown.yaml)
[](https://github.com/HariSekhon/SQL-scripts/actions/workflows/validate.yaml)
[](https://github.com/HariSekhon/SQL-scripts/actions/workflows/grype.yaml)
[](https://github.com/HariSekhon/SQL-scripts/actions/workflows/kics.yaml)
[](https://github.com/HariSekhon/SQL-scripts/actions/workflows/semgrep.yaml)
[](https://github.com/HariSekhon/SQL-scripts/actions/workflows/semgrep-cloud.yaml)
[](https://github.com/HariSekhon/SQL-scripts/actions/workflows/trivy.yaml)
[](https://github.com/HariSekhon/SQL-scripts#SQL-Scripts)
[](https://github.com/HariSekhon/SQL-scripts#SQL-Scripts)
[](https://github.com/HariSekhon/SQL-scripts)
[](https://gitlab.com/HariSekhon/SQL-scripts)
[](https://dev.azure.com/harisekhon/GitHub/_git/SQL-scripts)
[](https://bitbucket.org/HariSekhon/SQL-scripts)
[git.io/SQL](https://git.io/SQL)
Useful SQL scripts, split from [DevOps Bash tools](https://github.com/HariSekhon/DevOps-Bash-tools), for which this is now a submodule.
Hari Sekhon
Cloud & Big Data Contractor, United Kingdom
[](https://www.linkedin.com/in/HariSekhon/)
*(you're welcome to connect with me on LinkedIn)*
## Inventory
### DevOps / DBA
- `aws_athena_cloudtrail_ddl.sql` - [AWS Athena](https://aws.amazon.com/athena/) DDL to setup up integration to query [CloudTrail](https://aws.amazon.com/cloudtrail/) logs from Athena
- `bigquery_*.sql` - [Google BigQuery](https://cloud.google.com/bigquery) scripts:
- `bigquery_billing_*.sql` - billing queries for [GCP](https://cloud.google.com/) usage eg. highest cost services, most used GCP products, recent charges etc.
- `bigquery_info_*.sql` - information schema queries for datasets, tables, columns, partitioning, clustering etc.
- `mysql_*.sql`:
- [MySQL](https://www.mysql.com/) / [MariaDB](https://mariadb.org/) queries for DBA investigating + performance tuning
- `mysql_info.sql` - summary overview, useful to debug your `mysql.user` table auth effects
- (shows intended `USER()` vs actual `CURRENT_USER()`)
- tested on MySQL 5.5, 5.6, 5.7, 8.0 and MariaDB 5.5, 10.x
- `postgres_*.sql`:
- [PostgreSQL](https://www.postgresql.org/) queries for DBA investigating + performance tuning
- [postgres_info.sql](https://github.com/HariSekhon/SQL-scripts/blob/master/postgres_info.sql) - big summary overview, recommend you start here
- tested on PostgreSQL 8.4, 9.x, 10.x, 11.x, 12.x, 13.x
- `oracle_*.sql`:
- [Oracle](https://www.oracle.com) queries for DBA investigating
- tested on Oracle 9i, 10g, 11g, 19c
### Analytics
- `bigquery_*.sql` - [Google BigQuery](https://cloud.google.com/bigquery) scripts:
- `bigquery_billing_*.sql` - billing queries for [GCP](https://cloud.google.com/) usage eg. highest cost services, most used GCP products, recent charges etc.
- `bigquery_info_*.sql` - information schema queries for datasets, tables, columns, partitioning, clustering etc.
- [analytics/](https://github.com/HariSekhon/SQL-scripts/tree/master/analytics)`bigquery_*.sql` - ecommerce queries and [BigQuery ML](https://cloud.google.com/bigquery-ml/docs/bigqueryml-intro) machine learning classification logistic regression models and purchasing predictions
- for more [BigQuery](https://cloud.google.com/bigquery) examples, see [Data Engineering demos](https://github.com/GoogleCloudPlatform/training-data-analyst/tree/master/courses/data-engineering/demos)
### Database Knowledge Base
See the pages for:
- [SQL](https://github.com/HariSekhon/Knowledge-Base/blob/main/sql.md)
- [SQL Databases](https://github.com/HariSekhon/Knowledge-Base/blob/main/databases.md)
- [MySQL](https://github.com/HariSekhon/Knowledge-Base/blob/main/mysql.md)
- [PostgreSQL](https://github.com/HariSekhon/Knowledge-Base/blob/main/postgres.md)
- [Oracle](https://github.com/HariSekhon/Knowledge-Base/blob/main/oracle.md)
in the [HariSekhon/Knowledge-Base](https://github.com/HariSekhon/Knowledge-Base) repo:
[](https://github.com/HariSekhon/Knowledge-Base)
### DevOps SQL tooling
[](https://github.com/HariSekhon/DevOps-Bash-tools)
You can quickly test the PostgreSQL / MySQL scripts using `postgres.sh` / `mysqld.sh` / `mariadb.sh` in the [DevOps Bash tools](https://github.com/HariSekhon/DevOps-Bash-tools) repo, which boots a docker container and drops straight in to a `mysql` / `psql` shell with this directory mounted at `/sql` and used as `$PWD` for fast easy sourcing eg.
postgres:
```postgres-sql
\i /sql/postgres_query_times.sql
```
```postgres-sql
\i postgres_query_times.sql
```
mysql:
```mysql-sql
source /sql/mysql_sessions.sql
```
```mysql-sql
\. mysql_sessions.sql
```
### Related scripts
- [.psqlrc](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/.psqlrc) - advanced PostgreSQL psql client config
- [psql.sh](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/psql.sh) - quickly connect to PostgreSQL with command line switches inferred from environment variables
- [mysql.sh](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/mysql.sh) - quickly connect to MySQL / MariaDB with command line switches inferred from environment variables
- [postgres.sh](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/postgres.sh) - one-touch PostgreSQL, boots docker container and drops you in to `psql` shell. Version can be given as an argument
- [mysqld.sh](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/mysqld.sh) / [mariadb.sh](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/mariadb.sh) - one-touch MySQL / MariaDB, boots docker container and drops you in to `mysql` shell. Version can be given as an argument
- [postgres_foreach_table.sh](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/postgres/postgres_foreach_table.sh) / [mysql_foreach_table.sh](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/mysql/mysql_foreach_table.sh) - execute templated SQL queries/statements against all or a subset of tables
- [postgres_tables_row_counts.sh](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/postgres/postgres_tables_row_counts.sh) / [mysql_tables_row_counts.sh](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/mysql/mysql_tables_row_counts.sh) - get row counts for all or a subset of tables
- [sqlcase.pl](https://github.com/HariSekhon/DevOps-Perl-tools/blob/master/sqlcase.pl) - autocases your SQL code
- I use this a lot and call it via hotkey configured in my [.vimrc](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/configs/.vimrc)
- there are `*case.pl` specializations for most of the major RDBMS and distributed SQL systems, even several NoSQL systems, using each one's language specific keywords
- Hive & Impala SQL:
- [beeline.sh](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/bigdata/beeline.sh) - quickly connect to Hive, auto-determines HiveServer2 address, Kerberos & SSL options, ZooKeeper quorum
- [impala_shell.sh](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/bigdata/impala_shell.sh) - quickly connect to Impala, auto-determines a Hadoop worker node address and Kerberos options (can use an environment variable for a [Load Balancer](https://github.com/HariSekhon/HAProxy-configs/blob/master/bigdata/impala-jdbc.cfg) setup)
- [hive_foreach_table.sh](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/bigdata/hive_foreach_table.sh) / [impala_foreach_table.sh](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/bigdata/impala_foreach_table.sh) - execute templated SQL queries/statements against all or a subset of tables
- [hive_tables_row_counts.sh](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/bigdata/hive_tables_row_counts.sh) / [impala_tables_row_counts.sh](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/bigdata/impala_tables_row_counts.sh) - get row counts for all or a subset of tables
- [hive_tables_column_counts.sh](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/bigdata/hive_tables_column_counts.sh) / [impala_tables_column_counts.sh](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/bigdata/impala_tables_column_counts.sh) - get the column counts for big tables in Hive / Impala
- [hive_tables_metadata.sh](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/bigdata/hive_tables_metadata.sh) / [impala_tables_metadata.sh](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/bigdata/impala_tables_metadata.sh) / [hive_tables_locations.sh](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/bigdata/hive_tables_locations.sh) / [impala_tables_locations.sh](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/bigdata/impala_tables_locations.sh) - get Hive / Impala metadata for all or a subset of tables, eg. Location to determine where the external tables data is being stored (HDFS / S3 paths)
## Star History
[](https://star-history.com/#HariSekhon/SQL-scripts&Date)
[git.io/SQL](https://git.io/SQL)
## More Core Repos
### Knowledge
[](https://github.com/HariSekhon/Knowledge-Base)
[](https://github.com/HariSekhon/Diagrams-as-Code)
### DevOps Code
[](https://github.com/HariSekhon/DevOps-Bash-tools)
[](https://github.com/HariSekhon/DevOps-Python-tools)
[](https://github.com/HariSekhon/DevOps-Perl-tools)
[](https://github.com/HariSekhon/DevOps-Golang-tools)
### Containerization
[](https://github.com/HariSekhon/Kubernetes-configs)
[](https://github.com/HariSekhon/Dockerfiles)
### CI/CD
[](https://github.com/HariSekhon/GitHub-Actions)
[](https://github.com/HariSekhon/Jenkins)
### DBA - SQL
[](https://github.com/HariSekhon/SQL-scripts)
### DevOps Reloaded
[](https://github.com/HariSekhon/Nagios-Plugins)
[](https://github.com/HariSekhon/HAProxy-configs)
[](https://github.com/HariSekhon/Terraform)
[](https://github.com/HariSekhon/Packer-templates)
[](https://github.com/HariSekhon/Nagios-Plugin-Kafka)
### Templates
[](https://github.com/HariSekhon/Templates)
[](https://github.com/HariSekhon/Template-repo)
### Misc
[](https://github.com/HariSekhon/Spotify-tools)
[](https://github.com/HariSekhon/Spotify-playlists)
The rest of my original source repos are
[here](https://github.com/HariSekhon?tab=repositories&q=&type=source&language=&sort=stargazers).
Pre-built Docker images are available on my [DockerHub](https://hub.docker.com/u/harisekhon/).
