{"id":13775153,"url":"https://github.com/denzilribeiro/sqldbmonitoring","last_synced_at":"2025-05-11T07:32:01.810Z","repository":{"id":43861571,"uuid":"198336465","full_name":"denzilribeiro/sqldbmonitoring","owner":"denzilribeiro","description":"Azure SQL DB monitoring","archived":false,"fork":false,"pushed_at":"2022-08-01T16:23:38.000Z","size":59,"stargazers_count":28,"open_issues_count":5,"forks_count":35,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-02-26T22:23:58.224Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/denzilribeiro.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-07-23T02:29:20.000Z","updated_at":"2023-02-02T08:26:08.000Z","dependencies_parsed_at":"2022-08-25T22:51:33.087Z","dependency_job_id":null,"html_url":"https://github.com/denzilribeiro/sqldbmonitoring","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denzilribeiro%2Fsqldbmonitoring","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denzilribeiro%2Fsqldbmonitoring/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denzilribeiro%2Fsqldbmonitoring/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denzilribeiro%2Fsqldbmonitoring/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/denzilribeiro","download_url":"https://codeload.github.com/denzilribeiro/sqldbmonitoring/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253533773,"owners_count":21923514,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T17:01:34.585Z","updated_at":"2025-05-11T07:32:01.503Z","avatar_url":"https://github.com/denzilribeiro.png","language":"Shell","funding_links":[],"categories":["Tools"],"sub_categories":["Vectors"],"readme":"# Azure SQL Database / Managed Instance monitoring\nSolution for near-realtime monitoring on Azure SQL database and Azure SQL Managed instance using the [telegraf SQL plugin](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/sqlserver) . A previous version was covered under the blog [Near-realtime monitoring for managed instances](https://techcommunity.microsoft.com/t5/DataCAT/Real-time-performance-monitoring-for-Azure-SQL-Database-Managed/ba-p/305537)\n\n## VM Setup\nCreate an Ubuntu VM in a new or an existing resource group with a data disk and mount the attached data disk in the Linux OS. Other Linux distros will work as well, but commands may need to be adjusted accordingly.\n\nThe easiest way it to create an Ubuntu VM using the portal, as it will setup everything automatically for you. Just make sure you configure the NSG group to allow access inbound from specific client IPs to destination port 3000 (which is the port used by  Grafana UI) on this VM.\n\n**Example with Az CLI**\nCreate a resource group with the [az group create](https://docs.microsoft.com/en-us/cli/azure/group) command. Preferably, use the region where the monitored resources are located.\n\nCreate the Ubuntu VM with [az vm create](https://docs.microsoft.com/en-us/cli/azure/vm?view=azure-cli-latest#az-vm-create) : Sample CLI command here creates a VM with SSH authentication, as well as with an additional 1TB data disk which will be used to store the monitoring data. Typical customizations to the below command include:\n- Specify an existing public key using the `--ssh-key-values` argument\n- Specify an existing VNET and subnet using `--vnet-name` and `--subnet` arguments\n- Customize the admin username for the VM using `--admin-username`\n\n```\naz vm create --resource-group \u003cResourceGroupName\u003e --name \u003cVmName\u003e --image UbuntuLTS --size \u003cStandard_DS2_v2\u003e --generate-ssh-keys --nsg-rule SSH --data-disk-sizes-gb 1024\n```\n\n[Connect](https://docs.microsoft.com/en-us/azure/virtual-machines/linux/ssh-from-windows#connect-to-your-vm) to the VM and [mount the new disk to the VM.](https://docs.microsoft.com/en-us/azure/virtual-machines/linux/attach-disk-portal#connect-to-the-linux-vm-to-mount-the-new-disk). Please keep in mind\nthat the mount point for the data disk by default is `/datadrive` (as per the above doc).\nThe subsequent scripts in this repo assume a default of `/data/influxdb`. So either:\n- Use `/data/influxdb` as the mount point for the data disk, OR\n- Adjust the `INFLUXDB_HOST_DIRECTORY` setting mentioned later, to match the mount point used above.\n\n## Clone repo\nThe repo includes grafana dashboards for Azure SQL Database and Managed instances, and setup scripts for influxdb, telegraf and grafana.\n\n```\ncd $HOME\nsudo apt-get install git\ngit clone https://github.com/denzilribeiro/sqldbmonitoring.git\n```\n\n## Setup\nSetup will install docker.io, will install the telegraf latest release build and configure the firewall to open up port 3000 required for Grafana and pull and start the Grafana container.\n\n```\ncd $HOME/sqldbmonitoring\nsudo ./monitoringsetup.sh\n```\n\nIf you want to install the latest nightly build with most recent telegraf changes\n\n```\ncd $HOME/sqldbmonitoring \nsudo ./monitoringsetup.sh nightly\n```\n\n## Install, Configure and start InfluxDB\nAssuming you have mounted a separate data drive as specified in the Setup VM portion , edit the ./runinfluxdb.sh file and modify the INFLUXDB_HOST_DIRECTORY variable to point to the directory where you want the InfluxDB volume to be mounted, if it is something other than the default of /data/influxdb.\n```\ncd $HOME/sqldbmonitoring/influxdb\nsudo nano runinfluxdb.sh\n```\nPull the docker image and start InfluxDB:\n```\ncd $HOME/sqldbmonitoring/influxdb\nsudo ./runinfluxdb.sh\n```\n\n## Create Logins for each Managed Instance or SQL DB being monitored\n**Note:** Please ENSURE that you replace the sample passwords given below with\na truly strong password, before executing the same.\n\nFor Managed Instance:\n```\nUSE master;\nCREATE LOGIN telegraf WITH PASSWORD = N'StrongPassword1!', CHECK_POLICY = ON;\nGO\nGRANT VIEW SERVER STATE TO telegraf;\nGO\nGRANT VIEW ANY DEFINITION TO telegraf;\nGO\n```\n\nFor SQL DB create a database scoped user for each database being monitored:\n```\nCREATE USER [telegraf] WITH PASSWORD = N'Mystrongpassword1!';\nGO\nGRANT VIEW DATABASE STATE TO [telegraf];\nGO\n```\n\n## Edit the telegraf configuration file\nEdit the `/etc/telegraf/telegraf.conf` to add one connection string per each database you want to monitor. Optionally, modify the `/etc/telegraf/telegraf.conf` with the options specified [here](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/sqlserver)\n\n**Note:** For **Azure SQL Database**, you would need one connection string per database you are monitoring specifying the right database name and *not* `master`. An example snippet for monitoring two databases, each on a different logical SQL server, is shown below:\n```\n[[inputs.sqlserver]]\nservers = [\"Server=server1.database.windows.net;Port=1433;User Id=telegraf;Password=Mystrongpassword1!;database=myazuredb1;app name=telegraf;\"\n,\"Server=server2.database.windows.net;Port=1433;User Id=telegraf;Password=Mystrongpassword1!;database=myazuredb2;app name=telegraf;\"]\n\nquery_version = 2\nazuredb=true\n```\n\nBy default all the collectors are enabled. If you copy the sample telegraf.conf the Schedulers and SqlRequests collector are excluded as they are chattier than other collectors. You can control what collectors to exclude by including them in the excluded array in the conf file\n\n```\nexclude_query = [ 'Schedulers' , 'SqlRequests']\n```\n\nIf the influx DB docker container is on the same VM then the section below doesn’t have to change, if it is on another machine or non-default port, the url would have to be updated accordingly.\n\n```\nurls = [\"[http://127.0.0.1:8086](http://127.0.0.1:8086/)\"]\ndatabase = \"telegraf\"\n```\n\n**Note:**\n - Port for Azure SQL Managed Instance with public endpoint is 3342\n - If influxDB is hosted on a different VM have to modify the url for `outputs.influxdb`\n - Default polling interval is 10 seconds, if you want to change that have to add /change the Agent Interval value.\n\n## Start the telegraf service\nUse the commands below to start telegraf and check its status:\n```\nsudo systemctl start telegraf\nsudo systemctl status telegraf\n```\nTo Troubleshoot any failures you can look at the log entries using the command:\n```\nsudo journalctl -u telegraf --no-pager\n```\n\n## Create Telegraf database and retention policy\nWe want to set a retention policy in influxdb based on how long data needs to be kept.\nTo do so, you can use the ```create retention policy``` command within Influx (example below):\n```\nsudo docker exec -it influxdb influx\nuse telegraf;\nshow retention policies;\ncreate retention policy retain30days on telegraf duration 30d replication 1 default;\nquit\n```\n\n## Configure Grafana data source and dashboards\nThe Dashboards are located here:\n- [Azure SQL Database Dashboards](/dashboards/azuresqldb)\n- [Azure SQL Managed Instance Dashboards](/dashboards/azuresqlmi)\n\nBut first, we need to create the data source for InfluxDB in Grafana:\n- Browse to your Grafana instance - http://[GRAFANA_IP_ADDRESS_OR_SERVERNAME]:3000\n- First time you login into Grafana, login and password are set to: `admin`. Also take a look at the [Getting Started](https://grafana.com/docs/grafana/latest/guides/getting_started/) Grafana documentation.\n- Add a data source for InfluxDB. Detailed instructions are at in the [grafana data source docs](http://docs.grafana.org/features/datasources/influxdb/)\n\n    - Type: `InfluxDB`\n    - Name: `InfluxDB` (this is also the default)\n    - URL: http://[INFLUXDB_HOSTNAME_OR_IP_ADDRESS]:8086. (The default of http://localhost:8086 works if Grafana and InfluxDB are on the same machine; make sure to explicitly enter this URL in the field. )\n    - Database: `telegraf`\n    - Click \"Save \u0026 Test\". You should see the message \"Data source is working\".\n\n- Then, we can download Grafana dashboard JSON definitions from the repo [Azure SQL DB dashboards](/dashboards/azuresqldb) folder or [Azure SQL Managed Instance Dashboards](/dashboards/azuresqlmi) for all the dashboards, and then [import](http://docs.grafana.org/reference/export_import/#importing-a-dashboard) them into Grafana.\n\n** Note:** If you changed the name of the datasource above from the default value of `InfluxDB`, then you will need to edit the JSON for the dashboards and change `\"datasource\": \"InfluxDB\"` to `\"datasource\": \"YOURDATASOURCENAME\"`, where `YOURDATASOURCENAME` is a placeholder for the actual datasource name.\n\n## Dashboard samples\nAs mentioned above, we provide some sample dashboards, which can be customized by you as needed:\n- `AzureSQLDBEstate.json`:  This is for Azure SQL Databases and not for Managed Instances. This gives you a row per logical SQL Server, and a top level view of all the servers and databases, from which you can then drill-through to an individual database.\n- `AzureSQLDBPerformance.json`: This is a database level view of performance, going from high level metrics, to wait stats, to performance counters, and typically serves as the primary performance dashboard.\n- `AzureSQLDBStorage.json`:  This dashboard is primarily based on virtual file stats DMV, and gives you a view of storage latency for the respective databases.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdenzilribeiro%2Fsqldbmonitoring","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdenzilribeiro%2Fsqldbmonitoring","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdenzilribeiro%2Fsqldbmonitoring/lists"}