{"id":19993219,"url":"https://github.com/xuxinkun/kubesql","last_synced_at":"2025-10-16T12:15:28.370Z","repository":{"id":45528013,"uuid":"174909696","full_name":"xuxinkun/kubesql","owner":"xuxinkun","description":"A tool based on presto using sql to query the resources of kubernetes, such as pods, nodes and so on.","archived":false,"fork":false,"pushed_at":"2022-11-16T12:00:32.000Z","size":51,"stargazers_count":54,"open_issues_count":4,"forks_count":12,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-12T00:02:51.144Z","etag":null,"topics":["kubernetes","presto","presto-connector","presto-plugin","query","sql"],"latest_commit_sha":null,"homepage":"https://xuxinkun.github.io/2020/05/24/kubesql/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xuxinkun.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-03-11T02:15:45.000Z","updated_at":"2023-11-01T09:32:05.000Z","dependencies_parsed_at":"2023-01-23T08:00:58.550Z","dependency_job_id":null,"html_url":"https://github.com/xuxinkun/kubesql","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xuxinkun%2Fkubesql","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xuxinkun%2Fkubesql/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xuxinkun%2Fkubesql/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xuxinkun%2Fkubesql/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xuxinkun","download_url":"https://codeload.github.com/xuxinkun/kubesql/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252334224,"owners_count":21731362,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kubernetes","presto","presto-connector","presto-plugin","query","sql"],"created_at":"2024-11-13T04:52:31.880Z","updated_at":"2025-10-16T12:15:23.338Z","avatar_url":"https://github.com/xuxinkun.png","language":"Java","funding_links":[],"categories":["Java","Databases"],"sub_categories":[],"readme":"# kubesql\n\nkubesql is a tool to use sql to query the resources of kubernetes.\n\nThe resources of kubernetes such as nodes, pods and so on are handled as tables.\n\nFor example, all pods are easily to list from apiserver. But the number of pods on each node is not easy to caculate.\n\n```\npresto:kubesql\u003e select nodename, count(*) as pod_count from pods group by nodename;\n   nodename    | pod_count \n---------------+-----------\n 10.111.11.118 |         8 \n(1 row)\n\nQuery 20200513_094558_00006_7cycm, FINISHED, 1 node\nSplits: 49 total, 49 done (100.00%)\n0:00 [8 rows, 0B] [48 rows/s, 0B/s]\n```\n\n# deploy\n\n## docker\n\nIn case the kubeconfig file is located at `/root/.kube/config`. Just run and enjoy.\n\n```\ndocker run -it -d --name kubesql -v /root/.kube/config:/home/presto/config xuxinkun/kubesql:latest\ndocker exec -it kubesql presto --server localhost:8080 --catalog kubesql --schema kubesql\n```\n\n## kubernetes\n\nTODO\n\n# architecture\n\nkubesql makes use of **presto** to execute the sql. \n\nThere are three main modules in kubesql:\n\n![kubesql-arc](https://xuxinkun.github.io/img/kubesql/kubesql.png)\n\n\n- kubesql-watcher: monitor k8s api pod and node changes. And convert the structured data of pod and node into relational data.\n- kubecache: used to cache pod and node data.\n- kubesql-connector: as presto connector, accept calls from presto, query column information and corresponding data through kubecache, and return to presto about column and data information.\n\nSince all data is cached in memory, there is almost no disk requirement. But it also needs to provide larger memory according to the size of the cluster.\n\n## tables and columns\n\nTaking `pod` data as an example, the main data in the pod is divided into three parts, `metadata`, `spec`, and `status`.\n\nThe more difficult parts of metadata are labels and annotations. I flatten the label map, each key is used as a column.\n\n```\nlabels:\n    app: mysql\n    owner: xxx\n```\n\n`labels` word is used as the prefix, and put the keys of the labels as the column names. Thus two pieces of data are obtained:\n\n\n``` \nlabels.app: mysql\nlabels.owner: xxx\n```\n\nFor `pod A` there is an app label but `pod B` does not have the label, then for `pod B`, the value of this column `labels.app` is `null`.\n\n\nAnnotations are handled similarly. Thus, annotations can be used to filter pods.\n\nFor `spec`, the biggest difficulty lies in the handling of containers. Because there may be several containers in a pod, I will directly use containers as a new table. At the same time, a `uid` column is added to the containers table to indicate which pod this row of data comes from.\nThe fields in the containers are also added to the containers table.\nThe more important information in containers is request and limit.\n`requests.` is used as the prefix, and joined the resource name as a column name.\nFor example, `requests.cpu`,` requests.memory`, etc. Here, the CPU is processed as a double type and the unit is a core.\nFor example, 100m will be converted to 0.1. \nThe memory is bigint and the unit is B.\n\nFor status, conditions and containerStatus are more difficult to deal with. The conditions are a list, but the type of each condition is different.\nSo `type` is used as the prefix to generate column names for conditon. such as:\n\n``` \n  conditions:\n  - lastProbeTime: null\n    lastTransitionTime: 2020-04-22T09:03:10Z\n    status: \"True\"\n    type: Ready\n  - lastProbeTime: null\n    lastTransitionTime: 2020-04-22T09:03:10Z\n    status: \"True\"\n    type: ContainersReady\n```\n\nThen in the pod table, these columns can be found:\n\n|                Column                 |   Type    | Extra | Comment |\n|---------------------------------------|-----------|-------|---------|\n| containersready.lastprobetime         | timestamp |       |         |\n| containersready.lasttransitiontime    | timestamp |       |         |\n| containersready.message               | varchar   |       |         |\n| containersready.reason                | varchar   |       |         |\n| containersready.status                | varchar   |       |         |\n| ready.lastprobetime                   | timestamp |       |         |\n| ready.lasttransitiontime              | timestamp |       |         |\n| ready.message                         | varchar   |       |         |\n| ready.reason                          | varchar   |       |         |\n| ready.status                          | varchar   |       |         |\n\nIn this way, pods can be filtered for type ready and status True with condition \"ready.status\" = \"True\".\n\nSince containerStatus corresponds one-to-one with containers, ContainerStatus are merged with the containers table, and correspond one-to-one with the container name.\n\n## example\n\nShow tables.\n\n\u003e descibe {table name} is also supported.\n\n```\n[root@localhost kubesql]# docker exec -it kubesql presto --server localhost:8080 --catalog kubesql --schema kubesql\npresto:kubesql\u003e show tables;\n   Table    \n------------\n containers \n nodes      \n pods       \n(3 rows)\n```\n\n\nQuery the CPU resources of each pod (requests and limits).\n\n\n```\npresto:kubesql\u003e select pods.namespace,pods.name,sum(\"requests.cpu\") as \"requests.cpu\" ,sum(\"limits.cpu\") as \"limits.cpu\" from pods,containers where pods.uid = containers.uid group by pods.namespace,pods.name\n     namespace     |                 name                 | requests.cpu | limits.cpu \n-------------------+--------------------------------------+--------------+------------\n rrrrqq-test-01    | rrrrqq-test-01-202005151652391759    |          0.8 |        8.0 \n lll-nopassword-18 | lll-nopassword-18-202005211645264618 |          0.1 |        1.0 \n```\n\nQuery the remaining CPUs on each node.\n\n```\npresto:kubesql\u003e select nodes.name, nodes.\"allocatable.cpu\" - podnodecpu.\"requests.cpu\" from nodes, (select pods.nodename,sum(\"requests.cpu\") as \"requests.cpu\" from pods,containers where pods.uid = containers.uid group by pods.nodename) as podnodecpu where nodes.name = podnodecpu.nodename;\n    name     |       _col1        \n-------------+--------------------\n 10.11.12.29 | 50.918000000000006 \n 10.11.12.30 |             58.788 \n 10.11.12.32 | 57.303000000000004 \n 10.11.12.34 |  33.33799999999999 \n 10.11.12.33 | 43.022999999999996 \n```\n\nQuery all pods created after 2020-05-12.\n\n```\npresto:kube\u003e select name, namespace,creationTimestamp from pods where creationTimestamp \u003e date('2020-05-12') order by creationTimestamp desc;\n                         name                         |        namespace        |    creationTimestamp    \n------------------------------------------------------+-------------------------+-------------------------\n kube-api-webhook-controller-manager-7fd78ddd75-sf5j6 | kube-api-webhook-system | 2020-05-13 07:56:27.000 \n```\n\nQuery based on labels are also supported. Pods with the appid label of `springboot`, and have not been scheduled successfully.\n\n\nThe label appid exists in the `pods table` with the column name `labels.appid`. Use this column as a where condition to query pods.\n\n```\npresto:kubesql\u003e select namespace,name,phase from pods where phase = 'Pending' and \"labels.appid\" = 'springboot';\n     namespace      |     name     |  phase  \n--------------------+--------------+---------\n springboot-test-xx | v6ynsy3f73jn | Pending \n springboot-test-xx | mu4zktenmttp | Pending \n springboot-test-xx | n0yvpxxyvk4u | Pending \n springboot-test-xx | dd2mh6ovkjll | Pending \n springboot-test-xx | hd7b0ffuqrjo | Pending\n \n presto:kubesql\u003e select count(*) from pods where phase = 'Pending' and \"labels.appid\" = 'springboot';\n  _col0 \n -------\n      5 \n```\n\n# plan\n\nAt present, there are only pods and nodes resources. Compared with the various resources of k8s, it is only a small part. \nBut adding table for each resource requires adding a considerable amount of code. \nI'm also thinking about how to use openapi's swagger description to automatically generate code.\n\n\nThe deployment is now using docker to deploy, and the deployment method of kubernetes will be added soon, which will be more convenient.\n\n\nAt the same time, I am thinking that in the future, each worker in Presto will be responsible for a cluster cache. \nSuch a presto cluster can query all k8s cluster information. This feature also needs to be redesigned and considered.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxuxinkun%2Fkubesql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxuxinkun%2Fkubesql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxuxinkun%2Fkubesql/lists"}