{"id":15010316,"url":"https://github.com/perrygeo/postgres-extras-clj","last_synced_at":"2025-04-09T22:41:13.574Z","repository":{"id":247109631,"uuid":"824742331","full_name":"perrygeo/postgres-extras-clj","owner":"perrygeo","description":"The missing postgresql toolkit for Clojure developers","archived":false,"fork":false,"pushed_at":"2024-08-14T02:07:40.000Z","size":556,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-02T03:38:13.927Z","etag":null,"topics":["clojure","postgresql"],"latest_commit_sha":null,"homepage":"https://cljdoc.org/d/com.github.perrygeo/postgres-extras-clj/","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/perrygeo.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-05T20:43:21.000Z","updated_at":"2025-04-01T21:01:02.000Z","dependencies_parsed_at":"2024-08-14T03:59:25.967Z","dependency_job_id":null,"html_url":"https://github.com/perrygeo/postgres-extras-clj","commit_stats":null,"previous_names":["perrygeo/postgres-extras-clj"],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perrygeo%2Fpostgres-extras-clj","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perrygeo%2Fpostgres-extras-clj/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perrygeo%2Fpostgres-extras-clj/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perrygeo%2Fpostgres-extras-clj/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/perrygeo","download_url":"https://codeload.github.com/perrygeo/postgres-extras-clj/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248124848,"owners_count":21051757,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clojure","postgresql"],"created_at":"2024-09-24T19:33:30.615Z","updated_at":"2025-04-09T22:41:13.551Z","avatar_url":"https://github.com/perrygeo.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# postgres-extras-clj\n\nA Clojure toolbox for inspecting and diagnosing PostgreSQL databases.\n\n```clojure\ncom.github.perrygeo/postgres-extras-clj {:mvn/version \"0.1.23\"}\n```\n\n* [Clojars](https://clojars.org/com.github.perrygeo/postgres-extras-clj)\n* [Github](https://github.com/perrygeo/postgres-extras-clj)\n* [Documentation](https://cljdoc.org/d/com.github.perrygeo/postgres-extras-clj/)\n* [CI tests](https://github.com/perrygeo/postgres-extras-clj/actions/workflows/test.yml)\n* [Example Notebook](https://perrygeo.github.io/postgres-extras-clj/)\n\n\n## 🐘 Motivation\n\nPostgreSQL is a fantastic database but using it in production\nrequires some care, as do all databases.\nThe postgres [system catalogs](https://www.postgresql.org/docs/current/catalogs.html)\nallow us to monitor things like query performance, connection management,\nindex efficiency, disk usage, and MVCC bloat. But accessing that information\nrequires some arcane knowledge and reasonably hardcore SQL skills.\n\nThis project was inspired by the rustprooflabs [pgdd](https://github.com/rustprooflabs/pgdd)\nextension and by the phoenix web framework which ships with a developer-centric,\npostgres-specific dashboard based on [ecto_psql_extras](https://github.com/pawurb/ecto_psql_extras/tree/main).\nBoth projects demonstrate that high-level database tooling\ncan, and probably should, be built on top of the system catalogs.\nUnderstanding your database isn't optional.\n\n`postgres-extras-clj` provides this missing toolkit for Clojure developers.\n\nThe SQL logic lives in [`resources/sql/*.sql`](resources/sql/) and is\nformatted with pgFormat for consistency. SQL is \nannotated with [HugSQL](https://www.hugsql.org) comments to turn them\ninto clojure fns via macro magic. Instead of a web interface or postgres extension,\n`postgres-extras-clj.core` provides a clojure namespace with a few dozen useful \nfunctions that query system tables and **return diagnostics as plain data structures**. \nThe goal is to run with limited, SELECT-only privileges\nof system schemas and tables (with a few noted exceptions).\n\nAll you need is a JDBC connection and a REPL.\n\n\n### 📚 Data Dictionary\n \nThe data dictionary functionality is based on [pgdd](https://github.com/rustprooflabs/pgdd).\nThese include COMMENTS and are helpful for understanding the structure\nof your database, from a data modeling lens.\n\n| \u003cspan style=\"width:320px\"\u003eFunction\u003c/span\u003e  | Scenario |\n|---------|---------|\n| `columns` | List all database column objects |\n| `databases` | List all databases |\n| `functions` | List all function objects in current database |\n| `indexes` | List all index objects in current database |\n| `schemas` | List all shemas in current database |\n| `partition-children` | List all child partitions in current database |\n| `partition-parents` | List all parent partition tables in current database |\n| `tables` | List all table objects in current database |\n| `views` | List all view objects in current database |\n\nTo get a full map of data objects, use `(read-data-dictionary db)` which\nreturns a map, with keywords mirroring the above functions.\n\n### 🛠️ Operational Diagnostics\n\nDiagnostic stats based on [ecto_psql_extras](https://github.com/pawurb/ecto_psql_extras/tree/main).\nThese are valuable for looking at your database through an operations or DBA lens.\n\n| \u003cspan style=\"width:320px\"\u003eFunction\u003c/span\u003e | Scenario |\n|---------|---------|\n| `all-locks` | Queries with active locks |\n| `bloat` | Table and index \"bloat\" in your database ordered by most wasteful |\n| `blocking` | Queries holding locks other queries are waiting to be released | \n| `cache-hit` | Index and table hit rate |\n| `calls` | Queries that have the highest frequency of execution |\n| `connections` | Returns the list of all active database connections |\n| `db-settings` | Values of selected PostgreSQL settings |\n| `duplicate-indexes` | Multiple indexes that have the same set of columns, same opclass, expression and predicate |\n| `extensions` | Available and installed extensions |\n| `health-check` | Checks the db for liveliness |\n| `index-cache-hit` | Calculates your cache hit rate for reading indexes |\n| `index-size` | The size of indexes, descending by size |\n| `index-usage` | Index hit rate (effective databases are at 99% and up) |\n| `kill-all!` | Kill all the active database connections |\n| `locks` | Queries with active exclusive locks |\n| `long-running-queries` | All queries longer than the threshold by descending duration |\n| `null-indexes` | Find indexes with a high ratio of NULL values |\n| `outliers` | Queries that have longest execution time in aggregate. |\n| `records-rank` | All tables and the number of rows in each ordered by number of rows descending |\n| `seq-scans` | Count of sequential scans by table descending by order |\n| `table-cache-hit` | Calculates your cache hit rate for reading tables |\n| `table-indexes-size` | Total size of all the indexes on each table, descending by size |\n| `table-size` | Size of the tables (excluding indexes), descending by size |\n| `total-index-size` | Total size of all indexes in MB |\n| `total-table-size` | Size of the tables (including indexes), descending by size |\n| `unused-indexes` | Unused and almost unused indexes |\n| `vacuum-stats` | Dead rows and whether an automatic vacuum is expected to be triggered |\n\nTo get a full map of diagnostic stats, use `(read-stats db)` which\nreturns a map, with keywords mirroring the above functions.\n\nUse the `(diagnose (read-stats db))` and `(diagnose-warnings (read-stats db))` functions\nto evaluate the stats according to a set of heuristics. \n\n\n## Usage\n\nCheck out the [examples](./examples/pgbench_tutorial.clj) if you're looking to create a fresh namespace. \n\nThe following is a REPL demonstration of `postgres-extras-clj` with the `next.jdbc` adapter. \nRun `clj -M:dev` then evaluate the following forms\n\n\n```clojure\n(require '[postgres-extras-clj.core :as pgex] :reload-all)\n(require '[hugsql.core :as hugsql])\n(require '[hugsql.adapter.next-jdbc :as next-adapter])\n(require '[next.jdbc :as jdbc])\n\n(def db\n  (jdbc/get-datasource\n   \"jdbc:postgresql://localhost:5432/main?user=postgres\u0026password=password\"))\n\n(hugsql/set-adapter! (next-adapter/hugsql-adapter-next-jdbc))\n```\n\nFor this example, wer'e using a `next.jdbc` datasource but there are other options,\nsee the [HugSQL Adapters](https://www.hugsql.org/hugsql-adapters/) documentation.\n\nDo a quick health check\n\n```clojure\n(pgex/health-check db)\n; {:now #inst \"2024-07-05T18:04:57.506678000-00:00\",\n;  :version \"PostgreSQL 16.1 (Debian 16.1-1.pgdg110+1) on x86_64-pc-linux-gnu...\"}\n```\n\nGenerate a data dictionary summarizing all major objects in your database.\n\n```clojure\n(def dd (pgex/read-data-dictionary db))\n(keys dd)\n; (:databases\n;  :columns\n;  :functions\n;  :indexes\n;  :schemas\n;  :tables\n;  :views \n;  :partition-children\n;  :partition-parents)\n\n(rand-nth (:tables dd))\n; {:size_pretty \"16 kB\",\n;  :description nil,\n;  :owned_by \"postgres\",\n;  :size_plus_indexes \"48 kB\",\n;  :rows 1,\n;  :oid 19789,\n;  :data_type \"table\",\n;  :size_plus_indexes_bytes 49152,\n;  :s_name \"public\",\n;  :system_object false,\n;  :t_name \"users\",\n;  :size_bytes 16384,\n;  :bytes_per_row 16384}\n```\n\nCreate a full map of diagnostic stats. \n\n```clojure\n(def stats (pgex/read-stats db))\n(keys stats)\n; (:duplicate-indexes\n;  :db-settings\n;  :locks\n;  :vacuum-stats\n;  :index-usage\n;  :total-index-size\n;  :cache-hit\n;  :health-check\n;  :records-rank\n;  :null-indexes\n;  :index-cache-hit\n;  :all-locks\n;  :outliers\n;  :long-running-queries\n;  :extensions\n;  :total-table-size\n;  :unused-indexes\n;  :bloat\n;  :calls\n;  :table-size\n;  :connections\n;  :table-cache-hit\n;  :table-indexes-size\n;  :blocking\n;  :seq-scans\n;  :index-size)\n\n(rand-nth (:connections stats))\n; {:username \"postgres\"\n;  :client_address \"172.22.0.1/32\"\n;  :application_name \"psql\"}\n```\n\nAll of the stats and data dictionary keywords mirror the name of a public function in the\n`postgres-extras-clj.core` namespace so you can invoke them selectively,\ninstead of getting them from the full map.\n\n\n```clojure\n(rand-nth (pgex/tables db))\n; {:size_pretty \"16 kB\",\n;  :description nil,\n;  :owned_by \"postgres\",\n;  :size_plus_indexes \"48 kB\",\n;  :rows 1,\n;  :oid 19789,\n;  :data_type \"table\",\n;  :size_plus_indexes_bytes 49152,\n;  :s_name \"public\",\n;  :system_object false,\n;  :t_name \"users\",\n;  :size_bytes 16384,\n;  :bytes_per_row 16384}\n\n(rand-nth (pgex/connections db))\n; {:username \"postgres\"\n;  :client_address \"172.22.0.1/32\"\n;  :application_name \"psql\"}\n```\n\n\nRead stats and print the default diagnostics:\n\n```clojure\n;; warnings only\n(doseq [w (pgex/diagnose-warnings (pgex/read-stats db))]\n  (println (:message w)))\n```\n\nThe `default-diagnostic-fns` can be overridden.\nTo create your own diagnostics:\n\n```clojure\n(def unrealistic-expectations\n  {:table-cache-hit\n   {:pred #(\u003e (:ratio %) 0.999)\n    :onfalse \"The cache hit ratio is not as insanely high as I'd like.\"\n    :idfn :name}})\n\n(doseq [w (pgex/diagnose-warnings\n           (pgex/read-stats db)\n           :diagnostic-fns unrealistic-expectations)]\n  (println (:message w)))\n\n; ! Warning :table-cache-hit, message_topics, The cache hit ratio is not as insanely high as I'd like.\n; {:ratio 0.9806201550387597, :schema \"public\", :name \"message_topics\", :buffer_hits 253, :block_reads 5, :total_read 258}\n; ... many more\n```\n\n## Development\n\nTest runner with coverage\n\n    clj -X:test\n\nRun NREPL and interactive terminal REPL in one\n\n    clj -M:dev\n\n\nBuild a jar. Output in `./target/com.github.perrygeo/postgres-extras-clj-*.jar`\n\n    clj -T:build jar\n\n\nDeploy to Clojars.\nSet `CLOJARS_USERNAME` and `CLOJARS_PASSWORD` env vars.\nAssumes that `clj -T:build jar` has already been run.\n\n    clj -T:build deploy\n\n## License\n\nCopyright © 2024 Matthew T. Perry (`perrygeo`). \nDistributed under the MIT license.\n\nThe credit for the SQL query logic goes\nto the fantastic work done by these three projects:\n\n  * https://github.com/heroku/heroku-pg-extras\n  * https://github.com/pawurb/ecto_psql_extras\n  * https://github.com/rustprooflabs/pgdd\n\nTheir licenses (all MIT) are included in the SQL files.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperrygeo%2Fpostgres-extras-clj","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fperrygeo%2Fpostgres-extras-clj","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperrygeo%2Fpostgres-extras-clj/lists"}