{"id":13532768,"url":"https://github.com/ankane/dexter","last_synced_at":"2025-11-17T14:16:45.366Z","repository":{"id":37734138,"uuid":"95328271","full_name":"ankane/dexter","owner":"ankane","description":"The automatic indexer for Postgres","archived":false,"fork":false,"pushed_at":"2025-10-15T04:01:43.000Z","size":494,"stargazers_count":2036,"open_issues_count":2,"forks_count":53,"subscribers_count":19,"default_branch":"master","last_synced_at":"2025-10-15T08:40:38.785Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ankane.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2017-06-24T23:38:25.000Z","updated_at":"2025-10-15T04:01:47.000Z","dependencies_parsed_at":"2023-12-29T03:48:48.913Z","dependency_job_id":"58433517-38ff-4388-b749-979bd31f5a1d","html_url":"https://github.com/ankane/dexter","commit_stats":{"total_commits":398,"total_committers":7,"mean_commits":"56.857142857142854","dds":0.2587939698492462,"last_synced_commit":"7e6525a41701d03234787a7929d73bbd933c8af6"},"previous_names":[],"tags_count":34,"template":false,"template_full_name":null,"purl":"pkg:github/ankane/dexter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fdexter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fdexter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fdexter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fdexter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ankane","download_url":"https://codeload.github.com/ankane/dexter/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fdexter/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":284894654,"owners_count":27080744,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-17T02:00:06.431Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T07:01:13.614Z","updated_at":"2025-11-17T14:16:45.360Z","avatar_url":"https://github.com/ankane.png","language":"Ruby","readme":"# Dexter\n\nThe automatic indexer for Postgres\n\n[Read about how it works](https://ankane.org/introducing-dexter) or [watch the talk](https://www.youtube.com/watch?v=Mni_1yTaNbE)\n\n[![Build Status](https://github.com/ankane/dexter/actions/workflows/build.yml/badge.svg)](https://github.com/ankane/dexter/actions)\n\n## Installation\n\nFirst, install [HypoPG](https://github.com/HypoPG/hypopg) on your database server. This doesn’t require a restart.\n\n```sh\ncd /tmp\ncurl -L https://github.com/HypoPG/hypopg/archive/1.4.2.tar.gz | tar xz\ncd hypopg-1.4.2\nmake\nmake install # may need sudo\n```\n\nAnd enable it in databases where you want to use Dexter:\n\n```sql\nCREATE EXTENSION hypopg;\n```\n\nSee the [installation notes](#hypopg-installation-notes) if you run into issues.\n\nThen install the command line tool with:\n\n```sh\ngem install pgdexter\n```\n\nThe command line tool is also available with [Docker](#docker), [Homebrew](#homebrew), or as a [Linux package](guides/Linux.md).\n\n## How to Use\n\nDexter needs a connection to your database and a source of queries (like [pg_stat_statements](https://www.postgresql.org/docs/current/pgstatstatements.html)) to process.\n\n```sh\ndexter -d dbname --pg-stat-statements\n```\n\nThis finds slow queries and generates output like:\n\n```\nStarted\nProcessing 189 new query fingerprints\nIndex found: public.genres_movies (genre_id)\nIndex found: public.genres_movies (movie_id)\nIndex found: public.movies (title)\nIndex found: public.ratings (movie_id)\nIndex found: public.ratings (rating)\nIndex found: public.ratings (user_id)\n```\n\nTo be safe, Dexter will not create indexes unless you pass the `--create` flag. In this case, you’ll see:\n\n```\nIndex found: public.ratings (user_id)\nCreating index: CREATE INDEX CONCURRENTLY ON \"public\".\"ratings\" (\"user_id\")\nIndex created: 15243 ms\n```\n\n## Connection Options\n\nDexter supports the same connection options as psql.\n\n```\n-h host -U user -p 5432 -d dbname\n```\n\nThis includes URIs:\n\n```\npostgresql://user:pass@host:5432/dbname\n```\n\nand connection strings:\n\n```\nhost=localhost port=5432 dbname=mydb\n```\n\nAlways make sure your [connection is secure](https://ankane.org/postgres-sslmode-explained) when connecting to a database over a network you don’t fully trust.\n\n## Collecting Queries\n\nDexter can collect queries from a number of sources.\n\n- [Query stats](#query-stats)\n- [Live queries](#live-queries)\n- [Log files](#log-files)\n- [SQL files](#sql-files)\n\n### Query Stats\n\nEnable [pg_stat_statements](https://www.postgresql.org/docs/current/pgstatstatements.html) in your database.\n\n```psql\nCREATE EXTENSION pg_stat_statements;\n```\n\nAnd use:\n\n```sh\ndexter \u003cconnection-options\u003e --pg-stat-statements\n```\n\n### Live Queries\n\nGet live queries from the [pg_stat_activity](https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-PG-STAT-ACTIVITY-VIEW) view with:\n\n```sh\ndexter \u003cconnection-options\u003e --pg-stat-activity\n```\n\n### Log Files\n\nEnable logging for slow queries in your Postgres config file.\n\n```ini\nlog_min_duration_statement = 10 # ms\n```\n\nAnd use:\n\n```sh\ndexter \u003cconnection-options\u003e postgresql.log\n```\n\nSupports `stderr`, `csvlog`, and `jsonlog` formats.\n\nFor real-time indexing, pipe your logfile:\n\n```sh\ntail -F -n +1 postgresql.log | dexter \u003cconnection-options\u003e --stdin\n```\n\nAnd pass `--input-format csv` or `--input-format json` if needed.\n\n### SQL Files\n\nPass a SQL file with:\n\n```sh\ndexter \u003cconnection-options\u003e queries.sql\n```\n\nPass a single query with:\n\n```sh\ndexter \u003cconnection-options\u003e -s \"SELECT * FROM ...\"\n```\n\n## Collection Options\n\nTo prevent one-off queries from being indexed, specify a minimum number of calls before a query is considered for indexing\n\n```sh\ndexter --min-calls 100\n```\n\nYou can do the same for total time a query has run\n\n```sh\ndexter --min-time 10 # minutes\n```\n\nWhen streaming logs, specify the time to wait between processing queries\n\n```sh\ndexter --interval 60 # seconds\n```\n\n## Analyze\n\nFor best results, make sure your tables have been recently analyzed so statistics are up-to-date. You can ask Dexter to analyze tables it comes across that haven’t been analyzed in the past hour with:\n\n```sh\ndexter --analyze\n```\n\n## Tables\n\nYou can exclude large or write-heavy tables from indexing with:\n\n```sh\ndexter --exclude table1,table2\n```\n\nAlternatively, you can specify which tables to index with:\n\n```sh\ndexter --include table3,table4\n```\n\n## Debugging\n\nSee how Dexter is processing queries with:\n\n```sh\ndexter --log-sql --log-level debug2\n```\n\n## Hosted Postgres\n\nThe `hypopg` extension, which Dexter needs to run, is available on [these providers](https://github.com/ankane/dexter/issues/44).\n\nFor other providers, see [this guide](guides/Hosted-Postgres.md). To request a new extension:\n\n- Google Cloud SQL - vote or comment on [this page](https://issuetracker.google.com/issues/69250435)\n- DigitalOcean Managed Databases - vote or comment on [this page](https://ideas.digitalocean.com/managed-database/p/support-hypopg-for-postgres)\n\n## HypoPG Installation Notes\n\n### Postgres Location\n\nIf your machine has multiple Postgres installations, specify the path to [pg_config](https://www.postgresql.org/docs/current/app-pgconfig.html) with:\n\n```sh\nexport PG_CONFIG=/Applications/Postgres.app/Contents/Versions/latest/bin/pg_config\n```\n\nThen re-run the installation instructions (run `make clean` before `make` if needed)\n\n### Missing Header\n\nIf compilation fails with `fatal error: postgres.h: No such file or directory`, make sure Postgres development files are installed on the server.\n\nFor Ubuntu and Debian, use:\n\n```sh\nsudo apt-get install postgresql-server-dev-17\n```\n\nNote: Replace `17` with your Postgres server version\n\n## Additional Installation Methods\n\n### Docker\n\nGet the [Docker image](https://hub.docker.com/r/ankane/dexter) with:\n\n```sh\ndocker pull ankane/dexter\n```\n\nAnd run it with:\n\n```sh\ndocker run -ti ankane/dexter \u003cconnection-options\u003e\n```\n\nFor databases on the host machine, use `host.docker.internal` as the hostname (on Linux, this requires `--add-host=host.docker.internal:host-gateway`).\n\n### Homebrew\n\nWith Homebrew, you can use:\n\n```sh\nbrew install dexter\n```\n\n## Future Work\n\n[Here are some ideas](https://github.com/ankane/dexter/issues/45)\n\n## Upgrading\n\nRun:\n\n```sh\ngem install pgdexter\n```\n\nTo use master, run:\n\n```sh\ngem install specific_install\ngem specific_install https://github.com/ankane/dexter.git\n```\n\n## Thanks\n\nThis software wouldn’t be possible without [HypoPG](https://github.com/HypoPG/hypopg), which allows you to create hypothetical indexes, and [pg_query](https://github.com/lfittl/pg_query), which allows you to parse and fingerprint queries. A big thanks to Dalibo and Lukas Fittl respectively. Also, thanks to YugabyteDB for [this article](https://dev.to/yugabyte/explain-from-pgstatstatements-normalized-queries-how-to-always-get-the-generic-plan-in--5cfi) on how to explain normalized queries.\n\n## Research\n\nThis is known as the Index Selection Problem (ISP).\n\n## History\n\nView the [changelog](https://github.com/ankane/dexter/blob/master/CHANGELOG.md)\n\n## Contributing\n\nEveryone is encouraged to help improve this project. Here are a few ways you can help:\n\n- [Report bugs](https://github.com/ankane/dexter/issues)\n- Fix bugs and [submit pull requests](https://github.com/ankane/dexter/pulls)\n- Write, clarify, or fix documentation\n- Suggest or add new features\n\nTo get started with development, run:\n\n```sh\ngit clone https://github.com/ankane/dexter.git\ncd dexter\nbundle install\nbundle exec rake install\n```\n\nTo run tests, use:\n\n```sh\ncreatedb dexter_test\nbundle exec rake test\n```\n","funding_links":[],"categories":["Ruby","Compiled list","Awesome Ruby CLIs","Monitoring"],"sub_categories":["plv8:","Database"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fankane%2Fdexter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fankane%2Fdexter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fankane%2Fdexter/lists"}