{"id":30107202,"url":"https://github.com/datafusion-contrib/datafusion-postgres","last_synced_at":"2026-01-16T08:58:53.171Z","repository":{"id":232270682,"uuid":"782796446","full_name":"datafusion-contrib/datafusion-postgres","owner":"datafusion-contrib","description":"Postgres protocol frontend for DataFusion","archived":false,"fork":false,"pushed_at":"2025-08-07T03:24:11.000Z","size":427,"stargazers_count":74,"open_issues_count":2,"forks_count":16,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-08-07T03:27:33.123Z","etag":null,"topics":["datafusion","postgresql","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datafusion-contrib.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-04-06T04:06:55.000Z","updated_at":"2025-08-07T03:24:14.000Z","dependencies_parsed_at":"2024-04-08T22:50:21.029Z","dependency_job_id":"a9ece0ee-f2e3-4dc8-abbd-8cffb80b1c12","html_url":"https://github.com/datafusion-contrib/datafusion-postgres","commit_stats":{"total_commits":73,"total_committers":4,"mean_commits":18.25,"dds":0.410958904109589,"last_synced_commit":"42946fbebeabe9c9cf30cbe14c72db835274e4f3"},"previous_names":["sunng87/datafusion-postgres","datafusion-contrib/datafusion-postgres"],"tags_count":16,"template":false,"template_full_name":null,"purl":"pkg:github/datafusion-contrib/datafusion-postgres","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datafusion-contrib%2Fdatafusion-postgres","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datafusion-contrib%2Fdatafusion-postgres/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datafusion-contrib%2Fdatafusion-postgres/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datafusion-contrib%2Fdatafusion-postgres/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datafusion-contrib","download_url":"https://codeload.github.com/datafusion-contrib/datafusion-postgres/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datafusion-contrib%2Fdatafusion-postgres/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269663702,"owners_count":24455828,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-10T02:00:08.965Z","response_time":71,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["datafusion","postgresql","rust"],"created_at":"2025-08-10T02:01:16.200Z","updated_at":"2026-01-16T08:58:53.153Z","avatar_url":"https://github.com/datafusion-contrib.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"# datafusion-postgres\n\n[![Crates.io Version][crates-badge]][crates-url]\n[![Docs.rs Version][docs-badge]][docs-url]\n\n[crates-badge]: https://img.shields.io/crates/v/datafusion-postgres?label=datafusion-postgres\n[crates-url]: https://crates.io/crates/datafusion-postgres\n[docs-badge]: https://img.shields.io/docsrs/datafusion-postgres\n[docs-url]: https://docs.rs/datafusion-postgres/latest/datafusion_postgres\n\nA PostgreSQL-compatible server frontend for [Apache\nDataFusion](https://datafusion.apache.org). Available as both a library and CLI\ntool.\n\nBuilt on [pgwire](https://github.com/sunng87/pgwire) to provide PostgreSQL wire\nprotocol compatibility for analytical workloads. It was originally an example of\nthe [pgwire](https://github.com/sunng87/pgwire) project.\n\n## Scope of the Project\n\n- `datafusion-postgres`: Postgres frontend for datafusion, as a library.\n  - Serving Datafusion `SessionContext` with pgwire library\n  - Customizible/Optional authentication and Permission control\n- `datafusion-pg-catalog`: A Postgres compatible `pg_catalog` schema and\n  functions for datafusion backend.\n- `arrow-pg`: A data type mapping, encoding/decoding library for arrow and\n  postgres(pgwire) data types.\n- `datafusion-postgres-cli`: A cli tool starts a postgres compatible server for\n  datafusion supported file formats, just like python's `SimpleHTTPServer`.\n\n## Supported Database Clients\n\n- Database Clients\n  - [x] psql\n  - [x] DBeaver\n  - [x] pgcli\n  - [x] VSCode SQLTools\n  - [ ] Intellij Datagrip\n- BI \u0026 Visualization\n  - [x] Metabase\n  - [ ] PowerBI\n  - [x] Grafana\n\n## Quick Start\n\n### The Library `datafusion-postgres`\n\nThe high-level entrypoint of `datafusion-postgres` library is the `serve`\nfunction which takes a datafusion `SessionContext` and some server configuration\noptions.\n\n```rust\nuse std::sync::Arc;\nuse datafusion::prelude::SessionContext;\nuse datafusion_postgres::{serve, ServerOptions};\nuse datafusion_pg_catalog::setup_pg_catalog;\n\n// Create datafusion SessionContext\nlet session_context = Arc::new(SessionContext::new());\n// Configure your `session_context`\n// ...\n\n// Optional: setup pg_catalog schema\nsetup_pg_catalog(session_context, \"datafusion\")?;\n\n// Start the Postgres compatible server with SSL/TLS\nlet server_options = ServerOptions::new()\n    .with_host(\"127.0.0.1\".to_string())\n    .with_port(5432)\n    // Optional: setup tls\n    .with_tls_cert_path(Some(\"server.crt\".to_string()))\n    .with_tls_key_path(Some(\"server.key\".to_string()));\n\nserve(session_context, \u0026server_options).await\n```\n\n### Security Features\n\nThe server automatically includes:\n\n- User authentication (default postgres superuser)\n- Role-based access control with predefined roles:\n  - readonly: SELECT permissions\n  - readwrite: SELECT, INSERT, UPDATE, DELETE permissions\n  - dbadmin: Full administrative permissions\n- SSL/TLS encryption when certificates are provided\n- Query-level permission checking\n\n### The CLI `datafusion-postgres-cli`\n\nCommand-line tool to serve JSON/CSV/Arrow/Parquet/Avro files as PostgreSQL-compatible tables.\n\n```\ndatafusion-postgres-cli 0.6.1\nA PostgreSQL interface for DataFusion. Serve CSV/JSON/Arrow/Parquet files as tables.\n\nUSAGE:\n    datafusion-postgres-cli [OPTIONS]\n\nFLAGS:\n    -h, --help       Prints help information\n    -V, --version    Prints version information\n\nOPTIONS:\n        --arrow \u003carrow-tables\u003e...        Arrow files to register as table, using syntax `table_name:file_path`\n        --avro \u003cavro-tables\u003e...          Avro files to register as table, using syntax `table_name:file_path`\n        --csv \u003ccsv-tables\u003e...            CSV files to register as table, using syntax `table_name:file_path`\n    -d, --dir \u003cdirectory\u003e                Directory to serve, all supported files will be registered as tables\n        --host \u003chost\u003e                    Host address the server listens to [default: 127.0.0.1]\n        --json \u003cjson-tables\u003e...          JSON files to register as table, using syntax `table_name:file_path`\n        --parquet \u003cparquet-tables\u003e...    Parquet files to register as table, using syntax `table_name:file_path`\n    -p \u003cport\u003e                            Port the server listens to [default: 5432]\n        --tls-cert \u003ctls-cert\u003e            Path to TLS certificate file for SSL/TLS encryption\n        --tls-key \u003ctls-key\u003e              Path to TLS private key file for SSL/TLS encryption\n```\n\n#### 🔒 Security Options\n\n```bash\n# Run with SSL/TLS encryption\ndatafusion-postgres-cli \\\n  --csv data:sample.csv \\\n  --tls-cert server.crt \\\n  --tls-key server.key\n\n# Run without encryption (development only)\ndatafusion-postgres-cli --csv data:sample.csv\n```\n\n## 📋 Example Usage\n\n### Basic Example\n\nHost a CSV dataset as a PostgreSQL-compatible table:\n\n```bash\ndatafusion-postgres-cli --csv climate:delhiclimate.csv\n```\n\n```\nLoaded delhiclimate.csv as table climate\nTLS not configured. Running without encryption.\nListening on 127.0.0.1:5432 (unencrypted)\n```\n\n### Connect with psql\n\n\u003e **🔐 Authentication**: The default setup allows connections without authentication for development. For secure deployments, use `DfAuthSource` with standard pgwire authentication handlers (cleartext, MD5, or SCRAM). See `auth.rs` for implementation examples.\n\n```bash\npsql -h 127.0.0.1 -p 5432 -U postgres\n```\n\n```sql\npostgres=\u003e SELECT COUNT(*) FROM climate;\n count\n-------\n  1462\n(1 row)\n\npostgres=\u003e SELECT date, meantemp FROM climate WHERE meantemp \u003e 35 LIMIT 5;\n    date    | meantemp\n------------+----------\n 2017-05-15 |     36.9\n 2017-05-16 |     37.9\n 2017-05-17 |     38.6\n 2017-05-18 |     37.4\n 2017-05-19 |     35.4\n(5 rows)\n\npostgres=\u003e BEGIN;\nBEGIN\npostgres=\u003e SELECT AVG(meantemp) FROM climate;\n       avg\n------------------\n 25.4955206557617\n(1 row)\npostgres=\u003e COMMIT;\nCOMMIT\n```\n\n### 🔐 Production Setup with SSL/TLS\n\n```bash\n# Generate SSL certificates\nopenssl req -x509 -newkey rsa:4096 -keyout server.key -out server.crt \\\n  -days 365 -nodes -subj \"/C=US/ST=CA/L=SF/O=MyOrg/CN=localhost\"\n\n# Start secure server\ndatafusion-postgres-cli \\\n  --csv climate:delhiclimate.csv \\\n  --tls-cert server.crt \\\n  --tls-key server.key\n```\n\n```\nLoaded delhiclimate.csv as table climate\nTLS enabled using cert: server.crt and key: server.key\nListening on 127.0.0.1:5432 with TLS encryption\n```\n\n## Community\n\n### Developer Mailing List\n\nIf you like the idea of pgwire, datafusion-postgres and want to join the\ndevelopment of the library, or its ecosystem integrations, extensions, you are\nwelcomed to join our developer mailing list: https://groups.io/g/pgwire-dev/\n\n## License\n\nThis library is released under Apache license.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatafusion-contrib%2Fdatafusion-postgres","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatafusion-contrib%2Fdatafusion-postgres","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatafusion-contrib%2Fdatafusion-postgres/lists"}