{"id":28269207,"url":"https://github.com/danthegoodman1/sqlgateway","last_synced_at":"2025-07-04T13:06:54.468Z","repository":{"id":64301275,"uuid":"562557612","full_name":"danthegoodman1/SQLGateway","owner":"danthegoodman1","description":"HTTP Gateway for SQL databases with connection pooling and caching. An Edge Functions's best friend.","archived":false,"fork":false,"pushed_at":"2023-02-12T01:18:00.000Z","size":204,"stargazers_count":57,"open_issues_count":9,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-17T01:40:40.827Z","etag":null,"topics":["cockroachdb","golang","http","postgres","redis","transaction","wasm"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/danthegoodman1.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-06T18:06:26.000Z","updated_at":"2025-02-03T02:10:04.000Z","dependencies_parsed_at":"2024-06-20T01:42:38.356Z","dependency_job_id":null,"html_url":"https://github.com/danthegoodman1/SQLGateway","commit_stats":null,"previous_names":["danthegoodman1/psqlgateway"],"tags_count":10,"template":false,"template_full_name":"danthegoodman1/GoAPITemplate","purl":"pkg:github/danthegoodman1/SQLGateway","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danthegoodman1%2FSQLGateway","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danthegoodman1%2FSQLGateway/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danthegoodman1%2FSQLGateway/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danthegoodman1%2FSQLGateway/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/danthegoodman1","download_url":"https://codeload.github.com/danthegoodman1/SQLGateway/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danthegoodman1%2FSQLGateway/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263548653,"owners_count":23478808,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cockroachdb","golang","http","postgres","redis","transaction","wasm"],"created_at":"2025-05-20T15:14:14.070Z","updated_at":"2025-07-04T13:06:54.460Z","avatar_url":"https://github.com/danthegoodman1.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SQLGateway \u003c!-- omit in toc --\u003e\n\n\nAccess your SQL database over HTTP like it’s a SQL database but with superpowers. An edge function's best friend.\n\n**Superpowers include:**\n\n- HTTP access for SQL databases enable WASM-based runtimes to use TCP-connected DBs\n- Connection pooling protects from reconnects, wasted idle connections, and bursts of load\n- Automatic query and transaction tracing\n- Caching capabilities\n\n_Currently only the PSQL protocol is supported. Additional protocol support (like MySQL) is on the roadmap._\n\n- [Quick Start](#quick-start)\n- [Why This Exists](#why-this-exists)\n  - [Querying and Transactions](#querying-and-transactions)\n  - [Automatic query and transaction tracing](#automatic-query-and-transaction-tracing)\n  - [Caching (Coming Soon)](#caching-coming-soon)\n  - [Connection Pooling](#connection-pooling)\n  - [Database Throttling Under Load](#database-throttling-under-load)\n- [API](#api)\n  - [GET /hc](#get-hc)\n  - [POST /psql/query](#post-psqlquery)\n  - [/psql/begin](#psqlbegin)\n  - [/psql/commit](#psqlcommit)\n  - [/psql/rollback](#psqlrollback)\n  - [Error handling](#error-handling)\n- [Configuration](#configuration)\n- [Auth](#auth)\n- [Clustered vs. Single Node](#clustered-vs-single-node)\n- [Transactions](#transactions)\n- [Running distributed tests](#running-distributed-tests)\n\n## Quick Start\n\n\nPull this repo:\n\n```\ngit clone https://github.com/danthegoodman1/SQLGateway \u0026\u0026 cd SQLGateway\n```\n\nRun the `docker-compose.yml` file:\n\n```\ndocker compose up\n```\n\nThen in another terminal, run:\n\n```\ncurl --location --request POST 'http://localhost:8080/psql/query' \\\n--header 'Content-Type: application/json' \\\n--data-raw '{\n    \"Queries\": [\n        {\n            \"Statement\": \"SELECT 1 as num_one, NOW() as current_time\"\n        }\n    ]\n}'\n```\n\nYou should get the following back (with a different time):\n\n```\n{\"Queries\":[{\"Columns\":[\"num_one\",\"current_time\"],\"Rows\":[[1,\"2022-11-27T19:20:13.030114Z\"]],\"TimeNS\":958400}]}\n```\n\n## Why This Exists\n\nI wanted to use Cloudflare Workers, but also the Postgres ecosystem (specifically CockroachDB Serverless).\n\nThe idea was to keep the HTTP layer out of the way and make it feel like you are talking to a normal SQL database.\n\nNow we can connect the two worlds of WASM-runtimes and SQL databases without vendor lock-in!\n\nSome WASM runtimes that can now use SQL databases:\n\n- Cloudflare Workers\n- Vercel Edge Functions\n- Fastly Compute@Edge\n- Netlify Functions _note: [this](https://wasmedge.org/book/en/write_wasm/js/networking.html#tcp-client) seems to indicate that TCP connections may be supported, since they (at least used to) use WasmEdge. I have not bothered testing however :P_\n\nSome Databases that WASM runtimes can now use:\n\n- AWS RDS \u0026 Aurora\n- GCP Cloud SQL\n- CockroachDB Dedicated \u0026 Serverless\n- DigitalOcean managed databases\n- UpCloud Managed Databases\n\n### Querying and Transactions\n\nSend single queries, or send an array of queries to run atomically in a transaction.\n\nStart a transaction and go back and forth between the DB and your code just like normal. The nodes in the cluster will automatically route transaction queries to the correct node (coordinated through Redis). Abandoned transactions will be garbage collected.\n\n### Automatic query and transaction tracing\n\nMetric logs emitted on the performance of individual queries, as well as entire transactions. Build dashboards and create alerts to find slowdowns and hot-spots in your code.\n\nComing soon (maybe?): Alerting and dashboards (for now just use some logging provider)\n\n### Caching (Coming Soon)\n\nSpecify SELECTs that don’t need to be consistent you can have them cache and TTL with stale-while-revalidate support.\n\n### Connection Pooling\n\nPrevent constant session creation from creating unnecessary load on the DB, and burst execution environments from holding idle connections that won't be used again. \n\nUse HTTP Keep-Alive to keep connections warm for Lambda-like environments, but don’t risk overloading the DB with new connections or leaving tons of resource-intensive DB sessions idle.\n\n### Database Throttling Under Load\n\nWith a finite number of pool connections, you prevent uncapped load from hitting your database directly.\n\n## API\n\n### GET /hc\n\nHealth check endpoint, only guarantees that the HTTP server is running.\n\n### POST /psql/query\n\nRequest Body:\n\n_`*` indicates optional_\n```\n{\n  Queries: []{\n      Statement:   string\n      Params:      []any\n      Exec:        *bool // if provided, then no `Rows` or `Columns` will be returned for this query.\n      TxKey:       *string\n    }\n    \n  TxID:    *string\n}\n```\n\nExamples:\n```json\n{\n  \"Queries\": [\n    {\n      \"Statement\": \"SELECT $1::INT8 as a_number\",\n      \"Params\": [\n        42\n      ]\n    }\n  ]\n}\n```\n```json\n{\n  \"Queries\": [\n    {\n      \"Statement\": \"CREATE TABLE test_table IF NOT EXISTS ( id TEXT NOT NULL, val TEXT NOT NULL, PRIMARY KEY(id) )\",\n      \"Exec\": true\n    }\n  ]\n}\n```\n\n**Note:** Casting is probably required for parameters as due to the primitive type selection the SQL cannot always interpret which SQL type a JSON property should use.\n\nIf given a single query, it will be run directly on the connection.\n\nIf given multiple items, they will be run within the same transaction. You will receive the results of all that succeed,\nhowever if a single query fails then the entire transaction will fail, and all queries will remain un-applied regardless\nof whether there were rows returned. Rows will be returned for the successful queries of a failing transaction.\n\nIf a `TxID` is provided, then it will be run within a transaction, proxying if required.\n\nDO NOT CALL `COMMIT` OR `ROLLBACK` through here, that should be handled via the respective endpoints, or functions within the client libraries.\n\nResponse Body:\n\n```\n{\n    Queries []{\n        Columns:  []any\n        Rows:     [][]any\n        Error:    *string\n        TimeNS:   *int64 \n    }\n    \n    // Whether this was proxied to a remote node\n    Remote: bool\n}\n```\n\nAny query errors that occur will be included in the response body, rather than failing the request.\n\n### /psql/begin\n\nStarts a new transaction.\n\nRequest Body:\n\n```\n{\n    TxTimeoutSec: *int64 // sets the garbage collection timeout, default `30`\n}\n```\n\nReturns the transaction ID that must be carried through subsequent requests.\n\nResponse Body:\n\n```\n{\n    TxID: string\n}\n```\n\n### /psql/commit\n\nCommits an existing transaction. Returns status `200` and no content if successful.\n\nRequest Body:\n\n```\n{\n    TxID: string\n}\n```\n\n### /psql/rollback\n\nRolls back an existing transaction. Returns status `200` and no content if successful. \n\nRequest Body:\n\n```\n{\n    TxID: string\n}\n```\n\n### Error handling\n\nAll processing errors (not query errors) will return a 4XX/5XX error code, and as a `text/plain` response body.\n\n## Configuration\n\nConfiguration is done through environment variables\n\n| Env Var            | Description                                                                                                                | Required?                  | Default |\n|--------------------|----------------------------------------------------------------------------------------------------------------------------|----------------------------|---------|\n| `PG_DSN`           | PSQL wire protocol DSN. Used to connect to DB                                                                              | Yes                        |         |\n| `PG_POOL_CONNS`    | Number of pool connections to acquire                                                                                      | No                         | `2`     |\n| `REDIS_ADDR`       | Redis Address. Currently used in non-cluster mode (standard client).\u003cbr/\u003eIf omitted then clustering features are disabled. | No                         |         |\n| `REDIS_PASSWORD`   | Redis connection password                                                                                                  | No                         |         |\n| `REDIS_POOL_CONNS` | Number of pool connections to Redis.                                                                                       | No                         | `2`     |\n| `V_NAMESPACE`      | Virtual namespace for Redis. Sets the key prefix for Service discovery.                                                    | Yes (WIP, so No currently) |         |\n| `POD_URL`          | Direct URL that this pod/node can be reached at.\u003cbr/\u003eReplaces `POD_NAME` and `POD_BASE_DOMAIN` if exists.                  | Yes (conditional)          |         |\n| `POD_NAME`         | Name of the node/pod (k8s semantics).\u003cbr/\u003ePod can be reached at {POD_NAME}{POD_BASE_DOMAIN}                                | Yes (conditional)          |         |\n| `POD_BASE_DOMAIN`  | Base domain of the node/pod (k8s semantics).\u003cbr/\u003ePod can be reached at {POD_NAME}{POD_BASE_DOMAIN}                         | Yes (conditional)          |         |\n| `HTTP_PORT`        | HTTP port to run the HTTP(2) server on                                                                                     | No                         | `8080`  |\n| `POD_HTTPS`        | Indicates whether the pods should use HTTPS to contact each other.\u003cbr/\u003eSet to `1` if they should use HTTPS.                | No                         |         |\n| `TRACES`           | Indicates whether query trace information should be included in log contexts.\u003cbr/\u003eSet to `1` if they should be.            | No                         |         |\n| `DEBUG`            | Indicates whether the debug log level should be enabled.\u003cbr/\u003eSet to `1` to enable.                                         | No                         |         |\n| `PRETTY`           | Indicates whether pretty logs should be printed.\u003cbr/\u003eSet to `1` to enable.                                                 |                            |         |\n| `AUTH_USER`        | Sets the Basic Auth username required to connect. Requires that `AUTH_PASS` be set as well                                 | Yes (conditional)          |         |\n| `AUTH_PASS`        | Sets the Basic Auth password required to connect. Requires that `AUTH_USER` be set as well                                 | Yes (conditional)          |         |\n\n## Auth\n\nSQLGateway optionally supports Basic Auth in the style of `http(s)://USER:PASS@your.domain.tld`, just like a DB DSN.\n\nTo configure a username and password, set the env vars `AUTH_USER` and `AUTH_PASS`, then access with `http://{AUTH_USER}:{AUTH_PASS}@yourdomain`\n\n## Performance and Overhead\n\nWith some light testing on my laptop (2019 16\" MBP, 8 core 64GB ram) selecting 10,001 rows directly from `pgx` without processing takes ~8-9ms, while requesting that same query through SQLGateway takes ~12-13ms.\n\nSQLGateway currently uses the native JSON package, which is known to be very slow. The binding overhead can be optimized by \u003e10x with a more efficient JSON package for binding.\n\nHTTP overhead is ~80us per request, a bit more if you add in Basic Auth.\n\n## Clustered vs. Single Node\n\nSQLGateway can either be run in a cluster, or as a single node.\n\nIf running as a single node, ensure to omit the `REDIS_ADDR` env var.\n\nWhen running in clustered mode (`REDIS_ADDR` env var present), it will require that a connection to Redis can be established.\n\nWhen transactions are not found locally, a lookup to Redis will be attempted. If the transaction is found on a remote pod,\nthe request will be proxied to the remote pod.\n\nRedis Cluster mode support and etcd support are on the roadmap.\n\n## Transactions\n\nTransactions (and query requests) have a default timeout of 30 seconds.\n\nWhen any query in a transaction fails, the transaction is automatically rolled back and the pool connection released, meaning that the client that errors is not responsible for doing so.\n\nIf a transaction times out then it will also automatically roll back and release the pool connection.\n\nIf a pod crashes while it has a transaction, then the transaction will be immediately released, but may remain present within Redis.\nA special error is returned for this indicating this may be the case.\n\nIf Redis crashes while a transaction is still held on a pod, then other pods will not be able to route transaction queries to this pod.\nThe timeout will garbage collect these transactions, but the connection will remain held until it times out. \n\n## Running distributed tests\n\nRun 2 instances connected to a local CRDB/Postgres and Redis like the following (DSN and Redis env vars omitted):\n\n```POD_URL=\"localhost:8080\" HTTP_PORT=\"8080\" task```\n\n```POD_URL=\"localhost:8081\" HTTP_PORT=\"8081\" task```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanthegoodman1%2Fsqlgateway","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdanthegoodman1%2Fsqlgateway","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanthegoodman1%2Fsqlgateway/lists"}