{"id":26315337,"url":"https://github.com/onegreyonewhite/easyrest-plugin-postgres","last_synced_at":"2025-12-30T18:07:58.327Z","repository":{"id":282127861,"uuid":"947579775","full_name":"onegreyonewhite/easyrest-plugin-postgres","owner":"onegreyonewhite","description":"The EasyREST PostgreSQL Plugin is an external plugin for EasyREST that enables EasyREST to connect to and perform CRUD operations on PostgreSQL databases.","archived":false,"fork":false,"pushed_at":"2025-03-12T23:19:23.000Z","size":12,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-13T00:20:39.536Z","etag":null,"topics":["golang","postgres","postgresql","restapi"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/onegreyonewhite.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-12T23:13:15.000Z","updated_at":"2025-03-12T23:22:39.000Z","dependencies_parsed_at":"2025-03-13T02:31:41.973Z","dependency_job_id":null,"html_url":"https://github.com/onegreyonewhite/easyrest-plugin-postgres","commit_stats":null,"previous_names":["onegreyonewhite/easyrest-plugin-postgres"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onegreyonewhite%2Feasyrest-plugin-postgres","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onegreyonewhite%2Feasyrest-plugin-postgres/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onegreyonewhite%2Feasyrest-plugin-postgres/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onegreyonewhite%2Feasyrest-plugin-postgres/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/onegreyonewhite","download_url":"https://codeload.github.com/onegreyonewhite/easyrest-plugin-postgres/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243725628,"owners_count":20337671,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","postgres","postgresql","restapi"],"created_at":"2025-03-15T12:18:21.656Z","updated_at":"2025-12-30T18:07:58.319Z","avatar_url":"https://github.com/onegreyonewhite.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# EasyREST PostgreSQL Plugin\n\nThe **EasyREST PostgreSQL Plugin** is an external plugin for [EasyREST](https://github.com/onegreyonewhite/easyrest) that enables EasyREST to connect to and perform CRUD operations on PostgreSQL databases. This plugin implements the `easyrest.DBPlugin` interface using a PostgreSQL connection pool, session variable injection for context propagation, and transactional function execution.\n\n**Key Features:**\n\n- **CRUD Operations:** Supports SELECT, INSERT, UPDATE, and DELETE queries.\n- **Context Injection:** If query fields or conditions reference context variables (using the `erctx.` prefix), the plugin injects these values into session variables using `set_config`.\n- **Function Calls:** Executes functions within a transaction, rolling back on error.\n- **Connection Pooling:** Uses a PostgreSQL connection pool for optimal performance, with special handling for session variables to avoid race conditions.\n- **Deterministic SQL Generation:** Ensures predictable SQL statements by sorting map keys where necessary.\n- **COPY Support:** Uses PostgreSQL's COPY command for efficient bulk inserts.\n\n---\n\n## Table of Contents\n\n- [Prerequisites](#prerequisites)\n- [Performance Optimizations](#performance-optimizations)\n- [PostgreSQL Setup using Docker](#postgresql-setup-using-docker)\n- [SQL Schema and Functions](#sql-schema-and-functions)\n- [Environment Variables for EasyREST](#environment-variables-for-easyrest)\n- [Building the Plugin](#building-the-plugin)\n- [Running EasyREST Server with the Plugin](#running-easyrest-server-with-the-plugin)\n- [Testing API Endpoints](#testing-api-endpoints)\n- [Working with Tables, Views, and Context Variables](#working-with-tables-views-and-context-variables)\n- [License](#license)\n\n---\n\n## Prerequisites\n\n- [Docker](https://www.docker.com) installed on your machine.\n- [Go 1.23.6](https://golang.org/dl/) or later.\n- Basic knowledge of PostgreSQL and Docker.\n\n## Performance Optimizations\n\nThe plugin includes several performance optimizations:\n\n1. **Connection Pool Management:**\n   - Configurable maximum open connections (default: 100)\n   - Configurable idle connections (default: 25)\n   - Connection lifetime management\n   - Idle connection timeout\n\n2. **Bulk Operations:**\n   - Automatic COPY for large inserts (\u003e100 rows)\n   - Memory-efficient result handling\n   - Pre-allocated memory for large result sets\n   - Batch processing for large operations\n\n3. **Transaction Optimizations:**\n   - Read-only transactions for SELECT queries\n   - Configurable transaction timeouts\n   - Proper connection release\n\n4. **Query Parameters:**\n   - `maxOpenConns` - Maximum number of open connections (default: 100)\n   - `maxIdleConns` - Maximum number of idle connections (default: 25)\n   - `connMaxLifetime` - Connection reuse time in minutes (default: 5)\n   - `connMaxIdleTime` - Connection idle time in minutes (default: 10)\n   - `timeout` - Query timeout in seconds (default: 30)\n   - `bulkThreshold` - Number of rows threshold for using COPY command (default: 100)\n\nExample URI with all optimization parameters:\n\n- **Via Environment Variable:**\n  ```bash\n  export ER_DB_PG=\"postgres://postgres:root@localhost:5433/easyrestdb?maxOpenConns=100\u0026maxIdleConns=25\u0026connMaxLifetime=5\u0026connMaxIdleTime=10\u0026timeout=30\u0026bulkThreshold=100\u0026sslmode=disable\u0026search_path=public\"\n  ```\n- **Via Configuration File:**\n  ```yaml\n  plugins:\n    postgres: # Or any name you choose for the plugin instance\n      uri: postgres://postgres:root@localhost:5433/easyrestdb?maxOpenConns=100\u0026maxIdleConns=25\u0026connMaxLifetime=5\u0026connMaxIdleTime=10\u0026timeout=30\u0026bulkThreshold=100\u0026sslmode=disable\u0026search_path=public\n      path: ./easyrest-plugin-postgres # Path to the plugin binary\n  ```\n\n---\n\n## PostgreSQL Setup using Docker\n\nRun PostgreSQL in a Docker container on a non-standard port (e.g., **5433**) with a pre-created database. Open your terminal and execute:\n\n```bash\ndocker run --name pg-easyrest -p 5433:5432 \\\n  -e POSTGRES_PASSWORD=root \\\n  -e POSTGRES_DB=easyrestdb \\\n  -d postgres:15\n```\n\nThis command starts a PostgreSQL container:\n\n- **Container Name:** `pg-easyrest`\n- **Host Port:** `5433` (mapped to PostgreSQL's default port 5432 in the container)\n- **Password:** `root`\n- **Database Created:** `easyrestdb`\n\n---\n\n## SQL Schema and Functions\n\nCreate a SQL script (e.g., `schema.sql`) with tables, triggers, and functions. The script includes:\n\n1. A basic `users` table with auto-increment ID and timestamp\n2. A more complex `products` table with various field types and triggers\n3. Functions for business logic and data access\n4. Triggers for automatic timestamp updates and validation\n\nSee the complete `schema.sql` in the repository for the full implementation.\n\nTo execute this script on the running PostgreSQL server, run:\n\n```bash\ndocker exec -i pg-easyrest psql -U postgres -d easyrestdb \u003c schema.sql\n```\n\n---\n\n## Environment Variables for EasyREST\n\nConfigure EasyREST to use this PostgreSQL database via the PostgreSQL plugin. Set the following environment variables before starting the EasyREST server:\n\n```bash\n# --- Database Connection ---\n# The URI for the PostgreSQL database. The plugin will be selected when the URI scheme is 'postgres://'.\nexport ER_DB_PG=\"postgres://postgres:root@localhost:5433/easyrestdb?sslmode=disable\"\n\n# --- JWT Authentication ---\n# Secret for HS* algorithms OR path to public key file for RS*/ES*/PS* algorithms\nexport ER_TOKEN_SECRET=\"your-secret-key\"\n# export ER_TOKEN_PUBLIC_KEY=\"/path/to/public.key\"\n\n# Claim in the JWT token to use as the user identifier\nexport ER_TOKEN_USER_SEARCH=\"sub\"\n\n# --- General Settings ---\n# Default timezone for context propagation.\nexport ER_DEFAULT_TIMEZONE=\"GMT\"\n\n# Optional: Enable/disable scope checking (default: true if token secret/key is set)\n# export ER_CHECK_SCOPE=\"true\"\n```\n\n**Note:** If both a configuration file and environment variables are present, the configuration file settings take precedence for overlapping parameters. The `path` for the plugin can only be set via the configuration file.\n\n---\n\n## Building the Plugin\n\nClone the repository for the EasyREST PostgreSQL Plugin and build the plugin binary. In the repository root, run:\n\n```bash\ngo build -o easyrest-plugin-postgres postgres_plugin.go\n```\n\nThis produces the binary `easyrest-plugin-postgres` which must be in your PATH or referenced by the EasyREST server.\n\n---\n\n## Running EasyREST Server with the Plugin\n\nDownload and install the pre-built binary for the EasyREST Server from the [EasyREST Releases](https://github.com/onegreyonewhite/easyrest/releases) page. Once installed, ensure the configuration is set up (either via `config.yaml` or environment variables) and run the server binary.\n\n**Using a Configuration File (Recommended):**\n\n1. **Create the `config.yaml` file:** Save the configuration example above (or your customized version) as `config.yaml` (or any name you prefer).\n2. **Place the plugin binary:** Ensure the compiled `easyrest-plugin-postgres` binary exists at the location specified in the `path` field within `config.yaml` (e.g., in the same directory as the `easyrest-server` binary if `path: ./easyrest-plugin-postgres` is used).\n3. **Run the server:** Execute the EasyREST server binary, pointing it to your configuration file using the `--config` flag:\n\n    ```bash\n    ./easyrest-server --config config.yaml\n    ```\n\n    *   The server reads `config.yaml`.\n    *   It finds the `plugins.postgres` section.\n    *   It sees the `postgres://` scheme in the `uri` and knows to use a PostgreSQL plugin.\n    *   It uses the `path` field (`./easyrest-plugin-postgres`) to locate and execute the plugin binary.\n    *   The server then communicates with the plugin via RPC to handle requests to `/api/postgres/...`.\n\n**Using Environment Variables:**\n\n1. **Set Environment Variables:** Define the necessary `ER_` variables as shown previously.\n2. **Place the plugin binary:** The `easyrest-plugin-postgres` binary *must* be located either in the same directory as the `easyrest-server` binary or in a directory included in your system's `PATH` environment variable.\n3. **Run the server:**\n    ```bash\n    ./easyrest-server\n    ```\n    *   The server detects `ER_DB_PG` starting with `postgres://`.\n    *   It searches for an executable named `easyrest-plugin-postgres` in the current directory and in the system `PATH`.\n    *   If found, it executes the plugin binary and communicates via RPC.\n\n    **Limitation:** You cannot specify a custom path or name for the plugin binary when using only environment variables.\n\n---\n\n## Testing API Endpoints\n\n1. **Creating a JWT Token:**\n\n   EasyREST uses JWT for authentication. Use your preferred method to generate a token with the appropriate claims (including the subject claim specified by `ER_TOKEN_USER_SEARCH`). For example, you might generate a token using a script or an [online tool](https://jwt.io/) and then export it:\n\n   ```bash\n   export TOKEN=\"your_generated_jwt_token\"\n   ```\n\n2. **Calling the API:**\n\n   With the server running, you can test the endpoints. The API path depends on the name you gave the plugin instance in your `config.yaml` (e.g., `postgres`) or defaults to `postgres` if using `ER_DB_PG`. Assuming the name is `postgres`:\n\n   ```bash\n   # Fetch users\n   curl -H \"Authorization: Bearer $TOKEN\" \"http://localhost:8080/api/postgres/users/?select=id,name,created_at\"\n\n   # Create users (setup name from JWT-token 'sub' claim)\n   curl -X POST \"http://localhost:8080/api/postgres/users/\" \\\n     -H \"Authorization: Bearer $TOKEN\" \\\n     -H \"Content-Type: application/json\" \\\n     -d '[{\"name\": \"erctx.claims_sub\"},{\"name\": \"request.claims.sub\"}]'\n\n   # Call a function\n   curl -X POST \"http://localhost:8080/api/postgres/rpc/doSomething/\" \\\n     -H \"Authorization: Bearer $TOKEN\" \\\n     -H \"Content-Type: application/json\" \\\n     -d '{\"jsonParam\": \"test\"}'\n   ```\n\n---\n\n## Working with Tables, Views, and Context Variables\n\nThis plugin allows you to interact with your PostgreSQL database tables and views directly through the EasyREST API. It also provides a powerful mechanism for injecting context from the incoming request (like user information from a JWT) directly into your SQL session using PostgreSQL's `set_config` function.\n\n### Interacting with Tables and Views\n\nOnce the plugin is configured and connected to your database, EasyREST exposes API endpoints for your tables and views under the path `/api/{plugin_name}/{table_or_view_name}/` (e.g., `/api/postgres/users/`).\n\n-   **Read (SELECT):** Use `GET` requests. You can specify fields (`?select=col1,col2`), filters (`?where=...`), ordering (`?orderBy=...`), grouping (`?groupBy=...`), limit (`?limit=...`), and offset (`?offset=...`). Views are accessible via `GET` just like tables.\n-   **Create (INSERT):** Use `POST` requests with a JSON array of objects in the request body.\n-   **Update (UPDATE):** Use `PATCH` requests. Provide the data to update in the request body and specify the rows to update using `?where=...` query parameters.\n-   **Delete (DELETE):** Use `DELETE` requests, specifying rows to delete using `?where=...` query parameters.\n\n### Schema Introspection and Type Mapping\n\nThe plugin automatically introspects your database schema (tables and views) and makes it available via the `/api/{plugin_name}/schema` endpoint. This schema reflects the columns and their basic types.\n\nWhen the plugin reads the schema, it maps PostgreSQL data types to simpler, JSON-friendly types:\n\n| PostgreSQL Data Type Category | Example Types                               | Schema Type | Notes                                    |\n| :--------------------------- | :------------------------------------------ | :---------- | :--------------------------------------- |\n| Integer Types               | `INTEGER`, `SMALLINT`, `BIGINT`             | `integer`   |                                          |\n| Numeric Types              | `NUMERIC`, `DECIMAL`, `REAL`, `DOUBLE`      | `number`    |                                          |\n| Character Types            | `VARCHAR`, `CHAR`, `TEXT`                   | `string`    |                                          |\n| Binary Types               | `BYTEA`                                     | `string`    | Returned as base64 string                |\n| Date/Time Types            | `DATE`, `TIMESTAMP`, `TIME`                 | `string`    | Formatted as string (e.g., `YYYY-MM-DD`) |\n| Boolean                    | `BOOLEAN`                                   | `boolean`   |                                          |\n| JSON Types                 | `JSON`, `JSONB`                            | `string`    | Serialized as JSON string                |\n\n### Using Context Variables (`erctx.` / `request.`)\n\nThe plugin uses PostgreSQL's `set_config` function to set session variables for each request. These variables are available using `current_setting()` in your SQL code.\n\n**Example: Using Context in Functions**\n\n```sql\n-- Function that uses context variables\nCREATE OR REPLACE FUNCTION get_my_products()\nRETURNS JSON AS $$\nBEGIN\n    RETURN (\n        SELECT json_agg(row_to_json(p))\n        FROM (\n            SELECT id, sku, name, price\n            FROM products\n            WHERE created_by = current_setting('request.claims_sub', TRUE)\n        ) p\n    );\nEND;\n$$ LANGUAGE plpgsql;\n```\n\nCall this function via the API:\n```bash\ncurl -X POST \"http://localhost:8080/api/postgres/rpc/getMyProducts/\" \\\n  -H \"Authorization: Bearer $TOKEN\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{}'\n```\n\n---\n\n## License\n\nEasyREST PostgreSQL Plugin is licensed under the Apache License 2.0.  \nSee the file \"LICENSE\" for more information.\n\n© 2025 Sergei Kliuikov\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fonegreyonewhite%2Feasyrest-plugin-postgres","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fonegreyonewhite%2Feasyrest-plugin-postgres","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fonegreyonewhite%2Feasyrest-plugin-postgres/lists"}