{"id":34718443,"url":"https://github.com/tobilg/caddy-duckdb-module","last_synced_at":"2026-03-12T11:10:25.124Z","repository":{"id":326352283,"uuid":"1105170725","full_name":"tobilg/caddy-duckdb-module","owner":"tobilg","description":"A Caddy server module that provides a REST API for DuckDB database operations with built-in authentication and authorization.","archived":false,"fork":false,"pushed_at":"2025-11-27T18:22:47.000Z","size":144,"stargazers_count":39,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-11-30T02:27:24.507Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tobilg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-27T08:31:41.000Z","updated_at":"2025-11-29T22:52:26.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/tobilg/caddy-duckdb-module","commit_stats":null,"previous_names":["tobilg/caddy-duckdb-module"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/tobilg/caddy-duckdb-module","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tobilg%2Fcaddy-duckdb-module","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tobilg%2Fcaddy-duckdb-module/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tobilg%2Fcaddy-duckdb-module/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tobilg%2Fcaddy-duckdb-module/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tobilg","download_url":"https://codeload.github.com/tobilg/caddy-duckdb-module/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tobilg%2Fcaddy-duckdb-module/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28015427,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-24T02:00:07.193Z","response_time":83,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-12-25T01:25:59.827Z","updated_at":"2025-12-25T01:26:02.406Z","avatar_url":"https://github.com/tobilg.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Caddy DuckDB Module\n\nA Caddy server module that provides a REST API for DuckDB database operations with built-in authentication and authorization.\n\n## Features\n\n- **Dual Database Architecture**: Separate main database (file or in-memory) and internal auth database\n- **CRUD Operations**: RESTful API for Create, Read, Update, Delete operations on tables\n- **Raw SQL Queries**: Execute custom SQL queries with proper authorization\n- **Multi-Format Responses**: JSON, CSV, Parquet, Apache Arrow IPC\n- **Advanced Querying**: Pagination, sorting, and filtering\n- **Authentication**: API key-based authentication\n- **Authorization**: Role-based permissions at table level\n- **Transactional Writes**: All write operations are atomic\n- **SQL Injection Protection**: Query parameters and input validation\n- **Configurable Timeouts**: Query timeout protection\n\n## Quick Start\n\nGet up and running in under 2 minutes:\n\n```bash\n# Clone the repository\ngit clone https://github.com/tobilg/caddy-duckdb-module.git\ncd caddy-duckdb-module\n\n# Build the server and tools\nmake build-all\n\n# Initialize auth database and create an admin API key\nmake auth-init\nmake auth-add-key ROLE=admin\n\n# Start the server\nmake run\n```\n\nThe `auth-add-key` command will display your generated API key. Test the API:\n\n```bash\n# Replace YOUR_API_KEY with the key from setup output\ncurl -H \"X-API-Key: YOUR_API_KEY\" \\\n     -H \"Content-Type: application/json\" \\\n     -d '{\"sql\": \"SELECT 1 AS test\"}' \\\n     http://localhost:8080/duckdb/query\n```\n\nFor more end-to-end examples including filtering, pagination, updates, and deletes, see [EXAMPLE_QUERIES.md](EXAMPLE_QUERIES.md).\n\nFor Docker deployment, see [Docker](#docker) section below.\n\n## Building\n\n### Prerequisites\n\n- Go 1.24 or later\n- CGO enabled (required for DuckDB bindings)\n- C compiler (gcc, clang, or MSVC depending on platform)\n- xcaddy (for building Caddy with custom modules)\n\n### Important Notes\n\nThis module uses the official DuckDB Go driver (`github.com/duckdb/duckdb-go/v2`) which:\n- Requires CGO to be enabled\n- Downloads platform-specific binaries automatically during build\n- Supports Linux (amd64, arm64), macOS (amd64, arm64), and Windows (amd64)\n\n### Build with Go\n\nThe recommended way to build Caddy with the DuckDB module is using the provided build configuration:\n\n```bash\n# Clone the repository\ngit clone https://github.com/tobilg/caddy-duckdb-module.git\ncd caddy-duckdb-module\n\n# Download dependencies\ngo mod download\n\n# Build Caddy with DuckDB module\ngo build -o caddy ./cmd/caddy\n```\n\nThis will produce a `caddy` binary (~107MB) in the current directory with the DuckDB module included.\n\n### Alternative: Build with xcaddy\n\nYou can also use xcaddy for published versions:\n\n```bash\n# Install xcaddy\ngo install github.com/caddyserver/xcaddy/cmd/xcaddy@latest\n\n# Build from GitHub (CGO_ENABLED=1 is required!)\nCGO_ENABLED=1 xcaddy build --with github.com/tobilg/caddy-duckdb-module\n\n# Build from local source\nCGO_ENABLED=1 xcaddy build --with github.com/tobilg/caddy-duckdb-module=.\n```\n\n**Important:** You must set `CGO_ENABLED=1` when using xcaddy. Without it, the DuckDB C bindings won't compile and the build will fail with \"undefined: bindings.Type\" errors.\n\n### Module-Only Build (For Testing)\n\nTo verify the module compiles correctly without building the full Caddy binary:\n\n```bash\nCGO_ENABLED=1 go build\n```\n\nThis creates a package archive (not an executable) for testing purposes only.\n\n### Build Troubleshooting\n\n**Network Issues:**\nIf you encounter network errors downloading dependencies, ensure you have internet access. The DuckDB bindings will be downloaded automatically from GitHub on first build.\n\n**CGO Errors:**\nIf you get CGO-related errors, ensure:\n- CGO is enabled: `export CGO_ENABLED=1`\n- A C compiler is installed (gcc on Linux, clang on macOS, MSVC on Windows)\n\n**Platform-Specific Issues:**\n- **Linux**: Install build-essential: `apt-get install build-essential` (Debian/Ubuntu)\n- **macOS**: Install Xcode Command Line Tools: `xcode-select --install`\n- **Windows**: Install MinGW-w64 or use Visual Studio Build Tools\n\n## Configuration\n\n### Caddyfile\n\n```caddyfile\n:8080 {\n    route /duckdb/* {\n        duckdb {\n            # Main database path (optional, defaults to in-memory)\n            database_path /data/main.db\n\n            # Auth database path (required)\n            auth_database_path /data/auth.db\n\n            # Query timeout (default: 10s)\n            query_timeout 10s\n\n            # Max rows per page (default: 100)\n            max_rows_per_page 100\n\n            # Safety limit - max rows without pagination (default: 10000, 0 to disable)\n            absolute_max_rows 10000\n\n            # Number of threads (default: 4)\n            threads 4\n\n            # Access mode: read_only or read_write (default: read_write)\n            access_mode read_write\n\n            # Memory limit (optional, e.g., \"4GB\", \"512MB\". Default: 80% of RAM)\n            # memory_limit 4GB\n\n            # Enable object cache for faster repeated queries (optional, default: false)\n            # enable_object_cache true\n\n            # Temporary directory for spilling to disk (optional, uses system default if not set)\n            # temp_directory /tmp/duckdb-temp\n        }\n    }\n}\n```\n\n### JSON Configuration\n\n```json\n{\n  \"apps\": {\n    \"http\": {\n      \"servers\": {\n        \"srv0\": {\n          \"listen\": [\":8080\"],\n          \"routes\": [\n            {\n              \"match\": [{\"path\": [\"/duckdb/*\"]}],\n              \"handle\": [\n                {\n                  \"handler\": \"duckdb\",\n                  \"database_path\": \"/data/main.db\",\n                  \"auth_database_path\": \"/data/auth.db\",\n                  \"query_timeout\": \"10s\",\n                  \"max_rows_per_page\": 100,\n                  \"absolute_max_rows\": 10000,\n                  \"threads\": 4,\n                  \"access_mode\": \"read_write\",\n                  \"memory_limit\": \"4GB\",\n                  \"enable_object_cache\": true,\n                  \"temp_directory\": \"/tmp/duckdb-temp\"\n                }\n              ]\n            }\n          ]\n        }\n      }\n    }\n  }\n}\n```\n\n### Configuration Parameters\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `database_path` | string | `:memory:` | Path to main database file. Omit for in-memory database. |\n| `auth_database_path` | string | *required* | Path to authentication database (must be file-based). |\n| `query_timeout` | duration | `10s` | Maximum query execution time. |\n| `max_rows_per_page` | int | `100` | Default page size when pagination is used. |\n| `absolute_max_rows` | int | `10000` | Safety limit - max rows without pagination. Set to `0` to disable. |\n| `threads` | int | `4` | Number of threads for DuckDB query execution. |\n| `access_mode` | string | `read_write` | Database access mode: `read_only` or `read_write`. |\n| `memory_limit` | string | *80% of RAM* | Max memory DuckDB can use (e.g., `\"4GB\"`, `\"512MB\"`). Optional. |\n| `enable_object_cache` | bool | `false` | Enable DuckDB's object cache for faster repeated queries. Optional. |\n| `temp_directory` | string | *system default* | Directory for temporary files when spilling to disk. Optional. |\n\n**Performance Tuning:**\n- **`threads`**: Set to number of CPU cores for best performance\n- **`memory_limit`**: Prevent DuckDB from consuming too much memory\n- **`enable_object_cache`**: Useful for analytical workloads with repeated queries\n- **`temp_directory`**: Important for queries that exceed memory limits\n\n**Safety Features:**\n- **`absolute_max_rows`**: Prevents accidentally large responses when pagination is not specified\n- **`query_timeout`**: Protects against long-running queries\n\n## Docker\n\n### Pre-built Images\n\nOfficial Docker images are available on Docker Hub:\n\n```bash\ndocker pull tobilg/caddy-duckdb:latest\n```\n\nAvailable tags:\n- `latest` - Latest stable release from main branch\n- `x.y.z` - Specific version (e.g., `1.0.0`)\n- `x.y` - Minor version (e.g., `1.0`)\n\nSupported platforms: `linux/amd64`, `linux/arm64`\n\n### Quick Start with Docker\n\n```bash\n# Using pre-built image\ndocker run -d \\\n  --name caddy-duckdb \\\n  -p 8080:8080 \\\n  -v $(pwd)/data:/data \\\n  tobilg/caddy-duckdb:latest\n\n# Or build locally (~219MB)\ndocker build -t caddy-duckdb .\n\n# Create local data directory\nmkdir -p data\n\n# Initialize auth database and create API key (run locally)\nmake auth-init\nmake auth-add-key ROLE=admin\n# Save the displayed API key!\n\n# Start the container with local data directory mounted\ndocker run -d \\\n  --name caddy-duckdb \\\n  -p 8080:8080 \\\n  -v $(pwd)/data:/data \\\n  caddy-duckdb\n\n# Verify it's running\ncurl http://localhost:8080/duckdb/health\n# {\"status\":\"ok\"}\n\n# Test with your API key\ncurl -H \"X-API-Key: YOUR_API_KEY\" \\\n     -H \"Content-Type: application/json\" \\\n     -d '{\"sql\": \"SELECT 1 AS test\"}' \\\n     http://localhost:8080/duckdb/query\n```\n\n### Using Docker Compose\n\n```bash\n# Create local data directory, init auth DB, and create API key\nmkdir -p data\nmake auth-init\nmake auth-add-key ROLE=admin\n\n# Start with docker-compose\ndocker-compose up -d\n\n# View logs\ndocker-compose logs -f\n\n# Stop\ndocker-compose down\n```\n\n### Health Check\n\nThe container includes a health check endpoint at `/{DUCKDB_ROUTE_PREFIX}/health` (default: `/duckdb/health`):\n\n```bash\ncurl http://localhost:8080/duckdb/health\n# {\"status\":\"ok\"}\n```\n\nThis endpoint requires no authentication and is used by Docker's HEALTHCHECK.\n\n### Environment Variables\n\nAll settings can be configured via environment variables:\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `DUCKDB_PORT` | `8080` | Server port |\n| `DUCKDB_ROUTE_PREFIX` | `/duckdb` | API route prefix |\n| `DUCKDB_DATABASE_PATH` | `/data/main.db` | Main database path |\n| `DUCKDB_AUTH_DATABASE_PATH` | `/data/auth.db` | Auth database path |\n| `DUCKDB_THREADS` | `4` | Number of query threads |\n| `DUCKDB_MEMORY_LIMIT` | *(80% RAM)* | Memory limit (e.g., `4GB`) |\n| `DUCKDB_QUERY_TIMEOUT` | `10s` | Query timeout |\n| `DUCKDB_ACCESS_MODE` | `read_write` | `read_only` or `read_write` |\n| `DUCKDB_MAX_ROWS_PER_PAGE` | `100` | Default pagination size |\n| `DUCKDB_ABSOLUTE_MAX_ROWS` | `10000` | Max rows without pagination |\n\nExample with custom settings:\n\n```bash\ndocker run -d \\\n  --name caddy-duckdb \\\n  -p 8080:8080 \\\n  -e DUCKDB_THREADS=8 \\\n  -e DUCKDB_MEMORY_LIMIT=4GB \\\n  -e DUCKDB_QUERY_TIMEOUT=30s \\\n  -e DUCKDB_ROUTE_PREFIX=/api/v1 \\\n  -v $(pwd)/data:/data \\\n  caddy-duckdb\n```\n\nWith a custom route prefix, endpoints become `/api/v1/health`, `/api/v1/query`, `/api/v1/api/{table}`, etc.\n\n### Custom Caddyfile\n\nMount your own Caddyfile for advanced configuration:\n\n```bash\ndocker run -d \\\n  --name caddy-duckdb \\\n  -p 8080:8080 \\\n  -v $(pwd)/data:/data \\\n  -v $(pwd)/my-caddyfile:/etc/caddy/Caddyfile:ro \\\n  caddy-duckdb\n```\n\n## Quick Start Guide\n\n### Creating Tables\n\nUse the **raw SQL query endpoint** to create tables:\n\n```bash\ncurl -X POST http://localhost:8080/duckdb/query \\\n  -H \"X-API-Key: your-api-key\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"sql\": \"CREATE TABLE users (id INTEGER PRIMARY KEY, name VARCHAR, email VARCHAR, age INTEGER)\"\n  }'\n```\n\n**Requirements:**\n- Requires `can_query` permission (admin role by default)\n- Endpoint: `POST /duckdb/query`\n\n### Adding Records\n\nYou have **two options**:\n\n#### Option 1: CRUD API (Recommended for simple inserts)\n\n```bash\ncurl -X POST http://localhost:8080/duckdb/api/users \\\n  -H \"X-API-Key: your-api-key\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"name\": \"John Doe\",\n    \"email\": \"john@example.com\",\n    \"age\": 30\n  }'\n```\n\n**Response:**\n```json\n{\n  \"success\": true,\n  \"rows_affected\": 1\n}\n```\n\n- Requires `can_create` permission\n- Endpoint: `POST /duckdb/api/{table}`\n- Automatically handles parameterized queries\n\n#### Option 2: Raw SQL Query (For complex inserts)\n\n```bash\ncurl -X POST http://localhost:8080/duckdb/query \\\n  -H \"X-API-Key: your-api-key\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"sql\": \"INSERT INTO users (name, email, age) VALUES (?, ?, ?)\",\n    \"params\": [\"Jane Doe\", \"jane@example.com\", 28]\n  }'\n```\n\n- Requires `can_query` permission\n- Useful for batch inserts or complex SQL statements\n- Supports parameterized queries for safety\n\n### Key Points\n\n- **Authentication**: All endpoints require the `X-API-Key` header\n- **Permissions**: Role-based access control (admin, editor, reader)\n- **Security**: Uses parameterized queries and input validation to prevent SQL injection\n- **Transactions**: All write operations use transactions for atomicity\n\nThe CRUD API (`/duckdb/api/{table}`) is simpler for basic operations, while the raw SQL endpoint (`/duckdb/query`) gives you full DuckDB SQL power for complex operations.\n\n## Authentication \u0026 Authorization\n\n### Auth Database Setup\n\nThe auth database must be initialized **before** starting the server. Use the `auth-db` CLI tool:\n\n```bash\n# Build the CLI tool first (or use make targets which build automatically)\nmake build-tools\n\n# Initialize auth database with default roles (admin, editor, reader)\nmake auth-init\n\n# Or use the CLI directly\n./tools/auth-db init -d /path/to/auth.db\n\n# View all available commands\n./tools/auth-db --help\n```\n\n### Built-in Roles\n\n- **admin**: Full CRUD access + raw SQL queries on all tables\n- **editor**: CRUD access on all tables, no raw SQL\n- **reader**: Read-only access on all tables\n\n### Creating API Keys\n\nUse the `auth-db` CLI tool or Make targets:\n\n```bash\n# Using Make (recommended)\nmake auth-add-key ROLE=admin\n\n# Using CLI directly\n./tools/auth-db key add -d /path/to/auth.db -r admin\n\n# With custom key (instead of auto-generated)\n./tools/auth-db key add -d /path/to/auth.db -r admin -k my-secret-key\n\n# With expiration\n./tools/auth-db key add -d /path/to/auth.db -r admin -e 2025-12-31T23:59:59Z\n```\n\n### Managing API Keys\n\n```bash\n# List all API keys\nmake auth-list-keys\n\n# Remove an API key\n./tools/auth-db key remove -d /path/to/auth.db -k \u003capi-key\u003e\n```\n\n### Custom Roles\n\n```bash\n# Create a custom role\nmake auth-add-role NAME=analyst DESC=\"Data analyst with read and query access\"\n\n# Or using CLI\n./tools/auth-db role add -d /path/to/auth.db -n analyst --desc \"Data analyst\"\n\n# Grant permissions (operations: c=create, r=read, u=update, d=delete, q=query)\nmake auth-add-perm ROLE=analyst TABLE=reports OPS=r,q\n\n# Or using CLI\n./tools/auth-db permission add -d /path/to/auth.db -r analyst -t reports -o r,q\n\n# Grant all CRUD operations (no raw query)\n./tools/auth-db permission add -d /path/to/auth.db -r analyst -t \"*\" -o crud\n\n# List all roles and permissions\nmake auth-list-roles\nmake auth-list-perms\n```\n\n### Auth Database Info\n\n```bash\n# Show auth database statistics\nmake auth-info\n```\n\n## API Endpoints\n\n### CRUD Operations\n\nBase path: `/duckdb/api/{table}`\n\nAll requests require the `X-API-Key` header.\n\n#### Create (POST)\n\n```bash\ncurl -X POST http://localhost:8080/duckdb/api/users \\\n  -H \"X-API-Key: your-api-key\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"name\": \"John Doe\",\n    \"email\": \"john@example.com\",\n    \"age\": 30\n  }'\n```\n\nResponse:\n```json\n{\n  \"success\": true,\n  \"rows_affected\": 1\n}\n```\n\n#### Read (GET)\n\n```bash\n# Basic query\ncurl http://localhost:8080/duckdb/api/users \\\n  -H \"X-API-Key: your-api-key\"\n\n# With pagination\ncurl \"http://localhost:8080/duckdb/api/users?page=1\u0026limit=50\" \\\n  -H \"X-API-Key: your-api-key\"\n\n# With filters\ncurl \"http://localhost:8080/duckdb/api/users?filter=age:gt:18,status:eq:active\" \\\n  -H \"X-API-Key: your-api-key\"\n\n# With sorting\ncurl \"http://localhost:8080/duckdb/api/users?sort=created_at:desc,name:asc\" \\\n  -H \"X-API-Key: your-api-key\"\n\n# Combined\ncurl \"http://localhost:8080/duckdb/api/users?page=1\u0026limit=20\u0026filter=age:gt:18\u0026sort=name:asc\" \\\n  -H \"X-API-Key: your-api-key\"\n```\n\nResponse:\n```json\n{\n  \"data\": [\n    {\n      \"id\": 1,\n      \"name\": \"John Doe\",\n      \"email\": \"john@example.com\",\n      \"age\": 30\n    }\n  ],\n  \"pagination\": {\n    \"page\": 1,\n    \"limit\": 20,\n    \"total_rows\": 1,\n    \"total_pages\": 1\n  }\n}\n```\n\n##### HATEOAS Navigation Links\n\nAdd `links=true` to include navigation links in paginated responses:\n\n```bash\ncurl \"http://localhost:8080/duckdb/api/users?page=2\u0026limit=10\u0026links=true\" \\\n  -H \"X-API-Key: your-api-key\"\n```\n\nResponse with links:\n```json\n{\n  \"data\": [...],\n  \"pagination\": {\n    \"page\": 2,\n    \"limit\": 10,\n    \"total_rows\": 100,\n    \"total_pages\": 10\n  },\n  \"_links\": {\n    \"self\": \"/duckdb/api/users?limit=10\u0026page=2\u0026links=true\",\n    \"first\": \"/duckdb/api/users?limit=10\u0026page=1\u0026links=true\",\n    \"prev\": \"/duckdb/api/users?limit=10\u0026page=1\u0026links=true\",\n    \"next\": \"/duckdb/api/users?limit=10\u0026page=3\u0026links=true\",\n    \"last\": \"/duckdb/api/users?limit=10\u0026page=10\u0026links=true\"\n  }\n}\n```\n\nLink types:\n- `self`: Current page URL (always included)\n- `first`: First page URL (always included)\n- `last`: Last page URL (when total_pages \u003e 0)\n- `prev`: Previous page URL (when page \u003e 1)\n- `next`: Next page URL (when page \u003c total_pages)\n\n##### Filter Operators\n\n- `eq`: Equal\n- `ne`: Not equal\n- `gt`: Greater than\n- `gte`: Greater than or equal\n- `lt`: Less than\n- `lte`: Less than or equal\n- `like`: SQL LIKE pattern\n- `in`: IN clause (use pipe `|` to separate values)\n\nExample: `filter=status:in:active|pending`\n\n#### Update (PUT)\n\n```bash\ncurl -X PUT http://localhost:8080/duckdb/api/users \\\n  -H \"X-API-Key: your-api-key\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"where\": {\n      \"id\": 1\n    },\n    \"set\": {\n      \"age\": 31,\n      \"updated_at\": \"2025-01-01T00:00:00Z\"\n    }\n  }'\n```\n\nResponse:\n```json\n{\n  \"success\": true,\n  \"rows_affected\": 1\n}\n```\n\n#### Delete (DELETE)\n\n```bash\ncurl -X DELETE \"http://localhost:8080/duckdb/api/users?where=id:eq:1\" \\\n  -H \"X-API-Key: your-api-key\"\n```\n\nResponse:\n```json\n{\n  \"success\": true,\n  \"rows_affected\": 1\n}\n```\n\n### Raw SQL Queries\n\nEndpoint: `/duckdb/query` — Requires `can_query` permission (admin role by default).\n\n**POST Method** (supports parameterized queries):\n\n```bash\ncurl -X POST http://localhost:8080/duckdb/query \\\n  -H \"X-API-Key: your-api-key\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"sql\": \"SELECT * FROM users WHERE age \u003e ?\", \"params\": [18]}'\n```\n\n**GET Method** (read-only, bookmarkable):\n\n```bash\n# Pattern: /duckdb/query/{urlEncodedSQL}/result.{format}\ncurl \"http://localhost:8080/duckdb/query/SELECT%20*%20FROM%20users/result.json\" \\\n  -H \"X-API-Key: your-api-key\"\n\n# With jq for URL encoding\ncurl \"http://localhost:8080/duckdb/query/$(echo 'SELECT * FROM users' | jq -sRr @uri)/result.csv\" \\\n  -H \"X-API-Key: your-api-key\" -o users.csv\n```\n\nGET is limited to SELECT, SHOW, DESCRIBE, EXPLAIN queries.\n\n**Response** (same format as CRUD API):\n```json\n{\n  \"data\": [\n    {\"id\": 1, \"name\": \"John Doe\", \"email\": \"john@example.com\", \"age\": 30}\n  ]\n}\n```\n\n### Response Formats\n\nBoth `/api` and `/query` endpoints support multiple output formats:\n\n| Format | Accept Header | File module | Best For |\n|--------|---------------|----------------|----------|\n| JSON | `application/json` | `.json` | Web APIs, debugging |\n| CSV | `text/csv` | `.csv` | Spreadsheets, simple exports |\n| Parquet | `application/parquet` | `.parquet` | Analytics, data lakes (5-10x smaller) |\n| Arrow IPC | `application/vnd.apache.arrow.stream` | `.arrow` | Data pipelines, zero-copy transfers |\n\n```bash\n# Using Accept header (CRUD or POST query)\ncurl http://localhost:8080/duckdb/api/users -H \"X-API-Key: key\" -H \"Accept: text/csv\"\ncurl http://localhost:8080/duckdb/api/users -H \"X-API-Key: key\" -H \"Accept: application/parquet\" -o data.parquet\n\n# Using file module (GET query)\ncurl \"http://localhost:8080/duckdb/query/SELECT%20*%20FROM%20users/result.parquet\" -H \"X-API-Key: key\" -o data.parquet\n```\n\n**Reading exported files in Python:**\n```python\nimport pyarrow.parquet as pq\nimport pyarrow.ipc as ipc\n\n# Parquet\ndf = pq.read_table('data.parquet').to_pandas()\n\n# Arrow IPC\nwith open('data.arrow', 'rb') as f:\n    df = ipc.open_stream(f).read_all().to_pandas()\n```\n\n## Security Features\n\n1. **SQL Injection Protection**: All queries use parameterized statements\n2. **Input Validation**: Table and column names are sanitized\n3. **Internal Table Protection**: Auth tables cannot be accessed via API (hardened with SQL comment stripping and word-boundary matching)\n4. **API Key Hashing**: Keys are stored as bcrypt hashes\n5. **Transactional Writes**: All modifications are atomic\n6. **Query Timeouts**: Prevents long-running queries\n7. **Role-Based Access**: Fine-grained permissions at table level\n8. **Request ID Tracing**: All requests include a unique request ID for distributed tracing and log correlation\n\n### Rate Limiting\n\nRate limiting is intentionally **not** implemented in this module. Caddy has excellent rate limiting plugins that should be used instead:\n\n```caddyfile\n:8080 {\n    # Use Caddy's rate_limit directive (requires caddy-rate-limit plugin)\n    rate_limit {\n        zone duckdb_api {\n            key {remote_host}\n            events 100\n            window 1m\n        }\n    }\n\n    route /duckdb/* {\n        duckdb {\n            # ... your config\n        }\n    }\n}\n```\n\nRecommended plugins:\n- [caddy-rate-limit](https://github.com/mholt/caddy-ratelimit) - Token bucket rate limiting\n- [caddy-security](https://github.com/greenpau/caddy-security) - Comprehensive security including rate limiting\n\nThis separation of concerns allows you to configure rate limiting consistently across all your Caddy routes, not just the DuckDB endpoints.\n\n### Request ID Tracing\n\nAll API requests include a unique request ID for distributed tracing and log correlation:\n\n**How it works:**\n- If you provide an `X-Request-ID` header, the API uses your ID\n- If no header is provided, a UUID v4 is automatically generated\n- The request ID is always returned in the `X-Request-ID` response header\n- All log entries include the request ID for correlation\n\n**Example:**\n```bash\ncurl -H \"X-API-Key: your-api-key\" \\\n     -H \"X-Request-ID: my-trace-123\" \\\n     http://localhost:8080/duckdb/api/users\n```\n\nResponse headers:\n```\nX-Request-ID: my-trace-123\n```\n\nThe request ID is only in the response header, not in the JSON body. This keeps response payloads clean while still providing full traceability.\n\nThis enables you to:\n- Correlate client requests with server-side logs\n- Track requests across distributed systems\n- Debug issues by searching logs with the request ID\n- Build observability dashboards with request tracing\n\n### OpenAPI Specification\n\nA complete OpenAPI 3.0 specification is available at `/duckdb/openapi.json`. This endpoint is publicly accessible (no authentication required) to allow easy access to API documentation.\n\n```bash\n# Access the OpenAPI specification\ncurl http://localhost:8080/duckdb/openapi.json\n```\n\n**Features:**\n- Complete API documentation for all CRUD and query endpoints\n- Request/response schema definitions with examples\n- Security scheme documentation (API key authentication)\n- All filter operators, pagination parameters, and response formats documented\n- HATEOAS links and request ID tracing documented\n\n**Usage with API tools:**\n\n```bash\n# Download the spec for use with Swagger UI\ncurl http://localhost:8080/duckdb/openapi.json -o openapi.json\n\n# Import into Postman, Insomnia, or other API tools\n# Or generate client SDKs using openapi-generator-cli\nnpx @openapitools/openapi-generator-cli generate -i openapi.json -g python -o ./python-client\n```\n\n**Integration with Swagger UI:**\n\nYou can serve Swagger UI alongside your API for interactive documentation:\n\n```caddyfile\n:8080 {\n    # Serve Swagger UI (download swagger-ui-dist to /var/www/swagger)\n    handle /swagger/* {\n        root * /var/www/swagger\n        file_server\n    }\n\n    route /duckdb/* {\n        duckdb {\n            database_path /data/main.db\n            auth_database_path /data/auth.db\n        }\n    }\n}\n```\n\n## Example Usage\n\n### 1. Setup Auth Database\n\nInitialize the auth database and create an API key before starting the server:\n\n```bash\n# Initialize auth database with default roles\nmake auth-init\n\n# Create an admin API key\nmake auth-add-key ROLE=admin\n# Output: ✓ API key created successfully!\n#         API Key: \u003cyour-generated-key\u003e\n#         Role: admin\n```\n\n### 2. Start the Server\n\n```bash\nmake run\n```\n\n### 3. Create Tables\n\n```bash\ncurl -X POST http://localhost:8080/duckdb/query \\\n  -H \"X-API-Key: my-api-key-id\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"sql\": \"CREATE TABLE users (id INTEGER PRIMARY KEY, name VARCHAR, email VARCHAR, age INTEGER, created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP)\"\n  }'\n```\n\n### 4. Use CRUD Operations\n\nSee the API Endpoints section above for examples.\n\n## Development\n\n### Project Structure\n\n```\ncaddy-duckdb-module/\n├── module.go              # Main Caddy module\n├── config.go              # Configuration structs\n├── database/\n│   ├── manager.go         # Database connection manager\n│   └── operations.go      # CRUD operations\n├── auth/\n│   ├── models.go          # Auth data structures\n│   ├── authorizer.go      # Authorization logic\n│   └── middleware.go      # Auth middleware\n├── handlers/\n│   ├── crud.go            # CRUD handlers\n│   ├── query.go           # Query handler\n│   ├── params.go          # Parameter parsing\n│   └── openapi.go         # OpenAPI 3.0 specification handler\n├── formats/\n│   ├── json.go            # JSON formatter\n│   ├── csv.go             # CSV formatter\n│   ├── parquet.go         # Apache Parquet formatter\n│   └── arrow.go           # Apache Arrow IPC formatter\n└── examples/\n    └── Caddyfile          # Example configuration\n```\n\n### Make Targets\n\nThe project includes a Makefile for common development tasks:\n\n| Target | Description |\n|--------|-------------|\n| `make setup` | Full setup: check prerequisites, build, init auth DB |\n| `make build` | Build the Caddy binary |\n| `make build-tools` | Build the auth-db CLI tool |\n| `make run` | Build and run with example Caddyfile |\n| `make test` | Run all tests |\n| `make fmt` | Format code |\n| `make vet` | Run go vet |\n| `make lint` | Run fmt + vet |\n| `make clean` | Remove build artifacts |\n| `make docker-build` | Build Docker image |\n| `make docker-run` | Run with docker-compose |\n| `make help` | Show all available targets |\n\n**Auth Database Management:**\n\n| Target | Description |\n|--------|-------------|\n| `make auth-init` | Initialize auth database with default roles |\n| `make auth-add-key ROLE=\u003crole\u003e` | Add a new API key for a role |\n| `make auth-remove-key KEY=\u003ckey\u003e` | Remove an API key |\n| `make auth-list-keys` | List all API keys |\n| `make auth-add-role NAME=\u003cname\u003e` | Add a new role |\n| `make auth-remove-role NAME=\u003cname\u003e [FORCE=1]` | Remove a role |\n| `make auth-list-roles` | List all roles |\n| `make auth-add-perm ROLE=\u003crole\u003e TABLE=\u003ctable\u003e OPS=\u003cops\u003e` | Add permission |\n| `make auth-remove-perm ROLE=\u003crole\u003e TABLE=\u003ctable\u003e` | Remove a permission |\n| `make auth-list-perms` | List all permissions |\n| `make auth-info` | Show auth database statistics |\n\n### Running Tests\n\n```bash\nmake test\n# or directly:\ngo test ./...\n```\n\n## Concurrency and Multi-User Support\n\n### Concurrent Operations\n\nThis module is designed for **single-process concurrent access** and handles multiple simultaneous requests efficiently:\n\n**What Works Well:**\n- **Concurrent Reads**: Multiple users can read simultaneously without blocking each other\n- **Concurrent Inserts**: Multiple users can insert records into the same or different tables\n- **Mixed Read/Write**: Reads are never blocked by writes thanks to DuckDB's MVCC (Multi-Version Concurrency Control)\n- **Connection Pooling**: Configured to support `threads * 2` concurrent connections for optimal throughput\n\n**Transaction Conflict Handling:**\n- Write operations (INSERT/UPDATE/DELETE) automatically retry on conflicts with exponential backoff (up to 3 attempts)\n- When multiple users update the same row simultaneously, one succeeds and others retry automatically\n- Conflicts are rare for typical workloads but handled gracefully when they occur\n\n### Important Limitations\n\n**Single-Process Only:**\nDuckDB is designed for single-process access. This module works within one Caddy server instance but **does not support**:\n- Multiple Caddy instances writing to the same database file\n- Distributed deployments with shared database files\n- Multi-process concurrent writes\n\n**For Production Multi-Instance Deployments:**\nIf you need to run multiple Caddy instances:\n1. Use **read-only replicas**: Configure additional instances with `access_mode read_only`\n2. Use a **single writer** instance for all write operations\n3. Consider **horizontal partitioning**: Different instances handle different databases\n4. Use **external databases**: For true multi-writer scenarios, delegate writes to PostgreSQL, MySQL, or similar RDBMS\n\n### Performance Characteristics\n\n- **Read Performance**: Excellent - scales linearly with configured threads\n- **Write Performance**: Good for typical web workloads with automatic conflict resolution\n- **High-Contention Scenarios**: If many users frequently update the same rows, consider application-level locking or optimistic locking patterns\n\n## Limitations\n\n- **Multi-Process Writes**: Not supported - only one Caddy instance can write to a database file\n- Internal auth tables (`api_keys`, `roles`, `permissions`) cannot be queried via the API\n- DELETE operations require a WHERE clause for safety\n- Network connectivity required for initial dependency download\n\n## Contributing\n\nContributions are welcome! Please:\n\n1. Fork the repository\n2. Create a feature branch\n3. Make your changes\n4. Add tests\n5. Submit a pull request\n\n## License\n\n[MIT License](LICENSE)\n\n## Credits\n\nBuilt with:\n- [Caddy](https://caddyserver.com/)\n- [DuckDB](https://duckdb.org/)\n- [duckdb-go](https://github.com/duckdb/duckdb-go)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftobilg%2Fcaddy-duckdb-module","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftobilg%2Fcaddy-duckdb-module","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftobilg%2Fcaddy-duckdb-module/lists"}