{"id":24826663,"url":"https://github.com/DataZooDE/flapi","last_synced_at":"2025-10-14T00:30:38.121Z","repository":{"id":259683305,"uuid":"864915169","full_name":"DataZooDE/flapi","owner":"DataZooDE","description":"API Framework heavily relying on the power of DuckDB and DuckDB extensions. Ready to build performant and cost-efficient APIs on top of BigQuery or Snowflake  for AI Agents and Data Apps","archived":false,"fork":false,"pushed_at":"2025-06-19T07:27:27.000Z","size":1267,"stargazers_count":31,"open_issues_count":1,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-06-19T08:20:28.514Z","etag":null,"topics":["api","cpp","duckdb","duckdb-extension","openapi3","rest"],"latest_commit_sha":null,"homepage":"https://flapi.io","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DataZooDE.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-09-29T14:06:01.000Z","updated_at":"2025-06-19T07:27:31.000Z","dependencies_parsed_at":"2024-10-27T10:23:14.770Z","dependency_job_id":"cab5afaa-8640-4d45-bd3b-24e167368f85","html_url":"https://github.com/DataZooDE/flapi","commit_stats":null,"previous_names":["datazoode/flapi"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/DataZooDE/flapi","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataZooDE%2Fflapi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataZooDE%2Fflapi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataZooDE%2Fflapi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataZooDE%2Fflapi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DataZooDE","download_url":"https://codeload.github.com/DataZooDE/flapi/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataZooDE%2Fflapi/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279017353,"owners_count":26086052,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-13T02:00:06.723Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","cpp","duckdb","duckdb-extension","openapi3","rest"],"created_at":"2025-01-30T21:02:35.059Z","updated_at":"2025-10-14T00:30:38.115Z","avatar_url":"https://github.com/DataZooDE.png","language":"C++","readme":"# flAPI: Instant SQL based APIs\n\nflAPI is a powerful service that automatically generates read-only APIs for datasets by utilizing SQL templates. Built on top of [DuckDB](https://duckdb.org/) and leveraging its SQL engine and extension ecosystem, flAPI offers a seamless way to connect to various data sources and expose them as RESTful APIs.\n\n![overview of flAPI](https://i.imgur.com/m7UVZlR.png)\n\n## ⚡ Features\n\n- **Automatic API Generation**: Create APIs for your datasets without coding\n- **MCP (Model Context Protocol) Support**: Declarative creation of AI tools alongside REST endpoints\n- **Multiple Data Sources**: Connect to [BigQuery](https://github.com/hafenkran/duckdb-bigquery), SAP ERP \u0026 BW (via [ERPL](https://github.com/datazoode/erpl)), Parquet, [Iceberg](https://github.com/duckdb/duckdb_iceberg), [Postgres](https://github.com/duckdb/postgres_scanner), [MySQL](https://github.com/duckdb/duckdb_mysql), and more\n- **SQL Templates**: Use Mustache-like syntax for dynamic queries\n- **Caching**: DuckLake-backed cache with full refresh and incremental sync\n- **Security**: Implement row-level and column-level security with ease\n- **Easy deployment**: Deploy flAPI with a single binary file\n\n## 🛠 Quick Start\nThe easiest way to get started with flAPI is to use the pre-built docker image.\n\n#### 1. Pull the docker image from the Github Container Registry:\n\n```bash\n\u003e docker pull ghcr.io/datazoode/flapi:latest\n```\n\n\nThe image is pretty small and mainly contains the flAPI binary which is statically linked against [DuckDB v1.4.1](https://github.com/duckdb/duckdb/releases/tag/v1.4.1). Details about the docker image can be found in the [Dockerfile](./docker/development/Dockerfile).\n\n#### 2. Run flAPI:\nOnce you have downloaded the binary, you can run flAPI by executing the following command:\n\n```\n\u003e docker run -it --rm -p 8080:8080 -p 8081:8081 -v $(pwd)/examples/:/config ghcr.io/datazoode/flapi -c /config/flapi.yaml\n```\n\nThe different arguments in this docker command are:\n- `-it --rm`: Run the container in interactive mode and remove it after the process has finished\n- `-p 8080:8080`: Exposes port 8080 of the container to the host, this makes the REST API available at `http://localhost:8080`\n- `-p 8081:8081`: Exposes port 8081 for the MCP server (when enabled)\n- `-v $(pwd)/examples/:/config`: This mounts the local `examples` directory to the `/config` directory in the container, this is where the flAPI configuration file\nis expected to be found.\n- `ghcr.io/datazoode/flapi`: The docker image to use\n- `-c /config/flapi.yaml`: This is an argument to the flAPI application which tells it to use the `flapi.yaml` file in the `/config` directory as the configuration file.\n\n#### 2.1 Enable MCP Support:\nTo enable MCP support, you can either:\n\n**Option A: Use the command line flag**\n```\n\u003e docker run -it --rm -p 8080:8080 -p 8081:8081 -v $(pwd)/examples/:/config ghcr.io/datazoode/flapi -c /config/flapi.yaml --enable-mcp\n```\n\n**Option B: Configure in flapi.yaml**\n```yaml\nmcp:\n  enabled: true\n  port: 8081\n  # ... other MCP configuration\n```\n\n#### 3.1 Test the API server:\nIf everything is set up correctly, you should be able to access the API at the URL specified in the configuration file.\n\n```bash\n\u003e curl 'http://localhost:8080/'\n\n\n         ___\n     ___( o)\u003e   Welcome to\n     \\ \u003c_. )    flAPI\n      `---'    \n\n    Fast and Flexible API Framework\n    powered by DuckDB\n```\n\n#### 3.2 Get an overview of the available endpoints:\nThe flAPI server creates embedded Swagger UI at which provides an overview of the available endpoints and allows you to test them. It can be found at\n\n[`\u003e http://localhost:8080/doc`](http://localhost:8080/doc)\n\nYou should see the familiar Swagger UI page:\n\n![flAPI Swagger UI](https://i.imgur.com/HqjHMlA.png)\n\nThe raw yaml [Swagger 2.0](https://swagger.io/specification/) is also available at [`http://localhost:8080/doc.yaml`](http://localhost:8080/doc.yaml)\n\n#### 3.3 Test the MCP server:\nIf MCP is enabled, you can test the MCP server as well:\n\n```bash\n# Check MCP server health\n\u003e curl 'http://localhost:8081/mcp/health'\n\n{\"status\":\"healthy\",\"server\":\"flapi-mcp-server\",\"version\":\"0.3.0\",\"protocol_version\":\"2024-11-05\",\"tools_count\":0}\n\n# Initialize MCP connection\n\u003e curl -X POST http://localhost:8081/mcp/jsonrpc \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"jsonrpc\": \"2.0\", \"id\": 1, \"method\": \"initialize\"}'\n\n# List available tools\n\u003e curl -X POST http://localhost:8081/mcp/jsonrpc \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"jsonrpc\": \"2.0\", \"id\": 2, \"method\": \"tools/list\"}'\n```\n\n## 🤖 MCP (Model Context Protocol) Support\n\nflAPI now supports the **Model Context Protocol (MCP)** in a **unified configuration approach**. Every flAPI instance automatically runs both a REST API server and an MCP server concurrently, allowing you to create AI tools alongside your REST endpoints using the same configuration files and SQL templates.\n\n### Key Features\n\n- **Unified Configuration**: Single YAML files can define REST endpoints, MCP tools, and MCP resources\n- **Automatic Detection**: Configuration type is determined by presence of `url-path` (REST), `mcp-tool` (MCP tool), or `mcp-resource` (MCP resource)\n- **Shared Components**: MCP tools and resources use the same SQL templates, parameter validation, authentication, and caching as REST endpoints\n- **Concurrent Servers**: REST API (port 8080) and MCP server (port 8081) run simultaneously\n- **Declarative Definition**: Define everything using YAML configuration with SQL templatestocol\n- **Tool Discovery**: Automatic tool discovery and schema generation\n- **Security Integration**: Reuse existing authentication, rate limiting, and caching features\n\n### MCP Endpoints\n\n- `POST /mcp/jsonrpc` - Main JSON-RPC endpoint for tool calls\n- `GET /mcp/health` - Health check endpoint\n\n### Unified Configuration\n\n**MCP is now automatically enabled** - no separate configuration needed! Every flAPI instance runs both REST API and MCP servers concurrently.\n\nConfiguration files can define multiple entity types:\n\n#### REST Endpoint + MCP Tool (Unified)\n\n```yaml\n# Single configuration file serves as BOTH REST endpoint AND MCP tool\nurl-path: /customers/                    # Makes this a REST endpoint\nmcp-tool:                                # Also makes this an MCP tool\n  name: get_customers\n  description: Retrieve customer information by ID\n  result_mime_type: application/json\n\nrequest:\n  - field-name: id\n    field-in: query\n    description: Customer ID\n    required: false\n    validators:\n      - type: int\n        min: 1\n        max: 1000000\n        preventSqlInjection: true\n\ntemplate-source: customers.sql\nconnection: [customers-parquet]\n\nrate-limit:\n  enabled: true\n  max: 100\n  interval: 60\n\nauth:\n  enabled: true\n  type: basic\n  users:\n    - username: admin\n      password: secret\n      roles: [admin]\n```\n\n#### MCP Resource Only\n\n```yaml\n# MCP Resource example\nmcp-resource:\n  name: customer_schema\n  description: Customer database schema definition\n  mime_type: application/json\n\ntemplate-source: customer-schema.sql\nconnection: [customers-parquet]\n```\n\n### Using MCP Tools\n\nOnce MCP is enabled, you can interact with tools using JSON-RPC 2.0:\n\n```bash\n# Check MCP server health\ncurl 'http://localhost:8081/mcp/health'\n\n# Initialize MCP connection\ncurl -X POST http://localhost:8081/mcp/jsonrpc \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"jsonrpc\": \"2.0\", \"id\": 1, \"method\": \"initialize\"}'\n\n# List available tools (discovered from unified configuration)\ncurl -X POST http://localhost:8081/mcp/jsonrpc \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"jsonrpc\": \"2.0\", \"id\": 2, \"method\": \"tools/list\"}'\n\n# Call a tool (same SQL template used for both REST and MCP)\ncurl -X POST http://localhost:8081/mcp/jsonrpc \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"jsonrpc\": \"2.0\", \"id\": 3, \"method\": \"tools/call\", \"params\": {\"name\": \"get_customers\", \"arguments\": {\"id\": \"123\"}}}'\n```\n\n## 🎓 Example\n\nHere's a simple example of how to create an API endpoint using flAPI:\n\n### 1. Create a basic flAPI configuration\nflAPI uses the popular [YAML](https://yaml.org/) format to configure the API endpoints. A basic configuration file looks like this:\n\n```yaml\nproject_name: example-flapi-project\nproject_description: An example flAPI project demonstrating various configuration options\ntemplate:\n  path: './sqls'            # The path where SQL templates and API endpoint configurations are stored\n  environment-whitelist:    # Optional: List of regular expressions for whitelisting envvars which are available in the templates\n    - '^FLAPI_.*'\n\nduckdb:                     # Configuration of the DuckDB embedded into flAPI\n  db_path: ./flapi_cache.db # Optional: remove or comment out for in-memory database, we use this store also as cache\n  access_mode: READ_WRITE   # See the https://duckdb.org/docs/configuration/overview) for more details\n  threads: 8\n  max_memory: 8GB\n  default_order: DESC\n\nconnections:                # A YAML map of database connection configurations, a API endpoint needs to reference one of these connections\n   bigquery-lakehouse: \n                            # SQL commands to initialize the connection (e.g., e.g. installing, loading and configuring the BQ a DuckDB extension)\n      init: |\n         INSTALL 'bigquery' FROM 'http://storage.googleapis.com/hafenkran';\n         LOAD 'bigquery';\n      properties:           # A YAML map of connection-specific properties (accessible in templates via {{ context.conn.property_name }})\n         project_id: 'my-project-id'\n\n   customers-parquet: \n      properties:\n         path: './data/customers.parquet'\n\nheartbeat:\n  enabled: true            # The eartbeat worker is a background thread which can can be used to periodically trigger endpionts\n  worker-interval: 10      # The interval in seconds at which the heartbeat worker will trigger endpoints\n\nenforce-https:\n  enabled: false           # Whether to force HTTPS for the API connections, we strongly recommend to use a reverse proxy to do SSL termination\n  # ssl-cert-file: './ssl/cert.pem'\n  # ssl-key-file: './ssl/key.pem'\n\n```\n\nAfter that ensure that the template path (`./sqls` in this example) exists.\n\n### 1. Define your API endpoint (`./sqls/customers.yaml`):\nEach endpoint is at least defined by a YAML file and a corresponding SQL template in the template path. For our example we will create the file `./sqls/customers.yaml`:\n\n```yaml\nurl-path: /customers/      # The URL path at which the endpoint will be available\n\nrequest:                  # The request configuration for the endpoint, this defines the parameters that can be used in the query\n  - field-name: id\n    field-in: query       # The location of the parameter, other options are 'path', 'query' and 'body'\n    description: Customer ID # A description of the parameter, this is used in the auto-generated API documentation\n    required: false       # Whether the parameter is required\n    validators:           # A list of validators that will be applied to the parameter\n      - type: int\n        min: 1\n        max: 1000000\n        preventSqlInjection: true\n\ntemplate-source: customers.sql # The path to the SQL template that will be used to generate the endpoint\nconnection: \n  - customers-parquet          # The connection that will be used to execute the query\n\nrate-limit:\n  enabled: true           # Whether rate limiting is enabled for the endpoint\n  max: 100                # The maximum number of requests per interval\n  interval: 60            # The interval in seconds\n  \nauth:\n  enabled: true           # Whether authentication is enabled for the endpoint\n  type: basic             # The type of authentication, other options are 'basic' and 'bearer'\n  users:                  # The users that are allowed to access the endpoint\n    - username: admin\n      password: secret\n      roles: [admin]\n    - username: user\n      password: password\n      roles: [read]\n\nheartbeat:\n  enabled: true           # Whether the heartbeat worker if enabled will trigger the endpoint periodically\n  params:                 # A YAML map of parameters that will be passed by the heartbeat worker to the endpoint\n    id: 123\n\n```\nThere are many more configuration options available, see the [full documentation](link-to-your-docs) for more details.\n\n### 2. Configure the endpoints SQL template (`./sqls/customers.sql`):\nAfter the creation of the YAML endpoint configuration we need to connect the SQL template which connects the enpoint to the data connection.\nThe template files use the [Mustache templating language](https://mustache.github.io/) to dynamically generate the SQL query. \n\n```sql\nSELECT * FROM '{{{conn.path}}}'\nWHERE 1=1\n{{#params.id}}\n  AND c_custkey = {{{ params.id }}}\n{{/params.id}}\n```\n\nThe above template uses the `path` parameter defined in the connection configuration to directly query a local parquet file. If the id parameter is \nprovided, it will be used to filter the results.\n\n### 3. Send a request:\nTo test the endpoint and see if everything worked, we can use curl. We should also provide the correct basic auth credentials (`admin:secret` in this case). To make the JSON result easier to read, we pipe the output to [`jq`](https://jqlang.github.io/jq/).\n\n```bash\n\u003e curl -X GET -u admin:secret \"http://localhost:8080/customers?id=123\" | jq .\n\n{\n  \"next\": \"\",\n  \"total_count\": 1,\n  \"data\": [\n    {\n      \"c_mktsegment\": \"BUILDING\",\n      \"c_acctbal\": 5897.82999999999992724,\n      \"c_phone\": \"15-817-151-1168\",\n      \"c_address\": \"YsOnaaER8MkvK5cpf4VSlq\",\n      \"c_nationkey\": 5,\n      \"c_name\": \"Customer#000000123\",\n      \"c_comment\": \"ependencies. regular, ironic requests are fluffily regu\",\n      \"c_custkey\": 123\n    }\n  ]\n}\n\n```\n\n## ⁉️ DuckLake-backed caching (current implementation)\nflAPI uses the DuckDB DuckLake extension to provide modern, snapshot-based caching. You write the SQL to define the cached table, and flAPI manages schemas, snapshots, retention, scheduling, and audit logs.\n\n### Quick start: Full refresh cache\n1) Configure DuckLake globally (alias is `cache` by default):\n```yaml\nducklake:\n  enabled: true\n  alias: cache\n  metadata_path: ./examples/data/cache.ducklake\n  data_path: ./examples/data/cache.ducklake\n  data_inlining_row_limit: 10  # Enable data inlining for small changes (optional)\n  retention:\n    max_snapshot_age: 14d\n  compaction:\n    enabled: false\n  scheduler:\n    enabled: true\n```\n\n2) Add cache block to your endpoint (no `primary-key`/`cursor` → full refresh):\n```yaml\nurl-path: /publicis\ntemplate-source: publicis.sql\nconnection: [bigquery-lakehouse]\n\ncache:\n  enabled: true\n  table: publicis_cache\n  schema: analytics\n  schedule: 5m\n  retention:\n    max_snapshot_age: 14d\n  template_file: publicis/publicis_cache.sql\n```\n\n3) Write the cache SQL template (CTAS):\n```sql\n-- publicis/publicis_cache.sql\nCREATE OR REPLACE TABLE {{cache.catalog}}.{{cache.schema}}.{{cache.table}} AS\nSELECT\n  p.country,\n  p.product_category,\n  p.campaign_type,\n  p.channel,\n  sum(p.clicks) AS clicks\nFROM bigquery_scan('{{{conn.project_id}}}.landing__publicis.kaercher_union_all') AS p\nGROUP BY 1, 2, 3, 4;\n```\n\n4) Query from the cache in your main SQL:\n```sql\n-- publicis.sql\nSELECT\n  p.country,\n  p.product_category,\n  p.campaign_type,\n  p.channel,\n  p.clicks\nFROM {{cache.catalog}}.{{cache.schema}}.{{cache.table}} AS p\nWHERE 1=1\n```\n\nNotes:\n- The cache schema (`cache.analytics`) is created automatically if missing.\n- Regular GET requests never refresh the cache. Refreshes happen on warmup, on schedule, or via the manual API.\n- **Data Inlining**: When `data_inlining_row_limit` is configured, small cache changes (≤ specified row limit) are written directly to DuckLake metadata instead of creating separate Parquet files. This improves performance for small incremental updates.\n\n#### Data inlining (optional, for small changes)\n\nDuckLake supports writing very small inserts directly into the metadata catalog instead of creating a Parquet file for every micro-batch. This is called \"Data Inlining\" and can significantly speed up small, frequent updates.\n\n- **Enable globally**: configure once under the top-level `ducklake` block:\n\n  ```yaml\n  ducklake:\n    enabled: true\n    alias: cache\n    metadata_path: ./examples/data/cache.ducklake\n    data_path: ./examples/data/cache.ducklake\n    data_inlining_row_limit: 10  # inline inserts up to 10 rows\n  ```\n\n- **Behavior**:\n  - Inserts with rows ≤ `data_inlining_row_limit` are inlined into the catalog metadata.\n  - Larger inserts automatically fall back to normal Parquet file writes.\n  - Inlining applies to all caches (global setting), no per-endpoint toggle.\n\n- **Manual flush (optional)**: you can flush inlined data to Parquet files at any time using DuckLake’s function. Assuming your DuckLake alias is `cache`:\n\n  ```sql\n  -- Flush all inlined data in the catalog\n  CALL ducklake_flush_inlined_data('cache');\n\n  -- Flush only a specific schema\n  CALL ducklake_flush_inlined_data('cache', schema_name =\u003e 'analytics');\n\n  -- Flush only a specific table (default schema \"main\")\n  CALL ducklake_flush_inlined_data('cache', table_name =\u003e 'events_cache');\n\n  -- Flush a specific table in a specific schema\n  CALL ducklake_flush_inlined_data('cache', schema_name =\u003e 'analytics', table_name =\u003e 'events_cache');\n  ```\n\n- **Notes**:\n  - This feature is provided by DuckLake and is currently marked experimental upstream. See the DuckLake docs for details: [Data Inlining](https://ducklake.select/docs/stable/duckdb/advanced_features/data_inlining.html).\n  - If you don’t set `data_inlining_row_limit`, flAPI won’t enable inlining and DuckLake will use regular Parquet writes.\n\n### Advanced: Incremental append and merge\nThe engine infers sync mode from your YAML:\n- No `primary-key`, no `cursor` → full refresh (CTAS)\n- With `cursor` only → incremental append\n- With `primary-key` + `cursor` → incremental merge (upsert)\n\nExample YAMLs:\n```yaml\n# Incremental append\ncache:\n  enabled: true\n  table: events_cache\n  schema: analytics\n  schedule: 10m\n  cursor:\n    column: created_at\n    type: timestamp\n  template_file: events/events_cache.sql\n\n# Incremental merge (upsert)\ncache:\n  enabled: true\n  table: customers_cache\n  schema: analytics\n  schedule: 15m\n  primary-key: [id]\n  cursor:\n    column: updated_at\n    type: timestamp\n  template_file: customers/customers_cache.sql\n```\n\nCache template variables available to your SQL:\n- `{{cache.catalog}}`, `{{cache.schema}}`, `{{cache.table}}`, `{{cache.schedule}}`\n- `{{cache.snapshotId}}`, `{{cache.snapshotTimestamp}}` (current)\n- `{{cache.previousSnapshotId}}`, `{{cache.previousSnapshotTimestamp}}` (previous)\n- `{{cache.cursorColumn}}`, `{{cache.cursorType}}`\n- `{{cache.primaryKeys}}`\n- `{{params.cacheMode}}` is available with values `full`, `append`, or `merge`\n\nIncremental append example:\n```sql\n-- events/events_cache.sql\nINSERT INTO {{cache.catalog}}.{{cache.schema}}.{{cache.table}}\nSELECT *\nFROM source_events\nWHERE {{#cache.previousSnapshotTimestamp}} event_time \u003e TIMESTAMP '{{cache.previousSnapshotTimestamp}}' {{/cache.previousSnapshotTimestamp}}\n```\n\nIncremental merge example:\n```sql\n-- customers/customers_cache.sql\nMERGE INTO {{cache.catalog}}.{{cache.schema}}.{{cache.table}} AS t\nUSING (\n  SELECT * FROM source_customers\n  WHERE {{#cache.previousSnapshotTimestamp}} updated_at \u003e TIMESTAMP '{{cache.previousSnapshotTimestamp}}' {{/cache.previousSnapshotTimestamp}}\n) AS s\nON t.id = s.id\nWHEN MATCHED THEN UPDATE SET\n  name = s.name,\n  email = s.email,\n  updated_at = s.updated_at\nWHEN NOT MATCHED THEN INSERT (*) VALUES (s.*);\n```\n\n### When does the cache refresh?\n- Startup warmup: flAPI refreshes caches for endpoints with cache enabled.\n- Scheduled refresh: controlled by `cache.schedule` on each endpoint (e.g., `5m`).\n- Manual refresh: call the refresh API (see below).\n- Regular GET requests do not refresh the cache.\n\n### Audit, retention, compaction, and control APIs\nflAPI maintains an audit table inside DuckLake at `cache.audit.sync_events` and provides control endpoints:\n\n- Manual refresh:\n```bash\ncurl -X POST \"http://localhost:8080/api/v1/_config/endpoints/publicis/cache/refresh\"\n```\n\n- Audit logs (endpoint-specific and global):\n```bash\ncurl \"http://localhost:8080/api/v1/_config/endpoints/publicis/cache/audit\"\ncurl \"http://localhost:8080/api/v1/_config/cache/audit\"\n```\n\n- Garbage collection (retention):\nRetention can be configured per endpoint under `cache.retention`:\n```yaml\ncache:\n  retention:\n    max_snapshot_age: 7d     # time-based retention\n    # keep_last_snapshots: 3 # version-based retention (subject to DuckLake support)\n```\nThe system applies retention after each refresh and you can also trigger GC manually:\n```bash\ncurl -X POST \"http://localhost:8080/api/v1/_config/endpoints/publicis/cache/gc\"\n```\n\n- Compaction:\nIf enabled in global `ducklake.scheduler`, periodic file merging is performed via DuckLake `ducklake_merge_adjacent_files`.\n\n### Template authoring guide (reference)\nUse these variables inside your cache templates and main queries:\n\n- Identification\n  - `{{cache.catalog}}` → usually `cache`\n  - `{{cache.schema}}` → e.g., `analytics` (auto-created if missing)\n  - `{{cache.table}}` → your cache table name\n\n- Mode and scheduling\n  - `{{params.cacheMode}}` → `full` | `append` | `merge`\n  - `{{cache.schedule}}` → if set in YAML\n\n- Snapshots\n  - `{{cache.snapshotId}}`, `{{cache.snapshotTimestamp}}`\n  - `{{cache.previousSnapshotId}}`, `{{cache.previousSnapshotTimestamp}}`\n\n- Incremental hints\n  - `{{cache.cursorColumn}}`, `{{cache.cursorType}}`\n  - `{{cache.primaryKeys}}` → comma-separated list, e.g., `id,tenant_id`\n\nAuthoring tips:\n- Full refresh: use `CREATE OR REPLACE TABLE ... AS SELECT ...`.\n- Append: `INSERT INTO cache.table SELECT ... WHERE event_time \u003e previousSnapshotTimestamp`.\n- Merge: `MERGE INTO cache.table USING (SELECT ...) ON pk ...`.\n- Do not create schemas in templates; flAPI does that automatically.\n\n### Troubleshooting\n- Cache refresh happens on every request: by design this is disabled. Ensure you’re not calling the manual refresh endpoint from a client and that your logs show scheduled or warmup refreshes only.\n- Schema not found: verify `cache.schema` is set; flAPI will auto-create it.\n- Retention errors: use time-based `max_snapshot_age` first. Version-based retention depends on DuckLake support.\n\n## 🧩 YAML includes and environment variables\nflAPI extends plain YAML with lightweight include and environment-variable features so you can keep configurations modular and environment-aware.\n\n### Environment variables\n- Write environment variables as `{{env.VAR_NAME}}` anywhere in your YAML.\n- Only variables that match the whitelist in your root config are substituted:\n  ```yaml\n  template:\n    path: './sqls'\n    environment-whitelist:\n      - '^FLAPI_.*'     # allow all variables starting with FLAPI_\n      - '^PROJECT_.*'   # optional additional prefixes\n  ```\n- If the whitelist is empty or omitted, all environment variables are allowed.\n\nExamples:\n```yaml\n# Substitute inside strings\nproject_name: \"${{env.PROJECT_NAME}}\"\n\n# Build include paths dynamically\ntemplate:\n  path: \"{{env.CONFIG_DIR}}/sqls\"\n```\n\n### Include syntax\nYou can splice content from another YAML file directly into the current document.\n\n- Basic include: `{{include from path/to/file.yaml}}`\n- Section include: `{{include:top_level_key from path/to/file.yaml}}` includes only that key\n- Conditional include: append `if \u003ccondition\u003e` to either form\n\nConditions supported:\n- `true` or `false`\n- `env.VAR_NAME` (include if the variable exists and is non-empty)\n- `!env.VAR_NAME` (include if the variable is missing or empty)\n\nExamples:\n```yaml\n# Include another YAML file relative to this file\n{{include from common/settings.yaml}}\n\n# Include only a section (top-level key) from a file\n{{include:connections from shared/connections.yaml}}\n\n# Conditional include based on an environment variable\n{{include from overrides/dev.yaml if env.FLAPI_ENV}}\n\n# Use env var in the include path\n{{include from {{env.CONFIG_DIR}}/secrets.yaml}}\n```\n\nResolution rules and behavior:\n- Paths are resolved relative to the current file first; absolute paths are supported.\n- Includes inside YAML comments are ignored (e.g., lines starting with `#`).\n- Includes are expanded before the YAML is parsed.\n- Includes do not recurse: include directives within included files are not processed further.\n- Circular includes are guarded against within a single expansion pass; avoid cycles.\n\nTips:\n- Prefer section includes (`{{include:...}}`) to avoid unintentionally overwriting unrelated keys.\n- Keep shared blocks in small files (e.g., `connections.yaml`, `auth.yaml`) and include them where needed.\n\n\n\n## 🏭 Building from source\nThe source code of flAPI is written in C++ and closely resembles the [DuckDB build process](https://duckdb.org/docs/dev/building/overview). A good documentation of the build process is the GitHub action in [`build.yaml`](.github/workflows/build.yaml). In essecence a few prerequisites need to be met:\nIn essecence a few prerequisites need to be met:\n\n- Install the dependencies: `sudo apt-get install -y build-essential cmake ninja-build`\n- Checkout the repository and submodules: `git clone --recurse-submodules https://github.com/datazoode/flapi.git`\n- Build the project: `make release`\n\nThe build process will download and build DuckDB v1.1.2 and install the vcpkg package manager. We depend on the following vcpkg ports:\n\n- [`argparse`](https://github.com/p-ranav/argparse) - Command line argument parser\n- [`crow`](https://github.com/CrowCpp/Crow) - Our REST-Web framework and JSON handling\n- [`yaml-cpp`](https://github.com/jbeder/yaml-cpp) - YAML parser\n- [`jwt-cpp`](https://github.com/Thalhammer/jwt-cpp) - JSON Web Token library\n- [`openssl`](https://github.com/openssl/openssl) - Crypto library\n- [`catch2`](https://github.com/catchorg/Catch2) - Testing framework\n\n**Note**: MCP support is built-in and doesn't require additional dependencies beyond what's already included.\n\n## 📚 Documentation\n\nFor more detailed information, check out our [full documentation](link-to-your-docs).\n\n## 🤝 Contributing\n\nWe welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for more details.\n\n## 📄 License\n\nflAPI is licensed under the Apache License 2.0. See the [LICENSE](LICENSE) file for more details.\n\n## 🙋‍♀️ Support\n\nIf you have any questions or need help, please [open an issue](https://github.com/yourusername/flapi/issues) or join our [community chat](link-to-your-chat).\n\n---\n","funding_links":[],"categories":["C++","Libraries Powered by DuckDB"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDataZooDE%2Fflapi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FDataZooDE%2Fflapi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDataZooDE%2Fflapi/lists"}