{"id":28581761,"url":"https://github.com/tobilg/cloudflare-ducklake","last_synced_at":"2025-06-11T04:43:15.830Z","repository":{"id":297816816,"uuid":"991388680","full_name":"tobilg/cloudflare-ducklake","owner":"tobilg","description":"DuckLake running on top of Cloudflare Containers \u0026 R2","archived":false,"fork":false,"pushed_at":"2025-06-07T16:32:06.000Z","size":830,"stargazers_count":19,"open_issues_count":1,"forks_count":4,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-07T17:30:02.643Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tobilg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-27T14:48:04.000Z","updated_at":"2025-06-06T18:12:01.000Z","dependencies_parsed_at":"2025-06-07T17:41:09.360Z","dependency_job_id":null,"html_url":"https://github.com/tobilg/cloudflare-ducklake","commit_stats":null,"previous_names":["tobilg/cloudflare-ducklake"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tobilg%2Fcloudflare-ducklake","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tobilg%2Fcloudflare-ducklake/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tobilg%2Fcloudflare-ducklake/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tobilg%2Fcloudflare-ducklake/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tobilg","download_url":"https://codeload.github.com/tobilg/cloudflare-ducklake/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tobilg%2Fcloudflare-ducklake/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259201987,"owners_count":22820933,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-11T04:43:14.679Z","updated_at":"2025-06-11T04:43:15.820Z","avatar_url":"https://github.com/tobilg.png","language":"TypeScript","funding_links":[],"categories":["Projects Built with Cloudflare Containers","TypeScript","Projects build with Cloudflare Containers"],"sub_categories":[],"readme":"# cloudflare-ducklake\nRunning [DuckLake](https://ducklake.select/) on Cloudflare Containers, with DuckDB running as a Hono-based API.\n\n![Screenshot of DuckDB running as API on Cloudflare containers](docs/cf-ducklake.jpeg)\n\n## Build image locally\nBefore being able to build the Docker image locally, you have to download the DuckDB extensions we'd like to package into the image, so that they don't need to be downloaded on each container start:\n\n```bash\nscripts/download_extensions.sh\n```\n\nOnce this is done, you can run the following to build the image locally:\n```bash\nnpm run build:docker\n```\n\n## Run the image locally\nTo run the newly built image locally, run\n\n```bash\nnpm run dev:docker\n```\n\nTo query the DuckDB API within the locally running container, use\n\n```bash\ncurl --location 'http://localhost:8080/query' \\\n--header 'Content-Type: application/json' \\\n--data '{\n  \"query\": \"SELECT * FROM '\\''https://shell.duckdb.org/data/tpch/0_01/parquet/orders.parquet'\\'' LIMIT 1000\"\n}'\n```\n\n**Hint:**\nCurrently it's **not possible** to use `wrangler dev` during local development. I guess this will eventually change once Containers become GA.\n\n### DuckDB API\nDuckDB is exposed as a Hono.js-based API, that offers a few endpoints:\n\n* `GET /`: Will show a JSON welcome message\n* `GET /_health`: Enables potential container health checking (currently not used)\n* `POST /query`: Takes a `application/json` object body with a `query` property that contains the (encoded) SQL query. Returns the query result in `application/json` as well (see example above)\n\n### Securing the API\nYou can generate a unique API Token, e.g. with a tool like [Strong Password Generator](https://1password.com/password-generator), and deploy a Workers secret named `API_TOKEN` that will automatically be used to secure the `/query` endpoint once it's present with the below script:\n\n```bash\nscripts/setup_api_secrets.sh\n```\n\nIf you want to test this locally, too, you need to create a `.dev.vars` file in the root directory of this project. This will be appended for R2 Data Catalog \u0026 DuckLake usage later if you want to try these before deployment as well.\n\n### Create a R2 bucket\nIf you want to either deploy the DuckLake or the Apache Iceberg integration via R2 Data Catalog, you have to set up a R2 bucket first. To create a new R2 bucket, you can run the following:\n\n```bash\nscripts/setup_r2.sh YOUR-BUCKET-NAME\n```\nwhere `YOUR-BUCKET-NAME` is your desired name for the bucket. It will automatically add the respective environment variables to `.dev.vars`. You'll also need the bucket name later for settng up the DuckLake secrets.\n\nIf you'd want to set a [location hint](https://developers.cloudflare.com/r2/reference/data-location/#location-hints) or a [jurisdiction](https://developers.cloudflare.com/r2/reference/data-location/#available-jurisdictions), please edit the script accordingly before running it.\n\n## Deployment with DuckLake\n[DuckLake](https://ducklake.select/) was announced on 2025-05-27. It represents a new table format. Read more about it at:\n* [Manifesto](https://ducklake.select/manifesto/)\n* [Specification](https://ducklake.select/docs/stable/specification/introduction)\n* [Podcast episode](https://www.youtube.com/watch?v=zeonmOO9jm4)\n* [FAQs](https://ducklake.select/faq)\n\n### Setting up a catalog database\nAs we want to deploy it in a multi-user fashion, we need a transactional client-server database, such as MySQL or Postgres as backing store for the catalog data. \n\nWe chose [neon.tech](https://neon.tech), because it's serverless, and it offers a free tier of 0.5GB storage, which should be enough to get started.\n\nOnce you initially signed-up, you can choose the project name, cloud provider and the region:\n![Neon configuration](docs/neon-config.png)\n\nAfter you clicked on \"Create project\", you get directly taken to your Dashboard. The next step is to get the connection details. Therefore, click on the \"Connect\" button in the upper-right corner:\n\n![Neon parameters](docs/neon-parameters.png)\n\nNote the following connection parameters, and their equivalent environment variable/secret names (and add them to the `.dev.vars` file):\n* The user (`POSTGRES_USER`)\n* The password (`POSTGRES_PASSWORD`)\n* The hostname (`POSTGRES_HOST`)\n* The database name (`POSTGRES_DB`)\n\nYou can also create another user, and use this if you don't want to use the main instance user.\n\n### Getting a R2 Access Token\nPlease follow the instructions in the R2 docs on [how to create an API token](https://developers.cloudflare.com/r2/data-catalog/get-started/#3-create-an-api-token). \n\nYou'll need to store this token in a secure location (and eventually in `.dev.vars` if you want to use the R2 Data Catalog when running the Docker image locally), as you'll need later for the R2 Data Catalog deployment.\n\nAlso, please note the the S3 APIs Access Key (`R2_ACCESS_KEY_ID`) and Secret Key (`R2_SECRET_ACCESS_KEY`), as well as your Cloudflare account ID (`R2_ACCOUNT_ID`), you'll need them in the next step. Put them in `.dev.vars` as well if you want to run it locally.\n\n### Create secrets for DuckLake\nRunning the following script will create eight new Workers secrets needed for deployment:\n\n```bash\nscripts/setup_ducklake_secrets.sh\n```\n\n### Deployment\nIf you followed the above steps, you can now run the actual deployment. This will create a Worker, a DurableObject, build the Docker image locally and the upload it to the Cloudflare container registry.\n\n**Hint: You need access to Cloudflare containers beta to be able to deploy it to Cloudflare!**\n\n```bash\nnpm run deploy\n```\n\n### Running queries\n\n#### Workers service\nReplace the `WORKERS_URL` with the real URL, and the `API_TOKEN` with your real API token:\n\n```bash\ncurl --location 'https://WORKERS_URL/query' \\\n--header 'Content-Type: application/json' \\\n--header 'Authorization: Bearer API_TOKEN' \\\n--data '{\n  \"query\": \"CREATE TABLE ducklake.orders AS SELECT * FROM '\\''https://shell.duckdb.org/data/tpch/0_01/parquet/orders.parquet'\\''\"\n}'\n```\nThe response should look like this:\n```json\n[\n  {\"Count\":\"15000\"}\n]\n```\n\nGet the first row of the new table:\n```bash\ncurl --location 'https://WORKERS_URL/query' \\\n--header 'Content-Type: application/json' \\\n--header 'Authorization: Bearer API_TOKEN' \\\n--data '{\n  \"query\": \"SELECT * FROM ducklake.orders LIMIT 1\"\n}'\n```\n\nThe response should look like this:\n```json\n[\n  {\n    \"o_orderkey\": 1,\n    \"o_custkey\": 370,\n    \"o_orderstatus\": \"O\",\n    \"o_totalprice\": 172799.49,\n    \"o_orderdate\": {\n      \"days\": 9497\n    },\n    \"o_orderpriority\": \"5-LOW\",\n    \"o_clerk\": \"Clerk#000000951\",\n    \"o_shippriority\": 0,\n    \"o_comment\": \"nstructions sleep furiously among \"\n  }\n]\n```\n\n## Deployment with the additional R2 Data Catalog / Iceberg integration\nWith the `v1.3.0` release of DuckDB, it became possible to connect to the R2 Data Catalog. This means that you can now also read Iceberg data from R2's Object Storage directly from a SQL statement issued by DuckDB.\n\n### Enable R2 Data Catalog for existing R2 bucket\nYou can enable the Apache Iceberg support via the R2 Data Catalogby running the following command:\n\n```bash\nscripts/setup_r2_data_catalog.sh YOUR-BUCKET-NAME\n```\nwhere `YOUR-BUCKET-NAME` is the name for the bucket you used in the R2 bucket creation step before.\n\n### Getting the R2 Data Catalog information\nThe information about the R2 Data Catalog URI (`R2_ENDPOINT` secret) and warehouse name (`R2_CATALOG` secret) can be gathered by running:\n\n```bash\nscripts/get_catalog_info.sh YOUR-BUCKET-NAME\n```\nwhere `YOUR-BUCKET-NAME` is the name for the bucket you used in the R2 bucket creation step before.\n\nPlease also store this information, because you'll need in in the next step. Also, please add the three variables to `.dev.vars` if you want to use the R2 Data Catalog when running the Docker image locally.\n\n### Creating secrets for R2 Data Catalog\nTo create the necessary Workers secrets, run:\n\n```bash\nscripts/create_r2_data_catalog_secrets.sh\n```\nand copy \u0026 paste the respective values you noted in the last two steps.\n\n### Writing Iceberg example data\nAs we need some example data if we want to test the new Iceberg capabilities, we need to create this data manually with a [Python script](scripts/python/create-iceberg-data.py).\n\nFor convenience, we'll use [Marimo](https://marimo.io) for this. It requires a working Python installation on your machine.\n\n#### Setup Marimo\nTo setup Marimo, run the following npm task:\n\n```bash\nnpm run iceberg:setup\n```\n\nThis will create a new directory and install Marimo and some dependencies in a virtual environment.\n\n#### Create the Iceberg example data\nTo create the Iceberg example data, run the following npm task:\n\n```bash\nnpm run iceberg:create\n```\n\nThis will start Marimo, and load the respective Python script. You'll need to edit the variables for `WAREHOUSE`, `CATALOG_URI` and `TOKEN` with the values gathered in the last steps. This is also described in the [R2 Data Catalog docs](https://developers.cloudflare.com/r2/data-catalog/get-started/#6-create-a-python-notebook-to-interact-with-the-data-warehouse).\n\nAfter you did that, you can run the Python cells in the Marimo notebook, and should end up with some data created in the R2 Bucket. You can check in the Cloudflare Dashboard, or via `wrangler`.\n\n### Running DuckDB with Iceberg support\nOnce you created the secrets as outlined above (all of them!), the application will automatically create a DuckDB secret for accessing the R2 Data Catalog, and attach the catalog as well under the name of `r2lake` when it starts. The example table created in the last step is called `people` and was created in the `default` schema.\n\n### Deployment\nIf you followed the above steps, you can now run the deployment. This will update the existing deployment when you already deployed the DuckLake configuration.\n\n**Hint: You need access to Cloudflare containers beta to be able to deploy it to Cloudflare!**\n\n```bash\nnpm run deploy\n```\nThe `wrangler` deployment output shows the workers URL where the service can be reached. Note it down if you want to run test queries (`WORKERS_URL` in the step below).\n\n### Running queries\n\n#### Workers service\nReplace the `WORKERS_URL` with the real URL, and the `API_TOKEN` with your real API token:\n\n```bash\ncurl --location 'https://WORKERS_URL/query' \\\n--header 'Content-Type: application/json' \\\n--header 'Authorization: Bearer API_TOKEN' \\\n--data '{\n  \"query\": \"SELECT * FROM r2lake.default.people\"\n}'\n```\n\n#### Locally\nReplace the `API_TOKEN` with your real API token and run the following:\n\n```bash\ncurl --location 'http://localhost:8080/query' \\\n--header 'Content-Type: application/json' \\\n--header 'Authorization: Bearer API_TOKEN' \\\n--data '{\n  \"query\": \"SELECT * FROM r2lake.default.people\"\n}'\n```\n\nThis should return the response below:\n```json\n[\n  {\n    \"id\": \"1\",\n    \"name\": \"Alice\",\n    \"score\": 80\n  },\n  {\n    \"id\": \"2\",\n    \"name\": \"Bob\",\n    \"score\": 92.5\n  },\n  {\n    \"id\": \"3\",\n    \"name\": \"Charlie\",\n    \"score\": 88\n  }\n]\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftobilg%2Fcloudflare-ducklake","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftobilg%2Fcloudflare-ducklake","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftobilg%2Fcloudflare-ducklake/lists"}