{"id":20811440,"url":"https://github.com/quackscience/urlengine","last_synced_at":"2026-04-03T08:01:36.911Z","repository":{"id":259059918,"uuid":"861035974","full_name":"quackscience/urlengine","owner":"quackscience","description":"A pastie for SQL Tables. URL Engine for DuckDB \u0026 ClickHouse","archived":false,"fork":false,"pushed_at":"2024-11-22T20:39:03.000Z","size":174,"stargazers_count":9,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-03-27T10:51:47.284Z","etag":null,"topics":["clickhouse","duckdb","http","httpfs","storage","url","urlengine"],"latest_commit_sha":null,"homepage":"https://urleng.com","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/quackscience.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-09-21T20:32:55.000Z","updated_at":"2025-02-01T19:02:43.000Z","dependencies_parsed_at":"2025-05-07T09:46:50.044Z","dependency_job_id":null,"html_url":"https://github.com/quackscience/urlengine","commit_stats":null,"previous_names":["metrico/urlengine","metrico/duckdb-urlengine"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/quackscience/urlengine","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quackscience%2Furlengine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quackscience%2Furlengine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quackscience%2Furlengine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quackscience%2Furlengine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/quackscience","download_url":"https://codeload.github.com/quackscience/urlengine/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quackscience%2Furlengine/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31342699,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-03T06:57:30.245Z","status":"ssl_error","status_checked_at":"2026-04-03T06:57:29.849Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clickhouse","duckdb","http","httpfs","storage","url","urlengine"],"created_at":"2024-11-17T20:42:29.280Z","updated_at":"2026-04-03T08:01:36.833Z","avatar_url":"https://github.com/quackscience.png","language":"Go","readme":"\u003cimg src=\"https://github.com/user-attachments/assets/46a5c546-7e9b-42c7-87f4-bc8defe674e0\" width=250 /\u003e\n\n# DuckDB URL Storage Engine\nThe URL Engine provides format agnostic remote storage for DuckDB/Clickhouse with hive-style partitioning.\u003cbr\u003e\n\n### Demo\nA public demo instance is available at [https://urleng.glitch.me](https://urleng.glitch.me)\n\n##### Features\n- [x] INSERT Files via POST\n- [x] SELECT Files via GET/HEAD\n- [x] HTTP RANGE Support\n\n#### Usage\n##### Golang\nInstall and run the example GO service :\n```\ncd go/\ngo mod tidy\nPORT=80 go run server.go\n```\n\n### Examples\n#### 📦 DuckDB\n\nYou can COPY and SELECT from the URL Engine using extensions `json`,`csv`,`parquet`\n\n```sql\nD SET enable_http_write = 1;\n\nD COPY (SELECT version() as version, 9999 as number) TO 'https://urleng.glitch.me/test.json';\nD SELECT * FROM read_json_auto('https://urleng.glitch.me/test.json');\n┌─────────┬────────┐\n│ version │ number │\n│ varchar │ int64  │\n├─────────┼────────┤\n│ v1.1.0  │   9999 │\n└─────────┴────────┘\n\nD COPY (SELECT version() as version, 9999 as number) TO 'https://urleng.glitch.me/test.parquet';\nD SELECT * FROM read_parquet('https://urleng.glitch.me/test.parquet');\n┌─────────┬────────┐\n│ version │ number │\n│ varchar │ int64  │\n├─────────┼────────┤\n│ v1.1.0  │   9999 │\n└─────────┴────────┘\n\nD SELECT * FROM parquet_schema('https://urleng.glitch.me/test.parquet');\n┌──────────────────────┬───────────────┬────────────┬─────────────┬───┬────────────────┬───────┬───────────┬──────────┬──────────────┐\n│      file_name       │     name      │    type    │ type_length │ … │ converted_type │ scale │ precision │ field_id │ logical_type │\n│       varchar        │    varchar    │  varchar   │   varchar   │   │    varchar     │ int64 │   int64   │  int64   │   varchar    │\n├──────────────────────┼───────────────┼────────────┼─────────────┼───┼────────────────┼───────┼───────────┼──────────┼──────────────┤\n│ https://duckserver…  │ duckdb_schema │            │             │ … │                │       │           │          │              │\n│ https://duckserver…  │ version       │ BYTE_ARRAY │             │ … │ UTF8           │       │           │          │              │\n│ https://duckserver…  │ number        │ INT32      │             │ … │ INT_32         │       │           │          │              │\n├──────────────────────┴───────────────┴────────────┴─────────────┴───┴────────────────┴───────┴───────────┴──────────┴──────────────┤\n│ 3 rows                                                                                                        11 columns (9 shown) │\n└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘\n```\n\n##### HTTP/S HEADERS\nWhen Authentication is enabled, you can pass headers and other parameters using _SECRETS_\n\n```sql\nCREATE SECRET extra_http_headers (\n    TYPE HTTP, \n    EXTRA_HTTP_HEADERS MAP{\n\t\t'Authorization': 'Bearer ${SOME_TOKEN}',\n\t\t'CustomHeader': 'abc123'\n\t}\n);\n```\n\n##### Native format\n\nYou can also upload and attach a native DuckDB `.duckdb` database file and attach it to a read-only session\n```bash\ncurl --data-binary @/path/to/myduck.db https://urleng.glitch.me/myduck.db\n```\n```sql\nATTACH 'https://urleng.glitch.me/myduck.db' as remote; SELECT * FROM remote.table;\n```\n\n\n\n#### 📦 ClickHouse\n##### INSERT\n```sql\nINSERT INTO FUNCTION url('https://urleng.glitch.me/click.parquet', 'PARQUET', 'column1 String, column2 UInt32') VALUES (version(), 999);\n```\n##### SELECT\n```sql\nSELECT * FROM url('https://urleng.glitch.me/click.parquet', PARQUET) FORMAT Pretty;\n\n   ┏━━━━━━━━━━┳━━━━━━━━┓\n   ┃ version  ┃ number ┃\n   ┡━━━━━━━━━━╇━━━━━━━━┩\n1. │ 24.5.1.1 │    999 │\n   └──────────┴────────┘\n```\n\n##### DESCRIBE\n```sql\nDESCRIBE TABLE url('http://https://urleng.glitch.me/click.parquet', PARQUET) FORMAT Pretty;\n\n   ┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓\n   ┃ name    ┃ type             ┃ default_type ┃ default_expression ┃ comment ┃ codec_expression ┃ ttl_expression ┃\n   ┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩\n1. │ version │ Nullable(String) │              │                    │         │                  │                │\n   ├─────────┼──────────────────┼──────────────┼────────────────────┼─────────┼──────────────────┼────────────────┤\n2. │ number  │ Nullable(UInt32) │              │                    │         │                  │                │\n   └─────────┴──────────────────┴──────────────┴────────────────────┴─────────┴──────────────────┴────────────────┘\n```\n##### SET PARAM\n```sql\nSET param_url = 'https://urleng.glitch.me/your_secret_token';\nINSERT INTO FUNCTION url({url:String}, JSONEachRow, 'key String, value UInt64') VALUES ('hello', 1);\nSELECT * FROM url({url:String}, JSONEachRow);\n```\n\n\n\n### Design Flow\n\n```mermaid\nsequenceDiagram\n    autonumber\n    DuckDB-\u003e\u003eDuckServer: POST Request\n    loop Storage\n        DuckServer-\u003e\u003eDuckServer: WRITE FILE\n    end\n    DuckServer--\u003e\u003eDuckDB: POST Response\n    DuckDB-\u003e\u003eDuckServer: HEAD Request\n    loop Storage\n        DuckServer-\u003e\u003eDuckServer: READ FILE SIZE\n    end\n    DuckDB-\u003e\u003eDuckServer: GET RANGE Request\n    loop Storage\n        DuckServer-\u003e\u003eDuckServer: READ FILE RANGE\n    end\n    DuckServer--\u003e\u003eDuckDB: GET Response\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquackscience%2Furlengine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fquackscience%2Furlengine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquackscience%2Furlengine/lists"}