{"id":18732473,"url":"https://github.com/metrico/urleng","last_synced_at":"2026-01-29T05:45:35.038Z","repository":{"id":45809195,"uuid":"514691188","full_name":"metrico/urleng","owner":"metrico","description":"Distributed URL Engine for ClickHouse (public service)","archived":false,"fork":false,"pushed_at":"2023-07-08T17:10:14.000Z","size":16207,"stargazers_count":12,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-12-28T15:33:52.736Z","etag":null,"topics":["clickhouse","cloudflare-workers","deta","deta-base","serverless","url-engine","worker"],"latest_commit_sha":null,"homepage":"https://urleng.com","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/metrico.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"github":["lmangani","qxip"],"custom":["https://bunq.me/qxip"]}},"created_at":"2022-07-16T21:46:45.000Z","updated_at":"2024-09-20T15:35:21.000Z","dependencies_parsed_at":"2022-08-12T12:20:39.096Z","dependency_job_id":null,"html_url":"https://github.com/metrico/urleng","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metrico%2Furleng","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metrico%2Furleng/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metrico%2Furleng/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metrico%2Furleng/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/metrico","download_url":"https://codeload.github.com/metrico/urleng/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239605123,"owners_count":19666998,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clickhouse","cloudflare-workers","deta","deta-base","serverless","url-engine","worker"],"created_at":"2024-11-07T15:06:15.581Z","updated_at":"2025-11-13T11:30:13.335Z","avatar_url":"https://github.com/metrico.png","language":"HTML","readme":"![visitor badge](https://img.shields.io/endpoint?url=https://y3kp3s.deta.dev)\n\n# [Decentralized URL Engine for ClickHouse](https://urleng.com)\n\nCloudflare + Deta Worker serving a decentralized, stateful, serverless URL Engine platform for ClickHouse.\n\n#### About\n\nThe serverless API enables separate ClickHouse servers to access external *distributed tables* hosted on the cloud and share any type of data between nodes and 3rd party applications in real-time through the native `URL` Engine and Functions available in ClickHouse.\n\n![image](https://user-images.githubusercontent.com/1423657/179358754-92665ed1-c0f5-486f-9b94-e69144f8047a.png)\n\n#### Serverless Platforms\n\nThe distributed URL engine platform is running on Cloudflare Workers and utilizing Deta Cloud as backend.\n\n  - Cloudflare Worker NDJSON Proxy _(limited to 100k req/day)_ [*](https://developers.cloudflare.com/workers)\n  - Cloudflare Worker API Handler _(limited to 100k req/day)_ [*](https://developers.cloudflare.com/workers)\n  - Deta Micros + Base Storage _(unlimited requests, max 10k/inserts/query)_ [*](https://deta.sh)\n\n--------\n\n### Public Access Point\n\n - `https://urleng.com/{yoursupersecretkeygoeshere}`\n\n--------\n\n#### Usage\nThe following examples illustrates usage for distributed `INSERT` and `SELECT` statements\n\n##### INSERT\n```sql\n:) INSERT INTO FUNCTION url('https://urleng.com/supersecret', JSONEachRow, 'key String, value UInt64') VALUES ('hello', 1), ('world', 2)\n```\n##### SELECT\n```sql\n:) SELECT * FROM url('https://urleng.com/supersecret', JSONEachRow);\n\n┌─key───┬─value─┐\n│ hello │     1 │\n│ world │     2 │\n└───────┴───────┘\n\n2 rows in set. Elapsed: 0.126 sec. \n```\n\n##### URL ENGINE\n```sql\n:) CREATE TABLE default.url_engine_distributed\n   (\n       `key` String,\n       `value` UInt64,\n   )\n   ENGINE = URL('https://urleng.com/supersecret', 'JSONEachRow')\n```\n```sql\n:) INSERT INTO url_engine_distributed VALUES ('hello', 1), ('world', 2)\n:) SELECT * FROM url_engine_distributed\n\n┌─key───┬─value─┐\n│ hello │     1 │\n│ world │     2 │\n└───────┴───────┘\n\n2 rows in set. Elapsed: 0.185 sec. \n```\n###### Expiration\nItems can be set to expire by including an `__expires` key in the object carrying a future Unix timestamp:\n```sql\n:) CREATE TABLE default.url_engine_expire\n   (\n       `key` String,\n       `value` UInt64,\n       `__expires` UInt64 DEFAULT toUInt64(toUnixTimestamp(now() + interval 24 hour))\n   )\n   ENGINE = URL('https://urleng.com/supersecret', 'JSONEachRow')\n```\n\n##### clickhouse-local\nGet data into clickhouse-local with zero efforts:\n```bash\nclickhouse-local -q \"select count() from url('https://urleng.com/supersecret', JSONEachRow)\"\n```\n\n##### chdb\nGet data using [chdb]([https://chdb.dev](https://chdb.fly.dev/?#U0VMRUNUICogZnJvbSB1cmwoJ2h0dHBzOi8vdXJsZW5nLmNvbS94eHgnLCBKU09ORWFjaFJvdywgJ2tleSBTdHJpbmcsIHZhbHVlIFVJbnQ2NCcpIExJTUlUIDEwOw==)) in-memory engine:\n```bash\npython -m chdb \"SELECT * from url('https://urleng.com/xxx', JSONEachRow, 'key String, value UInt64');\" Pretty\n```\n\n##### CURL\n\nInsert and query data using curl or any other HTTP/S GET/POST capable client.\n\n###### POST ndjson\n```bash\ncurl -s -XPOST https://urleng.com/supersecret \\\n     -H 'Content-Type:application/x-ndjson' --data-binary @ndjson.txt\n```\n###### POST json\n```bash\ncurl -X POST https://url-engine.metrico.in/supersecret \\\n     -H 'Content-Type: application/json' -d '[{\"key\":\"curl\",\"value\":1}]'\n```\n###### GET\n```bash\ncurl -X GET https://urleng.com/supersecret\n```             \n\n-----\n\n#### Notes\n- `INSERTS` are updates. Existing data will be replaced. Runtime limit ~1000 records per INSERT.\n- `SELECT` pulls the full table, pending X-Header extensions to prefilter `WHERE` statements.\n-  Because there is no sign-up, the URL `/path` is essentially a *password*, so pick something unique.\n\n-------\n\n#### Examples\n- [URLENG Prometheus Metrics Storage + Exporter](https://github.com/metrico/distributed-clickhouse-prometheus)\n- [URLENG Stand Alone](https://github.com/metrico/clickhouse-node-url-engine) example for self-hosted caches _(in memory)_\n- [URLENG ClickHouse Fiddle](https://fiddle.clickhouse.com/2a71b82b-b7f8-4006-9e2e-01ee0519f0ca) example\n\n(C) 2022 QXIP BV, for more info visit the [qxip metrico](https://metrico.in) repository\n","funding_links":["https://github.com/sponsors/lmangani","https://github.com/sponsors/qxip","https://bunq.me/qxip"],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmetrico%2Furleng","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmetrico%2Furleng","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmetrico%2Furleng/lists"}