{"id":16788153,"url":"https://github.com/bobbyiliev/cf-url-shortener","last_synced_at":"2025-04-10T23:12:14.939Z","repository":{"id":56729349,"uuid":"473934624","full_name":"bobbyiliev/cf-url-shortener","owner":"bobbyiliev","description":"URL Shortener Cloudflare function that uses Upstash Redis and Kafka along with https://materialize.com","archived":false,"fork":false,"pushed_at":"2022-04-16T15:59:09.000Z","size":27,"stargazers_count":13,"open_issues_count":0,"forks_count":4,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-10T23:12:09.725Z","etag":null,"topics":["cloudflare-worker","kafka","materialize","nodejs","redis"],"latest_commit_sha":null,"homepage":"https://materialize.com/docs","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bobbyiliev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-03-25T08:51:15.000Z","updated_at":"2024-10-28T17:25:44.000Z","dependencies_parsed_at":"2022-08-16T00:40:21.323Z","dependency_job_id":null,"html_url":"https://github.com/bobbyiliev/cf-url-shortener","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bobbyiliev%2Fcf-url-shortener","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bobbyiliev%2Fcf-url-shortener/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bobbyiliev%2Fcf-url-shortener/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bobbyiliev%2Fcf-url-shortener/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bobbyiliev","download_url":"https://codeload.github.com/bobbyiliev/cf-url-shortener/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248312135,"owners_count":21082638,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloudflare-worker","kafka","materialize","nodejs","redis"],"created_at":"2024-10-13T08:17:05.405Z","updated_at":"2025-04-10T23:12:14.914Z","avatar_url":"https://github.com/bobbyiliev.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 👷 URL Shortener\n\nThis is a Node.js URL shortener app that uses [Cloudflare Workers](https://www.cloudflare.com/workers/).\n\nThe app is powered by [Cloudflare Workers](https://www.cloudflare.com/workers/) and [Upstash](https://upstash.com/) Redis for storing data and Kafka for storing the click events along with [Materialize](https://materialize.com/) for real-time data analytics.\n\n[Upstash](https://upstash.com/) offers Serverless, Low latency, and pay-as-you-go solutions for Kafka and Redis.\n\n[Materialize](https://materialize.com) is a streaming database for real-time applications. Materialize accepts input data from a variety of streaming sources (like Kafka), data stores and databases (like S3 and Postgres), and files (like CSV and JSON), and lets you query them using SQL.\n\n## App structure\n\nThe demo app has the following structure:\n\n- A serverless Cloudflare Worker that lets you add short links and redirect them to other URLs.\n- All data is stored in Upstash serverless Redis cluster as key-value pairs (short link -\u003e long link).\n- Every time you visit a short link, it triggers an event and stores it in Upstash Kafka.\n- We then get the data from Upstash Kafka and analyze it in Materialize in real-time.\n\nA demo of the app can be found here:\n\nhttps://cf-url-shortener.bobbyiliev.workers.dev/admin\n\n## Diagram\n\nThe following is a diagram of the app structure:\n\n![Diagram of Upstash and Materialize demo](https://user-images.githubusercontent.com/21223421/160150800-2d304712-13c1-4d15-910a-9f99b7b33771.png)\n\n## Demo\n\nHere is a quick demo of how the app works:\n\n![mz-upstash-demo](https://user-images.githubusercontent.com/21223421/160150872-58fca546-5a86-4132-8bb4-a989dc87ba83.gif)\n\n## Prerequisites\n\nBefore you get started, you need to make sure that you have the following\n\n- A Redis cluster and a Kafka cluster in Upstash.\n- A Kafka topic in Upstash called `visits-log`.\n- The Cloudflare CLI tool called `wrangler` on your local machine as described [here](https://developers.cloudflare.com/workers/cli-wrangler/install-update/)\n- A Materialize instance running on your local machine as described [here](https://materialize.com/docs/install/) or a [Materialize Cloud instance](https://cloud.materialize.com/deployments).\n\n## Running this demo\n\nOnce you have all the prerequisites, you can proceed with the following steps:\n\n- Clone the repository and run the following command:\n\n```bash\ngit clone https://github.com/bobbyiliev/cf-url-shortener.git\n```\n\n- Access the directory:\n\n```bash\ncd cf-url-shortener\n```\n\n- Install the `npm` dependencies:\n\n```bash\nnpm install\n```\n\n- Run the `wrangler` command to authenticate with Cloudflare:\n\n```bash\nwrangler login\n```\n\n- Then in the `wrangler.toml` file, update the `account_id` to match your Cloudflare account ID:\n\n```toml\naccount_id = \"YOUR_ACCOUNT_ID_HERE\"\n```\n\n- Set the following secrets in Cloudflare using the `wrangler` tool:\n\n```bash\nwrangler secret put UPSTASH_REDIS_REST_URL\nwrangler secret put UPSTASH_REDIS_REST_TOKEN\nwrangler secret put UPSTASH_KAFKA_REST_URL\nwrangler secret put UPSTASH_KAFKA_REST_USERNAME\nwrangler secret put UPSTASH_KAFKA_REST_PASSWORD\n```\n\n\u003e Make sure to use the **REST API URLs** and not the Broker details.\n\n- Run the following command to deploy the CF Worker:\n\n```bash\nwrangler deploy\n```\n\nWith the CF Worker deployed, you can visit the admin URL where you can add short links and redirect them to other URLs.\n\n## Setup Materialize\n\nOnce you've deployed the CF Worker, you can set up Materialize to analyze the data in Upstash Kafka in real-time.\n\nStart by creating a new Materialize instance in Materialize Cloud:\n\n- https://materialize.com/docs/cloud/get-started-with-cloud/\n\nOr alternatively, you can install Materialize locally:\n\n- https://materialize.com/docs/install/\n\nAfter you've created the instance, you can connect to it using the `psql` command as shown in the docs.\n\n### Create a [Kafka Source](https://materialize.com/docs/sql/create-source/kafka/)\n\nThe `CREATE SOURCE` statements allow you to connect Materialize to an external Kafka data source and lets you interact with its data as if the data were in a SQL table.\n\nTo create a new Kafka source in Materialize run the following statement:\n\n```sql\nCREATE SOURCE click_stats\n  FROM KAFKA BROKER 'UPSTASH_KAFKA_BROKER_URL' TOPIC 'visits-log'\n  WITH (\n      security_protocol = 'SASL_SSL',\n      sasl_mechanisms = 'SCRAM-SHA-256',\n      sasl_username = 'UPSTASH_KAFKA_BROKER_USERNAME',\n      sasl_password = 'UPSTASH_KAFKA_BROKER_PASSWORD'\n  )\nFORMAT BYTES;\n```\n\n\u003e Change the Kafka details to match your Upstash Kafka cluster Broker and credentials.\n\nNext, we will create a [NON-materialized View](https://materialize.com/docs/sql/create-view), which you can think of as kind of a reusable template to be used in other materialized views:\n\n```sql\nCREATE VIEW click_stats_v AS\n    SELECT\n        *\n    FROM (\n        SELECT\n            (data-\u003e\u003e'shortCode')::string AS short_code,\n            (data-\u003e\u003e'longUrl')::string AS long_url,\n            (data-\u003e\u003e'country')::string AS country,\n            (data-\u003e\u003e'city')::string AS city,\n            (data-\u003e\u003e'ip')::string AS ip\n        FROM (\n            SELECT CAST(data AS jsonb) AS data\n            FROM (\n                SELECT convert_from(data, 'utf8') AS data\n                FROM click_stats\n            )\n        )\n    );\n```\n\nFinally, create a [**materialized view**](https://materialize.com/docs/sql/create-materialized-view) to analyze the data in the Kafka source:\n\n```sql\nCREATE MATERIALIZED VIEW click_stats_m AS\n    SELECT\n        *\n    FROM click_stats_v;\n```\n\nThen you can query the materialized view just using standard SQL, but get the data in real-time, with sub-millisecond latency:\n\n```sql\nSELECT * FROM click_stats_m;\n```\n\nYou can stack up materialized views together, so let's order by the number of clicks per short link:\n\n```sql\nCREATE MATERIALIZED VIEW order_by_clicks AS\n    SELECT\n        short_code,\n        COUNT(*) AS clicks\n    FROM click_stats_m\n    GROUP BY short_code;\n```\n\nOne of the great features of Materialize is `TAIL`.\n\n`TAIL` streams updates from a source, table, or view as they occur.\n\nSo to stream the data from our materialized view using `TAIL`, we can use the following statement:\n\n```sql\nCOPY ( TAIL ( SELECT * FROM order_by_clicks ) ) TO STDOUT;\n```\n\nFor more information about `TAIL`, check out this blog post:\n\n\u003e [Subscribe to changes in a view with TAIL in Materialize](https://materialize.com/subscribe-to-changes-in-a-view-with-tail-in-materialize/)\n\n## Display the results in Metabase\n\nAs Materialize is Postgres-wire compatible, you can use BI tools like Metabase to create business intelligence dashboards using the real-time data streams in your Materialize instance.\n\nFor more information about Metabase + Materialize, check out the official documentation:\n\n\u003e [Metabase + Materialize](https://materialize.com/docs/third-party/metabase/)\n\nExample dashboard that shows the number of clicks per short link:\n\n![Materialize Metabase Dashboard](https://user-images.githubusercontent.com/21223421/162766444-5b78f011-9f0a-4515-9998-d8836040ddd7.png)\n\n## Conclusion\n\nUsing Materialize to analyze the data in your Upstash Kafka serverless instance is a great way to get real-time insights into your data.\n\nAs a next step, here are some other great resources to learn about Materialize and Upstash:\n\n- [Materialize](https://materialize.com/)\n- [Upstash](https://upstash.com/)\n- [Upstash Kafka](https://upstash.com/kafka/)\n- [Upstash Redis](https://upstash.com/redis/)\n- [Materialize Docs](https://materialize.com/docs/)\n\n\u003c!-- ### Kafka sink:\n\nAs of the time being inline avro schemas are not supported by Materialize.\n\nAn issue to track Upstash CSR can be found here:\nhttps://github.com/upstash/issues/issues/20\n\n```sql\nCREATE SINK stats_sink\n    FROM click_stats_m\n    INTO KAFKA BROKER 'UPSTASH_KAFKA_BROKER_URL' TOPIC 'stats-sink'\n    WITH (\n      security_protocol = 'SASL_SSL',\n      sasl_mechanisms = 'SCRAM-SHA-256',\n      sasl_username = 'UPSTASH_KAFKA_BROKER_USERNAME',\n      sasl_password = 'UPSTASH_KAFKA_BROKER_PASSWORD'\n    )\n    FORMAT AVRO USING SCHEMA '{\n        \"type\": \"record\",\n        \"name\": \"envelope\",\n        \"fields\": { \"name\": \"data\", \"type\": \"bytes\" }\n    }';\n``` --\u003e","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbobbyiliev%2Fcf-url-shortener","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbobbyiliev%2Fcf-url-shortener","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbobbyiliev%2Fcf-url-shortener/lists"}