{"id":23396433,"url":"https://github.com/mneedham/pizza-shop-workshop","last_synced_at":"2025-04-11T14:42:26.457Z","repository":{"id":65436522,"uuid":"590898421","full_name":"mneedham/pizza-shop-workshop","owner":"mneedham","description":"This is the repository for a workshop at JFokus 2023","archived":false,"fork":false,"pushed_at":"2023-05-31T09:51:16.000Z","size":541,"stargazers_count":6,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"odsc-europe-2023","last_synced_at":"2025-03-25T10:51:11.399Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mneedham.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2023-01-19T13:29:27.000Z","updated_at":"2023-07-05T20:43:52.000Z","dependencies_parsed_at":"2023-02-12T23:46:19.940Z","dependency_job_id":null,"html_url":"https://github.com/mneedham/pizza-shop-workshop","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mneedham%2Fpizza-shop-workshop","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mneedham%2Fpizza-shop-workshop/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mneedham%2Fpizza-shop-workshop/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mneedham%2Fpizza-shop-workshop/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mneedham","download_url":"https://codeload.github.com/mneedham/pizza-shop-workshop/tar.gz/refs/heads/odsc-europe-2023","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248420257,"owners_count":21100347,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-22T07:35:38.847Z","updated_at":"2025-04-11T14:42:26.437Z","avatar_url":"https://github.com/mneedham.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Pizza Shop Workshop\n\nThis is a workshop where attendees learn how to build a Real-Time analytics dashboard for an imaginary pizza service. It acts as an introduction to the components of the Real-Time Analytics Stack. \n\n## Pre Requisites\n\n* Install kcat - https://docs.confluent.io/platform/current/app-development/kafkacat-usage.html\n* Install jq - https://github.com/edenhill/kcat\n* Install psql - https://www.timescale.com/blog/how-to-install-psql-on-mac-ubuntu-debian-windows/\n\nI'm using Pygmentize to view each of the files, but you can just view them in an editor.\nYou can install Pygmentize by running the following:\n\n```bash\npip install Pygments\n```\n\n## Part 1\n\nSet up Kafka and MySQL\n\n```bash\npygmentize -O style=github-dark docker-compose-base.yml | less\n```\n\n```bash\ndocker-compose -f docker-compose-base.yml up\n```\n\nLook at the products\n\n```bash\ndocker exec -it mysql mysql -u mysqluser -p\n```\n\n(Password is `mysqlpw`)\n\n```sql\nSELECT name, description, category, price \nFROM pizzashop.products \nLIMIT 10;\n```\n\n```sql\nSELECT id, first_name, last_name, email, lat, lon\nFROM pizzashop.users \nLIMIT 10;\n```\n\nCreate an orders topic:\n\n```bash\ndocker exec -it kafka \\\n  kafka-topics \\\n  --create \\\n  --bootstrap-server localhost:9092 \\\n  --topic orders \\\n  --partitions 5\n```\n\nMetadata for that topic:\n\n```bash\nkcat -L -b localhost:29092 -t orders\n```\n\n```bash\ndocker exec -it kafka \\\n  kafka-run-class \\\n  kafka.tools.GetOffsetShell \\\n  --broker-list localhost:9092 \\\n  --topic orders\n```\n\n\n## Part 2\n\nLet's have a look at the Orders Service Simulator\n\n```bash\npygmentize -O style=github-dark orders-service/multiseeder.py | less\n```\n\nAnd its accompanying Dockerfile and docker-compose files:\n\n```bash\npygmentize -O style=github-dark orders-service/Dockerfile\npygmentize -O style=github-dark docker-compose-orders.yml | less\n```\n\nNow let's connect the simulator to our estate:\n\n```bash\ndocker compose -f docker-compose-orders.yml up -d\n```\n\n```bash\nkcat -C -b localhost:29092 -t orders | jq -c\nkcat -C -b localhost:29092 -t orders -c1 | jq\n```\n\n## Part 3\n\nNow let's get this data into Apache Pinot.\n\nView the schema:\n\n```bash\npygmentize -O style=github-dark pinot/config/orders/schema.json | less\n```\n\nView the table config:\n\n```bash\npygmentize -O style=github-dark pinot/config/orders/table.json | less\n```\n\nStart Pinot:\n\n```bash\ndocker compose -f docker-compose-pinot.yml up -d\n```\n\nCreate the table:\n\n```bash\ndocker run \\\n  -v $PWD/pinot/config:/config \\\n  --network pizza-shop \\\n  apachepinot/pinot:0.12.0-arm64 \\\n  AddTable \\\n  -schemaFile /config/orders/schema.json \\\n  -tableConfigFile /config/orders/table.json \\\n  -controllerHost pinot-controller \\\n  -exec\n```\n\nNavigate to the Pinot UI at http://localhost:9000/. \nLet's run some queries:\n\n```sql\nselect ts, id, price, productsOrdered, totalQuantity, userId\nfrom orders \norder by ts DESC\nlimit 10\n```\n\n```sql\nselect count(*), sum(price)\nfrom orders \nWHERE ts \u003e ago('PT1M')\norder by ts DESC\nlimit 10\n```\n\n## Part 4\n\nNow let's add a Streamlit dashboard populated by Pinot queries.\n\n```bash\ndocker compose -f docker-compose-dashboard.yml up -d\n```\n\nLet's first look at the basic dashboard:\n\n```bash\npygmentize -O style=github-dark streamlit/app_basic.py\n```\n\nNavigate to http://localhost:8501 to see the basic dashboard\n\nAnd now the auto refreshing dashboard:\n\n```bash\npygmentize -O style=github-dark streamlit/app.py\n```\n\nNavigate to http://localhost:8502 to see the auto-refreshing dashboard\n\n## Part 5\n\nAdd Debezium to the estate\n\n```bash\npygmentize -O style=github-dark docker-compose-debezium.yml | less\n```\n\n```bash\ndocker-compose -f docker-compose-debezium.yml up -d\n```\n\nStream MySQL changes into Kafka\n\n```bash\ncurl -X PUT -H  \"Content-Type:application/json\" http://localhost:8083/connectors/mysql/config \\\n    -d '{\n    \"connector.class\": \"io.debezium.connector.mysql.MySqlConnector\",\n    \"database.hostname\": \"mysql\",\n    \"database.port\": 3306,\n    \"database.user\": \"debezium\",\n    \"database.password\": \"dbz\",\n    \"database.server.name\": \"mysql\",\n    \"database.server.id\": \"223344\",\n    \"database.allowPublicKeyRetrieval\": true,\n    \"database.history.kafka.bootstrap.servers\": \"kafka:9092\",\n    \"database.history.kafka.topic\": \"mysql-history\",\n    \"schema.history.internal.kafka.bootstrap.servers\": \"kafka:9092\",    \n    \"schema.history.internal.kafka.topic\": \"mysql-schema-history\",\n    \"database.include.list\": \"pizzashop\",\n    \"time.precision.mode\": \"connect\",\n    \"topic.prefix\": \"mysql\",\n    \"include.schema.changes\": false\n}'\n```\n\nProducts will be written to the `mysql.pizzashop.products` topic. \n\n```bash\nkcat -C -b localhost:29092 -t mysql.pizzashop.products | jq\n```\n\n## Part 6\n\nAdding RisingWave to the estate\n\n```bash\npygmentize -O style=github-dark docker-compose-rwave.yml\n```\n\n```bash\ndocker compose -f docker-compose-rwave.yml up -d\n```\n\nConnect to the psql CLI:\n\n```bash\npsql -h localhost -p 4566 -d dev -U root\n```\n\nCreate orders table:\n\n```sql\nCREATE SOURCE IF NOT EXISTS orders (\n    id varchar,\n    createdAt TIMESTAMP,\n    userId integer,\n    status varchar,\n    price double,\n    items STRUCT \u003c\n      productId varchar,\n      quantity integer,\n      price double\n    \u003e[]\n)\nWITH (\n   connector='kafka',\n   topic='orders',\n   properties.bootstrap.server='kafka:9092',\n   scan.startup.mode='earliest',\n   scan.startup.timestamp_millis='140000000'\n)\nROW FORMAT JSON;\n```\n\nQuery orders:\n\n```sql\nWITH orderItems AS (\n    select unnest(items) AS \"orderItem\",\n           id AS \"orderId\", createdAt           \n    FROM orders\n)\nselect id, orders.createdat, orderItems.*\nFROM orders\nJOIN orderItems ON orderItems.\"orderId\" = orders.id\nLIMIT 10;\n```\n\nCreate products table:\n\n```sql\nCREATE SOURCE IF NOT EXISTS products (\n    id varchar,\n    name varchar,\n    description varchar,\n    category varchar,\n    price double,\n    image varchar\n)\nWITH (\n   connector='kafka',\n   topic='products',\n   properties.bootstrap.server='kafka:9092',\n   scan.startup.mode='earliest',\n   scan.startup.timestamp_millis='140000000'\n)\nROW FORMAT JSON;\n```\n\nQuery products:\n\n```sql\nSELECT * \nFROM Products;\n```\n\nJoin orders and products:\n\n```sql\nWITH orderItems AS (\n    select unnest(items) AS orderItem, \n           id AS \"orderId\", \"createdAt           |\n    FROM orders\n)\nSELECT \"orderId\", \"createdAt\",\n       ((orderItem).productid, (orderItem).quantity, (orderItem).price)::\n       STRUCT\u003cproductId varchar, quantity varchar, price varchar\u003e AS \"orderItem\",\n        (products.id, products.name, products.description, products.category, products.image, products.price)::\n        STRUCT\u003cid varchar, name varchar, description varchar, category varchar, image varchar, price varchar\u003e AS product\nFROM orderItems\nJOIN products ON products.id = (orderItem).productId\nLIMIT 10;\n```\n\nExport as materialized view:\n\n```sql\nCREATE MATERIALIZED VIEW orderItems_view AS\nWITH orderItems AS (\n    select unnest(items) AS orderItem, \n           id AS \"orderId\", createdAt AS \"createdAt\"\n    FROM orders\n)\nSELECT \"orderId\", \"createdAt\",\n       ((orderItem).productid, (orderItem).quantity, (orderItem).price)::\n       STRUCT\u003cproductId varchar, quantity varchar, price varchar\u003e AS \"orderItem\",\n        (products.id, products.name, products.description, products.category, products.image, products.price)::\n        STRUCT\u003cid varchar, name varchar, description varchar, category varchar, image varchar, price varchar\u003e AS product\nFROM orderItems\nJOIN products ON products.id = (orderItem).productId;\n```\n\nCreate sink:\n\n```sql\nCREATE SINK enrichedOrderItems_sink FROM orderItems_view \nWITH (\n   connector='kafka',\n   type='append-only',\n   properties.bootstrap.server='kafka:9092',\n   topic='enriched-order-items'\n);\n```\n\nWe can then query the `enriched-order-items` stream:\n\n```bash\nkcat -C -b localhost:29092 -t enriched-order-items -c1 | jq\n```\n\nQuery from the end of the stream:\n\n```bash\nkcat -C -b localhost:29092 -t enriched-order-items -o end | jq -c\n```\n\n## Part 7\n\nNow let's add an enhanched dashboard, but first we'll add the `order_items_enriched` table:\n\n```bash\npygmentize -O style=github-dark pinot/config/order_items_enriched/schema.json | less\npygmentize -O style=github-dark pinot/config/order_items_enriched/table.json | less\n```\n\n```bash\ndocker run \\\n  -v $PWD/pinot/config:/config \\\n  --network pizza-shop \\\n  apachepinot/pinot:0.12.0-arm64 \\\n  AddTable \\\n  -schemaFile /config/order_items_enriched/schema.json \\\n  -tableConfigFile /config/order_items_enriched/table.json \\\n  -controllerHost pinot-controller \\\n  -exec\n```\n\nAnd now the dashboard:\n\n```bash\ndocker compose -f docker-compose-dashboard-enhanced.yml up -d\n```\n\nNavigate to http://localhost:8503","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmneedham%2Fpizza-shop-workshop","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmneedham%2Fpizza-shop-workshop","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmneedham%2Fpizza-shop-workshop/lists"}