{"id":19361959,"url":"https://github.com/numaproj/numaflow-workshop","last_synced_at":"2026-03-02T11:31:27.803Z","repository":{"id":238642946,"uuid":"797097197","full_name":"numaproj/numaflow-workshop","owner":"numaproj","description":null,"archived":false,"fork":false,"pushed_at":"2024-05-07T15:42:01.000Z","size":370,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-11-18T11:06:09.051Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Dockerfile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/numaproj.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-07T07:33:48.000Z","updated_at":"2024-05-07T23:59:00.000Z","dependencies_parsed_at":"2024-11-10T07:26:28.453Z","dependency_job_id":"5c660dca-b716-4c47-8c6d-1ce45e4a3604","html_url":"https://github.com/numaproj/numaflow-workshop","commit_stats":null,"previous_names":["numaproj/numaflow-workshop"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/numaproj/numaflow-workshop","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/numaproj%2Fnumaflow-workshop","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/numaproj%2Fnumaflow-workshop/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/numaproj%2Fnumaflow-workshop/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/numaproj%2Fnumaflow-workshop/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/numaproj","download_url":"https://codeload.github.com/numaproj/numaflow-workshop/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/numaproj%2Fnumaflow-workshop/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30000029,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-02T11:09:27.951Z","status":"ssl_error","status_checked_at":"2026-03-02T11:08:53.255Z","response_time":60,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T07:26:01.606Z","updated_at":"2026-03-02T11:31:27.784Z","avatar_url":"https://github.com/numaproj.png","language":"Dockerfile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Numaflow Workshop\n\nThis workshop guides you through to install Numaflow in a local Kubernetes cluster, and run couple of pipelines for data processing.\n\n## Prerequisites\n\nSetup one of the tools to run container images:\n\n- [Docker Desktop](https://docs.docker.com/get-docker/)\n- [Rancher Desktop](https://rancherdesktop.io/)\n- [Podman](https://podman.io/)\n\n## A Local K8s Cluster\n\nInstall a local K8s cluster, if you don't have.\n\n- [k3d](https://k3d.io/)\n\n  ```shell\n    brew install k3d\n    k3d cluster create\n  ```\n\n- [kind](https://kind.sigs.k8s.io/)\n\n  ```shell\n  brew install kind\n  kind create cluster\n\n  # Install the metrics server\n  kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml\n  kubectl patch -n kube-system deployment metrics-server --type=json -p '[{\"op\":\"add\",\"path\":\"/spec/template/spec/containers/0/args/-\",\"value\":\"--kubelet-insecure-tls\"}]'\n  ```\n\n- [minikube](https://minikube.sigs.k8s.io/docs/start/)\n\n  ```shell\n  brew install minikube\n  minikube start\n  ```\n\n## Install Numaflow\n\n```shell\nkubectl create ns numaflow-system\nkubectl apply -k https://github.com/numaproj/numaflow-workshop/manifests/installation\n\n# Verify control plane pods are running in numaflow-system namespace\nkubectl -n numaflow-system get po\n\nNAME                                   READY   STATUS    RESTARTS   AGE\nnumaflow-controller-86c9475986-bkp4j   1/1     Running   0          2m18s\nnumaflow-dex-server-8656769d4c-5msjv   1/1     Running   0          2m18s\nnumaflow-server-65dd556cf7-v64s2       1/1     Running   0          2m18s\n```\n\nInstall an InterStepBufferService\n\n```shell\nkubectl apply -f https://raw.githubusercontent.com/numaproj/numaflow/stable/examples/0-isbsvc-jetstream.yaml\n```\n\nYou will see pods running like below in `default` namespace.\n\n```\nkubectl get po\n\nNAME                                     READY   STATUS             RESTARTS      AGE\nisbsvc-default-js-0                      3/3     Running            0             116s\nisbsvc-default-js-1                      3/3     Running            0             116s\nisbsvc-default-js-2                      3/3     Running            0             116s\n```\n\n## Run Your 1st Pipeline\n\n```shell\nkubectl apply -f https://raw.githubusercontent.com/numaproj/numaflow-workshop/main/manifests/1-simple-pipeline.yaml\n\nkubectl get pl # View pipelines\nkubectl get vtx # View vertices\nkubectl get po # Check if pods are running\n\n# Port forward the UI to https://localhost:8443/\nkubectl -n numaflow-system port-forward deployment/numaflow-server 8443:8443\n```\n\nNavigate to `default` namespace, and view the pipeline from the UI.\n\nFind the logs in the `out` vertex as below:\n\n![Out](assets/out-vertex-log.png)\n\nThe pipeline can be deleted by\n\n```shell\nkubectl delete -f https://raw.githubusercontent.com/numaproj/numaflow-workshop/main/manifests/1-simple-pipeline.yaml\n```\n\n## Build an Advanced Pipeline\n\nNow we will walk you through creating an advanced pipeline. In our example, this is called the `even-odd` pipeline, illustrated by the following diagram:\n\n![Pipeline Diagram](assets/even-odd-square.png)\n\nThere are five vertices in this example of an advanced pipeline. An `HTTP` source vertex which serves an HTTP endpoint to receive numbers as source data, a `UDF` vertex to tag the ingested numbers with the key `even` or `odd`, three `Log`sinks, one to print the `even` numbers, one to print the `odd` numbers, and the other one to print both the even and odd numbers.\n\nRun the following command to build the User-Defined Function image, it will import the built image to the local cluster.\n\n```shell\nmake image\n```\n\nRun the following command to create the `even-odd` pipeline.\n\n```shell\nkubectl apply -f https://raw.githubusercontent.com/numaproj/numaflow-workshop/main/manifests/2-even-odd-pipeline.yaml\n```\n\nYou may opt to view the list of pipelines you've created so far by running `kubectl get pipeline`. Otherwise, proceed to inspect the status of the pipeline, using `kubectl get pods`.\n\n```shell\n# Wait for pods to be ready\nkubectl get pods\n\nNAME                               READY   STATUS    RESTARTS   AGE\neven-odd-daemon-64d65c945d-vjs9f   1/1     Running   0          5m3s\neven-odd-even-or-odd-0-pr4ze       2/2     Running   0          30s\neven-odd-even-sink-0-unffo         1/1     Running   0          22s\neven-odd-in-0-a7iyd                1/1     Running   0          5m3s\neven-odd-number-sink-0-zmg2p       1/1     Running   0          7s\neven-odd-odd-sink-0-2736r          1/1     Running   0          15s\nisbsvc-default-js-0                3/3     Running   0          10m\nisbsvc-default-js-1                3/3     Running   0          10m\nisbsvc-default-js-2                3/3     Running   0          10m\n```\n\nYou also can view the pipeline from the UI at https://localhost:8443/.\n\n![Numaflow UI](assets/numaflow-ui-advanced-pipeline.png)\n\nNext, port-forward the HTTP endpoint, and make a `POST` request using `curl`.\n\n```shell\nkubectl port-forward svc/even-odd-in 8444:8443\n\n# Post data to the HTTP endpoint\ncurl -kq -X POST -d \"101\" https://localhost:8444/vertices/in\ncurl -kq -X POST -d \"102\" https://localhost:8444/vertices/in\ncurl -kq -X POST -d \"103\" https://localhost:8444/vertices/in\ncurl -kq -X POST -d \"104\" https://localhost:8444/vertices/in\n```\n\nNow you can watch the log for the `even` and `odd` vertices by running the commands below.\n\n```shell\n# Watch the log for the even vertex, or check the log from the even-sink vertex in the UI\nkubectl logs -f even-odd-even-sink-0-xxxxx\n2024/05/07 07:21:28 (even-sink)  Payload -  102  Keys -  []  EventTime -  1715066488069  Headers -  Content-Type: application/x-www-form-urlencoded, User-Agent: curl/8.4.0, Accept: */*, Content-Length: 3,\n2024/05/07 07:21:29 (even-sink)  Payload -  104  Keys -  []  EventTime -  1715066488835  Headers -  User-Agent: curl/8.4.0, Accept: */*, Content-Length: 3, Content-Type: application/x-www-form-urlencoded,\n\n# Watch the log for the odd vertex, or check the log from the odd-sink vertex in the UI\nkubectl logs -f even-odd-odd-sink-0-xxxxx\n2024/05/07 07:21:28 (odd-sink)  Payload -  101  Keys -  []  EventTime -  1715066488020  Headers -  Content-Length: 3, Content-Type: application/x-www-form-urlencoded, User-Agent: curl/8.4.0, Accept: */*,\n2024/05/07 07:21:28 (odd-sink)  Payload -  103  Keys -  []  EventTime -  1715066488133  Headers -  Accept: */*, Content-Length: 3, Content-Type: application/x-www-form-urlencoded, User-Agent: curl/8.4.0,\n\n```\n\n### Make Some Changes\n\nClone the repo, and update the logic in User-Defined Function [code](even-odd/main.go), re-run `make image`, repeat operations above to see if it reflects the changes.\n\nThe pipeline can be deleted by\n\n```shell\nkubectl delete -f https://raw.githubusercontent.com/numaproj/numaflow-workshop/main/manifests/2-even-odd-pipeline.yaml\n```\n\n## Clean Up\n\n```shell\nkubectl delete -f https://raw.githubusercontent.com/numaproj/numaflow/stable/examples/0-isbsvc-jetstream.yaml\nkubectl delete -k https://github.com/numaproj/numaflow-workshop/manifests/installation\nkubectl delete ns numaflow-system\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnumaproj%2Fnumaflow-workshop","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnumaproj%2Fnumaflow-workshop","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnumaproj%2Fnumaflow-workshop/lists"}