https://github.com/stuartleeks/dapr-pubsub-pipeline-ts-python
Example of a Dapr pubsub pipeline with TypeScript and Python
https://github.com/stuartleeks/dapr-pubsub-pipeline-ts-python
Last synced: about 1 year ago
JSON representation
Example of a Dapr pubsub pipeline with TypeScript and Python
- Host: GitHub
- URL: https://github.com/stuartleeks/dapr-pubsub-pipeline-ts-python
- Owner: stuartleeks
- Created: 2023-12-08T11:09:44.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-12-11T13:31:09.000Z (over 2 years ago)
- Last Synced: 2025-02-17T10:49:08.663Z (over 1 year ago)
- Language: Shell
- Size: 34.2 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# dapr-pubsub-pipeline-ts-python
Example of a Dapr pubsub pipeline with TypeScript and Python
- [dapr-pubsub-pipeline-ts-python](#dapr-pubsub-pipeline-ts-python)
- [Overview](#overview)
- [Running the project](#running-the-project)
- [Cleaning up Dapr logs](#cleaning-up-dapr-logs)
- [Running in Kubernetes](#running-in-kubernetes)
## Overview
The project is made up of a number of services as shown in the following diagram:
```
message individual
batch messages
┌─────────┐ ┌────────────────┐ ┌────────────────────┐
│ │ │ │ │ │
│ batcher ├──────────────►│ batch_receiver ├────────────────►│ message_processor │ ... further
│ │ │ │ │ │ processing here
└─────────┘ └────────────────┘ └──┬─────────────────┘
│
│ HTTP API
│ (rate-limited)
│
▼
┌────────────────────┐
│ │
│ processing_service │
│ │
└────────────────────┘
```
| Service | Language | Description |
| ------------------ | ---------- | ------------------------------------------------------------------------------------------------------------------ |
| batcher | TypeScript | Simulates periodic retrieval of message content for processing and puts the content onto the `input-batches` queue |
| batch_receiver | Python | Receives batches of messages from the `input-batches` queue and splits them before sending to the `messages` queue |
| message_processor | Python | Receives individual messages from the `messages` queue and processes them using the `processing_service` |
| processing_service | Python | A rate-limited HTTP API that simulates doing work |
## Running the project
The easiest way to run the project is to load it in Visual Studio Code using the Dev Containers extension as this automatically sets up the environment for you.
Once open as a dev container:
1. Run `dapr init` to initialise Dapr
2. Run `dapr run -f ./dapr.yaml` to start the services
3. ... TODO: how to view the results
## Cleaning up Dapr logs
When running, logs for each service are written to a `.dapr` folder under each service folder.
These can be cleaned up by running `find . -name ".dapr" | rm -rf` from the root of the project.
## Running in Kubernetes
To deploy to AKS, copy `sample.env` to `.env` and update the values.
Then run `just deploy`.
This will create a Service Bus namespace and queues, Container Registry, and AKS cluster.
It will then build and push the container images and deploy the services to AKS.
After deployment, run `kubectl get pods` to see the deployed pods.