https://github.com/kiranmantha/node-expressgateway-kafka
A sample express-gateway based application with kafka streams
https://github.com/kiranmantha/node-expressgateway-kafka
Last synced: 2 months ago
JSON representation
A sample express-gateway based application with kafka streams
- Host: GitHub
- URL: https://github.com/kiranmantha/node-expressgateway-kafka
- Owner: KiranMantha
- Created: 2024-03-06T08:47:11.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2024-05-21T13:42:12.000Z (about 1 year ago)
- Last Synced: 2024-05-22T13:39:34.367Z (about 1 year ago)
- Language: TypeScript
- Size: 179 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# nestjs micro-services with kafka based event driven architecture
Event-driven Microservices with nestjs, Kafka
## Install kafka
### Without docker
- download kafka tar file from [official site](https://kafka.apache.org/documentation/#quickstart)
- extract the downloaded tar file and `cd` to that location
- run zookeeper through `bin/zookeeper-server-start.sh config/zookeeper.properties`
- once zookeeper is up and running, in another terminal execute `bin/kafka-server-start.sh config/server.properties` to start kafka
- once kafka is up and running, it need to have one topic before consuming it to emit and read events. for this in a new terminal execute `bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092`
> this is a one-time action.
- this will create a topic named `quickstart-events`
- to list out all the topics, execute `bin/kafka-topics.sh --bootstrap-server localhost:9092 --list`
### With docker
- run `docker compose up` in root folder. this will pull latest `zookeeper` & `kafka` packages and run them on their respective ports.
## Install mongodb
### With homebrew
- https://medium.com/create-a-clocking-in-system-on-react/creating-a-local-mongodb-database-and-insert-a-document-c6a4a2102a22
- run below commands:
```bash
brew tap mongodb/brew
```
```bash
brew install mongodb-community
```
- after installation, to start mongodb, run `brew services start mongodb-community`.
- to check this: install mongodb vscode extension and add a connection with connection string: `mongodb://localhost`.
- to stop mongodb, run `brew services stop mongodb-community`.
### With docker
- install docker and start it
- run `docker run --name mymongodb -d -p 27017:27017 mongo`
- this will pull the latest `mongo` image and run a container named `mymongodb` on external port 27017 and internal port 27017 in detached and published mode
- to start using local mongodb, install mongodb vscode extension and add a connection with connection string: `mongodb://localhost:27017`.
## Folder structure:
```
root/
├── api-gateway (express endpoint)
├── billing (kafka MS)
├── users (kafka MS)
└── logs (kafka MS)
```
## kafka UI
https://medium.com/@penkov.vladimir/kafka-cluster-with-ui-and-metrics-easy-setup-d12d1b94eccf
## Docker tips
1. To create docker image from `Dockerfile`, run `docker build . -t `
2. to run the created docker image, run `docker run -d --publish `
3. To view the created image in the images list, run `docker image list`. this should list the created image
4. Once completed, to verify the contents of the image, run `docker run -ti bash`
5. To verify the contents of the container, run `docker exec -it bash`
6. To delete a docker image, run `docker image rm -f`
7. To create images from a `yml` file, run `docker compose build`. This will by default pick `docker-compose.yml` and `Dockerfile` files. To build image through a different config file then run `docker compose -f build`.
8. To run containers through `docker-compose.yaml` file, run `docker compose up`. For a different yaml file, run `docker compose -f up`
## Starting Application
### Prerequisites
- Docker
- Docker compose
### To run in local
1. start mongodb, kafka in docker by running `docker compose -f docker-compose.local.yml up`
2. start nest services by running `yarn start`. this will run all services using `.env.local`
### To run via docker
1. build docker images for all services by running `docker compose -f docker-build.yml build`
2. once above step is done, run `docker compose up`. this will run all services through `docker-compose.yml`
### To verify services
run contents in below bash file through postman and check terminal logs.
```bash
url:
http://localhost:3020/create-invoice
method:
POST
payload:
{
"userId": "uid-123",
"orderId": "oid-123",
"price": 123.45
}
```
once the request is made, check consoles of all the micro-services. each should log the data as per.
- all the kafka events will be logged into mongodb database by logs microservice through `mongoose`.
- logs table contains `id (primary key auto-generated)`, `serviceName (type String)`, `kafkaTopic (type String)` and `kafkaData (type Object)`
## Kubernetes Tips
1. To give an alias name for your kubectl: `alias k="microk8s kubectl"`
2. To create objects (service/deployment) from (multipls)config file: `kubectl create -f config1.yaml -f config2.yaml`
3. To delete objects defined in (multipls)config file: `kubectl delete -f config1.yaml -f config2.yaml`
4. To update objects by overriding (multipls)live configuration: `kubectl replace -f config1.yaml -f config2.yaml`
5. To delete pod/deploy/svc by name: `kubectl delete deploy/pod/svc `
6. To list out all pods, services, deployments: `kubectl get pods,deploy,svc`
7. To list out all pods, services, deployments for a given metadata label with key _app_:
- run `kubectl get pods,deploy,svc -l app=` or
- run `kubectl get pods,deploy,svc -l 'app in ()'`
- more info: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
8. To scale any deployment by name: `kubectl scale deployment/ --replicas=5`
9. To check if a deployment is working successfully or not:
- run `kubectl get pods`
- run `kubectl exec -it sh`
- run `nc -zv localhost ` => this should be success confirming that the pod is running as expected.
10. To start/pause/stop microk8s, run `microk8s start/pause/stop`.
## Reference links:
- https://medium.com/@haridasan/write-dynamic-dockerfile-for-nestjs-monorepo-183989f24fa8
- https://www.youtube.com/watch?v=dBrZzEpIOMk
- https://www.baeldung.com/ops/kafka-docker-setup
- https://gist.github.com/KiranMantha/5ba4a72fd8d01879ea642c6ca43194ed
- https://dzone.com/articles/how-to-deploy-apache-kafka-with-kubernetes
- https://kubernetes.io/docs/concepts/overview/working-with-objects/object-management/
- https://zeet.co/blog/kubernetes-deployment-vs-pod
- https://microk8s.io/docs/getting-started
- https://gist.github.com/dongjinleekr/fcadc20063553935cfb6536185421ca2
- https://robertbrem.github.io/Microservices_with_Kubernetes/19_CQRS_with_Kafka/01_Setup_Kafka/#setup-kafka-in-the-cluster
- https://www.youtube.com/watch?v=IS5AL3cQb1o
- https://github.com/aws-actions/amazon-ecr-login
- https://docs.github.com/en/actions/deployment/deploying-to-your-cloud-provider/deploying-to-amazon-elastic-container-service
- https://www.youtube.com/watch?v=DVrGXjjkpig
- https://gist.github.com/KiranMantha/da6fb40718eb34558c12ad8dff25b519