https://github.com/mihaitodor/knative-benthos
Batch sentiment analysis with benthos and Knative
https://github.com/mihaitodor/knative-benthos
batching benthos knative kubernetes sentiment-analysis
Last synced: 5 months ago
JSON representation
Batch sentiment analysis with benthos and Knative
- Host: GitHub
- URL: https://github.com/mihaitodor/knative-benthos
- Owner: mihaitodor
- License: mit
- Created: 2022-05-17T01:01:17.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2022-05-27T22:54:23.000Z (over 3 years ago)
- Last Synced: 2025-04-28T14:09:35.868Z (5 months ago)
- Topics: batching, benthos, knative, kubernetes, sentiment-analysis
- Language: Shell
- Homepage:
- Size: 1.06 MB
- Stars: 9
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Scalable sentiment analysis on Twitter data using Benthos and Knative
Demo project for the KnativeCon Europe 2022 KubeCon EU co-located event.
The talk recording is [here](https://www.youtube.com/watch?v=3OaRXwcRJJk&list=PLj6h78yzYM2Mv__LXlfzAAw5WMUTzIvS8) and the slides are [here](https://github.com/mihaitodor/Presentations/tree/master/knativecon_europe/2022-05-17).
Based on Carlos Santana's Knative on Kind (KonK): https://github.com/csantanapr/knative-kind
Benthos with sentiment analysis support: https://github.com/mihaitodor/benthos-vader
Twitter dataset source: https://www.kaggle.com/datasets/crowdflower/twitter-airline-sentiment
## Start Kind cluster and install Knative Serving
```shell
> ./install_serving.sh
> kubectl get pods -A
> watch kubectl get pods
```
## Deploy the benthos-vader lambda
```shell
> kubectl create configmap benthos-vader-config --from-file=benthos.yaml=./benthos/benthos_vader.yaml
> ./deploy_benthos_vader.sh
```
## Test sentiment analysis
```shell
> curl -v -X POST -d '{"text": "I love Benthos!"}' http://benthos-vader.default.127.0.0.1.sslip.io/post
```
## Run Postgres and populate the SOURCE table
```shell
> docker run --rm -it -p 5432:5432 -e POSTGRES_PASSWORD=password postgres
> # Populate SOURCE table in DB
> ./populate_db.sh
> # Connect to DB
> pgcli postgres://postgres:password@localhost:5432/postgres
>> SELECT COUNT(*) FROM SOURCE
```
## Run Prometheus
```shell
> prometheus --config.file=prometheus.yaml
> http://localhost:9090/graph?g0.expr=rate(output_sent{}[10s])&g0.tab=0&g0.range_input=5m
```
## Run Benthos batch workflow
```shell
> time benthos -c ./benthos/benthos_batch.yaml
```
## Teardown
```shell
> kind delete cluster --name knative
```