{"id":27248867,"url":"https://github.com/bakdata/common-kafka-streams-demo","last_synced_at":"2025-07-09T04:05:08.129Z","repository":{"id":40954061,"uuid":"270681872","full_name":"bakdata/common-kafka-streams-demo","owner":"bakdata","description":"Demo Kafka Streaming NLP pipeline for the blogpost \"Continuous NLP Pipelines with Python, Java, and Apache Kafka\"","archived":false,"fork":false,"pushed_at":"2022-12-08T10:15:05.000Z","size":116,"stargazers_count":9,"open_issues_count":2,"forks_count":3,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-10T23:48:23.332Z","etag":null,"topics":["kafka","kafka-streams","topologies"],"latest_commit_sha":null,"homepage":"https://medium.com/bakdata/continuous-nlp-pipelines-with-python-java-and-apache-kafka-f6903e7e429d","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bakdata.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-08T13:36:10.000Z","updated_at":"2024-04-20T15:22:18.000Z","dependencies_parsed_at":"2023-01-25T01:01:01.671Z","dependency_job_id":null,"html_url":"https://github.com/bakdata/common-kafka-streams-demo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/bakdata/common-kafka-streams-demo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fcommon-kafka-streams-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fcommon-kafka-streams-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fcommon-kafka-streams-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fcommon-kafka-streams-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bakdata","download_url":"https://codeload.github.com/bakdata/common-kafka-streams-demo/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fcommon-kafka-streams-demo/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264390667,"owners_count":23600560,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","kafka-streams","topologies"],"created_at":"2025-04-10T23:48:20.277Z","updated_at":"2025-07-09T04:05:08.110Z","avatar_url":"https://github.com/bakdata.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Streams Bootstrap and Faust Boostrap Demo\n\nKafka Streaming NLP pipeline demonstrating how we use Python and Java together for NLP on large text files. It uses the following libraries that introduce common configuration options to Faust and Kafka Streams applications:\n\n- [streams-bootstrap](https://github.com/bakdata/streams-bootstrap) (Java)\n- [faust-bootstrap](https://github.com/bakdata/faust-bootstrap) (Python)\n\nThe example consists of four different applications:\n\n* Producer: Loads files from a local source directory, uploads them to S3, and sends a record with the corresponding S3 pointer to a specific Kafka topic.\n* Loader: Stream processor that processes the records produced by the Producer, loads the content of the files from S3, and uses faust-s3-backed-serializer to serialize the message sent to the output topic.\n* spaCy Lemmatizer: Python `faust-bootstrap` application processes the records produced by the Loader, extracts non-stop-word lemmas using spaCy, and sends them as a list serialized with Avro to the output topic.\n* TfIDF-Application: Java `streams-bootstrap` application gets documents represented as a list of lemmas as input and calculates the corresponding TFIDF scores.\n\nYou can find a [blog post on medium](https://medium.com/bakdata/continuous-nlp-pipelines-with-python-java-and-apache-kafka-f6903e7e429d) with some examples and explanations of how this demo works.\n\n## Getting Started\n\nTo run the demo, you can use a Kubernetes cluster with Apache Kafka. To deploy Apache Kafka into a Kubernetes Cluster, you can use the [Confluent Platform Helm chart](https://github.com/confluentinc/cp-helm-charts).\nIf you test it locally, you can use [minikube](https://github.com/confluentinc/cp-helm-charts#start-minikube).\n\n### Preparation\n\nYou can get sample data by running `sh scrips/get-data.sh`, the data will be downloaded and extracted into `./data`. It will contain several text files.\n\nTo create the topics you can run `sh scripts/setup.sh \u003crelease-name\u003e`. It requires a running [kafka-client](https://github.com/confluentinc/cp-helm-charts/blob/master/examples/kafka-client.yaml) pod.\n\nAdd our Kafka application Helm charts:\n```bash\nhelm repo add bakdata-common https://raw.githubusercontent.com/bakdata/streams-bootstrap/master/charts/\n```\n\nMoreover, set up an Amazon S3 bucket to use the s3-backed SerDe. You will also need your AWS credentials for the deployment.\n\n### Deploy the Applications\n\nTo deploy the Producer, spaCy lemmatizer, and TFIDF Application, you can follow these steps:\n\n#### Build the Container Images\n\nTo build the container image for the Java applications, we use [jib](https://github.com/GoogleContainerTools/jib). \nRun the following in the `java`, and `loader` folder and set the image-names:\n\n```bash\n./gradlew jibBuildTar --info --image=\u003cimage-name\u003e\n```\n\nTo build the container image for the Python application, use the Dockerfile in the respective folder: \n\n```bash\ndocker build . -t spacy-lemmatizer\n```\n \n#### Push the images to your docker registry\n\nIf you run a local minikube cluster, you can refer to [this](https://minikube.sigs.k8s.io/docs/handbook/registry/).\n\n#### Setup the `values.yaml`\n\nSet up the parameters in the  `values.yaml` file for every application with the corresponding image name in your Docker registry. \n\n#### Deploy the application to your cluster:\n\nUsing the `bakdata-common/streams-app` Helm Chart makes it straightforward to deploy KafkaStreams and Faust applications with the proposed common configuration options using the following command:\n\n```bash\nhelm upgrade --debug --install --recreate-pods --wait --timeout=300 --force \\\n--values \u003capplication\u003e/values.yaml \u003crelease\u003e bakdata-common/streams-app\n``` \nIf you did not set the credentials in the `values.yaml` file you can additionally add the parameters when running helm upgrade:\n\n```bash\nhelm upgrade --debug --install --force --values values.yaml \u003crelease_name\u003e bakdata-common/streams-app \\\n--set env.AWS_ACCESS_KEY_ID=\u003cacces_key_id\u003e \\\n--set env.AWS_SECRET_ACCESS_KEY=\u003caccess_key\u003e \\\n--set env.AWS_REGION=\u003cregion\u003e\n```\n\n### Produce Messages\n\nFinally, you can start to produce messages. To do so, you can either deploy the Producer to the cluster or run it locally.\n(Remember to set up your local AWS credential chain.)\n\nInstall the dependencies from `requirements.txt` and run:\n\n```bash\npython producer.py --broker_url=\u003cborker_urls\u003e --data_path=../data --topic=input-topic --s3_bucket=\u003cs3-bucket-uri\u003e --s3_bucket_dir=input_data\n```\n\n## License\n\nThis project is licensed under the MIT license. Have a look at the [LICENSE](https://github.com/bakdata/common-kafka-streams-demo/blob/master/LICENSE) for more details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbakdata%2Fcommon-kafka-streams-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbakdata%2Fcommon-kafka-streams-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbakdata%2Fcommon-kafka-streams-demo/lists"}