https://github.com/build-on-aws/opensearch-using-kafka-golang

Ingesting Data into OpenSearch using Apache Kafka and Go
https://github.com/build-on-aws/opensearch-using-kafka-golang

go kafka opensearch serverless

Last synced: about 1 month ago
JSON representation

Ingesting Data into OpenSearch using Apache Kafka and Go

• Host: GitHub
• URL: https://github.com/build-on-aws/opensearch-using-kafka-golang
• Owner: build-on-aws
• License: mit-0
• Created: 2023-07-03T16:48:12.000Z (almost 2 years ago)
• Default Branch: main
• Last Pushed: 2023-07-11T23:06:49.000Z (almost 2 years ago)
• Last Synced: 2025-02-18T01:39:28.017Z (4 months ago)
• Topics: go, kafka, opensearch, serverless
• Language: Go
• Homepage: https://community.aws/tutorials/opensearch-kafka-golang-data-ingestion
• Size: 396 KB
• Stars: 2
• Watchers: 5
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE
  • Code of conduct: CODE_OF_CONDUCT.md

Awesome Lists containing this project

README

        
## Ingesting Data into OpenSearch using Apache Kafka and Go

Scalable data ingestion is a key aspect for a large-scale distributed search and analytics engine like [OpenSearch](https://opensearch.org/). Sometimes there are specific requirements in your data pipeline which might need you to write your own integration layer. 

[This blog post](https://community.aws/posts/opensearch-kafka-golang-data-ingestion) covers how to create a data pipeline wherein data written into Apache Kafka is ingested into [OpenSearch](https://opensearch.org/). We will make use of a custom [Go](http://go.dev/) application to ingest data using Go clients for [Kafka](https://github.com/twmb/franz-go) and [OpenSearch](https://opensearch.org/docs/latest/clients/go/). 

Here is a simplified version of the application architecture that outlines the components and how they interact with each other.

![High level architecture](images/arch.jpg)

The application consists of producer and consumer components, which are Go applications deployed to an `EC2` instance):

- The producer sends data to the MSK Serverless cluster.

- The consumer application receives data (`movie` information) from the MSK Serverless topic and uses the OpenSearch Go client to index data in the `movies` collection.

## Security

See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information.

## License

This library is licensed under the MIT-0 License. See the LICENSE file.