{"id":24789115,"url":"https://github.com/gotbordom/kafka-data-prepper-opensearch","last_synced_at":"2026-05-15T08:07:05.410Z","repository":{"id":245947021,"uuid":"819638817","full_name":"gotbordom/kafka-data-prepper-opensearch","owner":"gotbordom","description":"Simple implementation of Kafka writing to opensearch using data prepper and docker","archived":false,"fork":false,"pushed_at":"2024-06-25T20:39:17.000Z","size":29,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-29T17:16:07.320Z","etag":null,"topics":["docker","docker-compose","kafka","opensearch"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gotbordom.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-24T23:17:47.000Z","updated_at":"2025-01-17T00:08:06.000Z","dependencies_parsed_at":"2024-06-25T00:36:43.881Z","dependency_job_id":"c88724ab-2416-4a51-a5fe-539cd2d74c4b","html_url":"https://github.com/gotbordom/kafka-data-prepper-opensearch","commit_stats":null,"previous_names":["gotbordom/kafka-data-prepper-opensearch"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gotbordom%2Fkafka-data-prepper-opensearch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gotbordom%2Fkafka-data-prepper-opensearch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gotbordom%2Fkafka-data-prepper-opensearch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gotbordom%2Fkafka-data-prepper-opensearch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gotbordom","download_url":"https://codeload.github.com/gotbordom/kafka-data-prepper-opensearch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245267569,"owners_count":20587459,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","kafka","opensearch"],"created_at":"2025-01-29T17:16:08.993Z","updated_at":"2025-10-25T02:39:19.725Z","avatar_url":"https://github.com/gotbordom.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"#### Summary\nThis repo is intended for learning how to connect kafka to opensearch using a data-prepper. Given this it is using self signed keys for openssl. In a prod system this should not be done this way.\n\n#### Notes\n* Be aware that this was all done locally, and at this time all my paths are hard coded. So be aware that directly copying this and running the steps will not work copy and paste for you.\n* This relys on exposing the host machine's local IP addresss to docker to get around the issue of docker not correclty using localhost. For anyone else's implementation be sure to replace the following IP addresses with your own.\n    - pipelines.yaml: bootstrap_server: point this to xxx.xxx.xxx.1:19093 for your hostmachine.\n    - docker-compose.yml: KAFKA_ADVERTISED_LISTENERS for the ssl \nconnection, point this also the your hostmachine ip.\n    - kafka-1.cnf: whatever your hostname ip is needs to be added to the alt_names of the cert configuration BEFORE creating your certs, and keystores.\n* For the previous step, there are ways to just have a command grab this for you. I haven't written it yet, when I do I will likey update this.\n\n#### Dependencies\n1. docker\n2. docker-compose\n3. openssl\n\n#### Using this repo\n1. Clone the repo: `git clone git@github.com:gotbordom/kafka-data-prepper-opensearch.git`\n2. Step into the repo: `cd /path/to/kafka-data-prepper-opensearch/`\n3. Follow steps for creating self signed openssl certs, these steps are below\n4. Spin up all the containers: `docker compose up`\n5. Start a kafka producer in the docker container. If you have kafka locally it can be done there as well, so just adjust the docker command below accordingly.\n```bash\n// In the docker container\ndocker exec -it kafka-1 /bin/kafka-console-producer \\\n    --bootstrap-server localhost:19093 \\\n    --topic test-topic \\\n    --producer.config /etc/kafka/secrets/client-creds/client-ssl.properties\n```\n6. Should see the message written into the producer printed out by the terminal running all the containers.\n\n\n#### How the self signed certs were created for this work\n* Make sure to read the notes at the top about IP Addresses to use before running all of these steps.\n1. Authority key and cert\n```bash\nopenssl req -new -nodes \\\n   -x509 \\\n   -days 365 \\\n   -newkey rsa:2048 \\\n   -keyout /home/atracy/repositories/opensearch/secrets/ca.key \\\n   -out /home/atracy/repositories/opensearch/secrets/ca.crt \\\n   -config /home/atracy/repositories/opensearch/secrets/ca.cnf\n```\n2. Create a pem file from cery and key\n```bash\ncat /home/atracy/repositories/opensearch/secrets/ca.crt /home/atracy/repositories/opensearch/secrets/ca.key \u003e /home/atracy/repositories/opensearch/secrets/ca.pem\n```\n3. Server key and Authentication certificate\n```bash\nopenssl req -new \\\n    -newkey rsa:2048 \\\n    -keyout /home/atracy/repositories/opensearch/secrets/kafka-1-creds/kafka-1.key \\\n    -out /home/atracy/repositories/opensearch/secrets/kafka-1-creds/kafka-1.csr \\\n    -config /home/atracy/repositories/opensearch/secrets/kafka-1-creds/kafka-1.cnf \\\n    -nodes\n```\n4. Create and sign self signed cert\n```bash\nopenssl x509 -req \\\n    -days 3650 \\\n    -in /home/atracy/repositories/opensearch/secrets/kafka-1-creds/kafka-1.csr \\\n    -CA /home/atracy/repositories/opensearch/secrets/ca.crt \\\n    -CAkey /home/atracy/repositories/opensearch/secrets/ca.key \\\n    -CAcreateserial \\\n    -out /home/atracy/repositories/opensearch/secrets/kafka-1-creds/kafka-1.crt \\\n    -extfile /home/atracy/repositories/opensearch/secrets/kafka-1-creds/kafka-1.cnf \\\n    -extensions v3_req\n```\n5. Convert server cert to pkcs12 format\n```bash\nopenssl pkcs12 -export \\\n    -in /home/atracy/repositories/opensearch/secrets/kafka-1-creds/kafka-1.crt \\\n    -inkey /home/atracy/repositories/opensearch/secrets/kafka-1-creds/kafka-1.key \\\n    -chain \\\n    -CAfile /home/atracy/repositories/opensearch/secrets/ca.pem \\\n    -name kafka-1 \\\n    -out /home/atracy/repositories/opensearch/secrets/kafka-1-creds/kafka-1.p12 \\\n    -password pass:confluent\n```\n6. Create the broker's keystore and import keys\n```bash\nkeytool -importkeystore \\\n    -deststorepass confluent \\\n    -destkeystore /home/atracy/repositories/opensearch/secrets/kafka-1-creds/kafka.kafka-1.keystore.pkcs12 \\\n    -srckeystore /home/atracy/repositories/opensearch/secrets/kafka-1-creds/kafka-1.p12 \\\n    -deststoretype PKCS12  \\\n    -srcstoretype PKCS12 \\\n    -noprompt \\\n    -srcstorepass confluent\n```\n7. Make the client's truststore\n```bash\nkeytool -keystore /home/atracy/repositories/opensearch/secrets/kafka-1-creds/client-creds/kafka.client.truststore.pkcs12 \\\n    -alias CARoot \\\n    -import \\\n    -file /home/atracy/repositories/opensearch/secrets/ca.crt \\\n    -storepass confluent \\\n    -noprompt \\\n    -storetype PKCS12\n```\n8. Save the credentials for keystore and key\n```bash\nsudo tee /home/atracy/repositories/opensearch/secrets/kafka-1-creds/kafka-1_ssl_key_creds \u003c\u003c EOF \u003e/dev/null\nconfluent\nEOF\n\nsudo tee /home/atracy/repositories/opensearch/secrets/kafka-1-creds/kafka-1_keystore_creds \u003c\u003c EOF \u003e/dev/null\nconfluent\nEOF\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgotbordom%2Fkafka-data-prepper-opensearch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgotbordom%2Fkafka-data-prepper-opensearch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgotbordom%2Fkafka-data-prepper-opensearch/lists"}