{"id":20041174,"url":"https://github.com/santthosh/kafka-for-mac","last_synced_at":"2025-05-05T08:32:13.416Z","repository":{"id":37821368,"uuid":"123175540","full_name":"santthosh/kafka-for-mac","owner":"santthosh","description":"Getting Confluent Kafka components to work on Mac OS X","archived":false,"fork":false,"pushed_at":"2019-01-23T08:28:49.000Z","size":14,"stargazers_count":50,"open_issues_count":3,"forks_count":24,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-08T19:47:40.882Z","etag":null,"topics":["confluentinc","kafka-broker","kafka-rest","macos","schema-registry","zookeeper"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/santthosh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-02-27T19:12:07.000Z","updated_at":"2023-10-11T14:24:43.000Z","dependencies_parsed_at":"2022-08-19T11:21:03.818Z","dependency_job_id":null,"html_url":"https://github.com/santthosh/kafka-for-mac","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/santthosh%2Fkafka-for-mac","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/santthosh%2Fkafka-for-mac/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/santthosh%2Fkafka-for-mac/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/santthosh%2Fkafka-for-mac/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/santthosh","download_url":"https://codeload.github.com/santthosh/kafka-for-mac/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252466779,"owners_count":21752430,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["confluentinc","kafka-broker","kafka-rest","macos","schema-registry","zookeeper"],"created_at":"2024-11-13T10:45:38.555Z","updated_at":"2025-05-05T08:32:13.137Z","avatar_url":"https://github.com/santthosh.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# kafka-for-mac\n\nFor those who use macOS and docker, it can be quite challenging to get several Confluent Kafka components up and running. Support for Docker on macOS / Windows is not yet available as per Confluent. This repo is an attempt to remove the uncertainty and let people experiment with Confluent Kafka faster. Here are some references for you to understand why this challenging\n\n[I can't connect to Kafka on OS X](https://github.com/confluentinc/cp-docker-images/issues/265)\n\n[Note: Important Caveat - Images Not Tested for Docker for Mac or Windows](https://github.com/confluentinc/cp-docker-images)\n\n[Networking features in Docker for Mac](https://docs.docker.com/docker-for-mac/networking/)\n\n## Pre-requisites\n\nAs of this writing, my local is running\n\n`Docker version 17.12.0-ce, build c97c6d6r`\n\n## Getting Started\n\nThis docker-compose file has all the components necessary to bring up single node / single broker Kafka cluster,\n\n* Zookeeper\n* Kafka Broker\n* Schema Registry\n* Kafka REST Proxy\n\n### Bootstrap\n\n`docker-compose up -d`\n\nYou should see some results like this\n\n```\nCreating kafkaformac_kafka_1           ... done\nCreating kafkaformac_zookeeper_1       ...\nCreating kafkaformac_schema-registry_1 ... done\nCreating kafkaformac_schema-registry_1 ...\nCreating kafkaformac_kafka-rest_1      ... done\n```\n\n### Ensure the processes are running\n\n`docker-compose ps`\n\nYou should see some results like this\n```\n            Name                         Command            State                           Ports\n--------------------------------------------------------------------------------------------------------------------------\nkafkaformac_kafka-rest_1        /etc/confluent/docker/run   Up      0.0.0.0:29080-\u003e29080/tcp, 8082/tcp\nkafkaformac_kafka_1             /etc/confluent/docker/run   Up      0.0.0.0:29092-\u003e29092/tcp, 9092/tcp\nkafkaformac_schema-registry_1   /etc/confluent/docker/run   Up      0.0.0.0:29081-\u003e29081/tcp, 8081/tcp\nkafkaformac_zookeeper_1         /etc/confluent/docker/run   Up      2181/tcp, 2888/tcp, 0.0.0.0:32181-\u003e32181/tcp, 3888/tcp\n```\n\nCheck the health of zookeeper by monitoring its logs through this command\n\n`docker-compose logs zookeeper | grep -i binding`\n\nCheck if the kafka broker is healthy\n\n`docker-compose logs kafka | grep -i started`\n\nSimilarly inspect logs for schema-registry and kafka-rest\n\n`docker-compose logs schema-registry`\n\n`docker-compose logs kafka-rest`\n\n### Consuming and Producing using the Kafka clients\n* Add a mapping with your machine ip to  `docker.for.mac.host.internal` to your `/etc/hosts` file\n  * e.g. `10.0.4.162      docker.for.mac.host.internal`\n  * find your machine ip with `ifconfig | grep -e \"inet \"`\n  * (!) your machine ip changes if you change network, update the config and restart the container\n* Connect to the kafka cluster via `localhost:29092` (bootstrap server)\n\n### Publishing and Subscribing using Kafka REST\n\n[confluentinc/kafka-rest](https://github.com/confluentinc/kafka-rest) has full documentation around this, but the validate.sh script allows you to run all the same quickly, go ahead and run it\n\n`./validate.sh`\n\nYou should see the RESTful endpoints invoked in sequence for publishing and subscribing from your Kafka\n\n```\n# Get a list of topics\n[\"_schemas\",\"jsontest\"]\n# Produce a message with JSON data\n{\"offsets\":[{\"partition\":0,\"offset\":6,\"error_code\":null,\"error\":null}],\"key_schema_id\":null,\"value_schema_id\":null}\n# Get info about one topic\n{\"name\":\"jsontest\",\"configs\":{\"file.delete.delay.ms\":\"60000\",\"segment.ms\":\"604800000\",\"min.compaction.lag.ms\":\"0\",\"retention.bytes\":\"-1\",\"segment.index.bytes\":\"10485760\",\"cleanup.policy\":\"delete\",\"follower.replication.throttled.replicas\":\"\",\"message.timestamp.difference.max.ms\":\"9223372036854775807\",\"segment.jitter.ms\":\"0\",\"preallocate\":\"false\",\"segment.bytes\":\"1073741824\",\"message.timestamp.type\":\"CreateTime\",\"message.format.version\":\"0.11.0-IV2\",\"max.message.bytes\":\"1000012\",\"unclean.leader.election.enable\":\"false\",\"retention.ms\":\"604800000\",\"flush.ms\":\"9223372036854775807\",\"delete.retention.ms\":\"86400000\",\"leader.replication.throttled.replicas\":\"\",\"min.insync.replicas\":\"1\",\"flush.messages\":\"9223372036854775807\",\"compression.type\":\"producer\",\"min.cleanable.dirty.ratio\":\"0.5\",\"index.interval.bytes\":\"4096\"},\"partitions\":[{\"partition\":0,\"leader\":1,\"replicas\":[{\"broker\":1,\"leader\":true,\"in_sync\":true}]}]}\n# Create a consumer for JSON data, starting at the beginning of the topic's log. The consumer group is called \"my_json_consumer\" and the instance is \"my_consumer_instance\\  .\n{\"instance_id\":\"my_consumer_instance\",\"base_uri\":\"http://kr-host:8082/consumers/my_json_consumer/instances/my_consumer_instance\"}\n# Subscribe the consumer to a topic\nNo content in response\n# Then consume some data from a topic using the base URL in the first response.\n[]\n# Finally, close the consumer with a DELETE to make it leave the group and clean up its resources.\nNo content in response\n```\n\n### Tear down\n\n`docker-compose down`\n\n## Caveats\n\n* You really don't want bridge network on your docker compose, while its not well documented on why, its advised so. [confluentinc/cp-docker-images](https://github.com/confluentinc/cp-docker-images/wiki/Getting-Started)\n\n## Contributing\n\nFork this repository, send in a pull request\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsantthosh%2Fkafka-for-mac","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsantthosh%2Fkafka-for-mac","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsantthosh%2Fkafka-for-mac/lists"}