{"id":21847557,"url":"https://github.com/scribd/kafka-player","last_synced_at":"2025-04-14T13:33:58.022Z","repository":{"id":55375254,"uuid":"248819508","full_name":"scribd/kafka-player","owner":"scribd","description":"A simple scala app that plays messages from a file onto a Kafka topic.","archived":false,"fork":false,"pushed_at":"2021-01-04T17:47:25.000Z","size":364,"stargazers_count":6,"open_issues_count":0,"forks_count":1,"subscribers_count":8,"default_branch":"master","last_synced_at":"2023-04-10T07:46:18.649Z","etag":null,"topics":["cplat","kafka"],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/scribd.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-03-20T17:55:48.000Z","updated_at":"2023-04-10T07:46:18.649Z","dependencies_parsed_at":"2022-08-14T22:50:33.950Z","dependency_job_id":null,"html_url":"https://github.com/scribd/kafka-player","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scribd%2Fkafka-player","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scribd%2Fkafka-player/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scribd%2Fkafka-player/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scribd%2Fkafka-player/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/scribd","download_url":"https://codeload.github.com/scribd/kafka-player/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226832345,"owners_count":17689141,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cplat","kafka"],"created_at":"2024-11-27T23:18:38.103Z","updated_at":"2024-11-27T23:18:38.789Z","avatar_url":"https://github.com/scribd.png","language":"Scala","readme":"# kafka-player\n\nPlays a file onto a Kafka topic - one line == one message.\n\n### Build\n\nBuild the project by running `sbt assembly`. This produces an uber jar at `target/scala-2.12/kafka-player.jar`.\n\n### Invocation\n\nFor running against a local Kafka, invoke the player with a command like below:\n\n```\njava -jar $jar_file \\\n  --message-file $message_file \\\n  --num-messages $num_messages \\\n  --messages-per-second $messages_per_second \\\n  --topic $topic \\\n  --broker-string $broker_string\n```\n\nIf running against a TLS authenticated Kafka cluster, include additional parameters to specify the keystore location and keystore password:\n\n```\n  --keystore-location $keystore_location\n  --keystore-passphrase $keystore_passphrase\n```\n\n#### Parameters\n\n* `--message-file` - A file containing the lines to play onto Kafka. The player will play one message per line.\n* `--num-messages` - The number of messages to play. The default is 100,000 if not provided.\n* `--messages-per-second` - The number of messages to play per second. The default is 2 per second if not provided. Float values are supported (e.g. 0.5 to produce one message every two seconds).\n* `--topic` - The topic to play the messages onto.\n* `--broker-string` - The bootstrap broker string of the target cluster. The default is localhost:9092 if not provided.\n* `--keystore-location` - The location of the keystore to use for TLS authentication. Only set this if TLS authentication is desired.\n* `--keystore-passphrase` - The passphrase for the keystore. Only set this if TLS authentication is desired and a `keystore-location` is also provided.\n\n### Local Kafka\n\nA docker-compose file is included for local development for spinning up a single broker wurstmeister/kafka cluster (https://hub.docker.com/r/wurstmeister/kafka/) exposed to localhost.\nTopics required for specific scenarios may be added to the `KAFKA_CREATE_TOPICS` environment variable in `docker-compose.yml`.\nThe required format is described in https://github.com/wurstmeister/kafka-docker#automatically-create-topics.\n\n### Example\n\nThe `example` folder of this repository contains a data file called `example.json` and a launch script called `play-example.sh` to play the data file onto a local Kafka. Run the example with the following steps:\n\n```\n# start the kafka docker containers in the background - this will also create a topic called `example` on the cluster.\ndocker-compose up -d\n\n# play the example data file onto the local kafka\n./example/play-example.sh\n```\n\nOnce the player starts, run `kafkacat` in another terminal to verify your messages are coming in.\n\n```\nkafkacat -C -b localhost:9092 -t example\n```\n\nThe example plays 10,000 messages at a rate of one message every two seconds.\n\nThe `example.json` data file is an abridged and slightly transformed (into a line delimited format) version of American movies from https://github.com/jdorfman/awesome-json-datasets#movies.\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscribd%2Fkafka-player","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscribd%2Fkafka-player","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscribd%2Fkafka-player/lists"}