{"id":13783687,"url":"https://github.com/lensesio/kafka-connect-tools","last_synced_at":"2025-05-11T19:31:20.904Z","repository":{"id":52858838,"uuid":"54712511","full_name":"lensesio/kafka-connect-tools","owner":"lensesio","description":"Kafka Connect Tooling","archived":true,"fork":false,"pushed_at":"2021-04-17T11:41:48.000Z","size":12093,"stargazers_count":120,"open_issues_count":9,"forks_count":33,"subscribers_count":38,"default_branch":"master","last_synced_at":"2024-05-18T22:22:31.799Z","etag":null,"topics":["cli","kafka","kafka-connect"],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lensesio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-03-25T10:37:33.000Z","updated_at":"2023-10-16T12:30:48.000Z","dependencies_parsed_at":"2022-09-02T12:30:26.722Z","dependency_job_id":null,"html_url":"https://github.com/lensesio/kafka-connect-tools","commit_stats":null,"previous_names":["datamountaineer/kafka-connect-tools"],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lensesio%2Fkafka-connect-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lensesio%2Fkafka-connect-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lensesio%2Fkafka-connect-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lensesio%2Fkafka-connect-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lensesio","download_url":"https://codeload.github.com/lensesio/kafka-connect-tools/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253621043,"owners_count":21937468,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","kafka","kafka-connect"],"created_at":"2024-08-03T19:00:28.517Z","updated_at":"2025-05-11T19:31:20.450Z","avatar_url":"https://github.com/lensesio.png","language":"Scala","funding_links":[],"categories":["Operations"],"sub_categories":["Tools"],"readme":"\nConnect tools is Maven\n\n```bash\n\u003cdependency\u003e\n\t\u003cgroupId\u003ecom.datamountaineer\u003c/groupId\u003e\n\t\u003cartifactId\u003ekafka-connect-cli\u003c/artifactId\u003e\n\t\u003cversion\u003e1.0.7\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n## Requirements\n\n* Java 1.8\n* Gradle 5\n\nKafka Connect CLI\n=================\n\nThis is a tiny command line interface (CLI) around the [Kafka Connect REST Interface](http://docs.confluent.io/3.0.1/connect/userguide.html#rest-interface) to manage connectors.\nIt is used in a git like fashion where the first program argument indicates the command: it can be one of `[ps|get|rm|create|run|status|status|plugins|describe|validate|restart|pause|resume]`.\n\nThe CLI is meant to behave as a good unix citizen: input from `stdin`; output to `stdout`; \nout of band info to `stderr` and non-zero exit status on error. Commands \ndealing with configuration expect or produce data in .properties \nstyle: `key=value` lines and comments start with a `#`.\n\n    connect-cli 1.0.8\n    Usage: connect-cli [ps|get|rm|create|run|diff|status|plugins|describe|validate|restart|pause|resume] [options] [\u003cconnector-name\u003e]\n\n      --help\n            prints this usage text\n      -e \u003cvalue\u003e | --endpoint \u003cvalue\u003e\n            Kafka Connect REST URL, default is http://localhost:8083/\n      -f \u003cvalue\u003e | --format \u003cvalue\u003e\n            Format of the config, default is PROPERTIES. Valid options are 'properties' and 'json'.\n\n      Command: ps\n      list active connectors names.\n      \n      Command: get\n      get the configuration of the specified connector.\n      \n      Command: rm\n      remove the specified connector.\n      \n      Command: create\n      create the specified connector with the config from stdin; the connector cannot already exist.\n      \n      Command: run\n      create or update the specified connector with the config from stdin.\n      \n      Command: diff\n      diff the specified connector with the config from stdin.\n      \n      Command: status\n      get connector and it's task(s) state(s).\n      \n      Command: plugins\n      list the available connector class plugins on the classpath.\n      \n      Command: describe\n      list the configurations for a connector class plugin on the classpath.\n      \n      Command: pause\n      pause the specified connector.\n      \n      Command: restart\n      restart the specified connector.\n      \n      Command: resume\n      resume the specified connector.\n      \n      Command: validate\n      validate the connector config from stdin against a connector class plugin on the classpath.\n      \n      Command: task_ps\n      list the tasks belonging to a connector.\n      \n      Command: task_status\n      get the status of a connector task.\n      \n      Command: task_restart\n      restart the specified connector task.\n    \nYou can override the default endpoint by setting an environment variable `KAFKA_CONNECT_REST` i.e.\n\n    export KAFKA_CONNECT_REST=\"http://myserver:myport\"\n\nTo Build\n========\n\n```bash\ngradle buildCli\n```\n\n\nUsage\n=====\n\nClone this repository, do a `gradle buildCli` and run the jar in a way you prefer, for example with the provided `cli` shell script. The CLI can be used as follows.\n\nGet Active Connectors\n---------------------\n\nCommand: `ps`\n\nExample:\n\n    $ bin/connect-cli ps\n    twitter-source\n\nGet Connector Configuration\n---------------------------\n\nCommand: `get`\n\nExample:\n\n    $ bin/connect-cli get twitter-source\n    #Connector `twitter-source`:\n    name=twitter-source\n    tasks.max=1\n\n    (snip)\n\n    track.terms=test\n    #task ids: 0\n\nDelete a Connector\n------------------\n\nCommand: `rm`\n\nExample:\n\n    $ bin/connect-cli rm twitter-source\n\nCreate a New Connector\n----------------------\n\nThe connector cannot already exist.\n\nCommand: `create`\n\nExample:\n\n    $ bin/connect-cli create twitter-source \u003ctwitter.properties\n    #Connector `twitter-source`:\n    name=twitter-source\n    tasks.max=1\n\n    (snip)\n\n    track.terms=test\n    #task ids: 0\n\nCreate or Update a Connector\n----------------------------\n\nEither starts a new connector if it did not exist, or update an existing connector.\n\nCommand: `run`\n\nExample:\n\n    $ bin/connect-cli run twitter-source \u003ctwitter.properties\n    #Connector `twitter-source`:\n    name=twitter-source\n    tasks.max=1\n\n    (snip)\n\n    track.terms=test\n    #task ids: 0\n\nDiff a Connector\n----------------------------\n\nDiffs a connector config with the provided config.\n\nCommand: `diff`\n\nExample:\n\n    $ bin/connect-cli run twitter-source \u003ctwitter.properties\n    #Connector `twitter-source`:\n    name=twitter-source\n    tasks.max=1\n\n    (snip)\n\n    track.terms=test\n    #task ids: 0\n\nQuery Connector Status\n----------------------\n\nShows a connector's status and the state of its tasks.\n\nCommand: `status`\n\nExample:\n\n    bin/connect-cli status my-toy-connector\n    connectorState: RUNNING\n    numberOfTasks: 3\n    tasks:\n      - taskId: 0\n        taskState: RUNNING\n      - taskId: 1\n        taskState: FAILED\n        trace: java.lang.Exception: broken on purpose\n        at java.lang.Thread.run(Thread.java:745)\n      - taskId: 2\n        taskState: FAILED\n        trace: java.lang.Exception: broken on purpose\n        at java.lang.Thread.run(Thread.java:745)\n        \nCheck which Plugins are on the Classpath and available in the Connect Cluster\n-----------------------------------------------------------------------------\n        \n Shows which Connector classes are available on the classpath.\n        \nCommand: `plugins`\n        \nExample:  \n        \n        bin/connect-cli plugins\n        Class name: com.datamountaineeer.streamreactor.connect.blockchain.source.BlockchainSourceConnector\n        Class name: com.datamountaineer.streamreactor.connect.elastic.ElasticSinkConnector\n        Class name: com.datamountaineer.streamreactor.connect.druid.DruidSinkConnector\n        Class name: io.confluent.connect.hdfs.HdfsSinkConnector\n        Class name: io.confluent.connect.jdbc.JdbcSourceConnector\n        Class name: com.datamountaineer.streamreactor.connect.hbase.HbaseSinkConnector\n        Class name: org.apache.kafka.connect.file.FileStreamSourceConnector\n        Class name: com.datamountaineer.streamreactor.connect.hazelcast.sink.HazelCastSinkConnector\n        Class name: com.datamountaineer.streamreactor.connect.cassandra.sink.CassandraSinkConnector\n        Class name: com.datamountaineer.streamreactor.connect.rethink.source.ReThinkSourceConnector\n        Class name: com.datamountaineer.streamreactor.connect.jms.sink.JMSSinkConnector\n        Class name: com.datamountaineer.streamreactor.connect.influx.InfluxSinkConnector\n        Class name: com.datamountaineer.streamreactor.connect.redis.sink.RedisSinkConnector\n        Class name: com.datamountaineer.streamreactor.connect.bloomberg.BloombergSourceConnector\n        Class name: com.datamountaineer.streamreactor.connect.yahoo.source.YahooSourceConnector\n        Class name: com.datamountaineer.streamreactor.connect.kudu.sink.KuduSinkConnector\n        Class name: org.apache.kafka.connect.file.FileStreamSinkConnector\n        Class name: com.datamountaineer.streamreactor.connect.cassandra.source.CassandraSourceConnector\n        Class name: com.datamountaineer.streamreactor.connect.voltdb.VoltSinkConnector\n        Class name: com.datamountaineer.streamreactor.connect.mongodb.sink.MongoSinkConnector\n        Class name: com.datamountaineer.streamreactor.connect.rethink.sink.ReThinkSinkConnector\n        Class name: io.confluent.connect.hdfs.tools.SchemaSourceConnector\n        \nDescribe the configuration of a Connector\n-----------------------------------------\n\nDescribes the configuration parameters for a Connector.\n\nCommand: `describe`\n\nExample:\n\n    bin/connect-cli describe ReThinkSinkConnector\n    {\n      \"name\": \"com.datamountaineer.streamreactor.connect.rethink.sink.ReThinkSinkConnector\",\n      \"error_count\": 3,\n      \"groups\": [\"Common\", \"Connection\"],\n      \"configs\": [{\n        \"definition\": {\n          \"name\": \"connector.class\",\n          \"display_name\": \"Connector class\",\n          \"importance\": \"HIGH\",\n          \"order\": 2,\n          \"default_value\": \"\",\n          \"dependents\": [],\n          \"type\": \"STRING\",\n          \"required\": true,\n          \"group\": \"Common\"\n        },\n        \"value\": {\n          \"name\": \"connector.class\",\n          \"recommended_values\": [],\n          \"errors\": [\"Missing required configuration \\\"connector.class\\\" which has no default value.\"],\n          \"visible\": true\n        }\n      }, {\n    ...........\n    \n Validate a Connectors properties file against the required Configurations\n -------------------------------------------------------------------------\n      \n Given a properties file for an instance of a Connector validate it against the Connector configuration.\n      \n Command: `validate`\n      \n Example: \n      \n      bin/connect-cli validate ReThinkSinkConnector \u003c ../conf/quickstarts/rethink-sink.properties\n      ..............\n         \"definition\": {\n                \"name\": \"connect.rethink.sink.port\",\n                \"display_name\": \"connect.rethink.sink.port\",\n                \"importance\": \"MEDIUM\",\n                \"order\": 3,\n                \"default_value\": \"28015\",\n                \"dependents\": [],\n                \"type\": \"INT\",\n                \"required\": false,\n                \"group\": \"Connection\"\n              },\n              \"value\": {\n                \"name\": \"connect.rethink.sink.port\",\n                \"visible\": true,\n                \"errors\": [],\n                \"recommended_values\": [],\n                \"value\": \"28015\"\n              }\n            }]\n          }\n          Validation failed.\n          Missing required configuration \"connect.rethink.sink.export.route.query\" which has no default value.]: \n \nPause a Connector\n-----------------\n   \nCommand: `pause`\n\nExample: \n   \n    bin/connect-cli pause cassandra-sink\n    Waiting for pause\n    connectorState:  RUNNING\n    workerId: 10.0.0.9:8083\n    numberOfTasks: 1\n    tasks:\n     - taskId: 0\n       taskState: RUNNING\n       workerId: 10.0.0.9:8083\n       \nResume a Connector\n------------------\n   \nCommand: `resume`\n\nExample: \n   \n    bin/connect-cli resume cassandra-sink\n    Waiting for resume\n    connectorState:  RUNNING\n    workerId: 10.0.0.9:8083\n    numberOfTasks: 1\n    tasks:\n     - taskId: 0\n       taskState: RUNNING\n       workerId: 10.0.0.9:8083   \n       \nRestart a Connector\n-------------------\n   \nCommand: `restart`\n\nExample: \n   \n    bin/connect-cli restart cassandra-sink\n    Waiting for restart\n    connectorState:  RUNNING\n    workerId: 10.0.0.9:8083\n    numberOfTasks: 1\n    tasks:\n     - taskId: 0\n       taskState: RUNNING\n       workerId: 10.0.0.9:8083          \n\nList all Tasks of a Connector\n-----------------------------\n\nCommand: `task_ps`\n\nExample:\n\n    bin/connect-cli tasks_ps cassandra-sink\n    - cassandra-sink task 0\n      connector.class: com.datamountaineer.streamreactor.connect.cassandra.sink.CassandraSinkConnector\n      bootstrap.servers: kafka-broker1:6667,kafka-broker2:6667,kafka-broker3:6667\n      producer.schema.registry.url:http://schema-registry:8081\n      ...\n      \nGet the status of a Connector Task\n----------------------------------\n\nCommand: `task_status`\n\nExample:\n\n    bin/connect-cli task_status cassandra-sink 0\n    taskId: 0\n    taskState: RUNNING\n    workerId: 10.0.0.9:8083\n    \nRestart a Connector Task\n------------------------\n\nCommand: `task_restart`\n\nExample:\n\n    bin/connect-cli task_restart cassandra-sink 0\n\nMisc\n====\n\nContributions are encouraged, feedback to [rollulus](https://keybase.io/rollulus) at xs4all dot nl.\n\nThanks, enjoy!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flensesio%2Fkafka-connect-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flensesio%2Fkafka-connect-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flensesio%2Fkafka-connect-tools/lists"}