{"id":13566503,"url":"https://github.com/conduktor/kafka-security-manager","last_synced_at":"2025-04-10T01:06:32.803Z","repository":{"id":37989416,"uuid":"123456662","full_name":"conduktor/kafka-security-manager","owner":"conduktor","description":"Manage your Kafka ACL at scale","archived":false,"fork":false,"pushed_at":"2024-02-01T13:34:50.000Z","size":270,"stargazers_count":368,"open_issues_count":32,"forks_count":162,"subscribers_count":40,"default_branch":"master","last_synced_at":"2025-04-10T01:06:25.670Z","etag":null,"topics":["acl","acl-changes","broker","docker","kafka","ksm","security","zookeeper"],"latest_commit_sha":null,"homepage":"https://hub.docker.com/r/simplesteph/kafka-security-manager","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/conduktor.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-03-01T15:50:51.000Z","updated_at":"2025-03-03T08:25:20.000Z","dependencies_parsed_at":"2024-07-21T02:03:14.613Z","dependency_job_id":null,"html_url":"https://github.com/conduktor/kafka-security-manager","commit_stats":null,"previous_names":["simplesteph/kafka-security-manager"],"tags_count":23,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conduktor%2Fkafka-security-manager","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conduktor%2Fkafka-security-manager/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conduktor%2Fkafka-security-manager/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conduktor%2Fkafka-security-manager/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/conduktor","download_url":"https://codeload.github.com/conduktor/kafka-security-manager/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248137887,"owners_count":21053775,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["acl","acl-changes","broker","docker","kafka","ksm","security","zookeeper"],"created_at":"2024-08-01T13:02:10.917Z","updated_at":"2025-04-10T01:06:32.781Z","avatar_url":"https://github.com/conduktor.png","language":"Scala","funding_links":[],"categories":["Scala","Security"],"sub_categories":["Authentication \u0026 Authorization"],"readme":"[![Actions Status](https://github.com/conduktor/kafka-security-manager/workflows/ScalaCI/badge.svg)](https://github.com/conduktor/kafka-security-manager/actions)\n\n# An open-source project by [![Conduktor.io](https://docs.conduktor.io/assets/svgs/logo.svg)](https://conduktor.io/)\n\nThis project is sponsored by [conduktor.io](https://www.conduktor.io/). Conduktor provides a platform to help central teams defining global security and governance controls and developers have access to a central Kafka console and get a self-serve experience.\nIt also helps visualize your ACLs (attached to Service Accounts) in your Apache Kafka cluster!\n\n![Kafka Security Manager Diagram](https://github.com/conduktor/kafka-security-manager/assets/3936459/55fd98c6-63e1-4f77-a093-53ab9208ea16)\n\n# Kafka Security Manager\n\nKafka Security Manager (KSM) allows you to manage your Kafka ACLs at scale by leveraging an external source as the source of truth. \nZookeeper just contains a copy of the ACLs instead of being the source.\n\n![Kafka Security Manager Diagram](https://i.imgur.com/BuikeuB.png)\n\nThere are several advantages to this:\n- **Kafka administration is done outside of Kafka:** anyone with access to the external ACL source can manage Kafka Security\n- **Prevents intruders:** if someone were to add ACLs to Kafka using the CLI, they would be reverted by KSM within 10 seconds.\n- **Full auditability:** KSM provides the guarantee that ACLs in Kafka are those in the external source. Additionally, if for example your external source is GitHub, then PRs, PR approvals and commit history will provide Audit the full log of who did what to the ACLs and when\n- **Notifications**: KSM can notify external channels (such as Slack) in order to give feedback to admins when ACLs are changed. This is particularly useful to ensure that 1) ACL changes are correctly applied 2) ACL are not changed in Kafka directly.\n\nYour role is to ensure that Kafka Security Manager is never down, as it is now a custodian of your ACL.\n\n## Parsers\n\n### CSV\nThe csv parser is the default parser and also the fallback one in case no other parser is matched.\n\nThis is a sample CSV acl file:\n```\nKafkaPrincipal,ResourceType,PatternType,ResourceName,Operation,PermissionType,Host\nUser:alice,Topic,LITERAL,foo,Read,Allow,*\nUser:bob,Group,PREFIXED,bar,Write,Deny,12.34.56.78\nUser:peter,Cluster,LITERAL,kafka-cluster,Create,Allow,*\n```\n**Important Note**: As of KSM 0.4, a new column `PatternType` has been added to match the changes that happened in Kafka 2.0. This enables KSM to manage `LITERAL` and `PREFIXED` ACLs. See #28\n\n### YAML\nThe yaml parser will load ACLs from yaml instead, to activate the parser just provide files with `yml` or `yaml` extension.\n\nAn example YAML permission file might be:\n```yaml\nusers:\n  alice:\n    topics:\n      foo:\n        - Read\n      bar*:\n        - Produce\n  bob:\n    groups:\n      bar:\n        - Write,Deny,12.34.56.78\n      bob*:\n        - All\n    transactional_ids:\n      bar-*:\n        - All\n  peter:\n    clusters:\n      kafka-cluster:\n        - Create\n```\nThe YAML parser will handle automatically prefix patterns by simply appending a star to your resource name.\n\nIt also supports some helpers to simplify setup:\n- Consume (Read, Describe)\n- Produce (Write, Describe, Create, Cluster Create)\n\n## Sources\n\nCurrent sources shipping with KSM include:\n- File\n- GitHub\n- GitLab (using Personal Auth Tokens)\n- BitBucket\n- Amazon S3\n- Build your own (and contribute back!)\n\n# Building\n\n```\nsbt clean test\nsbt universal:stage\n```\nFat JAR:\n```\nsbt clean assembly\n```\nThis is a Scala app and therefore should run on the JVM like any other application\n\n# Artifacts\n\nBy using the JAR dependency, you can create your own `SourceAcl`.\n\nRELEASES artifacts are deployed to [Maven Central](http://search.maven.org/#search%7Cga%7C1%7Cio.conduktor):\n\n`build.sbt` (see [Maven Central](http://search.maven.org/#search%7Cga%7C1%7Cio.conduktor) for the latest `version`)\n```\nlibraryDependencies += \"io.conduktor\" %% \"kafka-security-manager\" % \"version\"\n```\n\n# Configuration\n\n## Security configuration - Zookeeper client\n\nMake sure the app is using a property file and launch options similar to your broker so that it can\n1. Authenticate to Zookeeper using secure credentials (usually done with JAAS)\n2. Apply Zookeeper ACL if enabled\n\n*Kafka Security Manager does not connect to Kafka.*\n\nSample run for a typical SASL Setup:\n```\ntarget/universal/stage/bin/kafka-security-manager -Djava.security.auth.login.config=conf/jaas.conf\n```\n\nWhere `conf/jaas.conf` contains something like:\n```\nClient {\n    com.sun.security.auth.module.Krb5LoginModule required\n    useKeyTab=true\n    storeKey=true\n    keyTab=\"/etc/kafka/secrets/zkclient1.keytab\"\n    principal=\"zkclient/example.com@EXAMPLE.COM\";\n};\n```\n\n## Security configuration - Admin client\nWhen configured authorizer class is `io.conduktor.ksm.compat.AdminClientAuthorizer`,\n`kafka-security-manager` will use kafka admin client instead of direct zookeeper connection.\nConfiguration example would be\n```\nKafkaClient {\n  org.apache.kafka.common.security.plain.PlainLoginModule required\n  username=\"admin\"\n  password=\"admin-secret\";\n};\n```\n\n## Configuration file\n\nFor a list of configuration see [application.conf](src/main/resources/application.conf). You can customise them using environment variables or create your own `application.conf` file and pass it at runtime doing:\n```\ntarget/universal/stage/bin/kafka-security-manager -Dconfig.file=path/to/config-file.conf\n```\n\nOverall we use the [lightbend config](https://github.com/lightbend/config) library to configure this project.\n\n## Environment variables\nThe [default configurations](src/main/resources/application.conf) can be overwritten using the following environment variables:\n\n- `KSM_READONLY=false`: enables KSM to synchronize from an External ACL source. The default value is `true`, which prevents KSM from altering ACLs in Zookeeper\n- `KSM_EXTRACT_ENABLE=true`: enable extract mode (get all the ACLs from Kafka formatted as a CSV or YAML)\n- `KSM_EXTRACT_FORMAT=csv`: selects which format to extract the ACLs with (defaults to csv, supports also yaml)\n- `KSM_REFRESH_FREQUENCY_MS=10000`: how often to check for changes in ACLs in Kafka and in the Source. 10000 ms by default. If it's set to `0` or negative value, for example `-1`, then KMS executes ACL synchronization just once and exits\n- `KSM_NUM_FAILED_REFRESHES_BEFORE_NOTIFICATION=1`: how many times that the refresh of a Source needs to fail (e.g. HTTP timeouts) before a notification is sent. Any value less than or equal to `1` here will notify on every failure to refresh.\n- `AUTHORIZER_CLASS`: authorizer class for ACL operations. Default is `SimpleAclAuthorizer`, configured with\n  - `AUTHORIZER_ZOOKEEPER_CONNECT`: zookeeper connection string\n  - `AUTHORIZER_ZOOKEEPER_SET_ACL=true` (default `false`): set to true if you want your ACLs in Zookeeper to be secure (you probably do want them to be secure) - when in doubt set as the same as your Kafka brokers.\n\n  No-zookeeper authorizer class on top of Kafka Admin Client is bundled with KSM as `io.conduktor.ksm.compat.AdminClientAuthorizer`,\n  configured with options for `org.apache.kafka.clients.admin.AdminClientConfig`:\n  - `ADMIN_CLIENT_ID` - `client.id`, an id to pass to the server when making requests, for tracing/audit purposes, default `kafka-security-manager`\n  Properties below are not provided to client unless environment variable is set:\n  - `ADMIN_CLIENT_BOOTSTRAP_SERVERS` - `bootstrap.servers`\n  - `ADMIN_CLIENT_SECURITY_PROTOCOL` - `security.protocol`\n  - `ADMIN_CLIENT_SASL_JAAS_CONFIG` -`sasl.jaas.config` - alternative to system jaas configuration\n  - `ADMIN_CLIENT_SASL_MECHANISM` - `sasl.mechanism`\n  - `ADMIN_CLIENT_SSL_KEY_PASSWORD` - `ssl.key.password`\n  - `ADMIN_CLIENT_SSL_KEYSTORE_LOCATION` - `ssl.keystore.location`\n  - `ADMIN_CLIENT_SSL_KEYSTORE_PASSWORD` - `ssl.keystore.password`\n  - `ADMIN_CLIENT_SSL_TRUSTSTORE_LOCATION` - `ssl.truststore.location`\n  - `ADMIN_CLIENT_SSL_TRUSTSTORE_PASSWORD` - `ssl.truststore.password`\n\n- `SOURCE_CLASS`: Source class. Valid values include\n    - `io.conduktor.ksm.source.NoSourceAcl` (default): No source for the ACLs. Only use with `KSM_READONLY=true`\n    - `io.conduktor.ksm.source.FileSourceAcl`: get the ACL source from a file on disk. Good for POC\n    - `io.conduktor.ksm.source.GitHubSourceAcl`: get the ACL from GitHub. Great to get started quickly and store the ACL securely under version control.\n    - `io.conduktor.ksm.source.GitLabSourceAcl`: get the ACL from GitLab using personal access tokens. Great to get started quickly and store the ACL securely under version control.\n      - `SOURCE_GITLAB_REPOID` GitLab project id\n      - `SOURCE_GITLAB_FILEPATH` Path to the ACL file in GitLab project\n      - `SOURCE_GITLAB_BRANCH` Git Branch name\n      - `SOURCE_GITLAB_HOSTNAME` GitLab Hostname\n      - `SOURCE_GITLAB_ACCESSTOKEN` GitLab Personal Access Token. See [Personal access tokens\n](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) to authenticate with the GitLab API.\n    - `io.conduktor.ksm.source.S3SourceAcl`: get the ACL from S3. Good for when you have a S3 bucket managed by Terraform or Cloudformation. This requires `region`, `bucketname` and `objectkey`. See [Access credentials](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html) for credentials management.\n      - `SOURCE_S3_REGION` AWS S3 Region\n      - `SOURCE_S3_BUCKETNAME` AWS S3 Bucket name\n      - `SOURCE_S3_OBJECTKEY` The Object containing the ACL CSV in S3\n    - `io.conduktor.ksm.source.BitbucketServerSourceAcl`: get the ACL from Bitbucket Server using the v1 REST API. Great if you have private repos in Bitbucket.\n    - `io.conduktor.ksm.source.BitbucketCloudSourceAcl`: get the ACL from Bitbucket Cloud using the Bitbucket Cloud REST API v2.\n    - `io.conduktor.ksm.source.HttpSourceAcl`: get the ACL from an HTTP endpoint. You can enable [Google OAuth OIDC Token Authentication](https://cloud.google.com/docs/authentication/production).\n        - `SOURCE_HTTP_URL` HTTP endpoint to retrieve ACL data.\n        - `SOURCE_HTTP_METHOD` HTTP Method. Default is `GET`.\n        - `SOURCE_HTTP_AUTH_TYPE` To enable Http Authentication. `googleiam` for Google IAM. Default is `NONE`.\n        - `SOURCE_HTTP_AUTH_GOOGLEIAM_SERVICE_ACCOUNT` Google Service Account name.\n        - `SOURCE_HTTP_AUTH_GOOGLEIAM_SERVICE_ACCOUNT_KEY` Google Service Account Key in JSON string encoded. If not the key isn't configured, it'll try to get the token from environment.\n        - `SOURCE_HTTP_AUTH_GOOGLEIAM_TARGET_AUDIENCE` Google Target Audience for token authentication.\n\n- `NOTIFICATION_CLASS`: Class for notification in case of ACL changes in Kafka.\n    - `io.conduktor.ksm.notification.ConsoleNotification` (default): Print changes to the console. Useful for logging\n    - `io.conduktor.ksm.notification.SlackNotification`: Send notifications to a Slack channel (useful for devops / admin team)\n- `ACL_PARSER_CSV_DELIMITER`: Change the delimiter character for the CSV Parser (useful when you have SSL)\n\n# Running on Docker\n\n## Building the image\n\n```\n./build-docker.sh\n```\n\n## Docker Hub\n\nAlternatively, you can get the automatically built Docker images on [Docker Hub](https://hub.docker.com/r/conduktor/kafka-security-manager)  \n\n## Running\n\n(read above for configuration details)\n\nThen apply to the docker run using for example (in EXTRACT mode):\n\n```\ndocker run -it -e AUTHORIZER_ZOOKEEPER_CONNECT=\"zookeeper-url:2181\" -e KSM_EXTRACT_ENABLE=true \\\n            conduktor/kafka-security-manager:latest\n```\n\nAny of the environment variables described above can be used by the docker run command with the `-e ` options.\n\n## Example\n\n```\ndocker-compose up -d\ndocker-compose logs kafka-security-manager\n# view the logs, have fun changing example/acls.csv\ndocker-compose down\n```\n\nFor full usage of the docker-compose file see [kafka-security-manager](https://github.com/conduktor/kafka-security-manager)\n\n## Extracting ACLs\n\nYou can initially extract all your existing ACL in Kafka by running the program with the config `ksm.extract.enable=true` or `export KSM_EXTRACT_ENABLE=true`\n\nOutput should look like:\n```\n[2018-03-06 21:49:44,704] INFO Running ACL Extraction mode (ExtractAcl)\n[2018-03-06 21:49:44,704] INFO Getting ACLs from Kafka (ExtractAcl)\n[2018-03-06 21:49:44,704] INFO Closing Authorizer (ExtractAcl)\n\nKafkaPrincipal,ResourceType,PatternType,ResourceName,Operation,PermissionType,Host\nUser:bob,Group,PREFIXED,bar,Write,Deny,12.34.56.78\nUser:alice,Topic,LITERAL,foo,Read,Allow,*\nUser:peter,Cluster,LITERAL,kafka-cluster,Create,Allow,*\n```\n\nYou can then use place this CSV anywhere and use it as your source of truth.\n\n# Compatibility\n\nKSM Version | Kafka Version | Notes\n--- | --- | ---\n1.1.0-SNAPSHOT | 2.8.x | updated log4j dependency\n1.0.1 | 2.8.x | updated log4j dependency\n0.11.0 | 2.5.x | renamed packages to `io.conduktor`. Breaking change on extract config name\n0.10.0 | 2.5.x | YAML support\u003cbr\u003eAdd configurable num failed refreshes before notification \n0.9 | 2.5.x | Upgrade to Kafka 2.5.x\n0.8 | 2.3.1 | Add a \"run once\" mode\n0.7 | 2.1.1 | Kafka Based ACL refresher available (no zookeeper dependency)\n0.6 | 2.0.0 | important stability fixes - please update\n0.5 | 2.0.0 |\n0.4 | 2.0.0 | important change: added column 'PatternType' in CSV\n0.3 | 1.1.x |\n0.2 | 1.1.x | upgrade to 0.3 recommended\n0.1 | 1.0.x | might work for earlier versions\n\n# Contributing\n\nYou can break the API / configs as long as we haven't reached 1.0. Each API break would introduce a new version number.\n\nPRs are welcome, especially with the following:\n- Code refactoring  / cleanup / renaming\n- External Sources for ACLs (JDBC, Microsoft AD, etc...)\n- Notification Channels (Email, etc...)\n\nPlease open an issue before opening a PR.\n\n# Release process\n\n- update version in [build.sbt] (make sure to use format vX.Y.Z)\n- update [README.md] and [CHANGELOG.md]\n- push the tag (eg: `v0.10.0`)\n- update version in [build.sbt] to the next snapshot version\n\nThat's it !\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconduktor%2Fkafka-security-manager","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fconduktor%2Fkafka-security-manager","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconduktor%2Fkafka-security-manager/lists"}