{"id":20046434,"url":"https://github.com/okdp/okdp-spark-auth-filter","last_synced_at":"2025-05-05T09:31:35.408Z","repository":{"id":224897788,"uuid":"764523830","full_name":"OKDP/okdp-spark-auth-filter","owner":"OKDP","description":"Oauth2/OIDC Authentication filter for Apache Spark Apps/History UIs","archived":false,"fork":false,"pushed_at":"2025-04-01T07:16:50.000Z","size":905,"stargazers_count":9,"open_issues_count":0,"forks_count":8,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-04-08T20:51:35.360Z","etag":null,"topics":["apache-spark","docker","k8s-spark","kubernetes","oauth2","oidc-client","openid-connect","spark-kubernetes"],"latest_commit_sha":null,"homepage":"https://okdp.io","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OKDP.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-28T08:29:55.000Z","updated_at":"2025-04-02T08:49:58.000Z","dependencies_parsed_at":"2024-06-25T16:42:11.313Z","dependency_job_id":"fbfb261a-373b-404f-a0c9-1bef18084cd6","html_url":"https://github.com/OKDP/okdp-spark-auth-filter","commit_stats":null,"previous_names":["okdp/okdp-spark-auth-filter"],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OKDP%2Fokdp-spark-auth-filter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OKDP%2Fokdp-spark-auth-filter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OKDP%2Fokdp-spark-auth-filter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OKDP%2Fokdp-spark-auth-filter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OKDP","download_url":"https://codeload.github.com/OKDP/okdp-spark-auth-filter/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252471480,"owners_count":21753191,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-spark","docker","k8s-spark","kubernetes","oauth2","oidc-client","openid-connect","spark-kubernetes"],"created_at":"2024-11-13T11:23:52.354Z","updated_at":"2025-05-05T09:31:35.384Z","avatar_url":"https://github.com/OKDP.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Build](https://github.com/okdp/okdp-spark-auth-filter/actions/workflows/ci.yml/badge.svg?branch=main)\n[![License Apache2](https://img.shields.io/hexpm/l/plug.svg)](http://www.apache.org/licenses/LICENSE-2.0)\n\n[Apache Spark](https://spark.apache.org/) extension filter to enable Oauth2/OpenID Connect based authentication for Spark UIs and Spark History.\n\n![Architecture](docs/images/project-components.png)\n\nThe project consists of two main components:\n\n1. `Authentication filter`: Authenticates the user against an Oauth2/OIDC provider by implementing the [Authorization Code grant flow](https://datatracker.ietf.org/doc/html/rfc6749).\n The filter supports all the providers compliant with Oauth2 and OpenID standards.\n3. `Authorization provider`: An additional optional layer, on top of the `Authentication filter`, authorizes Spark UI/History UI user access \nby comparing the user email and/or groups and/or roles returned by the Oauth2/OIDC provider during the authentication phase with the configured [spark ACLs](https://spark.apache.org/docs/latest/security.html#authentication-and-authorization)\n\n# Authorization Grant support matrix\n\n| Authorization Grant | Support | Description |\n|:--------------------------------------|:------------------:|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| `Authorization Code` | :heavy_check_mark: | Confidential clients (server side apps/trusted environments) - [Authorization Code standard flow](https://datatracker.ietf.org/doc/html/rfc6749#section-4.1). |\n| `Authorization Code + PKCE extension` | :heavy_check_mark: | Confidential clients (Server side apps/trusted environments) - [Authorization Code flow with PKCE extension for Confidential clients](https://datatracker.ietf.org/doc/html/draft-ietf-oauth-v2-1-10). \u003c/br\u003e [OAuth 2.1 requirement](https://oauth.net/2.1/). |\n| `Authorization Code + PKCE extension` | :heavy_check_mark: | Public clients (SPAs/Native apps/Untrusted environments use cases) - [Authorization Code flow with PKCE extension for Public clients](https://datatracker.ietf.org/doc/html/rfc7636) | \n\nThe following authorization grants are not suitable, hence not supported:\n\n| Authorization Grant | Support | Description |\n|:---------------------------------------|:-------:|------------------------------------------------------------------------------------|\n| `Implicit` | NA | Deprecated (replaced by `Authorization Code + PKCE extension for public clients`). |\n| `Resource Owner Password Credentials` | NA | Not suitable. |\n| `Client Credentials` | NA | Not suitable. |\n\n\n# Installation\n\nThe different releases are published to [Maven Central Repository](https://repo1.maven.org/maven2/io/okdp/okdp-spark-auth-filter/).\n\nPlease, check the [latest release note](https://github.com/OKDP/okdp-spark-auth-filter/releases/latest) and download the latest version from [Maven Central Repository](https://central.sonatype.com/artifact/io.okdp/okdp-spark-auth-filter/versions).\n\n1. Using Docker\n\n```shell\nADD https://repo1.maven.org/maven2/io/okdp/okdp-spark-auth-filter/1.3.4/okdp-spark-auth-filter-1.3.4.jar ${SPARK_HOME}/jars\n```\n\n2. Using Maven\n\n```xml\n\u003cdependency\u003e\n \u003cgroupId\u003eio.okdp\u003c/groupId\u003e\n \u003cartifactId\u003eokdp-spark-auth-filter\u003c/artifactId\u003e\n \u003cversion\u003e1.3.4\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n3. Spark on Yarn/Standalone mode\n\nCopy the jar https://repo1.maven.org/maven2/io/okdp/okdp-spark-auth-filter/1.3.4/okdp-spark-auth-filter-1.3.4.jar into `${SPARK_HOME}/jars/` in the different spark nodes\n\n# Configuration\n\n## Create an Oauth2/OIDC client\n\nCreate an Oauth2/OIDC client with an `Authorization Code grant` flow (confidential client).\n\nSet the redirect URL to a valid spark UI or Spark History UI home page.\n\nFor [keycloak](https://www.keycloak.org/docs/latest/server_admin/#_oidc_clients):\n1. Confidential Clients:\n * Access Type: `Confidential`\n * Standard Flow Enabled: `Enabled`\n * Implicit Flow Enabled: `Disabled`\n * Direct Access Grants Enabled: `Disabled`\n2. Public Clients:\n * Access Type: `Public`\n * Standard Flow Enabled: `Enabled`\n * Implicit Flow Enabled: `Disabled`\n * Direct Access Grants Enabled: `Disabled`\n\nOnce done, save the `client_id` and `client_secret` (Confidential clients only) into your secret management vault.\n\n## Configure the authentication filter\n\nThe filter relies on the spark [spark.ui.filters](https://spark.apache.org/docs/latest/configuration.html) configuration property.\n\n| Property | Equivalent env variable | Default | Description |\n|:---------------------------|-------------------------------|:----------------------------------------:|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| `issuer-uri` | `AUTH_ISSUER_URI` | - | OIDC Provider issuer URL\u003c/br\u003eThis is used to discover OIDC endpoints |\n| `client-id` | `AUTH_CLIENT_ID` | - | The Oauth2/OIDC client Id |\n| `client-secret` | `AUTH_CLIENT_SECRET` | - | The Oauth2/OIDC client secret \u003c/br\u003e * Mandatory for Confidential Clients. \u003c/br\u003e * Optional for Public clients. \u003c/br\u003e |\n| `redirect-uri` | `AUTH_REDIRECT_URI` | - | Spark UI/History home page\u003c/br\u003eex.: https://spark-history.example.com/home |\n| `scope` | `AUTH_SCOPE` | - | The scope(s) requested by the Authorization Request.\u003c/br\u003eExample: `openid+profile+email+roles+offline_access` |\n| `use-pkce` | `AUTH_USE_PKCE` | auto | * `true`: Force the usage of PKCE (The OIDC provider should support it). \u003c/br\u003e* `false`: Disable the usage of PKCE for confidential clients. \u003c/br\u003e * `auto`: Detect if OIDC provider supports PKCE and use it, otherwise use Authorization Code standard flow. |\n| `cookie-max-age-minutes` | `AUTH_COOKE_MAX_AGE_MINUTES` | 12 * 60 | The maximum spark-cookie cookie duration in minutes |\n| `cookie-cipher-secret-key` | `AUTH_COOKIE_ENCRYPTION_KEY` | - | Cookie encryption key\u003c/br\u003e Can be generated using: `openssl enc -aes-128-cbc -k \u003cPASS PHRASE\u003e -P -md sha1 -pbkdf2` |\n| `cookie-is-secure` | `AUTH_COOKE_IS_SECURE` | true | When enabled, the cookie is transmitted over a secure connection only (HTTPS).\u003c/br\u003e Disable the option if your run with a non secure connection (HTTP) |\n| `user-id` | `AUTH_USER_ID` | email | * `email`: set the id seen by spark acls as the email filled in the access token. \u003c/br\u003e * `sub`: set the id seen by spark acls as the sub filled in the access. \u003c/br\u003e * `google`: set the id to the sub sent by google but remove the prefix 'account.google.com:'. |\n| `jwt-header` | `JWT_HEADER` | jwt_token | Header that may contain the JWT Token that will be used for authentication. If not present, it will fall back with the default autentication workflow with a redirection on the login page. |\n| `jwt-header-signing-alg` | `JWT_HEADER_SIGNING_ALG` | RS256, ES256 | Signature algorithm used to verify the JWT Token provided. |\n| `jwt-header-issuer` | `JWT_HEADER_ISSUER` | issuer-uri from well known configuration | Issuer if different from the default issuer uri retrieved from the well known configuration fetched with 'issuer-uri' parameter. |\n| `jwt-header-jwks-uri` | `JWT_HEADER_JWKS_URI` | jwks uri from well known configuration | JWKS URI used to retrieve the key needed to verify the JWT token signature. By default will use the JWKS URI filled in the well known configuration fetched with 'issuer-uri' parameter. |\n| `ignore-refresh-token` | `IGNORE_REFRESH_TOKEN` | false | * `true`: Ignore refresh token storage in the cookie (Prevent exceeding the cookie size limit). \u003c/br\u003e* `false`: Store the refresh token in the cookie. |\n\n\u003e [!NOTE]\n\u003e 1. `issuer-uri` property or `AUTH_ISSUER_URI` env variable\n\u003e \n\u003e Try to access the endpoint `\u003cissuer-uri\u003e/.well-known/openid-configuration` (public access) to check if the `issuer-uri` is valid. \n\u003e \n\u003e This should return the different authentication endpoints (authorization, access token, user info endpoints, supported scopes etc.).\n\u003e \n\u003e For keycloack, the default `issuer-uri` is at `https://\u003ckeycloak.example.com\u003e/auth/realms/master/` and `https://\u003ckeycloak.example.com\u003e/auth/realms/master/.well-known/openid-configuration` is the well known configuration endpoint.\n\u003e \n\u003e \n\u003e 2. `cookie-cipher-secret-key` property or `AUTH_COOKIE_ENCRYPTION_KEY` env variable\n\u003e \n\u003e Generate the cookie encryption key by issuing the command:\n\u003e\n\u003e ```shell \n\u003e openssl enc -aes-128-cbc -k \u003cYOUR_PASS_PHRASE\u003e -P -md sha1 -pbkdf2\n\u003e ```\n\u003e \n\u003e \n\u003e 3. `scope` property or `AUTH_SCOPE` env variable\n\u003e \n\u003e The minimum required scope to turn on authentication is: `openid+profile+email`\n\u003e\n\u003e Add offline scope `offline_access` to enable the refresh token\n\u003e\n\u003e Add the `roles` and/or `groups` scope to enable role/groups based authorization\n\u003e\n\u003e It is not necessary to add groups and/or roles `scope` if you only need the authentication, and you don't need the authorization.\n\u003e\n\u003e N.B.: Please, note that the `groups` scope is not supported by the most OIDC providers. You can check the supported scopes at `\u003cissuer-uri\u003e/.well-known/openid-configuration` url.\n\u003e \n\u003e N.B.: Keycloack supports returning the groups for a user by adding `Group Membership mapper` to your client\n\u003e \n\u003e \n\u003e 4. `use-pkce` property or `AUTH_USE_PKCE` env variable\n\u003e \n\u003e The default value is `auto` to automatically detect if the OIDC provider supports PKCE.\n\u003e \n\u003e When the OIDC provider supports PKCE, the filter uses the PKCE flow automatically.\n\u003e \n\u003e Other values: `true` or `false` to enforce the property manually.\n\u003e \n\u003e N.B.: The OIDC provider must support PKCE extension in order to use Public Clients.\n\u003e \n\u003e \n\u003e 5. `cookie-is-secure` property or `AUTH_COOKE_IS_SECURE` env variable\n\u003e \n\u003e It's recommended to secure the connection to your spark UIs by enabling HTTPS. Although, the spark cookie is encrypted, it's recommended to send it over an encrypted connection.\n\u003e \n\u003e By default, the property is enabled. Disable the property if your connection is not secure otherwise the cookie will not be sent\n\u003e \n\n### Basic configuration\n\nThe filter can be enabled either by setting the properties globally in the `spark-defaults.properties`\n\n```properties\nspark.ui.filters=io.okdp.spark.authc.OidcAuthFilter\nspark.io.okdp.spark.authc.OidcAuthFilter.param.issuer-uri=\u003cissuer-uri\u003e\nspark.io.okdp.spark.authc.OidcAuthFilter.param.client-id=\u003cclient-id\u003e\n\n# Comment this line if your client-id is public and your OIDC provider support PKCE\nspark.io.okdp.spark.authc.OidcAuthFilter.param.client-secret=\u003cclient-secret\u003e\n\nspark.io.okdp.spark.authc.OidcAuthFilter.param.redirect-uri=\u003credirect-uri\u003e\nspark.io.okdp.spark.authc.OidcAuthFilter.param.scope=\u003cscope\u003e\n\n# Keep the default value 'auto'\n# spark.io.okdp.spark.authc.OidcAuthFilter.param.use-pkce=\u003ctrue|false|auto\u003e\n\nspark.io.okdp.spark.authc.OidcAuthFilter.param.cookie-max-age-minutes=480\nspark.io.okdp.spark.authc.OidcAuthFilter.param.cookie-cipher-secret-key=\u003ccookie-cipher-secret-key\u003e\nspark.io.okdp.spark.authc.OidcAuthFilter.param.cookie-is-secure=\u003ctrue|false\u003e\nspark.io.okdp.spark.authc.OidcAuthFilter.param.user-id=\u003csub|email\u003e\n```\n\nOr during the job submission like the following:\n\n```shell\nspark-submit --conf spark.ui.filters=io.okdp.spark.authc.OidcAuthFilter \\\n--conf spark.io.okdp.spark.authc.OidcAuthFilter.param.issuer-uri=\u003cissuer-uri\u003e \\\n--conf spark.io.okdp.spark.authc.OidcAuthFilter.param.client-id=\u003cclient-id\u003e \\\n--conf spark.io.okdp.spark.authc.OidcAuthFilter.param.client-secret=\u003cclient-secret\u003e \\\n--conf spark.io.okdp.spark.authc.OidcAuthFilter.param.redirect-uri=\u003credirect-uri\u003e \\\n--conf spark.io.okdp.spark.authc.OidcAuthFilter.param.scope=\u003cscope\u003e \\\n--conf spark.io.okdp.spark.authc.OidcAuthFilter.param.cookie-max-age-minutes=480 \\\n--conf spark.io.okdp.spark.authc.OidcAuthFilter.param.cookie-cipher-secret-key=\u003ccookie-cipher-secret-key\u003e \\\n--conf spark.io.okdp.spark.authc.OidcAuthFilter.param.cookie-is-secure=\u003ctrue|false\u003e \\\n--conf spark.io.okdp.spark.authc.OidcAuthFilter.param.user-id=\u003csub|email\u003e \\\n--class ...\n```\n\nRemove the following configuration if your client id is public and your OIDC provider supports PKCE:\n\n```shell\n--conf spark.io.okdp.spark.authc.OidcAuthFilter.param.client-secret=\u003cclient-secret\u003e\n```\n\n### Kubernetes configuration\n\nThe properties can also be passed by their equivalent env variables. \n\nYou can save the client_id, client secret and the cookie encryption key in a kubernetes secret and reference it as an env variable like the following:\n\n```yaml\nenv:\n- name: AUTH_ISSUER_URI\n value: \u003cissuer-uri\u003e\n- name: AUTH_CLIENT_ID\n valueFrom:\n secretKeyRef:\n- name: AUTH_REDIRECT_URI\n value: \u003credirect-uri\u003e\n- name: AUTH_SCOPE\n value: openid+profile+email+roles+offline_access\n# Keep the default value of AUTH_USE_PKCE as auto, other values: true, false\n#- name: AUTH_USE_PKCE\n# value: auto\n# Remove the AUTH_CLIENT_SECRET if your client id is public and your OIDC provider supports PKCE\n- name: AUTH_CLIENT_SECRET\n valueFrom:\n secretKeyRef:\n- name: AUTH_COOKIE_ENCRYPTION_KEY\n valueFrom:\n secretKeyRef:\n```\n\n\n## Configure the authorization provider (optional)\n### Overview\n\nAn example of a raw access token returned by the Oauth2/OIDC provider during a successful authentication is like the following:\n\n```json\n{\n \"access_token\": \"eyJhbGciOiJI6-auxZsE6...\",\n \"token_type\": \"bearer\",\n \"expires_in\": 86399,\n \"refresh_token\": \"ChlvaWJmNXBuaG1rdWN0e...\",\n \"id_token\": \"eyJhbGciOiJSUzI1NiIsImtpZCI6IjBkZWEw...\"\n}\n```\n\nThe token payload after [decoding](https://jwt.io/) the `access_token` is as follows:\n\n```json\n{\n \"iss\": \"\u003cissuer-uri\u003e\",\n \"sub\": \"CgNib2ISBGxkYXA\",\n \"aud\": \"\u003cclient-id\u003e\",\n \"exp\": 1708476719,\n \"iat\": 1708390319,\n \"at_hash\": \"x_kKHrjGfnSfkjDwIGPPbg\",\n \"email\": \"bob@example.org\",\n \"email_verified\": true,\n \"groups\": [\"admins\", \"team1\", \"/team2\"],\n \"roles\": [\"role-team1\", \"admin-role\"],\n \"name\": \"bob\"\n}\n```\n\nThe `\"email\"`, `\"groups\"` and/or `\"roles\"` can be mapped in Spark ACLs to consequently grant or denies access.\n\n### Configuration\n\nA basic configuration properties to enable the provider globally, in `spark-defaults.conf`, are:\n\n```properties\nspark.user.groups.mapping=io.okdp.spark.authz.OidcGroupMappingServiceProvider\nspark.acls.enable=true\nspark.history.ui.acls.enable=true\n# Comma separated list of admin groups (view all applications)\nspark.history.ui.admin.acls.groups=admins,team1\n```\n\nThese properties should be set before the spark history starts.\n\nYou can also decide globally which users, roles and/or groups you grant access to your applications individually in spark history UI \nor your spark ui by adding the properties in `spark-defaults.conf`:\n\nSelect the properties to enable the authorization for:\n\n```properties\n#Comma separated list of groups\nspark.admin.acls.groups=admins,admin-role\nspark.modify.acls.groups=team1\nspark.ui.view.acls.groups=/team2,role-team1\n\n# Comma separated list of users\nspark.admin.acls=bob@example.org\nspark.modify.acls=bob@example.org\nspark.ui.view.acls=bob@example.org,bill@example.org\n```\n\nOr at spark job submission time (select the properties to enable):\n\n```shell\nspark-submit -conf spark.admin.acls.groups=admins,admin-role \\\n--conf spark.modify.acls.groups=team1 \\\n--conf spark.ui.view.acls.groups=/team2,role-team1 \\\n--conf spark.admin.acls=bob@example.org \\\n--conf spark.modify.acls=bob@example.org \\\n--conf spark.ui.view.acls=bob@example.org,bill@example.org \\\n ...\n```\n\n# Use cases, limitation and future work\n\nThe filter is designed to address a basic use cases where you don't need to deploy extra components in order to secure your Spark History UIs.\n\nThe filter can also be used to secure Spark UIs, but note that, in real world kubernetes integration, each spark application submission creates its own ingress endpoint. With hundreds of running spark applications, it becomes very difficult to track all the endpoints and configure them.\n\nAnother limitation, is that depending on the oidc provider, the number of redirect URIs per oidc client can be limited and the usage of URIs pattern is also prohibited by [OAuth 2.1](https://oauth.net/2.1/).\n\nA new central portal UI is under development to simplify dynamic discovery, log/monitoring tracking and provides shortcuts to easily navigate and filter the spark applications.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fokdp%2Fokdp-spark-auth-filter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fokdp%2Fokdp-spark-auth-filter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fokdp%2Fokdp-spark-auth-filter/lists"}