https://github.com/salrashid123/istio_external_authorization_server
Tutorial to setup a simple Istio external authorization server
https://github.com/salrashid123/istio_external_authorization_server
envoyproxy istio
Last synced: about 1 year ago
JSON representation
Tutorial to setup a simple Istio external authorization server
- Host: GitHub
- URL: https://github.com/salrashid123/istio_external_authorization_server
- Owner: salrashid123
- License: apache-2.0
- Created: 2020-03-06T19:33:53.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2024-11-13T12:51:36.000Z (over 1 year ago)
- Last Synced: 2025-04-24T03:46:47.123Z (about 1 year ago)
- Topics: envoyproxy, istio
- Language: Go
- Homepage:
- Size: 765 KB
- Stars: 53
- Watchers: 5
- Forks: 16
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# External Authorization Server with Istio
Tutorial to setup an external authorization server for istio. In this setup, the `ingresss-gateway` will first send the inbound request headers to another istio service which check the header values submitted by the remote user/client. If the header values passes some criteria, the external authorization server will instruct the authorization server to proceed with the request upstream.
The check criteria can be anything (kerberos ticket, custom JWT) but in this example, it is the simple presence of the header value match as defined in configuration.
In this setup, it is important to ensure the authorization server is always (and exclusively) called by the ingress gateway and that the upstream services must accept the custom JWT token issued by the authorization server.
To that end, this configuration sets up `mTLS`, `RBAC` and `ORIGIN` authentication. RBAC ensures service->service traffic flows between the gateway, authorization server and the upstream systems. Each upstream service will only allow `ORIGIN` JWT tokens issued by the authorization server.
This tutorial is a continuation of the [istio helloworld](https://github.com/salrashid123/istio_helloworld) application.
>> `12/11/24`: Use minikube
>> `11/25/21`: Updated for example to NOT use an actual service account. Instead, use the istio built [gen-jwtpy](https://istio.io/v1.10/docs/tasks/security/authentication/authn-policy/#end-user-authentication) in JWT issuers
>> `3/20/21`: Updated for [istio 1.9: Integrate external authorization system (e.g. OPA, oauth2-proxy, etc.) with Istio using AuthorizationPolicy](https://istio.io/latest/blog/2021/better-external-authz/). Part of the upgrade is to use the `v3` API (`go-control-plane/envoy/config/core/v3`, `go-control-plane/envoy/service/auth/v3`)
### References
- [Envoy External Authorization](https://www.envoyproxy.io/docs/envoy/latest/api-v2/config/filter/http/ext_authz/v2/ext_authz.proto)
- [Envoy External Authorization server (envoy.ext_authz) HelloWorld](https://github.com/salrashid123/envoy_external_authz)
- [Istio Security](https://istio.io/docs/concepts/security/)
- [External authorization with custom action](https://istio.io/latest/docs/tasks/security/authorization/authz-custom/)
### Setup
The following setup uses a minikube and a convenient JWK endpoint provided by an Istio sample JWT authentication tutorial.
First install istio
```bash
minikube start --driver=kvm2 --cpus=4 --kubernetes-version=v1.28 --host-only-cidr 192.168.39.1/24
minikube addons enable metallb
## in a new window
minikube dashboard
## get the IP, for me it was the following
$ minikube ip
192.168.39.1
## setup a loadbalancer metallb, enter the ip range shown below
minikube addons configure metallb
# -- Enter Load Balancer Start IP: 192.168.39.104
# -- Enter Load Balancer End IP: 192.168.39.110
## download and install istio
export ISTIO_VERSION=1.24.0
export ISTIO_VERSION_MINOR=1.24
wget -P /tmp/ https://github.com/istio/istio/releases/download/$ISTIO_VERSION/istio-$ISTIO_VERSION-linux-amd64.tar.gz
tar xvf /tmp/istio-$ISTIO_VERSION-linux-amd64.tar.gz -C /tmp/
rm /tmp/istio-$ISTIO_VERSION-linux-amd64.tar.gz
export PATH=/tmp/istio-$ISTIO_VERSION/bin:$PATH
istioctl install --set profile=demo \
--set meshConfig.enableAutoMtls=true \
--set values.gateways.istio-ingressgateway.runAsRoot=true \
--set meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY \
--set meshConfig.defaultConfig.gatewayTopology.forwardClientCertDetails=SANITIZE_SET
```
### Build and push images
You can use the following prebuilt containers for this tutorial if you want to.
If you would rather build and stage your own, the `Dockerfile` for each container is provided in this repo.
The images we will use here has the following endpoints enabled:
* `salrashid123/svc`: Frontend service
- `/version`: Displays a static "version" number for the image. If using `salrashid123/svc:1` then the version is `1`. If using `salrashid123/svc:2` the version is `2`
- `/backend`: Makes an HTTP Ret call to the backend service's `/backend` and `/headerz` endpoints.
* `salrashid123/besvc`: Backend Service
- `/headerz`: Displays the http headers
- `/backend`: Displays the pod name
* `salrashid123/ext-authz-server`: External Authorization gRPC Server
- gRPC Authorization server running in namespace `authz-ns` as service `authz`
- Authorization server reads an environment variable that lists the set of authorized (eg `authzallowedusers: "alice,bob"`)
This server will read the "Authorization: Bearer " header value from the incoming request to determine the username
To build your own, create a public dockerhub images with the names specified below:
- Build External Authorization Server (you can ofcourse use your own dockerhub repo!)
```bash
cd authz_server/
docker build -t salrashid123/ext-authz-server .
docker push salrashid123/ext-authz-server
```
- Build Frontend
```bash
cd frontend
docker build --build-arg VER=1 -t salrashid123/svc:1 .
docker build --build-arg VER=2 -t salrashid123/svc:2 .
docker push salrashid123/svc:1
docker push salrashid123/svc:2
```
- Build Backend
```bash
cd backend
docker build --build-arg VER=1 -t salrashid123/besvc:1 .
docker build --build-arg VER=1 -t salrashid123/besvc:2 .
docker push salrashid123/besvc:1
docker push salrashid123/besvc:2
```
### Verify istio is installed
```bash
kubectl label namespace default istio-injection=enabled
kubectl get no,po,rc,svc,ing,deployment -n istio-system
```
### Deploy Istio Gateway and services
```bash
kubectl apply -f istio-lb-certs.yaml
sleep 10
## create the ingress gateway
kubectl apply -f istio-ingress-gateway.yaml
## kill and restart the ingress pod since the LB cert's may not have been loaded
INGRESS_POD_NAME=$(kubectl get po -n istio-system | grep ingressgateway\- | awk '{print$1}'); echo ${INGRESS_POD_NAME};
kubectl delete po/$INGRESS_POD_NAME -n istio-system
kubectl apply -f istio-app-config.yaml
export GATEWAY_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo $GATEWAY_IP
```
#### Debugging ingress-gateway
```bash
INGRESS_POD_NAME=$(kubectl get po -n istio-system | grep ingressgateway\- | awk '{print$1}'); echo ${INGRESS_POD_NAME};
kubectl exec --namespace=istio-system $INGRESS_POD_NAME -c istio-proxy -- curl -X POST http://localhost:15000/logging\?level\=debug
kubectl logs $INGRESS_POD_NAME -n istio-system
```
### Deploy application
Deploy the baseline application without the external authorization server
```bash
$ kubectl apply -f app-deployment.yaml
$ kubectl get po,svc
NAME READY STATUS RESTARTS AGE
pod/be-v1-8589f84d6-ll82f 2/2 Running 0 74s
pod/be-v2-6ff75fccd8-chj92 2/2 Running 0 74s
pod/svc1-bdb4d7c59-fgfk5 2/2 Running 0 74s
pod/svc2-7f65cc98f-hxcw9 2/2 Running 0 74s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/be ClusterIP 10.116.6.105 8080/TCP 74s
service/kubernetes ClusterIP 10.116.0.1 443/TCP 5m26s
service/svc1 ClusterIP 10.116.9.247 8080/TCP 75s
service/svc2 ClusterIP 10.116.12.54 8080/TCP 74s
```
### Send Traffic
Verify traffic for the frontend and backend services. (we're using [jq](https://stedolan.github.io/jq/download/) to help parse the response)
```bash
# Access the frontend for svc1,svc2
curl -s --cacert certs/CA_crt.pem -w " %{http_code}\n" --resolve svc1.example.com:443:$GATEWAY_IP https://svc1.example.com/version
curl -s --cacert certs/CA_crt.pem -w " %{http_code}\n" --resolve svc2.example.com:443:$GATEWAY_IP https://svc2.example.com/version
# Access the backend through svc1,svc2
curl -s --cacert certs/CA_crt.pem -w " %{http_code}\n" --resolve svc1.example.com:443:$GATEWAY_IP https://svc1.example.com/backend | jq '.'
curl -s --cacert certs/CA_crt.pem -w " %{http_code}\n" --resolve svc2.example.com:443:$GATEWAY_IP https://svc2.example.com/backend | jq '.'
```
If you would rather run this in a loop:
```bash
for i in {1..1000}; do curl -s -w " %{http_code}\n" --cacert certs/CA_crt.pem --resolve svc1.example.com:443:$GATEWAY_IP https://svc1.example.com/version; sleep 1; done
```
##### Kiali Dashboard
If you want, launch the kiali dashboard (default password is `admin/admin`). In a new window, run:
```bash
echo $ISTIO_VERSION
echo $ISTIO_VERSION_MINOR
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-$ISTIO_VERSION_MINOR/samples/addons/prometheus.yaml
sleep 20
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-$ISTIO_VERSION_MINOR/samples/addons/kiali.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-$ISTIO_VERSION_MINOR/samples/addons/grafana.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-$ISTIO_VERSION_MINOR/samples/addons/jaeger.yaml
### in a new window, install prometheus, kaili, jager and grafana
## open a tunnel and access the kiali dashboard at http://localhost:20001/kiali (admin/admi)
kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app.kubernetes.io/name=kiali -o jsonpath='{.items[0].metadata.name}') 20001:20001
```
### Generate Authz config
First we need to setup the auth* configs to use a convenient JWT/JWK issuer istio provides (you can use any jWT issuer, ofcourse; this is just a demo...do not use this in production!!!)
##### Use Istio's sample JWT issuer script
Istio provides a convenient JWT issuer, JWK and script the gateway will for authentication. You are certainly supposed to use your own JWK/JWT issuer; we're just using this one since it has a convenient JWK endpoint to verify the tokens with
We will use following script to issue a JWT and verify the JWK. This will be the same key that the external authorization server uses.
```bash
wget --no-verbose https://raw.githubusercontent.com/istio/istio/release-$ISTIO_VERSION_MINOR/security/tools/jwt/samples/gen-jwt.py
wget --no-verbose https://raw.githubusercontent.com/istio/istio/release-$ISTIO_VERSION_MINOR/security/tools/jwt/samples/key.pem
# may need pip3 install jwcrypto
python3 gen-jwt.py -aud some.audience -expire 3600 key.pem
```
```json
{
"alg": "RS256",
"kid": "DHFbpoIUqrY8t2zpA2qXfCmr5VO5ZEr4RzHU_-envvQ",
"typ": "JWT"
}
{
"aud": "some.audience",
"exp": 1635174518,
"iat": 1635170918,
"iss": "testing@secure.istio.io",
"sub": "testing@secure.istio.io"
}
```
You can also see that its `kid` key-id is visible too `"DHFbpoIUqrY8t2zpA2qXfCmr5VO5ZEr4RzHU_-envvQ"`
The JWK endpoint istio will use to validate a JWT issued by the authorization server is:
```json
$ curl -s https://raw.githubusercontent.com/istio/istio/release-1.10/security/tools/jwt/samples/jwks.json | jq '.'
{
"keys": [
{
"e": "AQAB",
"kid": "DHFbpoIUqrY8t2zpA2qXfCmr5VO5ZEr4RzHU_-envvQ",
"kty": "RSA",
"n": "xAE7eB6qugXyCAG3yhh7pkDkT65pHymX-P7KfIupjf59vsdo91bSP9C8H07pSAGQO1MV_xFj9VswgsCg4R6otmg5PV2He95lZdHtOcU5DXIg_pbhLdKXbi66GlVeK6ABZOUW3WYtnNHD-91gVuoeJT_DwtGGcp4ignkgXfkiEm4sw-4sfb4qdt5oLbyVpmW6x9cfa7vs2WTfURiCrBoUqgBo_-4WTiULmmHSGZHOjzwa8WtrtOQGsAFjIbno85jp6MnGGGZPYZbDAa_b3y5u-YpW7ypZrvD8BgtKVjgtQgZhLAGezMt0ua3DRrWnKqTZ0BJ_EyxOGuHJrLsn00fnMQ"
}
]
}
```
NOTE: we will not be issuing these JWTs. The external authorization server will use the private key to reissue a JWT intended for a given service.
Apply the preset environment variables to `ext_authz_filter.yaml`:
```bash
export SERVICE_ACCOUNT_EMAIL="testing@secure.istio.io"
wget --no-verbose https://raw.githubusercontent.com/istio/istio/release-$ISTIO_VERSION_MINOR/security/tools/jwt/samples/key.pem
export KEY_ID="DHFbpoIUqrY8t2zpA2qXfCmr5VO5ZEr4RzHU_-envvQ"
export SVC_ACCOUNT_KEY=`base64 -w 0 key.pem && echo`
echo $SERVICE_ACCOUNT_EMAIL
echo $KEY_ID
echo $SVC_ACCOUNT_KEY
```
### Apply Authz rules
```bash
envsubst < "ext_authz_rules.yaml.tmpl" > "ext_authz_rules.yaml"
kubectl apply -f ext_authz_rules.yaml
```
This will cause a 'deny' for everyone since we specified some headers that cannot be met (since we didnt' even deploy the authzserver in the first place that'd issue the JWT we just declared above!)
### Deploy ExtAuthz server
Edit mesh-config
```bash
kubectl edit configmap istio -n istio-system
```
append the section for `extensionProviders` to the top of the `mesh` definition as such (remember to delete the definition of `extensionProviders` already set with `envoyOtelAls`)
```yaml
apiVersion: v1
data:
mesh: |-
extensionProviders:
- name: "my-ext-authz-grpc"
envoyExtAuthzGrpc:
service: "authz.authz-ns.svc.cluster.local"
port: "50051"
- name: otel
envoyOtelAls:
port: 4317
service: opentelemetry-collector.istio-system.svc.cluster.local
```
please note the name for the provider: `"my-ext-authz-grpc"`. This is defined in the `ext_authz.yaml` provider filter
```yaml
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: ext-authz
namespace: istio-system
spec:
selector:
matchLabels:
istio: ingressgateway
action: CUSTOM
provider:
name: "my-ext-authz-grpc"
rules:
- to:
- operation:
paths: ["/*"]
```
Reload the gateway:
```bash
INGRESS_POD_NAME=$(kubectl get po -n istio-system | grep ingressgateway\- | awk '{print$1}'); echo ${INGRESS_POD_NAME};
kubectl delete po/$INGRESS_POD_NAME -n istio-system
```
Apply the authz config
```bash
envsubst < "ext_authz.yaml.tmpl" > "ext_authz.yaml"
kubectl apply -f ext_authz.yaml
```
```bash
$ kubectl get PeerAuthentication,RequestAuthentication,AuthorizationPolicy -n authz-ns
NAME MODE AGE
peerauthentication.security.istio.io/ing-authzserver-peer-authn-policy STRICT 13s
NAME ACTION AGE
authorizationpolicy.security.istio.io/deny-all-authz-ns 13s
authorizationpolicy.security.istio.io/ing-authzserver-authz-policy ALLOW 13s
$ kubectl get PeerAuthentication,RequestAuthentication,AuthorizationPolicy -n default
NAME AGE
requestauthentication.security.istio.io/ing-svc1-request-authn-policy 109s
requestauthentication.security.istio.io/ing-svc2-request-authn-policy 109s
requestauthentication.security.istio.io/svc-be-v1-request-authn-policy 109s
requestauthentication.security.istio.io/svc-be-v2-request-authn-policy 109s
NAME ACTION AGE
authorizationpolicy.security.istio.io/deny-all-default 109s
authorizationpolicy.security.istio.io/ing-svc1-authz-policy ALLOW 109s
authorizationpolicy.security.istio.io/ing-svc2-authz-policy ALLOW 109s
authorizationpolicy.security.istio.io/svc1-be-v1-authz-policy ALLOW 109s
authorizationpolicy.security.istio.io/svc1-be-v2-authz-policy ALLOW 109s
```
### Access Frontend
The static/demo configuration here uses two users (`alice`, `bob`), two frontend services (`svc1`,`svc2`) one backend service with two labled versions (`be`, `version=v1`,`version=v2`).
The following conditions are coded into the authorization server:
- If the authorization server sees `alice`, it issues a JWT token with `svc1` and `be` as the targets (multiple audiences)
- If the authorization server sees `bob`, it issues a JWT token with `svc2` as the target
- If the authorization server sees `carol`, it issues a JWT token with `svc1` as the target only.
```golang
var aud []string
if token == "alice" {
aud = []string{"http://svc1.default.svc.cluster.local:8080/", "http://be.default.svc.cluster.local:8080/"}
} else if token == "bob" {
aud = []string{"http://svc2.default.svc.cluster.local:8080/"}
} else if token == "carol" {
aud = []string{"http://svc1.default.svc.cluster.local:8080/"}
} else {
aud = []string{}
}
```
The net effect of that is `alice` can view `svc1`, `bob` can view `svc2` using `ORIGIN` authentication.
As Alice:
```bash
export USER=alice
curl -s \
--cacert certs/CA_crt.pem --resolve svc1.example.com:443:$GATEWAY_IP \
-H "Authorization: Bearer $USER" \
-w " %{http_code}\n" \
https://svc1.example.com/version
curl -s \
--cacert certs/CA_crt.pem --resolve svc2.example.com:443:$GATEWAY_IP \
-H "Authorization: Bearer $USER" \
-w " %{http_code}\n" \
https://svc2.example.com/version
```
```
>>> 1 200
>>> Audiences in Jwt are not allowed 403
```
If you want to view the authz logs
```bash
AUTHZ_POD_NAME=$(kubectl get po -n authz-ns | grep authz\- | awk '{print$1}'); echo ${AUTHZ_POD_NAME};
kubectl logs -n authz-ns $AUTHZ_POD_NAME -c authz-container
```
You should see some debug logs as well as the actual reissued JWT header
```log
2024/11/13 12:33:41 Starting gRPC Server at :50051
2024/11/13 12:34:01 >>> Authorization called check()
2024/11/13 12:34:01 Authorization Header Bearer alice
2024/11/13 12:34:01 Using Claim {alice [http://svc1.default.svc.cluster.local:8080/ http://be.default.svc.cluster.local:8080/] {testing@secure.istio.io testing@secure.istio.io [] 2024-11-13 12:35:01.446845225 +0000 UTC m=+79.858574856 2024-11-13 12:34:01.446845119 +0000 UTC m=+19.858574750 }}
2024/11/13 12:34:01 Issuing outbound Header eyJhbGciOiJSUzI1NiIsImtpZCI6IkRIRmJwb0lVcXJZOHQyenBBMnFYZkNtcjVWTzVaRXI0UnpIVV8tZW52dlEiLCJ0eXAiOiJKV1QifQ.eyJ1aWQiOiJhbGljZSIsImF1ZCI6WyJodHRwOi8vc3ZjMS5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsOjgwODAvIiwiaHR0cDovL2JlLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWw6ODA4MC8iXSwiaXNzIjoidGVzdGluZ0BzZWN1cmUuaXN0aW8uaW8iLCJzdWIiOiJ0ZXN0aW5nQHNlY3VyZS5pc3Rpby5pbyIsImV4cCI6MTczMTUwMTMwMSwiaWF0IjoxNzMxNTAxMjQxfQ.Q-itZwt9AkSjpR3JjHxAyMMdhUOMwythPdB3IH-kZspwP4PH87BGyKNs71WzRqTCbOkd9U9EoiGEO16blV_EFpBQHKkHCIp-T070D2eyJu262MXr3tCwsrp0YHl-tx7qyICoLtMe77LNduI8bWj_2Q61fwALnAOslyH0Cj47u7Gq1FQ0-dFXssR8oMXM8eNSaF30oU2SHf_FGrd56TgJ8gCcl0Qhik6qC11ihjKl8S3_ccw1D48iCX8JlA8cWR5JMTqHhwQEEdTZtMJAR7HB0DuSAMKxWu2ENuyE6_lLDQmbLPbwTW6dy1nJa4JQGA9Eo6JtfWf3FHlAc7QuFfvz3Q
2024/11/13 12:34:01 >>> Authorization called check()
2024/11/13 12:34:01 Authorization Header Bearer alice
2024/11/13 12:34:01 Using Claim {alice [http://svc1.default.svc.cluster.local:8080/ http://be.default.svc.cluster.local:8080/] {testing@secure.istio.io testing@secure.istio.io [] 2024-11-13 12:35:01.504861628 +0000 UTC m=+79.916591235 2024-11-13 12:34:01.50486156 +0000 UTC m=+19.916591168 }}
2024/11/13 12:34:01 Issuing outbound Header eyJhbGciOiJSUzI1NiIsImtpZCI6IkRIRmJwb0lVcXJZOHQyenBBMnFYZkNtcjVWTzVaRXI0UnpIVV8tZW52dlEiLCJ0eXAiOiJKV1QifQ.eyJ1aWQiOiJhbGljZSIsImF1ZCI6WyJodHRwOi8vc3ZjMS5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsOjgwODAvIiwiaHR0cDovL2JlLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWw6ODA4MC8iXSwiaXNzIjoidGVzdGluZ0BzZWN1cmUuaXN0aW8uaW8iLCJzdWIiOiJ0ZXN0aW5nQHNlY3VyZS5pc3Rpby5pbyIsImV4cCI6MTczMTUwMTMwMSwiaWF0IjoxNzMxNTAxMjQxfQ.Q-itZwt9AkSjpR3JjHxAyMMdhUOMwythPdB3IH-kZspwP4PH87BGyKNs71WzRqTCbOkd9U9EoiGEO16blV_EFpBQHKkHCIp-T070D2eyJu262MXr3tCwsrp0YHl-tx7qyICoLtMe77LNduI8bWj_2Q61fwALnAOslyH0Cj47u7Gq1FQ0-dFXssR8oMXM8eNSaF30oU2SHf_FGrd56TgJ8gCcl0Qhik6qC11ihjKl8S3_ccw1D48iCX8JlA8cWR5JMTqHhwQEEdTZtMJAR7HB0DuSAMKxWu2ENuyE6_lLDQmbLPbwTW6dy1nJa4JQGA9Eo6JtfWf3FHlAc7QuFfvz3Q
```
note JWT headers include cliams and audiences
```json
{
"alg": "RS256",
"kid": "DHFbpoIUqrY8t2zpA2qXfCmr5VO5ZEr4RzHU_-envvQ",
"typ": "JWT"
}
{
"uid": "alice",
"aud": [
"http://svc1.default.svc.cluster.local:8080/",
"http://be.default.svc.cluster.local:8080/"
],
"exp": 1635173568,
"iat": 1635173508,
"iss": "testing@secure.istio.io",
"sub": "testing@secure.istio.io"
}
```
As Bob:
```bash
export USER=bob
curl -s \
--cacert certs/CA_crt.pem --resolve svc1.example.com:443:$GATEWAY_IP \
-H "Authorization: Bearer $USER" \
-w " %{http_code}\n" \
https://svc1.example.com/version
curl -s \
--cacert certs/CA_crt.pem --resolve svc2.example.com:443:$GATEWAY_IP \
-H "Authorization: Bearer $USER" \
-w " %{http_code}\n" \
https://svc2.example.com/version
```
```
>>> Audiences in Jwt are not allowed 403
>>> 2 200
```
As Carol
```bash
export USER=carol
curl -s \
--cacert certs/CA_crt.pem --resolve svc1.example.com:443:$GATEWAY_IP \
-H "Authorization: Bearer $USER" \
-w " %{http_code}\n" \
https://svc1.example.com/version
curl -s \
--cacert certs/CA_crt.pem --resolve svc2.example.com:443:$GATEWAY_IP \
-H "Authorization: Bearer $USER" \
-w " %{http_code}\n" \
https://svc2.example.com/version
```
```
>>> 1 200
>>> Audiences in Jwt are not allowed 403
```
>> note, it seems the traffic from the gateway to the authorization server isn't correctly detected to be associated with the ingress-gateway (maybe a bug or some label is missing)
### Access Backend
The configuration also defines Authorization policies on the `svc1`-> `be` traffic using **BOTH** `PEER` and `ORIGIN`.
- `PEER`:
This is done using normal RBAC service identities:
```yaml
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
name: svc1-be-v1-authz-policy
namespace: default
spec:
action: ALLOW
selector:
matchLabels:
app: be
version: v1
rules:
- from:
- source:
principals: ["cluster.local/ns/default/sa/svc1-sa"]
to:
- operation:
methods: ["GET"]
```
#### Backend PEER and ORIGIN
Note the `from->source->principals` denotes the service account `svc1` runs as.
- `ORIGIN`
THis step is pretty unusual and requires some changes to application code to _forward_ its inbound authentication token.
Recall the inbound JWT token to `svc1` for `alice` includes two audiences:
```golang
aud = []string{"http://svc1.default.svc.cluster.local:8080/", "http://be.default.svc.cluster.local:8080/"}
```
This means we can use the same JWT token on the backend service if we setup an authentication and authz rule:
```yaml
## svc --> be-v1
apiVersion: security.istio.io/v1
kind: RequestAuthentication
metadata:
name: svc-be-v1-request-authn-policy
namespace: default
spec:
selector:
matchLabels:
app: be
version: v1
jwtRules:
- issuer: "$SERVICE_ACCOUNT_EMAIL"
audiences:
- "http://be.default.svc.cluster.local:8080/"
jwksUri: "https://raw.githubusercontent.com/istio/istio/release-$ISTIO_VERSION_MINOR/security/tools/jwt/samples/jwks.json"
outputPayloadToHeader: x-jwt-payload
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
name: svc1-be-v1-authz-policy
namespace: default
spec:
action: ALLOW
selector:
matchLabels:
app: be
version: v1
rules:
- from:
- source:
principals: ["cluster.local/ns/default/sa/svc1-sa"]
to:
- operation:
methods: ["GET"]
when:
- key: request.auth.claims[iss]
values: ["$SERVICE_ACCOUNT_EMAIL"]
- key: request.auth.claims[aud]
values: ["http://be.default.svc.cluster.local:8080/"]
```
The `RequestAuthentication` accepts a JWT token signed by the external authz server and must also include the audience of the backend (which alice's token has). The second authorization (redundantly) rule further parses out the token and looks for the same.
Istio does not automatically forward the inbound token (though it maybe possible with `SIDECAR_INBOUND`->`SIDECAR_OUTBOUND` forwarding somehow...)...to achieve this requres some application code changes. The folloing snippet is the code within `frontend/app.js` which take the token and uses it on the backend api call.
>> `4/27/20`: update on the comment "(though it maybe possble with `SIDECAR_INBOUND`->`SIDECAR_OUTBOUND` forwarding somehow...)" Its not; envoy doens't carry state from the filters forward like this. You need to either accept and forward the header in code as shown below:
```javascript
var resp_promises = []
var urls = [
'http://' + host + ':' + port + '/backend',
'http://' + host + ':' + port + '/headerz',
]
out_headers = {};
if (FORWARD_AUTH_HEADER == 'true') {
var auth_header = request.headers['authorization'];
logger.info("Got Authorization Header: [" + auth_header + "]");
out_headers = {
'authorization': auth_header,
};
}
urls.forEach(element => {
resp_promises.push( getURL(element,out_headers) )
});
```
Or configure istio to make an `OUTBOUND` ext_authz filter call. The external authz filter will return a new Authorization server token intended for ust `svcb`.
You will also need to set [allowed_client_headers](https://www.envoyproxy.io/docs/envoy/latest/api-v2/config/filter/http/ext_authz/v2/ext_authz.proto#envoy-api-msg-config-filter-http-ext-authz-v2-authorizationresponse) so that the auth token returned by ext-authz server is sent to the upstream (in this case, upstream is `svcb`)
I think the config would be _something_ like this:
```yaml
apiVersion: networking.istio.io/v1
kind: EnvoyFilter
metadata:
name: ext-authz-service
namespace: default
spec:
workloadLabels:
app: svc1
filters:
- listenerMatch:
listenerType: OUTBOUND # <<<< OUTBOUND svc1->*
listenerProtocol: HTTP
insertPosition:
index: FIRST
filterName: envoy.ext_authz
filterType: HTTP
filterConfig:
grpc_service:
envoy_grpc:
cluster_name: patched.authz.authz-ns.svc.cluster.local
authorization_response:
allowed_client_headers:
patterns:
- exact: "Authorization"
```
(ofcourse changes are needed to ext-authz server as provided in this repo..)
>> Note: i added both ORIGIN and PEER just to demonstrate this...Until its easier forward the token by envoy/istio, i woudn't recommend doing this bit..
Anwyay, to test all this out
```bash
export USER=alice
curl -s \
--cacert certs/CA_crt.pem --resolve svc1.example.com:443:$GATEWAY_IP \
-H "Authorization: Bearer $USER" \
-w " %{http_code}\n" \
https://svc1.example.com/backend | jq '.'
export USER=bob
curl -s \
--cacert certs/CA_crt.pem --resolve svc2.example.com:443:$GATEWAY_IP \
-H "Authorization: Bearer $USER" \
-w " %{http_code}\n" \
https://svc2.example.com/backend | jq '.'
export USER=carol
curl -s \
--cacert certs/CA_crt.pem --resolve svc1.example.com:443:$GATEWAY_IP \
-H "Authorization: Bearer $USER" \
-w " %{http_code}\n" \
https://svc1.example.com/backend | jq '.'
```
Sample output
-Alice
Alice's TOKEN issued by the authorization server includes two audiences:
```golang
aud = []string{"http://svc1.default.svc.cluster.local:8080/", "http://be.default.svc.cluster.local:8080/"}
```
Which is allowed by backend services `RequestAuthentication` policy.
```bash
export USER=alice
curl -s \
--cacert certs/CA_crt.pem --resolve svc1.example.com:443:$GATEWAY_IP \
-H "Authorization: Bearer $USER" \
-w " %{http_code}\n" \
https://svc1.example.com/backend | jq '.'
[
{
"url": "http://be.default.svc.cluster.local:8080/backend",
"body": "pod: [be-v2-64d9cf5fb4-mpsq5] node: [gke-istio-1-default-pool-b516bc56-xz2c]",
"statusCode": 200
},
{
"url": "http://be.default.svc.cluster.local:8080/headerz",
"body": "{\"host\":\"be.default.svc.cluster.local:8080\",\"x-forwarded-proto\":\"http\",\"x-request-id\":\"bb31942c-f04e-9b12-ba69-d68603a520af\",\"content-length\":\"0\",\"x-forwarded-client-cert\":\"By=spiffe://cluster.local/ns/default/sa/be-sa;Hash=2e0f9ca7bea6ac081f4c256de79ffdb4db2e55968b0ded2526e95cb89f4c36ac;Subject=\\\"\\\";URI=spiffe://cluster.local/ns/default/sa/svc1-sa\",\"x-b3-traceid\":\"cda6d87c8d342998ee1f797471592dff\",\"x-b3-spanid\":\"6dc54e848db21050\",\"x-b3-parentspanid\":\"ee1f797471592dff\",\"x-b3-sampled\":\"1\"}",
"statusCode": 200
}
]
```
- Bob
Bob's token does not include the backend service
```golang
aud = []string{"http://svc2.default.svc.cluster.local:8080/"}
```
Which means the `RequestAuthentication` will fail. Bob is only allowed to invoke `svc2` anyway
```bash
export USER=bob
curl -s \
--cacert certs/CA_crt.pem --resolve svc2.example.com:443:$GATEWAY_IP \
-H "Authorization: Bearer $USER" \
-w " %{http_code}\n" \
https://svc2.example.com/backend | jq '.'
[
{
"url": "http://be.default.svc.cluster.local:8080/backend",
"body": "Audiences in Jwt are not allowed",
"statusCode": 403
},
{
"url": "http://be.default.svc.cluster.local:8080/headerz",
"body": "Audiences in Jwt are not allowed",
"statusCode": 403
}
]
```
- Carol
Carol's token is allowed to invoke `svc1` but does not include the issuer to pass the `RequestAuthentication` policy
```golang
aud = []string{"http://svc1.default.svc.cluster.local:8080/"}
```
```bash
export USER=carol
curl -s \
--cacert certs/CA_crt.pem --resolve svc1.example.com:443:$GATEWAY_IP \
-H "Authorization: Bearer $USER" \
-w " %{http_code}\n" \
https://svc1.example.com/backend | jq '.'
[
{
"url": "http://be.default.svc.cluster.local:8080/backend",
"body": "Audiences in Jwt are not allowed",
"statusCode": 403
},
{
"url": "http://be.default.svc.cluster.local:8080/headerz",
"body": "Audiences in Jwt are not allowed",
"statusCode": 403
}
]
```
If you would rather run these tests in a loop
```bash
for i in {1..1000}; do curl -s \
--cacert certs/CA_crt.pem --resolve svc1.example.com:443:$GATEWAY_IP \
-H "Authorization: Bearer $USER" \
-w " %{http_code}\n" \
https://svc1.example.com/version; sleep 1; done
```
---
At this point, the system is setup to to always use mTLS, `ORIGIN` and `PEER` authentication plus `RBAC`. If you want to verify any component of `PEER`, change the policy and change the service account that is the target service authorization policy accepts and reapply the config.
Change either the settings `RequestAuthentication` _or_ `AuthorizationPolicy` depending on which layer you are testing
(remember to replace the value for `$ISTIO_VERSION_MINOR` )
```yaml
## svc --> be-v1
apiVersion: security.istio.io/v1
kind: RequestAuthentication
metadata:
name: svc-be-v1-request-authn-policy
namespace: default
spec:
selector:
matchLabels:
app: be
version: v1
jwtRules:
- issuer: "$SERVICE_ACCOUNT_EMAIL"
audiences:
- "http://be.default.svc.cluster.local:8080/" ## or CHANGE ORIGIN <<<< "Audiences in Jwt are not allowed"
jwksUri: "https://raw.githubusercontent.com/istio/istio/release-$ISTIO_VERSION_MINOR/security/tools/jwt/samples/jwks.json"
# forwardOriginalToken: true
outputPayloadToHeader: x-jwt-payload
---
## svc --> be-v2
apiVersion: security.istio.io/v1
kind: RequestAuthentication
metadata:
name: svc-be-v2-request-authn-policy
namespace: default
spec:
selector:
matchLabels:
app: be
version: v2
jwtRules:
- issuer: "$SERVICE_ACCOUNT_EMAIL"
audiences:
- "http://be.default.svc.cluster.local:8080/" ## or CHANGE ORIGIN <<<< "Audiences in Jwt are not allowed"
jwksUri: "https://raw.githubusercontent.com/istio/istio/release-$ISTIO_VERSION_MINOR/security/tools/jwt/samples/jwks.json"
# forwardOriginalToken: true
outputPayloadToHeader: x-jwt-payload
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: svc1-be-v1-authz-policy
namespace: default
spec:
action: ALLOW
selector:
matchLabels:
app: be
version: v1
rules:
- from:
- source:
principals: ["cluster.local/ns/default/sa/svc1-sa"] # CHANGE PEER <<<< "RBAC: access denied"
to:
- operation:
methods: ["GET"]
when:
- key: request.auth.claims[iss]
values: ["$SERVICE_ACCOUNT_EMAIL"] ## or CHANGE ORIGIN at Authz <<<< "RBAC: access denied"
- key: request.auth.claims[aud]
values: ["http://be.default.svc.cluster.local:8080/"]
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
name: svc1-be-v2-authz-policy
namespace: default
spec:
action: ALLOW
selector:
matchLabels:
app: be
version: v2
rules:
- from:
- source:
principals: ["cluster.local/ns/default/sa/svc1-sa"] # CHANGE PEER <<<< "RBAC: access denied"
to:
- operation:
methods: ["GET"]
when:
- key: request.auth.claims[iss]
values: ["$SERVICE_ACCOUNT_EMAIL"] ## or CHANGE ORIGIN at Authz <<<< "RBAC: access denied"
- key: request.auth.claims[aud]
values: ["http://be.default.svc.cluster.local:8080/"]
```
then reapply the config and access the backend as `alice`
```bash
export USER=alice
curl -s \
--cacert certs/CA_crt.pem --resolve svc1.example.com:443:$GATEWAY_IP \
-H "Authorization: Bearer $USER" \
-w " %{http_code}\n" \
https://svc1.example.com/backend | jq '.'
[
{
"url": "http://be.default.svc.cluster.local:8080/backend",
"body": "RBAC: access denied",
"statusCode": 403
},
{
"url": "http://be.default.svc.cluster.local:8080/headerz",
"body": "RBAC: access denied",
"statusCode": 403
}
]
```
Finally, the external server is attached to the ingress gateway but you could also attach it to a sidecar for an endpoint. In this mode, the authorization decision is done not at the ingress gateway but locally on a service's sidecar. To use that mode, define the `EnvoyFilter` workloadLabel and listenerType. eg:
```yaml
apiVersion: networking.istio.io/v1
kind: EnvoyFilter
metadata:
name: svc1-authz-filter
namespace: default
spec:
workloadSelector:
labels:
app: svc1
configPatches:
- applyTo: HTTP_FILTER
match:
context: SIDECAR_INBOUND
listener:
filterChain:
filter:
name: "envoy.filters.network.http_connection_manager"
subFilter:
name: "envoy.filters.http.router"
patch:
operation: INSERT_FIRST
value:
name: "envoy.filters.http.ext_authz"
config:
grpc_service:
envoy_grpc:
cluster_name: patched.authz.authz-ns.svc.cluster.local
```
If you do this, you will have to setup PEER policies that allow the service to connect and use the authorization server.
---
### Debugging
You can debug issues using these resources
- [Debugging Envoy and Istio](https://istio.io/docs/ops/diagnostic-tools/proxy-cmd/)
- [Security Problems](https://istio.io/docs/ops/common-problems/security-issues/)
To set the log level higher and inspect a pod's logs:
```bash
istioctl manifest apply --set values.global.proxy.accessLogFile="/dev/stdout"
```
- Ingress pod
```bash
INGRESS_POD_NAME=$(kubectl get po -n istio-system | grep ingressgateway\- | awk '{print$1}'); echo ${INGRESS_POD_NAME};
kubectl exec -ti $INGRESS_POD_NAME -n istio-syste -- /bin/bash
istioctl proxy-config log $INGRESS_POD_NAME --level debug
kubectl logs -f --tail=0 $INGRESS_POD_NAME -n istio-system
istioctl dashboard envoy $INGRESS_POD_NAME.istio-system
istioctl experimental authz check $INGRESS_POD_NAME.istio-system
```
```bash
$ istioctl experimental authz check $INGRESS_POD_NAME.istio-system
Checked 2/2 listeners with node IP 10.48.2.5.
LISTENER[FilterChain] CERTIFICATE mTLS (MODE) JWT (ISSUERS) AuthZ (RULES)
0.0.0.0_80 none no (none) no (none) no (none)
0.0.0.0_443 /etc/istio/ingressgateway-certs/tls.crt no (none) no (none) no (none)
$ istioctl authn tls-check $INGRESS_POD_NAME.istio-system authz.authz-ns.svc.cluster.local
HOST:PORT STATUS SERVER CLIENT AUTHN POLICY DESTINATION RULE
authz.authz-ns.svc.cluster.local:50051 AUTO STRICT - /default -
```
- Authz pod
```bash
AUTHZ_POD_NAME=$(kubectl get po -n authz-ns | grep authz\- | awk '{print$1}'); echo ${AUTHZ_POD_NAME};
istioctl proxy-config log $AUTHZ_POD_NAME -n authz-ns --level debug
kubectl logs -f --tail=0 $AUTHZ_POD_NAME -c authz-container -n authz-ns
istioctl dashboard envoy $AUTHZ_POD_NAME.authz-ns
istioctl experimental authz check $AUTHZ_POD_NAME.authz-ns
```
- SVC1 pod
```bash
SVC1_POD_NAME=$(kubectl get po -n default | grep svc1\- | awk '{print$1}'); echo ${SVC1_POD_NAME};
$ istioctl authn tls-check $SVC1_POD_NAME.default be.default.svc.cluster.local
HOST:PORT STATUS SERVER CLIENT AUTHN POLICY DESTINATION RULE
be.default.svc.cluster.local:8080 OK STRICT ISTIO_MUTUAL /default default/be-destination
```
- SVC2 pod
```bash
SVC2_POD_NAME=$(kubectl get po -n default | grep svc2\- | awk '{print$1}'); echo ${SVC2_POD_NAME};
$ istioctl authn tls-check $SVC2_POD_NAME.default be.default.svc.cluster.local
HOST:PORT STATUS SERVER CLIENT AUTHN POLICY DESTINATION RULE
be.default.svc.cluster.local:8080 OK STRICT ISTIO_MUTUAL /default default/be-destination
```
### Using Google OIDC ORIGIN authentication at Ingress
If you want to use OIDC JWT authentication at the ingress gateway and then have that token forwarded to the
external authz service, apply the `RequestAuthentication` policies on the ingress gateway as shown in the equivalent
Envoy configuration [here](https://github.com/salrashid123/envoy_iap/blob/master/envoy_google.yaml#L32).
You can generate an `id-token` using the script found under `jwt_client/` folder.
### Debugging
```bash
kubectl get pods -n istio-system -o name -l istio=ingressgateway | sed 's|pod/||' | while read -r pod; do istioctl proxy-config log "$pod" -n istio-system --level rbac:debug; done
```