1. a managed Kubernetes cluster (AWS, Google Cloud Platform),

2. and an Identity Provider (Okta, Microsoft Entra ID, Google Workspace, JumpCloud)

Why OIDC into Kubernetes?

Kubernetes Authentication Recap

kind: RoleBinding
subjects:
- kind: User
  name: jane
- kind: Group
  name: engineers
roleRef:
  name: pod-reader

• Google Cloud’s Kubernetes Engine uses Google Workspace identities and provides the gke-gcloud-auth-plugin since v1.26.

• AWS EKS uses the aws-iam-authenticator by default (see here). This drops an aws-auth ConfigMap object in you cluster to map a user ARN to a Kubernetes user string.

• Azure provides the “Azure RBAC for Kubernetes Authorization” (see here).

1. Configure your Identity Provider (IdP). Your IdP must support OpenID Connect Discovery.

2. Enable and configure the OIDC plugin in Kubernetes

3. Instructions for your engineers to set up the kubectl client

Step 1: Configure IdP

OIDC Flow Considerations

Why are we not using refresh tokens?

Why we are not using client credentials?

Why are we using PKCE?

Configure Application

• issuer: this is the address Kubernetes will contact to obtain the IdP’s public key to verify the identity token

• client ID: a unique identifier of the application. If you want to treat different Kubernetes clusters as different applications, then a new application and a new client ID should be generated for each cluster.

• client secret: Google Workspace requires a client secret. This is omitted for other IdPs.

• user claim: “email” for all IdPs

• groups claim: “groups” for all IdPs

Okta

Microsoft Entra ID

Claims

{
  "aud": "c44b1f19-...",
  "iss": "https://login.microsoftonline.com/{tenantId}/v2.0",
  "iat": 1697421579,
  "nbf": 1697421579,
  "exp": 1697425479,
  "groups": [
    "8a8dcf11-8629-4d60-9d00-b48e27d5afc8",
    "7162ac13-5d2c-4f5e-aacc-da9f4f19a905",
...
    "9b469762-902f-4df5-8573-c584809e6640"
  ],
  "name": "John Smith",
  "nonce": "qXu...M",
  "oid": "63419c9a-f9a9-4b70-ba40-19e68c23d24c",
  "preferred_username": "john.smith@example.com",
  "rh": "0.AVkBIDdDh4Zd-EW4H6kqBH2F6hkfS7TzYg1NnE655pu2dqadAJg.",
  "sub": "iMWPm2Vu2BnE9oJ3kZMwuNikQD99MOJOlAj0Lzf__oc",
  "tid": "83433721-5d86-45f8-b81f-a92af47d85ea",
  "uti": "7OCmTU_V2EK8erPRbgnyAA",
  "ver": "2.0"
}

Google Workspace

JumpCloud

• Check “Authorization Token” and uncheck “Refresh Token”

• “Client Authentication Type” is “Public (None PKCE)”

• Add http://localhost:8080 as the “Redirect URI” - note, the scheme is http

• The “Login URL” isn’t relevant, since there is no a service-provider initiated login flow. Enter http://localhost:8080 as well since the field is required.

• In the “Attribute Mapping” section check “Email” and “Profile”

• Check “Include group attribute” and name the attribute as groups

Step 2: Configure the Kubernetes Cluster

1. Issuer: in addition to the OAuth 2.0 endpoints, must also support OpenID Connect Discovery. All identity providers featured in this article support OIDC Discovery.

2. Client ID: generated by the identity provider

3. User claim: which JWT claim contains the value that is used as the user in Kubernetes

4. Group claim: which JWT claim contains the value that is used as the group in Kubernetes

1. User prefix: we use a different prefix for each OIDC provider. okta: for Okta, google: for Google Workspace, and ms: for Microsoft Entra ID.

2. Group prefix: set to the same as user prefix

kind: RoleBinding
subjects:
- kind: User
  name: okta:jane
- kind: Group
  name: okta:engineers
roleRef:
  name: pod-reader

AWS Elastic Kubernetes Service

AWS_REGION=${region} aws eks associate-identity-provider-config --cluster-name=${cluster} --oidc "identityProviderConfigName=OidcDemoConfig,issuerUrl=${issuer},clientId=${clientId},usernameClaim=email,usernamePrefix=${prefix},groupsClaim=groups,groupsPrefix=${prefix}"

Google Kubernetes Engine (GKE)

$ kubectl get services -n anthos-identity-service
NAME               TYPE           CLUSTER-IP  EXTERNAL-IP   PORT(S)              AGE
gke-oidc-envoy     LoadBalancer   1.1.1.125   1.1.1.91      443:32031/TCP        9m
gke-oidc-service   ClusterIP      1.1.1.17    <none>        8443/TCP,10390/TCP   9m

1. Configure Ingress on the Load Balancer

metadata:
  annotations:
    cloud.google.com/load-balancer-type: Internal
    cloud.google.com/neg: '{"ingress":true}'

metadata:
  annotations:
    cloud.google.com/load-balancer-type: External
    cloud.google.com/neg: '{"ingress":true,"exposed_ports": {"443":{}}}'

kubectl patch service gke-oidc-envoy -n anthos-identity-service --type merge --patch-file envoy-service-patch.yaml

2. Configure Egress on the Cluster

3. Prepare for Credential Distribution

apiVersion: authentication.gke.io/v2alpha1
kind: ClientConfig
metadata:
  name: default
  namespace: kube-public
spec:
  certificateAuthorityData: <redacted>
  name: oidc-demo-gke-cluster
  server: https://1.1.1.1:443

spec:
  authentication:
  - name: oidc
    oidc:
      clientID: ${clientId}
      cloudConsoleRedirectURI: https://console.cloud.google.com/kubernetes/oidc
      extraParams: prompt=consent
      issuerURI: https://p0dev-ext.okta.com/oauth2/${audience}
      kubectlRedirectURI: http://localhost:8000
      scopes: openid,profile,email,groups,offline_access
      userClaim: email
      groupsClaim: groups
      userPrefix: "${prefix}"
      groupPrefix: "${prefix}"

cliendId=YOUR_CLIENT_ID audience=YOUR_AUDIENCE prefix=YOUR_PREFIX envsubst < $(echo client-config-patch.yaml) > client-config-patch.yaml.tmp
kubectl patch clientconfig default -n kube-public --type merge --patch-file client-config-patch.yaml.tmp
rm client-config-patch.yaml.tmp

Troubleshooting

• Add the -v=1 option to the kube config:

kubectl config set-credentials ... --exec-arg=-v=1

• Inspect the logs of the pods of the gke-oidc-service in the anthos-identity-service namespace. Three pods are deployed, to view logs from all pods:

kubectl logs --tail 500 -l k8s-app=gke-oidc-service --all-containers -n anthos-identity-service

Azure Kubernetes Service

Step 3: Configure Developer Access

1. Any kubectl command will automatically trigger the login flow if the user is not authenticated:

kubectl get pods

1. The browser opens your IdP’s login page. If credentials are still cached in the browser developers don’t have to provide password / MFA again.

2. After successful authentication the browser will display:

1. The command kubectl command in the CLI is automatically executed after successful login.

Setup Instructions

1. Engineers must install the kubelogin Kubernetes client plugin. For example, on a Mac:

brew install int128/kubelogin/kubelogin

1. Engineers must configure the client the plugins with the appropriate arguments. Distribute the instructions on how to do that, which depends on the cloud provider where the cluster is hosted. We describe the commands below, and you can also find handy scripts to generate these commands automatically from your Terraform state in the accompanying repository.

AWS

cluster=OidcDemoCluster
region=us-west-2

idpConfig=$(aws eks describe-identity-provider-config --region $region --cluster-name $cluster --identity-provider-config type=oidc,name=OidcDemoConfig --output json)

issuer=$(echo $idpConfig | jq -r .identityProviderConfig.oidc.issuerUrl)
clientId=$(echo $idpConfig | jq -r .identityProviderConfig.oidc.clientId)
server=$(aws eks describe-cluster --region $region --name $cluster --query "cluster.endpoint" --output text)
caData=$(aws eks describe-cluster --region $region --name $cluster --query "cluster.certificateAuthority.data" --output text)

kubectl config set-credentials ${cluster} --exec-command=kubectl --exec-api-version=client.authentication.k8s.io/v1beta1 \
--exec-arg="oidc-login" \
--exec-arg="get-token" \
--exec-arg="--oidc-issuer-url=${issuer}" \
--exec-arg="--oidc-client-id=${clientId}" \
--exec-arg="--oidc-extra-scope=openid profile email groups offline_access"

kubectl config set-cluster ${cluster} --server=${server}
kubectl config set clusters.${cluster}.certificate-authority-data "${caData}"

kubectl config set-context ${cluster} --cluster=${cluster} --user=${cluster}

kubectl config use-context ${cluster}

Google Cloud Platform (GCloud)

cluster=oidc-demo

clientConfig=$(kubectl get clientconfig default -n kube-public -o json)

issuer=$(echo $clientConfig | jq -r '.spec.authentication[0] | select(.name == "oidc") | .oidc.issuerURI')
clientId=$(echo $clientConfig | jq -r '.spec.authentication[0] | select(.name == "oidc") | .oidc.clientID')
clientSecret=$(echo $clientConfig | jq -r '.spec.authentication[0] | select(.name == "oidc") | .oidc.clientSecret')
server=$(echo $clientConfig | jq -r '.spec.server')
caData=$(echo $clientConfig | jq -r '.spec.certificateAuthorityData')

kubectl config set-credentials ${cluster} --exec-command=kubectl --exec-api-version=client.authentication.k8s.io/v1beta1 \
--exec-arg="oidc-login" \
--exec-arg="get-token" \
--exec-arg="--oidc-issuer-url=${issuer}" \
--exec-arg="--oidc-client-id=${clientId}" \
--exec-arg="--oidc-client-secret=${clientSecret}" \
--exec-arg="--oidc-extra-scope=openid profile email groups offline_access"

kubectl config set-cluster ${cluster} --server=${server}
kubectl config set clusters.${cluster}.certificate-authority-data "${caData}"

kubectl config set-context ${cluster} --cluster=${cluster} --user=${cluster}

kubectl config use-context ${cluster}

Troubleshooting

kubectl oidc-login setup --oidc-issuer-url=${issuer} --oidc-client-id=${clientId} --oidc-extra-scope="openid profile email groups offline_access"

{
  "sub": "1234456789",
  "name": "John Smith",
  "email": "john.smith@example.com",
  "ver": 1,
  "iss": "https://example.okta.com/oauth2/<REDACTED>",
  "aud": "<REDACTED>",
  "iat": 1697392001,
  "exp": 1697395601,
  "jti": "<REDACTED>",
  "amr": [
    "mfa",
    "otp",
    "pwd"
  ],
  "idp": "<REDACTED>",
  "nonce": "<REDACTED>",
  "preferred_username": "john.smith@example.com",
  "auth_time": 1697391436,
  "at_hash": "<REDACTED>",
  "groups": [
    "k8s-oidc-demo-group"
  ]
}

Appendix

Execute the PKCE flow manually

1. /authorize

$ python3 -m http.server
Serving HTTP on :: port 8000 (http://[::]:8000/) ...

export ISSUER=
export CLIENT_ID= 
export CODE_CHALLENGE=
export CODE_VERIFIER= 
open $(echo "${ISSUER}/authorize?access_type=offline&client_id=${CLIENT_ID}&code_challenge_method=S256&code_challenge=${CODE_CHALLENGE}&redirect_uri=http%3A%2F%2Flocalhost%3A8000&response_type=code&scope=openid%20profile%20email" | envsubst)

export AUTH_CODE=<copied value>

2. /token

curl -X POST "${ISSUER}/token" \
--form grant_type="authorization_code" \
--form code="${AUTH_CODE}" \
--form client_id="${CLIENT_ID}" \
--form redirect_uri="http://localhost:8000" \
--form code_verifier="${CODE_VERIFIER}" \
--form scope="profile openid email"

{
    "token_type": "Bearer",
    "scope": "email openid profile",
    "expires_in": 3764,
    "ext_expires_in": 3764,
    "access_token": "ey...XyHtA",
    "id_token": "ey...Deaw"
}

{
    "error": "invalid_grant",
    "error_description": "AADSTS501481: The Code_Verifier does not match the code_challenge supplied in the authorization request ...",
    "error_codes": [
        501481
    ],
    ...
}