Deploying External Secrets Operator into Kubernetes

Documentation

Kubernetes

External Secrets Operator (ESO)

Authentication Notes

On the ESO cluster, ESO runs with the external-secrets ServiceAccount. This is the account used when making requests to the external vault.

On the vault side we will be configuring a kubernetes authentication method and role which uses short-lived jwt tokens.

On the ESO cluster we need to grant an additional role to the external-secrets ServiceAccount via a ClusterRoleBinding (role-tokenreview-binding). The system:auth-delegator role gives the ability for the external-secrets-sa ServiceAccount to request short-lived tokens from the ESO cluster. This is required by the auth method we have configured on vault.

Because the ServiceAccount has this role, it can now request short-lived-tokens from the ESO cluster. The cluster will issue a short-lived token and sign it using the CA certificate of the ESO cluster.

On the vault side, we add the ESO’s certificate to the auth method and disable use of local CA and service account JWT thus instructing vault to trust any tokens singed by the ESO cluster’s CA certificate and not expect the tokens to be signed by the vault cluster’s own CA certificate.

Need to also create a role in the auth method that is bound to the ServiceAccount name es-external and bound to the namespace external-secrets

Firewall rules

ESO Cluster IPs need access the vault-ui https endpoint.

Vault Cluster IPs need access the the ESO Cluster’s Kubernetes https API endpoint.

Install

Installation wil be done via helm:

helm repo add external-secrets https://charts.external-secrets.io
helm show values external-secrets/external-secrets > external-secrets-values.yaml
helm install external-secrets \
   external-secrets/external-secrets \
    -n external-secrets \
    --create-namespace \
  --values ./external-secrets-values.yaml

Create ClusterRoleBinding

cat <<EOF > external-secrets-clusterrolebinding.yaml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
  namespace: external-secrets
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: external-secrets
  namespace: external-secrets\
EOF
kubectl apply -f external-secrets-clusterrolebinding.yaml

Things should look like this:

kubectl get namespaces|grep -e NAME -e external-secrets
# NAME               STATUS   AGE
# external-secrets   Active   22m


kubectl -n external-secrets get pods
# NAME                                                READY   STATUS    RESTARTS   AGE
# external-secrets-569d5f796f-k4pn2                   1/1     Running   0          23m
# external-secrets-cert-controller-69b5cc454f-c6ct5   1/1     Running   0          23m
# external-secrets-webhook-6c7b888b5d-m98r2           1/1     Running   0          23m


kubectl -n external-secrets describe pod external-secrets-569d5f796f-k4pn2|grep Service\ Account
# Service Account:  external-secrets-sa


kubectl get ClusterRoleBindings|grep role-tokenreview-binding
# role-tokenreview-binding                               ClusterRole/system:auth-delegator                                  26m


kubectl describe ClusterRoleBinding role-tokenreview-binding
# Name:         role-tokenreview-binding
# Labels:       <none>
# Annotations:  <none>
# Role:
#   Kind:  ClusterRole
#   Name:  system:auth-delegator
# Subjects:
#   Kind            Name                 Namespace
#   ----            ----                 ---------
#   ServiceAccount  external-secrets-sa  external-secrets

Configuring vault

Setup environment

export VAULT_ADDR=https://vault.mogocloud.ca:443 #vault-endpoint
kctx mogo-ops-cluster
ESOCLUSTER_CA=$(k config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)
ESOCLUSTER_HOST=$(k config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')
ESO_SA=external-sercrets
ESO_NS=external-secrets

pass -c com/github/momelod/vault ; vault login --method=github
# Copied com/github/momelod/vault to clipboard. Will clear in 45 seconds.
# GitHub Personal Access Token (will be hidden):
# Success! You are now authenticated. The token information displayed below
# is already stored in the token helper. You do NOT need to run "vault login"
# again. Future Vault requests will automatically use this token.
# 
# Key                    Value
# ---                    -----
# token                  hvs.xyz
# token_accessor         xyz
# token_duration         768h
# token_renewable        true
# token_policies         ["default" "devops-admin-policy"]
# identity_policies      []
# policies               ["default" "devops-admin-policy"]
# token_meta_org         mogofinancial
# token_meta_username    momelod

Create the kubernetes auth method for this new cluster.

vault auth enable -path=mogo-ops-cluster kubernetes
# Success! Enabled kubernetes auth method at: mogo-ops-cluster/

Configure the new auth method with the CA cert and HOST of our ESO Cluster:

vault write auth/mogo-ops-cluster/config \
     kubernetes_host="$ESOCLUSTER_HOST" \
     kubernetes_ca_cert="$ESOCLUSTER_CA" \
     disable_local_ca_jwt=true
# Success! Data written to: auth/mogo-ops-cluster/config

Configure a new auth role within the auth method

vault write auth/mogo-ops-cluster/role/external-secret \
        bound_service_account_names=external-secrets \
        bound_service_account_namespaces=external-secrets \
        policies=oke-policy \
        kubernetes_host="$ESOCLUSTER_HOST" \
        kubernetes_ca_cert="$ESOCLUSTER_CA"
# Success! Data written to: auth/mogo-ops-cluster/role/external-secret

Configure a SecretStore

ClusterSecretStore and SecretStore objects manage the authentication to a secret backend. In our case vault. ClusterSecretStore is available cluster wide, whereas SecretStore is namespace scoped.

cat <<EOF > /tmp/SecretStore.yaml
---
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: external-secrets
spec:
  provider:
    vault:
      server: "https://vault.mogocloud.ca"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "mogo-ops-cluster"
          role: "external-secret"
          serviceAccountRef:
            name: "external-secrets"
EOF
k apply -f /tmp/SecretStore.yaml
# secretstore.external-secrets.io/vault-prod-backend created


k get SecretStores
# NAME            AGE   STATUS   CAPABILITIES   READY
# vault-backend   35m   Valid    ReadWrite      True

Create an ExternalSecret

The external secret object uses the ClusterSecretStore we created for authentication and uses that to fetch a named secret in vault. With that secret it creates a local copy as a kubernetes secret. The operator will update the local kubernetes secret anytime the copy in vault is changed.

cat <<EOF > /tmp/ExternalSecret.yaml
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: test-secret
  namespace: external-secrets
spec:
  refreshInterval: "15s"
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: test-secret
  data:
  - secretKey: DD_API_URL
    remoteRef:
      key: secrets/devops/kv/data/datadog-shared-devops-key
      property: DD_API_URL
EOF
k -n external-secrets apply -f /tmp/ExternalSecret.yaml
# externalsecret.external-secrets.io/datadog-secret created


k -n external-secrets get ExternalSecret test-secret
# NAME          STORE           REFRESH INTERVAL   STATUS         READY
# test-secret   vault-backend   15s                SecretSynced   True


k -n external-secrets get secret datadog-secret
# NAME             TYPE     DATA   AGE
# test-secret      Opaque   2      38m

Profit!

k -n external-secrets get secret test-secret -o json |jq -r '.data.["DD_API_URL"]'|base64 -d
# https://api.datadoghq.com

Debugging

Why are we getting the SecretSyncedError? Need to get on a vault-N pod and monitor the vault logs for errors:

vault login 
vault monitor --log-level=debug

Uninstallation

kubectl get SecretStores,ClusterSecretStores,ExternalSecrets --all-namespaces
helm delete external-secrets --namespace external-secrets
kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n external-secrets

Deploying netbox into Kubernetes Deploying redis into Kubernetes