Secrets Management with External Secrets, Argo CD and GitOps

Introduction

The previous tutorial’s introduction discussed the benefits of managing secrets with Argo CD and the Argo CD Vault Plugin via GitOps. This guide explores using the alternative secrets management tool, External Secrets Operator¹ (ESO). Although the implementation of these tools differ, both essential solve the same problem. How to manage secrets via GitOps.

How does External Secrets Operator differ from Argo CD Vault Plugin?

Prerequisites

For this guide you’ll need the following:

• A secret management system such as HashiCorp Vault (used in this guide), AWS Secrets Manager or Google Cloud Secret Manager (ESO supports many more systems)
• Beginner to intermediate knowledge of Argo CD, Kubernetes & Git/GitHub
• Terraform ≥ v0.15

How External Secrets Operator (ESO) Works

Simply put, the External Secrets Operator uses a controller to fetch secrets from an external secrets manager and create a Kubernetes secret inside a cluster.

ESO relies on a few Custom Resources to perform this function:

SecretStore - This resource determines how the secret manager of your choice is accessed. It’s namespaced and multiple SecretStore’s can be used to access additional secret managers.

ExternalSecret - This resource functions as a template for creating a secret. It references the SecretStore for access details and declares what secret to fetch.

ClusterSecretStore - Acts like the SecretStore resource except it’s cluster scoped and can be accessed by ExternalSecrets in any namespace.

Here’s a high-level overview of what this guide aims to achieve; ESO working with Argo CD to automate the creation and updating of secrets in a Kubernetes cluster via GitOps:

1. Argo CD syncs the ExternalSecret manifest from a git repo to create an ExternalSecret resource inside the Kubernetes cluster.
2. The ESO Controller inspects the configuration of the synced ExternalSecret to determine which SecretStore to use and the path in the external API (Secrets Manager e.g. Vault) the secret it located on.
3. The Controller then queries the SecretStore for the credentials required to access the external API.
4. ESO fetches the secret(s) from the secrets manager at the path declared in the ExternalSecret.
5. Finally, ESO creates/syncs an actual Kubernetes secret resource inside the cluster.

Install External Secrets Operator on a Kubernetes Cluster

Install the External Secrets Operator either via Helm (using the helm command) or via Argo CD as an App:

Helm

Add the External Secretes repo:

helm repo add external-secrets https://charts.external-secrets.io

Install the External Secrets Operator:

helm install external-secrets \
   external-secrets/external-secrets \
  -n external-secrets \
  --create-namespace \
  --set installCRDs=true

Argo CD (my preference)

If you haven’t already, download and install the Argo CD CLI² then login to your Argo CD installation using the argocd command:

argocd login argocd.local
Username: admin
Password:
'admin:login' logged in successfully
Context 'argocd.local' updated

Create the external-secrets app using the following argocd command:

argocd app create external-secrets \
 --repo https://charts.external-secrets.io \
 --helm-chart external-secrets \
 --revision 0.5.9 \
 --dest-namespace external-secrets \
 --dest-server https://kubernetes.default.svc

And then sync it to complete the installation on the cluster:

argocd app sync external-secrets

Create a secret containing your Secret Manager’s credentials

Before creating a SecretStore, you’ll need to create a secret containing the credentials the SecretStore will use to authenticate with your Secrets Manager. HashiCorp’s Vault is the secrets manager used in this guide. With token-based authentication.

Run the following command to store your Vault token in a file named vault_token.txt:

echo -n 'VAULT_TOKEN' > ./vault_token.txt

Then use kubectl to create a secret named vault-token containing the credentials stored in the vault_token.txt file:

kubectl -n external-secrets create secret generic vault-token \
 --from-file=token=./vault_token.txt

Create a ClusterSecretStore

As previously mentioned, ESO offers two types of resources for defining how secret managers are accessed, SecretStore and ClusterSecretStore.

For this guide we’ll create a CluserSecretStore:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "https://vault.local"
      path: "secret"
      version: "v2"
      auth:
        # points to a secret that contains a vault token
        # https://www.vaultproject.io/docs/auth/token
        tokenSecretRef:
          name: "vault-token"
          key: "token"
          namespace: external-secrets

The above ClusterSeceretStore resource defines the URL of the Vault server (line 8). It also references the name (vault-token, line 15) and namespace (external-secrets, line 17) of the secret containing the static token required to authenticate against Vault.

Create a Secret in Vault

Create an example secret in Vault using the following command:

vault kv put secret/demo-secrets tokenA=too_many_secrets tokenB=not_enough_secrets

Key                Value
---                -----
created_time       2022-08-18T23:48:59.493381453Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Create an ExternalSecret (using Git and Argo CD)

Create a git repository containing the following example ExternalSecret manifest (here’s my example repo):

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: demo-secret
  namespace: default
spec:
  refreshInterval: "15s"
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: demo-secret
  data:
  - secretKey: token
    remoteRef:
      key: secret/demo-secrets
      property: tokenA

The ESO documentation example provides a detailed description of each parameter in an ExternalSecret resource, but the important lines from above are:

• 5. Namespace to create the secret resource in (default).
• 9. Name of the `ClusterSecretStore` created earlier (vault-backend).
• 12. Name of created secret.
• 14. Key for the secret value.
• 16. Path of the secret in the Vault server.
• 17. Value of the secret in the Vault server.

Sync this ExternalSecret from the git repository to the cluster by creating an app named demo-secret in Argo CD:

argocd app create demo-secret --repo https://github.com/colinwilson/argocd-micro-example-apps.git --path external-secrets/ExternalSecret --dest-namespace default --dest-server https://kubernetes.default.svc --sync-policy automated

A breakdown of the flags specified in the above command:

• argocd app create demo-secret creates an Argo CD application named ‘demo-secret’
• --repo https://github.com/colinwilson/argocd-micro-example-apps.git specifies the URL of your git repository
• --path external-secrets/ExternalSecret specifies the path within the git repo to use as the app directory (my example)
• --dest-namespace default the namespace to deploy the app to (default)
• --dest-server https://kubernetes.default.svc deploy the app internally i.e. the same Kubernetes cluster Argo CD is installed on
• --sync-policy automated set the sync policy to automated (optional)

If successful you can view the demo-secret app in the Argo CD UI:

Click on app to see further details:

You can see from the UI that the secret has been successfully created. You can confirm the value of the generated secret matches the value in Vault by running the kubectl command below:

kubectl -n default get secret demo-secret -o jsonpath="{.data.token}" | base64 -d; echo

too_many_secrets

Automatic Updates

Argo CD

Now the demo-secret app has been set up and configured with the automated sync policy enabled, any changes to the ExternalSecret in your git repository will be picked up by Argo CD’s polling and reflected in the cluster. Secrets can be automatically created in the cluster simply by pushing additional ExternalSecret manifests to the same repository path monitored by Argo CD.

External Secrets Operator

The ESO controller can monitor your Secret Manager(s) at regular intervals for changes in secret values and update the cluster secrets accordingly. This interval is set by spec.refreshInterval in the ExternalSecret. In the case of this guide, the interval for demo-secret is set to 15 seconds.

Summary

Once Argo CD has been configured with ESO and an external secrets manager, the workflow for managing secrets becomes streamlined.

• Pushing a new ExternalSecret manifest to your Argo CD monitored repository will result in a secret being created on the cluster
• Updating an existing ExternalSecret in your git repository results in synchronisation of those changes on the cluster
• Updates to secrets in your secret manager are automatically reflected in the secrets on the cluster

This is GitOps! Syncing configurations in your Git repository with your infrastructure. All without any secrets being stored in the repository itself.

This guide covered the basic feature of ESO. Check out both the External Secrets and Argo CD documentation for more info on other features.

References

• External Secrets Operator Documentation
• Argo CD Documentation

Footnotes