GitOps - Part 2

In the previous post, we looked at how to use fluxv2 for deploying apps through helm and kustomization. In this we will take a brief look at how to secure your app secrets using Mozilla SOPS. There are many ways for securing the secrets and one of the ways was to use the secret store CSI driver which was covered in this post and the other popular way is to use sealed secrets for which details can be found here. These approaches come with a step of installing the extra component which acts as a controller for securing the secret.

We will look at Mozilla SOPS which provides the CLI to encrypt secrets with all the major cloud provider without having to install additional component and we will see the steps for using the same in Azure.

Create a variables which will be used for the scripts.

export GITHUB_TOKEN=xxxxxx
export RESOURCE_GROUP_NAME=gitopssops
export LOCATION=southeastasia
export CLUSTER_NAME=gitopssops
export GITHUB_USER=girishgouda
export GITHUB_REPO=githubsops        

Creation of AKS cluster with managed identity enabled

az group create -n $RESOURCE_GROUP_NAME -l $LOCATION
az aks create -g $RESOURCE_GROUP_NAME -n $CLUSTER_NAME --enable-managed-identity
az aks get-credentials -g $RESOURCE_GROUP_NAME -n $CLUSTER_NAME
kubectl get nodes        

Install or bootstrap flux in the cluster.

kubectl create ns flux-system
flux bootstrap github \
--owner=$GITHUB_USER \
--repository=$GITHUB_REPO \
--branch=master \
--path=./clusters/$CLUSTER_NAME        

Create a kustomization which will point to manifests folder to synchronize yaml files to Kubernetes. This will be used to have encrypted secret.

flux create kustomization demoapp \
  --namespace=flux-system \
  --source=flux-system \
  --path="./manifests" \
  --prune=true \
  --validation=client \
  --interval=5m \
  --export > ./clusters/$CLUSTER_NAME/demoapp-kustomization.yaml        

Create a manifests directory

mkdir manifests        

Create a namespace called demoapp where the secret will be created

cat > ./manifests/namespace.yaml <<EOF
---
apiVersion: v1
kind: Namespace
metadata:
  name: demoapp
EOF
        

Install the AAD pod identity using helm in a namespace aad-pod-idnetity as shown below.

cat > ./clusters/gitops/aad-pod-identity.yaml <<EOF
---
apiVersion: v1
kind: Namespace
metadata:
  name: aad-pod-identity
---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  name: aad-pod-identity
  namespace: aad-pod-identity
spec:
  url: https://raw.githubusercontent.com/Azure/aad-pod-identity/master/charts
  interval: 10m
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: aad-pod-identity
  namespace: aad-pod-identity
spec:
  interval: 5m
  chart:
    spec:
      chart: aad-pod-identity
      version: 4.0.0
      sourceRef:
        kind: HelmRepository
        name: aad-pod-identity
        namespace: aad-pod-identity
      interval: 1m
  values:
    nmi:
      allowNetworkPluginKubenet: true
EOF        

Push the changes to git repo and verify that the aad-pod-identity and demoapp are created.

Create a azure key vault and create a key which will be used for encryption and decryption of secret

export KEY_VAULT_NAME=gitopsazurevault
az keyvault create --name $KEY_VAULT_NAME --resource-group $RESOURCE_GROUP_NAME --location $LOCATION
az keyvault key create --name sops-key --vault-name $KEY_VAULT_NAME --protection software --ops encrypt decrypt
az keyvault set-policy --name $KEY_VAULT_NAME --resource-group $RESOURCE_GROUP_NAME --object-id $OBJECT_ID --key-permissions encrypt decrypt
az keyvault key show --name sops-key --vault-name $KEY_VAULT_NAME --query key.kid        

The last command will provide a key "https://gitopsazurevault.vault.azure.net/keys/sops-key/xxxxx" which will be used for encryption and decryption.

Next, we need to tell kustomize controller to use the managed identity of the cluster to access the key for decryption.

First, we create a azureidentity with the details of the cluster manged identity and then bind the identity to selector called sops-aks-decryptor as shown.

cat > ./clusters/gitops/sops-identity.yaml <<EOF
---
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentity
metadata:
  name: sops-akv-decryptor
  namespace: flux-system
spec:
  clientID: xxxxxx #client id of  the cluster managed identity 
  resourceID: /subscriptions/subId/resourcegroups/MC_gitopssops_gitopssops_southeastasia/providers/Microsoft.ManagedIdentity/userAssignedIdentities/gitopssops-agentpool
  type: 0 # user-managed identity
---
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentityBinding
metadata:
  name: sops-akv-decryptor-binding
  namespace: flux-system
spec:
  azureIdentity: sops-akv-decryptor
  selector: sops-akv-decryptor  # kustomize-controller label will match this name
EOF        

Note: we need to specify the clientId of the managed identity which was created with the cluster creation and also the resourceId of the managed identity.

Next, we patch the kustoimze controller by modifying the kustomization.yaml to include patch yaml as shown below.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- gotk-components.yaml
- gotk-sync.yaml
patchesStrategicMerge:
- sops-kustomize-patch.yaml        

The patch file will tell the kustomize controller to use azure msi for authentication.

--
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kustomize-controller
  namespace: flux-system
spec:
  template:
    metadata:
      labels:
        aadpodidbinding: sops-akv-decryptor  # match the AzureIdentityBinding selector
    spec:
      containers:
      - name: manager
        env:
        - name: AZURE_AUTH_METHOD
          value: msi-        

Note: The same selector sops-aks-decryptor is used for aadpodidbinding

Finally, we need to tell kustomize-controller to use SOPS as decryption provider as shown below.

Install SOPS and create a SOPS conf file which will be used for encrypting the secret. The configuration file contains the details of path and the sections of the files to be encrypted using the key as shown below.

cat > .sops.yaml <<EOF
creation_rules:
  - path_regex: .*.yaml
    encrypted_regex: ^(data|stringData)$
    azure_keyvault: https://gitopsdemokeyvault.vault.azure.net/keys/sops-key/xxxxxx
EOF        

Create a temporary secret file which will not be pushed to git.

cat > ./secret.yaml <<EOF
---
apiVersion: v1
kind: Secret
metadata:
  name: demoapp-credentials
  namespace: demoapp
type: Opaque
stringData:
  username: admin
  password: password
EOF        

Note : SOPS CLI will use the logged in user in AZ CLI and hence make sure the logged in user has "encrypt" and "decrypt" key permissions.

Encrypt the secret.yam file as shown below.

sops --encrypt secret.yaml > ./manifests/secret.enc.yaml        

The encrypted file will be as shown below.

apiVersion: v1
kind: Secret
metadata:
    name: demoapp-credentials
    namespace: demoapp
type: Opaque
stringData:
    username: ENC[AES256_GCM,data:BYkU34g=,iv:DMTnv3Ucd5xSGHGbEIBIeypKydHwxTnYl3enK0nKdjw=,tag:vagn0DVSfjOSaDGVRfLdMg==,type:str]
    password: ENC[AES256_GCM,data:Fey4wZyIvB4Jvg==,iv:gFtSzrtuC5MfC9OVkPNIYZ4MiYGX9Xsy81fZPUsZzIM=,tag:GW4hSN9U+YcYYwytCZKHZQ==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv:
        - vault_url: https://gitopsazurevault.vault.azure.net
          name: sops-key
          version: xxxxxxxxx
          created_at: "2021-09-14T05:25:57Z"
          enc: ODebW_-5BhT2LdEzWrC9NFxA4A6INoTEVesUmT_9ECA0hKb4-nSEj-QDcHigjJZpGh3qUm6jZSGbjQpcPdE8eHYLlpmgb8sU4nDn3P8_Gm9KBB1txSg3fDTgEl0HEs6Re3zu7VQo0gVTbnZN-qyLRZbVvqN6NnK7npWB2SXLJNDucHiqDqBDluJxkKhM0ngTh2qvPp7MzQZ1L9A9P0tLYrE8btBJYLzGUKRFc_pbfTPMPxtoei00TShWjFhtO3WNQ9tzqJz0Ajy7ITn5ntu2YDpbE9jqsAe3k6yqeKRIZ8TbHPHm329dNhlUiQFujpVVy-ypUkyg
    hc_vault: []
    age: []
    lastmodified: "2021-09-14T05:26:00Z"
    mac: ENC[AES256_GCM,data:P/54bG7GUxilkhJ2gbHlW2jUA9wcL7a8Wp4cyupyaOHdyB9hRtWYkFsnBblQLi7t5QMe7XJEOisZR45l1HOHZVgvZUg5ghI/4xLmc0eC/krkqv5CNeH5JyAEuq3yr9o/nLy0+EjgNIrc7tLl2Xo8Qn3VdR1KytWgLXpK0FORwcE=,iv:1FylnUvw3RBn/mR+ZhBwukIh+TbBxvD6irsoYUTRVCE=,tag:KI3qIxwyJWKAkkX9lctG4w==,type:str]
    pgp: []
    encrypted_regex: ^(data|stringData)$
    version: 3.7.1

        

We can now remove the secret.yaml and push the secret.enc.yaml to git

We can finally verify the sceret being created by using the folowing command

kubectl describe secret -n demoapp demoapp-credentials        

This is how we can use sops in Fluxv2 to safely push the secrets to git. In the future articles, we will look at some more interesting features.