Kubernetes GitOps with FluxCD - Part 3 - Automated Image Updates

Table of Contents

In our previous post, we explored how to manage secrets in FluxCD with SOPS. Building on our GitOps foundation from part 1 and secure secrets handling from part 2, this article will focus on image update automation - a powerful FluxCD feature that automatically updates your deployments when new container images are available, maintaining GitOps principles while eliminating manual image version updates.

1. Setup image repository

To demonstrate image automation, we’ll create a sample application using Quarkus, a Kubernetes-native Java framework.

First, let’s create a GitHub repository.

Let’s implement a simple REST controller for our sample application.

package com.kiriyard;

import jakarta.ws.rs.GET;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;

@Path("/")
public class K8SampleApplication {

    private static final int VERSION = 1;

    @GET
    @Produces(MediaType.TEXT_PLAIN)
    public String index() {
        return "Greetings From K8S App : Version %d".formatted(VERSION);
    }
}

Now, let’s create a GitHub Action workflow that builds and publishes this image to the GitHub Container Registry.

name: Build K8S App Image

on:
  push:
    branches:
      - main

jobs:
  build-image:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      checks: write
      id-token: write
      packages: write
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Login to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Set up JDK 21
        uses: actions/setup-java@v4
        with:
          distribution: temurin
          java-version: 21
          cache: 'maven'

      - name: Generate build ID
        id: prep
        run: |
          branch=${GITHUB_REF##*/}
          sha=${GITHUB_SHA::8}
          ts=$(date +%s)
          echo "BUILD_ID=${branch}-${sha}-${ts}" >> $GITHUB_OUTPUT          

      - name: Build and push Docker image
        run: |
          ./mvnw install \
            -Dquarkus.container-image.build=true \
            -Dquarkus.container-image.registry=ghcr.io \
            -Dquarkus.container-image.group="" \
            -Dquarkus.container-image.name=${{ github.repository }} \
            -Dquarkus.container-image.tag=${{ steps.prep.outputs.BUILD_ID }} \
            -Dquarkus.container-image.push=true

After pushing our changes, we can verify the image has been created in the GitHub Container Registry.

We’ll note the full image path for future reference: ghcr.io/kiriapurv/k8s-sample-app:main-3cbb3902-1740556397

2. Setup private repository pull secrets

Since we’re using a private container registry, we need to configure GitHub authentication credentials as Kubernetes pull secrets for our deployment.

We’ll generate a Personal Access Token (PAT) and configure a Kubernetes secret using SOPS, following the encryption approach we established in our previous post.

First, let’s create the secret YAML manifest:

kubectl create secret docker-registry github-registry-secret \
  --docker-server=ghcr.io \
  --docker-username=** \
  --docker-password=** \
  --namespace=default \
  --dry-run=client -o yaml > github-registry-secret.yaml

Now, let’s encrypt this secret using SOPS:

sops encrypt --in-place github-registry-secret.yaml

After pushing these changes to our Git repository, we’ll wait for FluxCD to perform reconciliation:

flux get kustomizations flux-system --watch

NAME            REVISION                SUSPENDED       READY   MESSAGE                              
flux-system     main@sha1:893edbfc      False           True    Applied revision: main@sha1:893edbfc
flux-system     main@sha1:893edbfc      False   Unknown Reconciliation in progress
flux-system     main@sha1:893edbfc      False   Unknown Reconciliation in progress
flux-system     main@sha1:893edbfc      False   Unknown Reconciliation in progress
flux-system     main@sha1:893edbfc      False   Unknown Reconciliation in progress
flux-system     main@sha1:893edbfc      False   True    Applied revision: main@sha1:c2636b99
flux-system     main@sha1:c2636b99      False   True    Applied revision: main@sha1:c2636b99

Let’s verify that our secret has been successfully created in the cluster:

kubectl get secrets

NAME                     TYPE                             DATA   AGE
github-registry-secret   kubernetes.io/dockerconfigjson   1      65s
samplesecret             Opaque                           1      20h

3. Create deployment

Now let’s create a Kubernetes deployment using our container image.

First, let’s create apps/sample-app/deployment.yaml

apiVersion: apps/v1
metadata:
  name: sample-app
  namespace: default
kind: Deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      imagePullSecrets:
        - name: github-registry-secret
      containers:
      - name: sample-app
        image: ghcr.io/kiriapurv/k8s-sample-app:main-3cbb3902-1740556397
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "128Mi"
            cpu: "50m"
          limits:
            memory: "128Mi"
            cpu: "50m"

Next, create a Kustomization file to manage our resources at apps/sample-app/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml

Finally, create a FluxCD Kustomization to deploy our application at cluster/default/sample-app.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: sample-app
  namespace: flux-system
spec:
  interval: 5m
  path: ./apps/sample-app
  prune: true
  retryInterval: 2m
  sourceRef:
    kind: GitRepository
    name: flux-system
  targetNamespace: default
  timeout: 3m
  wait: true

After pushing these changes, let’s monitor the FluxCD reconciliation process:

flux get kustomizations sample-app --watch

NAME            REVISION        SUSPENDED       READY   MESSAGE                    
sample-app                      False           Unknown Reconciliation in progress
sample-app              False   Unknown Reconciliation in progress
sample-app              False   True    Applied revision: main@sha1:939bb8b3
sample-app      main@sha1:939bb8b3      False   True    Applied revision: main@sha1:939bb8b3

Let’s verify that our deployment has been created:

kubectl get pods
NAME                                READY   STATUS    RESTARTS      AGE
nginx-deployment-854f6c9499-wxdjx   1/1     Running   1 (11m ago)   45h
sample-app-9477cc684-fs876          1/1     Running   0             48s

To validate that our application is working correctly, let’s set up port forwarding and test the endpoint:

kubectl port-forward deployments/sample-app 8080:8080 
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

Let’s send a request to verify our application is responding:

curl http://localhost:8080

Greetings From K8S App : Version 1

Perfect! Our application is working as expected.

4. Setup FluxCD Image Automation

Now that we have our application deployed, we’ll configure FluxCD’s image automation capabilities. This requires three key components:

  1. An ImageRepository resource that monitors our container registry for new images
  2. An ImagePolicy resource that defines the criteria for selecting new images
  3. An ImageUpdateAutomation resource that configures the automation process

We’ll configure ImageRepository and ImagePolicy in same cluster/default/sample-app.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: sample-app
  namespace: flux-system
spec:
  interval: 5m
  path: ./apps/sample-app
  prune: true
  retryInterval: 2m
  sourceRef:
    kind: GitRepository
    name: flux-system
  targetNamespace: default
  timeout: 3m
  wait: true
---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
  name: sample-app
  namespace: default
spec:
  image: ghcr.io/kiriapurv/k8s-sample-app
  interval: 5m
  secretRef:
    name: github-registry-secret
---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
  name: sample-app-main-policy
  namespace: default
spec:
  imageRepositoryRef:
    name: sample-app
  filterTags:
    pattern: '^main-[a-fA-F0-9]+-(?P<ts>.*)'
    extract: '$ts'
  policy:
    numerical:
      order: asc

Understanding the ImagePolicy Configuration

The ImagePolicy resource is crucial for defining how FluxCD selects the latest version of an image. Let’s break down its key components:

Tag Filtering with filterTags

The filterTags section uses regular expressions to identify and extract meaningful information from image tags:

  • pattern: Defines a regex pattern to match against image tags. In our case, '^main-[a-fA-F0-9]+-(?P<ts>.*)' means:

  • ^main-: The tag must start with “main-”

  • [a-fA-F0-9]+: Followed by one or more hexadecimal characters (our Git commit SHA)

  • (?P<ts>.*): Captures the remaining part of the tag as a named group “ts” (timestamp)

  • extract: Specifies which part of the pattern to extract for version comparison. The '$ts' refers to the named capture group "ts" from our pattern, which contains the Unix timestamp.

This extraction is essential because the FluxCD automation needs to know which part of the tag contains the version information to compare.

Version Selection with policy

The policy section defines how FluxCD determines which image version is the “latest”:

  • numerical: Indicates we’re dealing with numeric versioning (as opposed to semantic versioning)

  • order: asc: Specifies that higher numbers are considered newer (ascending order)

In our case, we’re using Unix timestamps in our image tags, which naturally increase over time. By setting order: asc, we ensure that FluxCD selects the image with the most recent timestamp.

More details on sortable image tag is available in official documentation - https://fluxcd.io/flux/guides/sortable-image-tags/

Next, we need to annotate our deployment manifest to indicate which image policy should be applied. Let’s update

We’ll update apps/sample-app/deployment.yaml

apiVersion: apps/v1
metadata:
  name: sample-app
  namespace: default
kind: Deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      imagePullSecrets:
        - name: github-registry-secret
      containers:
      - name: sample-app
        image: ghcr.io/kiriapurv/k8s-sample-app:main-3cbb3902-1740556397 # {"$imagepolicy": "default:sample-app-main-policy"}
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "128Mi"
            cpu: "50m"
          limits:
            memory: "128Mi"
            cpu: "50m"

Note that we’ve added a special comment annotation after the image value: # {"$imagepolicy": “default:sample-app-main-policy”}.

This tells FluxCD which policy to use when determining image updates.

Finally, we need to create an ImageUpdateAutomation resource to orchestrate the update process.

Let’s create cluster/default/image-update-automation.yaml:

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageUpdateAutomation
metadata:
  name: default-image-update
  namespace: default
spec:
  interval: 30m
  sourceRef:
    kind: GitRepository
    name: flux-system
    namespace: flux-system
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        email: [email protected]
        name: fluxcdbot
      messageTemplate: '{{range .Changed.Changes}}{{print .OldValue}} -> {{println .NewValue}}{{end}}'
    push:
      branch: main
  update:
    path: ./
    strategy: Setters

This configuration tells FluxCD to:

  • Check for updates every 30 minutes
  • Commit changes using the FluxCD bot identity
  • Use a message template that shows which images were updated
  • Push changes to the main branch
  • Use the “Setters” strategy to update image references throughout the repository

Let’s push these changes and monitor FluxCD events to verify our automation is working:

flux events -n default --watch
LAST SEEN               TYPE    REASON                  OBJECT                                  MESSAGE                                             
31s (x24 over 2m38s)    Warning DependencyNotReady      ImagePolicy/sample-app-main-policy      referenced ImageRepository has not been scanned yet
30s (x2 over 30s)       Normal  Succeeded       ImagePolicy/sample-app-main-policy      Latest image tag for 'ghcr.io/kiriapurv/k8s-sample-app' resolved to main-3cbb3902-1740556397
30s     Normal  Succeeded       ImageRepository/sample-app      successful scan: found 1 tags

Let’s confirm that our image resources are properly configured:

flux get -n default images all
NAME                            LAST SCAN                       SUSPENDED       READY   MESSAGE                       
imagerepository/sample-app      2025-02-26T14:08:00+05:30       False           True    successful scan: found 1 tags

NAME                                    LATEST IMAGE                                                    READY   MESSAGE                                                                                      
imagepolicy/sample-app-main-policy      ghcr.io/kiriapurv/k8s-sample-app:main-3cbb3902-1740556397       True    Latest image tag for 'ghcr.io/kiriapurv/k8s-sample-app' resolved to main-3cbb3902-1740556397

5. Updating Image

Now let’s test our automation by updating our application code and pushing the change.

We’ll increment the version number in our controller class:

package com.kiriyard;

import jakarta.ws.rs.GET;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;

@Path("/")
public class K8SampleApplication {

-    private static final int VERSION = 1;
+    private static final int VERSION = 2;

    @GET
    @Produces(MediaType.TEXT_PLAIN)
    public String index() {
        return "Greetings From K8S App : Version %d".formatted(VERSION);
    }
}

After committing and pushing this change, GitHub Actions will build a new container image. Once the image is published, FluxCD should detect it, update our Git repository, and then apply the changes to our cluster. Let’s monitor the FluxCD events:

flux -n default events 

6s                      Normal  Succeeded               ImageUpdateAutomation/default-image-update      pushed commit '91fce99' to branch 'main'                                                                                                   
                                                                                                        ghcr.io/kiriapurv/k8s-sample-app:main-3cbb3902-1740556397 -> ghcr.io/kiriapurv/k8s-sample-app:main-29e96d2a-1740565130

We can also verify the changes in our Git repository:

Now let’s check if the deployment in our cluster has been updated:

kubectl -n default describe pod sample-app-649d676845-gcpwk 
Name:             sample-app-649d676845-gcpwk
Namespace:        default
Priority:         0
Service Account:  default
Node:             192.168.1.19/192.168.1.19
Start Time:       Wed, 26 Feb 2025 16:18:50 +0530
Labels:           app=sample-app
                  pod-template-hash=649d676845
Annotations:      <none>
Status:           Running
IP:               10.42.0.15
IPs:
  IP:           10.42.0.15
Controlled By:  ReplicaSet/sample-app-649d676845
Containers:
  sample-app:
    Container ID:   containerd://c505519d6dd3d013fa2be3275e2b5085df4486a4c3ebe75a50688b37e49eb7ff
    Image:          ghcr.io/kiriapurv/k8s-sample-app:main-29e96d2a-1740565130
    Image ID:       ghcr.io/kiriapurv/k8s-sample-app@sha256:8447a6a128b96474ee620dfc692c15d7c22309f973663917191d2b87b8f3db72
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Wed, 26 Feb 2025 16:18:57 +0530
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     50m
      memory:  128Mi
    Requests:
      cpu:        50m
      memory:     128Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-l2rxk (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  kube-api-access-l2rxk:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Guaranteed
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  4m42s  default-scheduler  Successfully assigned default/sample-app-649d676845-gcpwk to 192.168.1.19
  Normal  Pulling    4m41s  kubelet            Pulling image "ghcr.io/kiriapurv/k8s-sample-app:main-29e96d2a-1740565130"
  Normal  Pulled     4m35s  kubelet            Successfully pulled image "ghcr.io/kiriapurv/k8s-sample-app:main-29e96d2a-1740565130" in 5.3s (5.3s including waiting). Image size: 166698164 bytes.
  Normal  Created    4m35s  kubelet            Created container sample-app
  Normal  Started    4m35s  kubelet            Started container sample-app

As we can see, the pod is now running with the new image tag: main-29e96d2a-1740565130.

Finally, let’s test our application to confirm the version has been updated:

curl http://localhost:8080

Greetings From K8S App : Version 2

Success! Our application has been automatically updated to version 2, demonstrating the complete GitOps-based image automation flow with FluxCD.

What next ?

Future posts will explore advanced GitOps patterns with FluxCD, including:

  • Helm chart automation
  • Notification and alerting configuration

Stay tuned for each of these topics.

References