Kubernetes GitOps with FluxCD - Part 4 - Helm Chart Automation - Kube Prometheus Stack

Table of Contents

In our previous post, we explored how Flux CD enables automated image updates while maintaining GitOps principles. Building on our foundation of basic Flux CD setup, SOPS-based secret management, and image update automation, this article focuses on Helm Chart Automation - a sophisticated capability that further enhances your GitOps workflow by declaratively managing Helm releases.

Helm serves as Kubernetes’ package manager, simplifying the deployment of complex applications through charts that bundle related Kubernetes resources. While Helm itself is powerful, managing Helm releases manually contradicts GitOps principles of full automation and declarative configuration. Flux CD bridges this gap with its Helm Controller, allowing teams to define, deploy, and update Helm releases through Git-based declarations rather than imperative commands.

This article demonstrates advanced techniques for automating Helm chart deployments with Flux CD, using the Kube Prometheus Stack as a practical example. We’ll explore managing chart dependencies, version control, and environment-specific configurations while adhering to GitOps best practices.

1. Setup Helm Source and Release

First, let’s define a Helm Repository and Release at cluster/default/kps-helm.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: prometheus-comminty
  namespace: monitoring
spec:
  interval: 30m
  url: https://prometheus-community.github.io/helm-charts
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: kube-prometheus-stack
  namespace: monitoring
spec:
  releaseName: kube-prometheus-stack
  interval: 10m
  chart:
    spec:
      chart: kube-prometheus-stack
      version: "65.8.1"
      interval: 10m
      sourceRef:
        kind: HelmRepository
        name: prometheus-comminty
        namespace: monitoring
  install:
    crds: CreateReplace
  upgrade:
    crds: CreateReplace
  values:
    grafana:
      defaultDashboardsTimezone: "UTC+5:30"
    prometheus:
      prometheusSpec:
        ruleSelector: {}
        ruleNamespaceSelector: {}
        ruleSelectorNilUsesHelmValues: false
        serviceMonitorSelector: {}
        serviceMonitorNamespaceSelector: {}
        serviceMonitorSelectorNilUsesHelmValues: false
        podMonitorSelector: {}
        podMonitorNamespaceSelector: {}
        podMonitorSelectorNilUsesHelmValues: false
        retention: 30d
        enableOTLPReceiver: true
        enableRemoteWriteReceiver: true
        enableFeatures: [otlp-write-receiver]

FluxCD is capable of managing Custom Resource Definitions (CRDs), which is why we’ve configured crds: CreateReplace. This directive instructs FluxCD to automatically create and replace CRDs during subsequent upgrades—a capability we’ll validate later in this post.

Let’s push these changes and monitor the reconciliation process to apply this Helm release:

flux -n monitoring events --watch 

LAST SEEN       TYPE    REASON                  OBJECT                                  MESSAGE                                                                                                                      
48s             Normal  HelmChartCreated        HelmRelease/kube-prometheus-stack       Created HelmChart/monitoring/monitoring-kube-prometheus-stack with SourceRef 'HelmRepository/monitoring/prometheus-comminty'
47s     Normal  NoSourceArtifact        HelmChart/monitoring-kube-prometheus-stack      no artifact available for HelmRepository source 'prometheus-comminty'
45s     Normal  ChartPullSucceeded      HelmChart/monitoring-kube-prometheus-stack      pulled 'kube-prometheus-stack' chart with version '65.8.1'
44s     Normal  ArtifactUpToDate        HelmChart/monitoring-kube-prometheus-stack      artifact up-to-date with remote revision: '65.8.1'
47s     Normal  NewArtifact     HelmRepository/prometheus-comminty      stored fetched index of size 5.487MB from 'https://prometheus-community.github.io/helm-charts'
0s      Normal  InstallSucceeded        HelmRelease/kube-prometheus-stack       Helm install succeeded for release monitoring/kube-prometheus-stack.v1 with chart [email protected]

Next, let’s access Grafana to verify the deployment:

kubectl -n monitoring port-forward services/kube-prometheus-stack-grafana 3000:80

Forwarding from 127.0.0.1:3000 -> 3000
Forwarding from [::1]:3000 -> 3000

Let’s confirm that the default Kubernetes metrics dashboards are functioning correctly:

As we can observe, everything is operational and properly configured.

2. Upgrade helm release

We initially configured our Helm release at version 65.8.1. At the time of writing this post, the latest available version is 69.5.2.

According to the official release documentation, one would typically need to manually upgrade the Custom Resource Definitions before performing a Helm upgrade. However, since FluxCD can manage this process automatically, let’s put this feature to the test.

Our approach will be methodical: first, we’ll capture the current CRD state in the cluster, then upgrade the Helm release, and finally compare the new CRD definitions to validate the upgrade process.

Let’s begin by fetching the current CRD:

kubectl describe crd prometheuses.monitoring.coreos.com > beforeupgrade.yaml
head beforeupgrade.yaml 

Name:         prometheuses.monitoring.coreos.com
Namespace:    
Labels:       helm.toolkit.fluxcd.io/name=kube-prometheus-stack
              helm.toolkit.fluxcd.io/namespace=monitoring
Annotations:  controller-gen.kubebuilder.io/version: v0.16.1
              operator.prometheus.io/version: 0.77.2
API Version:  apiextensions.k8s.io/v1
Kind:         CustomResourceDefinition
Metadata:
  Creation Timestamp:  2025-02-27T07:49:22Z

For the next step, we’ll update our Helm Release manifest to version 69.5.2 and push these changes to our Git repository. Then we’ll monitor the reconciliation process:

flux -n monitoring events  --watch

LAST SEEN       TYPE    REASON                  OBJECT                                  MESSAGE 
14s     Normal  ChartPullSucceeded      HelmChart/monitoring-kube-prometheus-stack      pulled 'kube-prometheus-stack' chart with version '69.5.2'
21m     Normal  NewArtifact     HelmRepository/prometheus-comminty      stored fetched index of size 5.487MB from 'https://prometheus-community.github.io/helm-charts'
0s      Normal  UpgradeSucceeded        HelmRelease/kube-prometheus-stack       Helm upgrade succeeded for release monitoring/kube-prometheus-stack.v2 with chart [email protected]

Now, let’s examine if the CRDs were successfully updated during the process:

kubectl describe crd prometheuses.monitoring.coreos.com > afterupgrade.yaml
head afterupgrade.yaml 

Name:         prometheuses.monitoring.coreos.com
Namespace:    
Labels:       helm.toolkit.fluxcd.io/name=kube-prometheus-stack
              helm.toolkit.fluxcd.io/namespace=monitoring
Annotations:  controller-gen.kubebuilder.io/version: v0.17.1
              operator.prometheus.io/version: 0.80.1
API Version:  apiextensions.k8s.io/v1
Kind:         CustomResourceDefinition
Metadata:
  Creation Timestamp:  2025-02-27T07:49:22Z

The results confirm the upgrade was successful. We can observe that the controller-gen version has been updated from v0.16.1 to v0.17.1, and the Prometheus operator version has been upgraded from 0.77.2 to 0.80.1.

Additionally, we can verify that all dependencies have been upgraded as well (as seen in the Grafana version displayed in the screenshot below):

What next ?

In forthcoming articles, we’ll explore more advanced GitOps patterns with FluxCD, including:

  • Notification and alerting configuration
  • Push based reconciliation triggers with Webhook receivers

Stay tuned for each of these topics.

References