Kubernetes GitOps with FluxCD - Part 6 - Push-based Reconciliation with Webhook Receivers

Table of Contents

In our previous post, we set up Discord alerts to get notifications about our cluster’s status. Building on what we’ve learned so far—basic FluxCD setup, SOPS-based secret management, image update automation, Helm chart automation, and alerts—this article focuses on webhook receivers.

By default, FluxCD polls Git repositories at the configured frequency to detect changes. With webhook receivers, your Git provider can notify FluxCD immediately when changes occur, ensuring the reconciliation process doesn’t have to wait for the next scheduled polling interval. This significantly expedites deployments and enhances overall cluster efficiency.

Let’s configure webhook receivers for our Kubernetes cluster.

1. Setup Webhook Secret

First, we’ll generate a cryptographically secure random secret which will be configured in both the FluxCD webhook receiver and GitHub webhooks.

We’ll create cluster/default/github-webhook-token.yaml

TOKEN=$(openssl rand 24 | sha256sum -b | awk '{print $1}')

kubectl create secret generic github-webhook-token \
  --namespace=flux-system \
  --from-literal=token=$TOKEN \
  --dry-run=client -o yaml > github-webhook-token.yaml

Take note of the TOKEN variable (e.g., using echo $TOKEN) which we’ll later configure in GitHub.

The generated YAML file would look like:

apiVersion: v1
kind: Secret
metadata:
  name: github-webhook-token
  namespace: flux-system
data:
  token: <base64 encoded token>

Now let’s encrypt this secret with SOPS and publish changes to our Git repository.

Let’s confirm if the secret is created:

kubectl -n flux-system get secrets 

NAME                     TYPE     DATA   AGE
discord-webhook-secret   Opaque   1      3d20h
flux-system              Opaque   3      4d21h
github-webhook-token     Opaque   1      9s
sops-gpg                 Opaque   1      5d21h

As we can see, the token secret has been successfully created.

Note
For detailed instructions on secret management, refer to Kubernetes GitOps with FluxCD - Part 2 - Secret Management using SOPS.

2. Setup FluxCD Webhook Receiver

The next step is to configure the webhook receiver in FluxCD.

Let’s define it for our cluster’s flux-system repository.

Create cluster/default/flux-system/webhook-receiver.yaml

apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
  name: flux-system
  namespace: flux-system
spec:
  type: github
  events:
    - "ping"
    - "push"
  secretRef:
    name: github-webhook-token
  resources:
    - kind: GitRepository
      name: flux-system

Modify cluster/default/flux-system/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - gotk-components.yaml
  - gotk-sync.yaml
+ - webhook-receiver.yaml
patches:
  - patch: |-
      apiVersion: kustomize.toolkit.fluxcd.io/v1
      kind: Kustomization
      metadata:
        name: flux-system
        namespace: flux-system
      spec:
        decryption:
          provider: sops
          secretRef:
            name: sops-gpg

Let’s push these changes and observe if the webhook receiver is registered:

flux get receivers 

NAME            SUSPENDED       READY   MESSAGE                                                                                               
flux-system     False           True    Receiver initialized for path: /hook/2524a3ab747c79f26105e00e224d9d212558c1145ad3192ee59e1d6529f0bf0

Take note of the path, which we’ll configure in GitHub.

As we can see, the receiver is successfully initialized and the webhook path is configured.

3. Setup Ingress

So far we have configured the receiver, but GitHub webhooks require a public endpoint for our webhook receiver. Let’s configure an ingress for the flux webhook receiver service.

Create cluster/default/flux-system/ingress.yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: webhook-receiver
  namespace: flux-system
  annotations:
    external-dns.alpha.kubernetes.io/target: "XXXXXX"
    cert-manager.io/cluster-issuer: "cluster-issuer"
spec:
  tls:
  - hosts:
    - flux-wh.XXXXX
    secretName: flux-ingress-cert
  rules:
  - host: flux-wh.XXXX
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: webhook-receiver
            port:
              number: 80
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-acme-issuer
  namespace: flux-system
spec:
  podSelector:
    matchLabels:
      acme.cert-manager.io/http01-solver: "true"
    ingress:
    - from:
      - namespaceSelector: {}
    policyTypes:
      - Ingress

Since the flux-system namespace implements strict NetworkPolicies, we need to allow our ACME resolver to be reachable through ingress. Hence, we’ve configured an additional NetworkPolicy.

Note
For detailed instructions on automated DNS configuration refer to Kubernetes - Setup External DNS, and TLS management refer to Kubernetes - Setup Cert Manager for Automated TLS Management.

Modify cluster/default/flux-system/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - gotk-components.yaml
  - gotk-sync.yaml
  - webhook-receiver.yaml
+ - ingress.yaml
patches:
  - patch: |-
      apiVersion: kustomize.toolkit.fluxcd.io/v1
      kind: Kustomization
      metadata:
        name: flux-system
        namespace: flux-system
      spec:
        decryption:
          provider: sops
          secretRef:
            name: sops-gpg

Let’s push these changes and verify if the ingress is operational:

kubectl -n flux-system get ingress

NAME               CLASS   HOSTS                   ADDRESS                     PORTS     AGE
webhook-receiver   nginx   flux-wh.XXX   192.168.1.19,192.168.1.24   80, 443   69m

As we can see, the ingress is up and running. Now let’s check if our domain is reachable:

curl https://flux-wh.XXX

404 page not found

We can successfully reach our webhook endpoint. The 404 response is expected since we’re not hitting a specific webhook path.

4. Setup Github Webhooks

Now let’s use the secret token, endpoint, and webhook path we’ve configured to set up a webhook in GitHub:

5. Verify Setup

Let’s modify our sample app and increase the CPU limits in apps/sample-app/deployment.yaml:

apiVersion: apps/v1
metadata:
  name: sample-app
  namespace: default
kind: Deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      imagePullSecrets:
        - name: github-registry-secret
      containers:
        - name: sample-app
          image: ghcr.io/kiriapurv/k8s-sample-app:main-29e96d2a-1740565130 # {"$imagepolicy": "default:sample-app-main-policy"}
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
          resources:
            requests:
              memory: "128Mi"
              cpu: "50m"
            limits:
              memory: "128Mi"
-             cpu: "50m"
+             cpu: "150m"

Let’s push our changes and verify if the webhook is working:

The webhook has been delivered successfully, and reconciliation has been triggered immediately, demonstrating the push-based model in action.

6. Setup Image Repository

We can follow similar steps to set up a webhook for the image repository of our sample app. This ensures that the automated image updates we implemented in Kubernetes GitOps with FluxCD - Part 3 - Automated Image Updates can be instantly reconciled.

For the image repository, here’s the YAML configuration:

apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
  name: sample-app
  namespace: default
spec:
  type: github
  events:
    - "ping"
    - "package"
  secretRef:
    name: github-webhook-token
  resources:
    - kind: ImageRepository
      name: sample-app

This configuration listens for package events, which are triggered when new container images are published to the GitHub Container Registry.

What next ?

Future posts will explore advanced GitOps patterns with FluxCD, including:

  • Miscellaneous FluxCD Configurations

Stay tuned for each of these topics.

References