Kubernetes GitOps with FluxCD - Part 5 - Implementing Discord Alerts

Table of Contents

In previous post, we explored how to setup helm chart automation with FluxCD, Building on our foundation of basic Flux CD setup, SOPS-based secret management, image update automation and Helm Chart Automation, this article focuses on implementing FluxCD Alerts with Discord integration to enhance GitOps observability stack.

Alerts provide real-time notifications about the state of our GitOps operations, helping us respond promptly to issues and stay informed about successful reconciliations. Let’s dive into configuring in our Kubernetes cluster.

1. Obtaining the Discord Webhook URL

First, we need to acquire a Webhook URL from Discord:

  1. Navigate to your Discord server settings
  2. Select “Integrations”
  3. Click on “Webhooks”
  4. Create a new webhook or use an existing one
  5. Copy the generated webhook URL

2. Securely Store the Webhook URL

Next, we’ll securely store the webhook URL as a Kubernetes secret by creating cluster/default/discord-webhook-secret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: discord-webhook-secret
  namespace: flux-system
type: Opaque
stringData:
  address: https://discordapp.com/api/webhooks/XXXXXX/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Following our established security practices, we’ll encrypt this sensitive information with SOPS and commit the changes to our Git repository as detailed in our SOPS-based secret management post.

3. Configuring Alert Resources

With our webhook secured, we can now configure the alert resources. Create cluster/default/discord-alert.yaml with the following content:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: discord
  namespace: flux-system
spec:
  type: discord
  secretRef:
    name: discord-webhook-secret
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: discord-info-alert
  namespace: flux-system
spec:
  eventMetadata:
    cluster: "default"
  providerRef:
    name: discord
  eventSeverity: info
  eventSources:
    - kind: GitRepository
      name: '*'
    - kind: Kustomization
      name: '*'
    - kind: HelmRelease
      name: '*'
    - kind: ImageUpdateAutomation
      name: '*'

This configuration creates two resources:

  • A Provider that specifies Discord as our notification destination
  • An Alert that defines which events should trigger notifications

As of the time of writing, FluxCD supports the following event sources (as referenced at https://pkg.go.dev/github.com/fluxcd/notification-controller/api/v1#CrossNamespaceObjectReference )

  • Bucket
  • GitRepository
  • Kustomization
  • HelmRelease
  • HelmChart
  • HelmRepository
  • ImageRepository
  • ImagePolicy
  • ImageUpdateAutomation
  • OCIRepository

4. Verifying the Configuration

After committing these changes to Git and allowing FluxCD to reconcile them, let’s verify that our alerts are properly configured:

flux get alerts

NAME                    SUSPENDED       READY   MESSAGE        
discord-info-alert      False           True    Alert is Ready

The READY status confirms that our alert configuration has been successfully applied.

5. Testing the Alert System

To validate that our alerts are functioning as expected, let’s trigger an event by removing a previously created resource. In this case, we’ll delete the nginx deployment we created during our initial FluxCD setup.

rm cluster/default/nginx.yaml

After committing and pushing this change, we should observe a notification in our Discord channel:

Success! We have received the notification, confirming that our alert system is operational and correctly integrated with Discord.

Best Practices for FluxCD Alerts

When implementing alerts in your GitOps workflow, consider these best practices:

  • Filter by Severity: Configure different alerts for various severity levels (info, error, warning) to avoid alert fatigue
  • Use Namespaces: Organize alerts by namespace to better manage notifications in larger clusters
  • Integration Options: Beyond Discord, consider integrating with other platforms like Slack, Microsoft Teams, or PagerDuty for comprehensive coverage
  • Alert Routing: Route critical alerts to urgent channels and informational alerts elsewhere

What next ?

In forthcoming articles, we’ll explore more advanced GitOps patterns with FluxCD, including:

  • Push based reconciliation triggers with Webhook receivers

Stay tuned for each of these topics.

References