Alert

The GitOps Toolkit Custom Resource Definitions documentation.

The Alert API defines how events are filtered by severity and involved object, and what provider to use for dispatching.

Specification

Spec:

type AlertSpec struct {
	// Send events using this provider.
	// +required
	ProviderRef meta.LocalObjectReference `json:"providerRef"`

	// Filter events based on severity, defaults to ('info').
	// +kubebuilder:validation:Enum=info;error
	// +optional
	EventSeverity string `json:"eventSeverity,omitempty"`

	// Filter events based on the involved objects.
	// +required
	EventSources []CrossNamespaceObjectReference `json:"eventSources"`

	// A list of Golang regular expressions to be used for excluding messages.
	// +optional
	ExclusionList []string `json:"exclusionList,omitempty"`

	// Short description of the impact and affected cluster.
	// +optional
	Summary string `json:"summary,omitempty"`

	// This flag tells the controller to suspend subsequent events dispatching.
	// Defaults to false.
	// +optional
	Suspend bool `json:"suspend,omitempty"`
}

Status:

// ProviderStatus defines the observed state of Provider
type ProviderStatus struct {
	// +optional
	Conditions []Condition `json:"conditions,omitempty"`
}

Status condition types:

const (
	// ReadyCondition represents the fact that a given object has passed
	// validation and was acknowledge by the controller.
	ReadyCondition string = "Ready"
)

Example

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Alert
metadata:
  name: webapp
  namespace: default
spec:
  providerRef: 
    name: on-call-slack
  eventSeverity: info
  eventSources:
    - kind: GitRepository
      name: webapp
    - kind: Bucket
      name: secrets
    - kind: Kustomization
      name: webapp-backend
    - kind: Kustomization
      name: webapp-frontend

The event severity can be set to info or error.

To target all resources of a particular kind in a namespace, you can use the * wildcard:

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Alert
metadata:
  name: all-kustomizations
  namespace: default
spec:
  providerRef: 
    name: dev-msteams
  eventSeverity: error
  eventSources:
    - kind: Kustomization
      namespace: default
      name: '*'
  suspend: false

You can target resources with a particular set of labels using spec.EventSource[].matchLabels. It is only possible when the name of the event source is set to a wildcard *.

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Alert
metadata:
  name: dev-kustomizations
  namespace: default
spec:
  providerRef:
    name: dev-msteams
  eventSeverity: error
  eventSources:
    - kind: Kustomization
      namespace: default
      name: '*'
      matchLabels:
        app: dev
  suspend: false

If you don’t specify an event source namespace, the alert namespace will be used.

Note that on multi-tenant clusters, platform admins can disable cross-namespace references with the --no-cross-namespace-refs=true flag. When this flag is set, alerts can only refer to event sources in the same namespace as the alert object, preventing tenants from subscribing to another tenant’s events.

You can add a summary to describe the impact of an event:

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Alert
metadata:
  name: ingress
  namespace: nginx
spec:
  summary: "Ingress traffic affected in production (us-west-2)"
  providerRef: 
    name: on-call-slack
  eventSeverity: error
  eventSources:
    - kind: HelmRelease
      name: nginx-ingress

Skip alerting if the message matches a Go regex from the exclusion list:

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Alert
metadata:
  name: flux-system
  namespace: flux-system
spec:
  providerRef: 
    name: on-call-slack
  eventSeverity: error
  eventSources:
    - kind: GitRepository
      name: flux-system
  exclusionList:
    - "waiting.*socket"

The above definition will not send alerts for transient Git clone errors like:

unable to clone 'ssh://git@ssh.dev.azure.com/v3/...', error: SSH could not read data: Error waiting on socket