OCI cheatsheet

How to build, publish and consume OCI Artifacts with Flux.

How Flux OCI works?

With Flux, you can distribute and reconcile Kubernetes configuration packaged as OCI artifacts. Instead of connecting Flux to a Git repository where the application desired state is defined, you can connect Flux to a container registry where you’ll push the application deploy manifests, right next to the application container images.

OCI vs Git

Using OCI instead of Git is particularity useful when the Git repository doesn’t contain the final Kubernetes manifests. If you are using cuelang, jsonnet or any other tool that generates Kubernetes resources in YAML format, you can run the generators in CI and publish the resulting manifests as OCI artifacts for Flux to consume.

Authoring artifacts

On the client-side, the Flux CLI offers commands for packaging Kubernetes configs into OCI artifacts and pushing these artifact to container registries.

The Flux CLI commands for managing OCI artifacts are:

flux push artifact
flux pull artifact
flux tag artifact
flux list artifacts

OCI Artifact content

While this guide refers to Kubernetes YAMLs as the manifests which are bundled in an OCI artifact, any type of configuration can be published with the Flux CLI. For example the tf-controller (A Flux controller created by Weaveworks) expects for Terraform files to be included in the OCI artifact.

Consuming artifacts

On the server-side, Flux pulls OCI artifacts from container registries, extracts the Kubernetes manifests and reconciles them on the cluster.

With OCIRepository you tell Flux which artifacts to pull from a container registry and how to authenticate in order to download and make available the artifacts’ content inside the cluster.

The OCIRepository source can be used in the same way as a GitRepository. You can be notified when new artifacts are pulled (Flux Alert), you can trigger a pull with webhooks (Flux Receiver), and you can apply the OCI artifact content on the cluster (Flux Kustomization).

Example:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 10m
  url: oci://ghcr.io/stefanprodan/manifests/podinfo
  ref:
    tag: latest
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 10m
  targetNamespace: default
  sourceRef:
    kind: OCIRepository
    name: podinfo
  path: ./

Helm OCI

For Helm users, Flux comes with support for defining Helm releases with charts stored in container registries.

Example:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 10m
  type: oci
  url: oci://ghcr.io/stefanprodan/charts
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 10m
  targetNamespace: default
  chart:
    spec:
      chart: podinfo
      sourceRef:
        kind: HelmRepository
        name: podinfo

Unlike Helm, the Flux OCI artifacts don’t have a custom media type and they can be stored in any container registry. The artifacts created by Flux can contain any type of configuration besides Kubernetes manifests.

The two artifact types can be used together, for example you could bundle a Namespace, a HelmRepository and a HelmRelease into a Flux OCI artifact where the HelmRepository points to a container registry where the chart is.

Workflow examples

Following are examples for deploying a demo application to Kubernetes using manifests stored in Git.

flowchart LR

A((User)) --> B(Git Repository)
B --> C((CI Job))
C --> D[Container Registry]
D --> E((Flux))
E --> F[Staging]
E --> G[Production]

We’ll use the main branch to deploy to staging and Git tags for deploying on production. When pushing changes to the main branch, a CI job generates the Kubernetes manifests for staging and pushes them to a container registry using the Flux CLI. Flux running on the staging cluster detects the new artifact digest, pulls the manifests and applies them.

When cutting a release from the main branch by tagging a commit with a semver version, the CI job generates the manifests for production and pushes them using the Git tag as the OCI artifact tag. The Flux controllers running on the production cluster detects the new semver tag, pulls the manifests and applies them.

GitHub Actions

If your are using GitHub for CI, please see the these examples on how to automate the publishing of OCI Artifacts in your workflows with the Flux GitHub Action.

Deploy latest on staging

Push the latest changes from Git to the container registry:

git clone https://github.com/stefanaprodan/podinfo.git && cd podinfo

flux push artifact oci://ghcr.io/stefanprodan/manifests/podinfo:$(git rev-parse --short HEAD) \
	--path="./kustomize" \
	--source="$(git config --get remote.origin.url)" \
	--revision="$(git branch --show-current)/$(git rev-parse HEAD)"

The output is similar to:

► pushing artifact to ghcr.io/stefanprodan/manifests/podinfo:b3b00fe
✔ artifact successfully pushed to ghcr.io/stefanprodan/manifests/podinfo@sha256:4f90664660b3a567287e6957fa0481f347541b5908f6f797ec665255a399aed6

Tag the current commit SHA as latest:

$ flux tag artifact oci://ghcr.io/stefanprodan/manifests/podinfo:$(git rev-parse --short HEAD) \
  --tag latest

The output is similar to:

► tagging artifact
✔ artifact tagged as ghcr.io/stefanprodan/manifests/podinfo:latest

Pull the latest build on the staging cluster:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 5m
  url: oci://ghcr.io/stefanprodan/manifests/podinfo
  ref:
    tag: latest

Reconcile the latest manifests from the OCI artifact:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: podinfo
  namespace: flux-system
spec:
  sourceRef:
    kind: OCIRepository
    name: podinfo
  interval: 60m
  retryInterval: 5m
  path: ./
  prune: true
  wait: true
  timeout: 2m
  targetNamespace: default
  patches:
    - patch: |
        - op: add
          path: /metadata/annotations/env
          value: staging        
      target:
        name: podinfo

Deploy stable on production

Push the latest release from Git to the container registry:

git checkout 6.1.0

flux push artifact oci://ghcr.io/stefanprodan/manifests/podinfo:$(git tag --points-at HEAD) \
	--path="./kustomize" \
	--source="$(git config --get remote.origin.url)" \
	--revision="$(git tag --points-at HEAD)/$(git rev-parse HEAD)"

Tag the release as stable:

flux tag artifact oci://ghcr.io/stefanprodan/manifests/podinfo:$(git tag --points-at HEAD) \
  --tag stable

Deploy the latest stable build on the production cluster:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 5m
  url: oci://ghcr.io/stefanprodan/manifests/podinfo
  ref:
    tag: stable

Or deploy the latest version by semver:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 5m
  url: oci://ghcr.io/stefanprodan/manifests/podinfo
  ref:
    semver: ">=1.0.0"

Authentication

Flux works with Docker Hub, GitHub and GitLab Container Registry, ACR, ECR, GCR, Artifactory, Harbor, self-hosted Docker Registry and any other registry which is compatible with the OCI Distribution Specification.

For authentication purposes, the flux <verb> artifact commands are using the ~/.docker/config.json config file and the Docker credential helpers.

echo ${GITHUB_PAT} | docker login ghcr.io -u ${GITHUB_USER} --password-stdin

To pull artifacts in Kubernetes clusters, Flux can authenticate to container registries using image pull secrets or IAM role bindings to the source-controller service account.

Generate an image pull secret for GitHub Container Registry example:

flux create secret oci ghcr-auth \
  --url=ghcr.io \
  --username=flux \
  --password=${GITHUB_PAT}

Then reference the secret in the OCIRepository with:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 5m
  url: oci://ghcr.io/stefanprodan/manifests/podinfo
  provider: generic
  secretRef:
    name: ghcr-auth

When running Flux on managed Kubernetes clusters like EKS, AKS or GKE, you can set the provider field to azure, aws or gcp and Flux will use the Kubernetes node credentials to pull artifacts without needing an image pull secret.

For more details on how to setup authentication for Azure, AWS and Google Cloud please see the documentation.

Monitoring

Configure alerting for when new artifacts are pulled and reconciled:

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Alert
metadata:
  name: podinfo
  namespace: flux-system
spec:
  summary: "podinfo in production"
  providerRef:
    name: slack
  eventSeverity: info
  eventSources:
    - kind: OCIRepository
      name: 'podinfo'
    - kind: Kustomization
      name: 'podinfo'

Or watch the Kubernetes events:

$ kubectl alpha events -n flux-system --for OCIRepository/podinfo --watch

stored artifact with digest '04db795c5e8f039ee06e7f388e90ef9d16b713506dc100faed1773e0f8410d07'
from 'oci://ghcr.io/stefanprodan/manifests/podinfo',
origin source 'https://github.com/stefanprodan/podinfo.git',
origin revision '6.1.5/6b869d1a184969f7e24e6e4ad30be0b18f8b7416'

Tracing

When publishing artifacts with flux push it is important to specify the Git source and revision with:

flux push artifact oci://<repo url> --path=<manifests dir> \
	--source="$(git config --get remote.origin.url)" \
	--revision="$(git branch --show-current)/$(git rev-parse HEAD)"

The Git source and the revision are stored in the container registry as annotations in the OCI artifact manifest, and they are reflected in-cluster under the status.artifact.metadata of the OCIRepository object.

You can determine the Git origin of an artifact pulled by an OCIRepository by looking up its status:

$ kubectl -n flux-system describe ocirepository podinfo

Status:
  Artifact:
    Revision:          04db795c5e8f039ee06e7f388e90ef9d16b713506dc100faed1773e0f8410d07
    Last Update Time:  2022-08-09T18:29:11Z
    Metadata:
      org.opencontainers.image.created:   2022-08-08T12:31:05+03:00
      org.opencontainers.image.revision:  6.1.5/6b869d1a184969f7e24e6e4ad30be0b18f8b7416
      org.opencontainers.image.source:    https://github.com/stefanprodan/podinfo.git                           
  Conditions:
    Last Transition Time:     2022-08-09T18:29:12Z
    Message:                  stored artifact for digest '04db795c5e8f039ee06e7f388e90ef9d16b713506dc100faed1773e0f8410d07'
    Observed Generation:      9
    Reason:                   Succeeded
    Status:                   True
    Type:                     Ready

For any Kubernetes resources managed by Flux, you can trace it back to its source with flux trace:

$ flux -n default trace deployment podinfo

Object:          Deployment/podinfo
Namespace:       default
Status:          Managed by Flux
---
Kustomization:   podinfo
Namespace:       flux-system
Target:          default
Path:            ./kustomize
Revision:        dbdb109711ffb3be77504d2670dbe13c24dd63d8d7f1fb489d350e5bfe930dd3
Status:          Last reconciled at 2022-08-10 14:40:28 +0200 CEST
Message:         Applied revision: dbdb109711ffb3be77504d2670dbe13c24dd63d8d7f1fb489d350e5bfe930dd3
---
OCIRepository:   podinfo
Namespace:       flux-system
URL:             oci://ghcr.io/stefanprodan/manifests/podinfo
Digest:          sha256:dbdb109711ffb3be77504d2670dbe13c24dd63d8d7f1fb489d350e5bfe930dd3
Revision:        dbdb109711ffb3be77504d2670dbe13c24dd63d8d7f1fb489d350e5bfe930dd3
Origin Revision: 6.1.6/450796ddb2ab6724ee1cc32a4be56da032d1cca0
Origin Source:   https://github.com/stefanprodan/podinfo.git
Status:          Last reconciled at 2022-08-10 14:40:22 +0200 CEST
Message:         stored artifact for digest 'dbdb109711ffb3be77504d2670dbe13c24dd63d8d7f1fb489d350e5bfe930dd3'

Automated updates to Git

If you wish to track the OCI artifacts revisions in Git, you can use Flux image automation to patch the artifacts tags in the YAML manifests stored in the Git repository used at bootstrap.

First we’ll configure Flux to clone the bootstrap repository and push commits to the main branch:

apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: flux-system
  git:
    push:
      branch: main
    commit:
      author:
        email: fluxcdbot@users.noreply.github.com
        name: fluxcdbot
      messageTemplate: '{{range .Updated.Images}}{{println .}}{{end}}'
  update:
    path: ./clusters/my-cluster
    strategy: Setters

You can also configure Flux to push the change to a new branch and open a Pull Request, for more details please see the image automation guide and GitHub Actions Auto Pull Request.

Flux OCI automation

Define an image repository and a semver policy for the OCI artifact:

apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageRepository
metadata:
  name: podinfo-oci
  namespace: flux-system
spec:
  image: ghcr.io/stefanprodan/manifests/podinfo
  interval: 5m
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImagePolicy
metadata:
  name: podinfo-oci
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: podinfo-oci
  policy:
    semver:
      range: 6.1.x

Then add the policy marker to the OCIRepository manifest in Git:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 5m
  url: oci://ghcr.io/stefanprodan/manifests/podinfo
  ref:
    tag: 6.1.0 # {"$imagepolicy": "flux-system:podinfo-oci:tag"}

Based on the above configuration, Flux will scan the container registry every five minutes, and when it finds a newer version, it will update the OCIRepository.spec.ref.tag and will push the change to Git.

Helm OCI automation

Define an image repository and a semver policy for the Helm chart:

apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageRepository
metadata:
  name: podinfo-chart
  namespace: flux-system
spec:
  image: ghcr.io/stefanprodan/charts/podinfo
  interval: 5m
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImagePolicy
metadata:
  name: podinfo-chart
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: podinfo-chart
  policy:
    semver:
      range: 6.1.x

Then add the policy marker to the HelmRelease manifest in Git:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 10m
  targetNamespace: default
  chart:
    spec:
      chart: podinfo
      version: 6.1.0 # {"$imagepolicy": "flux-system:podinfo-chart:tag"}
      sourceRef:
        kind: HelmRepository
        name: podinfo

Based on the above configuration, Flux will scan the container registry every five minutes, and when it finds a newer Helm chart version, it will update the HelmRelease.spec.chart.spec.chart.version and will push the change to Git.

Last modified 2022-08-11: Add Automated updates to Git to OCI docs (261230a)