Build container image with buildah, unprivileged

This "living" document is trying to document the different ways to build container images using buildah, in an unprivileged way. Please note it is also possible to build container images without using buildah and any container in container setup (document soon to be written and published).

Current status

Before starting to dig into solutions, let’s "asses" what is the default behavior in some OpenShift Pipelines versions (as this might change over time).

OpenShift Pipelines 1.8

In 1.8, the default service account (by default) is pipeline SA and has a custom pipelines-scc SCC (Security Context Constraints). This SCC is very similar to anyuid ; the two differences are the SETFCAP allowed capability and the MustRunAs on fsGroup instead of RunAsAny.

--- anyuid.yaml 2022-11-15 15:31:07.961911083 +0100
+++ pipelines-scc.yaml  2022-11-15 15:31:16.433975322 +0100
@@ -5,11 +5,12 @@
 allowHostPorts: false
 allowPrivilegeEscalation: true
 allowPrivilegedContainer: false
-allowedCapabilities: null
+allowedCapabilities:
+- SETFCAP
 apiVersion: security.openshift.io/v1
 defaultAddCapabilities: null
 fsGroup:
-  type: RunAsAny
+  type: MustRunAs
 groups:
 - system:cluster-admins
 kind: SecurityContextConstraints

This means that buildah task in the cluster can run as root — and to work they need to run as root.

Run as the build user

As documented on the buildah repository, the buildah image comes with a specific user, build (uid 1000) that has everything setup to be able to run build inside the container. This is true for both the supported buildah image as well as the community one.

To be able to run this on OpenShift Pipelines, you need to be able to ask for a specific user in your Task and, well, that’s pretty much it.

In 1.8, the pipeline SA already allow to use a user id outside of the namespace range, but we can make sure this will work no matter what OpenShift Pipelines ships by defining our own SCC and ServiceAccount.

apiVersion: v1 (1)
kind: ServiceAccount
metadata:
  name: pipelines-sa-userid-1000
---
kind: SecurityContextConstraints (2)
metadata:
  annotations:
  name: pipelines-scc-userid-1000
allowHostDirVolumePlugin: false
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegeEscalation: false
allowPrivilegedContainer: false
allowedCapabilities: null
apiVersion: security.openshift.io/v1
defaultAddCapabilities: null
fsGroup:
  type: MustRunAs
groups:
- system:cluster-admins
priority: 10
readOnlyRootFilesystem: false
requiredDropCapabilities:
- MKNOD
runAsUser: (3)
  type: MustRunAs
  uid: 1000
seLinuxContext:
  type: MustRunAs
supplementalGroups:
  type: RunAsAny
users: []
volumes:
- configMap
- downwardAPI
- emptyDir
- persistentVolumeClaim
- projected
- secret
---
apiVersion: rbac.authorization.k8s.io/v1 (4)
kind: ClusterRole
metadata:
  name: pipelines-scc-userid-1000-clusterrole
rules:
- apiGroups:
  - security.openshift.io
  resourceNames:
  - pipelines-scc-userid-1000
  resources:
  - securitycontextconstraints
  verbs:
  - use
---
apiVersion: rbac.authorization.k8s.io/v1 (5)
kind: RoleBinding
metadata:
  name: pipelines-scc-userid-1000-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: pipelines-scc-userid-1000-clusterrole
subjects:
- kind: ServiceAccount
  name: pipelines-sa-userid-1000
1 This is the service account we’ll use later on.
2 This is the custom SCC, based of restricted with only the runAsUser change
3 This is the most important part of the SCC. Here we are "enforcing" any Pod that will get this SCC attached (through the ServiceAccount) to run as userid 100 (and not anything else).
4 This is the ClusterRole the will use our SCC.
5 This binds our ClusterRole (that uses our SCC) to the ServiceAccount we created earlier.

With this setup, any Pod that runs with the pipelines-sa-userid-1000 service account will be able to run as userid 1000, and only that userid.

The next step is to define our buildah Task to use the build (1000 userid) user. We are copying the ClusterTask that OpenShift Pipelines ships and do small modifications. Ideally, this would also be shipped with OpenShift Pipelines, somehow.

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: buildah-as-user
  # […]
spec:
  description: >-
    Buildah task builds source into a container image and
    then pushes it to a container registry.
    Buildah Task builds source into a container image using Project Atomic's
    Buildah build tool.It uses Buildah's support for building from Dockerfiles,
    using its buildah bud command.This command executes the directives in the
    Dockerfile to assemble a container image, then pushes that image to a
    container registry.
  params:
  - name: IMAGE
    description: Reference of the image buildah will produce.
  - name: BUILDER_IMAGE
    description: The location of the buildah builder image.
    default: registry.redhat.io/rhel8/buildah@sha256:99cae35f40c7ec050fed3765b2b27e0b8bbea2aa2da7c16408e2ca13c60ff8ee
  - name: STORAGE_DRIVER
    description: Set buildah storage driver
    default: vfs
  - name: DOCKERFILE
    description: Path to the Dockerfile to build.
    default: ./Dockerfile
  - name: CONTEXT
    description: Path to the directory to use as context.
    default: .
  - name: TLSVERIFY
    description: Verify the TLS on the registry endpoint (for push/pull to a non-TLS registry)
    default: "true"
  - name: FORMAT
    description: The format of the built container, oci or docker
    default: "oci"
  - name: BUILD_EXTRA_ARGS
    description: Extra parameters passed for the build command when building images.
    default: ""
  - description: Extra parameters passed for the push command when pushing images.
    name: PUSH_EXTRA_ARGS
    type: string
    default: ""
  - description: Skip pushing the built image
    name: SKIP_PUSH
    type: string
    default: "false"
  results:
  - description: Digest of the image just built.
    name: IMAGE_DIGEST
    type: string
  workspaces:
  - name: source
  steps:
  - name: build
    securityContext:
      runAsUser: 1000 (1)
    image: $(params.BUILDER_IMAGE)
    workingDir: $(workspaces.source.path)
    script: |
      echo "Running as USER ID `id`" (2)
      buildah --storage-driver=$(params.STORAGE_DRIVER) bud \
        $(params.BUILD_EXTRA_ARGS) --format=$(params.FORMAT) \
        --tls-verify=$(params.TLSVERIFY) --no-cache \
        -f $(params.DOCKERFILE) -t $(params.IMAGE) $(params.CONTEXT)
      [[ "$(params.SKIP_PUSH)" == "true" ]] && echo "Push skipped" && exit 0
      buildah --storage-driver=$(params.STORAGE_DRIVER) push \
        $(params.PUSH_EXTRA_ARGS) --tls-verify=$(params.TLSVERIFY) \
        --digestfile $(workspaces.source.path)/image-digest $(params.IMAGE) \
        docker://$(params.IMAGE)
      cat $(workspaces.source.path)/image-digest | tee /tekton/results/IMAGE_DIGEST
    volumeMounts:
    - name: varlibcontainers
      mountPath: /home/build/.local/share/containers
    volumeMounts:
    - name: varlibcontainers
      mountPath: /home/build/.local/share/containers
  volumes:
  - name: varlibcontainers
    emptyDir: {}
1 This is where we explicitly ask to run the container as the user id 1000 which correspond to the build user in the buildah image.
2 We print the the user id, just to showcase we are running the process as user id 1000.

Now, we can start a TaskRun or integrate it with a PipelineRun.

TaskRun

apiVersion: v1
data:
  Dockerfile: |
    ARG BASE_IMG=registry.access.redhat.com/ubi8/ubi
    FROM $BASE_IMG AS buildah-runner
    RUN dnf -y update && \
        dnf -y install git && \
        dnf clean all
    CMD git
kind: ConfigMap
metadata:
  name: dockerfile (1)
---
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: buildah-as-user-1000
spec:
  serviceAccountName: pipelines-sa-userid-1000
  params:
  - name: IMAGE
    value: image-registry.openshift-image-registry.svc:5000/test/buildahuser
  taskRef:
    kind: Task
    name: buildah-as-user
  workspaces:
  - configMap:
      name: dockerfile (2)
    name: source
1 In this example, we only want to run a TaskRun, so we won’t have any prior task that fetches some sources with a Dockerfile, so we will use a configmap instead.
2 Thanks to the workspace, we can mount a configmap as the source workspace for our buildah-as-user Task.

PipelineRun

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: pipeline-buildah-as-user-1000
spec:
  params:
  - name: IMAGE
  - name: URL
  workspaces:
  - name: shared-workspace
  - name: sslcertdir
    optional: true
  tasks:
  - name: fetch-repository (1)
    taskRef:
      name: git-clone
      kind: ClusterTask
    workspaces:
    - name: output
      workspace: shared-workspace
    params:
    - name: url
      value: $(params.URL)
    - name: subdirectory
      value: ""
    - name: deleteExisting
      value: "true"
  - name: buildah
    taskRef:
      name: buildah-as-user (2)
    runAfter:
    - fetch-repository
    workspaces:
    - name: source
      workspace: shared-workspace
    - name: sslcertdir
      workspace: sslcertdir
    params:
    - name: IMAGE
      value: $(params.IMAGE)
---
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: pipelinerun-buildah-as-user-1000
spec:
  serviceAccountName: pipelines-sa-userid-1000
  params:
  - name: URL
    value: https://github.com/openshift/pipelines-vote-api
  - name: IMAGE
    value: image-registry.openshift-image-registry.svc:5000/test/buildahuser
  taskRef:
    kind: Pipeline
    name: pipeline-buildah-as-user-1000
  workspaces:
  - name: shared-workspace (3)
    volumeClaimTemplate:
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 100Mi
1 In this example, we will use the git-clone ClusterTask to fetch the source containing a Dockerfile and then use that new buildah task to build it.
2 We are refering to our modified buildah Task
3 We are using a PVC, automatically created by the controller, to share data between the git-clone Task and our modified buildah task

Known issues

This approach works relatively well with most Dockerfile. However, there is some cases where a build will fail: - Using the --mount=type=cache will likely fail due to permissions issues, see here - Using the --mount=type=secret is bound to fail as well as it will try to mount something, and this requires additionnal capabilities that are not provided by our SCC (and that are closer to privileged capabilities).