Build container image with buildah, unprivileged
This "living" document is trying to document the different ways to build container images using buildah, in an unprivileged way. Please note it is also possible to build container images without using buildah and any container in container setup (document soon to be written and published).
Current status
Before starting to dig into solutions, let’s "asses" what is the default behavior in some OpenShift Pipelines versions (as this might change over time).
OpenShift Pipelines 1.8
In 1.8, the default service account (by default) is pipeline
SA and has a custom pipelines-scc
SCC (Security Context Constraints). This SCC is very similar to anyuid ; the two differences are the SETFCAP
allowed capability and the MustRunAs
on fsGroup
instead of RunAsAny
.
--- anyuid.yaml 2022-11-15 15:31:07.961911083 +0100
+++ pipelines-scc.yaml 2022-11-15 15:31:16.433975322 +0100
@@ -5,11 +5,12 @@
allowHostPorts: false
allowPrivilegeEscalation: true
allowPrivilegedContainer: false
-allowedCapabilities: null
+allowedCapabilities:
+- SETFCAP
apiVersion: security.openshift.io/v1
defaultAddCapabilities: null
fsGroup:
- type: RunAsAny
+ type: MustRunAs
groups:
- system:cluster-admins
kind: SecurityContextConstraints
This means that buildah
task in the cluster can run as root
— and to work they need to run as root
.
Run as the build
user
As documented on the buildah repository, the buildah
image comes with a specific user, build
(uid 1000
) that has everything setup to be able to run build inside the container. This is true for both the supported buildah image as well as the community one.
To be able to run this on OpenShift Pipelines, you need to be able to ask for a specific user in your Task
and, well, that’s pretty much it.
In 1.8, the pipeline
SA already allow to use a user id outside of the namespace range, but we can make sure this will work no matter what OpenShift Pipelines ships by defining our own SCC and ServiceAccount.
apiVersion: v1 (1)
kind: ServiceAccount
metadata:
name: pipelines-sa-userid-1000
---
kind: SecurityContextConstraints (2)
metadata:
annotations:
name: pipelines-scc-userid-1000
allowHostDirVolumePlugin: false
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegeEscalation: false
allowPrivilegedContainer: false
allowedCapabilities: null
apiVersion: security.openshift.io/v1
defaultAddCapabilities: null
fsGroup:
type: MustRunAs
groups:
- system:cluster-admins
priority: 10
readOnlyRootFilesystem: false
requiredDropCapabilities:
- MKNOD
runAsUser: (3)
type: MustRunAs
uid: 1000
seLinuxContext:
type: MustRunAs
supplementalGroups:
type: RunAsAny
users: []
volumes:
- configMap
- downwardAPI
- emptyDir
- persistentVolumeClaim
- projected
- secret
---
apiVersion: rbac.authorization.k8s.io/v1 (4)
kind: ClusterRole
metadata:
name: pipelines-scc-userid-1000-clusterrole
rules:
- apiGroups:
- security.openshift.io
resourceNames:
- pipelines-scc-userid-1000
resources:
- securitycontextconstraints
verbs:
- use
---
apiVersion: rbac.authorization.k8s.io/v1 (5)
kind: RoleBinding
metadata:
name: pipelines-scc-userid-1000-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: pipelines-scc-userid-1000-clusterrole
subjects:
- kind: ServiceAccount
name: pipelines-sa-userid-1000
1 | This is the service account we’ll use later on. |
2 | This is the custom SCC, based of restricted with only the runAsUser change |
3 | This is the most important part of the SCC. Here we are "enforcing" any Pod that will get this SCC attached (through the ServiceAccount) to run as userid 100 (and not anything else). |
4 | This is the ClusterRole the will use our SCC. |
5 | This binds our ClusterRole (that uses our SCC) to the ServiceAccount we created earlier. |
With this setup, any Pod that runs with the pipelines-sa-userid-1000
service account will be able to run as userid 1000
, and only that userid.
The next step is to define our buildah
Task to use the build (1000
userid) user. We are copying the ClusterTask
that OpenShift Pipelines ships and do small modifications. Ideally, this would also be shipped with OpenShift Pipelines, somehow.
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: buildah-as-user
# […]
spec:
description: >-
Buildah task builds source into a container image and
then pushes it to a container registry.
Buildah Task builds source into a container image using Project Atomic's
Buildah build tool.It uses Buildah's support for building from Dockerfiles,
using its buildah bud command.This command executes the directives in the
Dockerfile to assemble a container image, then pushes that image to a
container registry.
params:
- name: IMAGE
description: Reference of the image buildah will produce.
- name: BUILDER_IMAGE
description: The location of the buildah builder image.
default: registry.redhat.io/rhel8/buildah@sha256:99cae35f40c7ec050fed3765b2b27e0b8bbea2aa2da7c16408e2ca13c60ff8ee
- name: STORAGE_DRIVER
description: Set buildah storage driver
default: vfs
- name: DOCKERFILE
description: Path to the Dockerfile to build.
default: ./Dockerfile
- name: CONTEXT
description: Path to the directory to use as context.
default: .
- name: TLSVERIFY
description: Verify the TLS on the registry endpoint (for push/pull to a non-TLS registry)
default: "true"
- name: FORMAT
description: The format of the built container, oci or docker
default: "oci"
- name: BUILD_EXTRA_ARGS
description: Extra parameters passed for the build command when building images.
default: ""
- description: Extra parameters passed for the push command when pushing images.
name: PUSH_EXTRA_ARGS
type: string
default: ""
- description: Skip pushing the built image
name: SKIP_PUSH
type: string
default: "false"
results:
- description: Digest of the image just built.
name: IMAGE_DIGEST
type: string
workspaces:
- name: source
steps:
- name: build
securityContext:
runAsUser: 1000 (1)
image: $(params.BUILDER_IMAGE)
workingDir: $(workspaces.source.path)
script: |
echo "Running as USER ID `id`" (2)
buildah --storage-driver=$(params.STORAGE_DRIVER) bud \
$(params.BUILD_EXTRA_ARGS) --format=$(params.FORMAT) \
--tls-verify=$(params.TLSVERIFY) --no-cache \
-f $(params.DOCKERFILE) -t $(params.IMAGE) $(params.CONTEXT)
[[ "$(params.SKIP_PUSH)" == "true" ]] && echo "Push skipped" && exit 0
buildah --storage-driver=$(params.STORAGE_DRIVER) push \
$(params.PUSH_EXTRA_ARGS) --tls-verify=$(params.TLSVERIFY) \
--digestfile $(workspaces.source.path)/image-digest $(params.IMAGE) \
docker://$(params.IMAGE)
cat $(workspaces.source.path)/image-digest | tee /tekton/results/IMAGE_DIGEST
volumeMounts:
- name: varlibcontainers
mountPath: /home/build/.local/share/containers
volumeMounts:
- name: varlibcontainers
mountPath: /home/build/.local/share/containers
volumes:
- name: varlibcontainers
emptyDir: {}
1 | This is where we explicitly ask to run the container as the user id 1000 which correspond to the build user in the buildah image. |
2 | We print the the user id, just to showcase we are running the process as user id 1000 . |
Now, we can start a TaskRun
or integrate it with a PipelineRun
.
TaskRun
apiVersion: v1
data:
Dockerfile: |
ARG BASE_IMG=registry.access.redhat.com/ubi8/ubi
FROM $BASE_IMG AS buildah-runner
RUN dnf -y update && \
dnf -y install git && \
dnf clean all
CMD git
kind: ConfigMap
metadata:
name: dockerfile (1)
---
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
name: buildah-as-user-1000
spec:
serviceAccountName: pipelines-sa-userid-1000
params:
- name: IMAGE
value: image-registry.openshift-image-registry.svc:5000/test/buildahuser
taskRef:
kind: Task
name: buildah-as-user
workspaces:
- configMap:
name: dockerfile (2)
name: source
1 | In this example, we only want to run a TaskRun , so we won’t have any prior task that fetches some sources with a Dockerfile , so we will use a configmap instead. |
2 | Thanks to the workspace , we can mount a configmap as the source workspace for our buildah-as-user Task . |
PipelineRun
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
name: pipeline-buildah-as-user-1000
spec:
params:
- name: IMAGE
- name: URL
workspaces:
- name: shared-workspace
- name: sslcertdir
optional: true
tasks:
- name: fetch-repository (1)
taskRef:
name: git-clone
kind: ClusterTask
workspaces:
- name: output
workspace: shared-workspace
params:
- name: url
value: $(params.URL)
- name: subdirectory
value: ""
- name: deleteExisting
value: "true"
- name: buildah
taskRef:
name: buildah-as-user (2)
runAfter:
- fetch-repository
workspaces:
- name: source
workspace: shared-workspace
- name: sslcertdir
workspace: sslcertdir
params:
- name: IMAGE
value: $(params.IMAGE)
---
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
name: pipelinerun-buildah-as-user-1000
spec:
serviceAccountName: pipelines-sa-userid-1000
params:
- name: URL
value: https://github.com/openshift/pipelines-vote-api
- name: IMAGE
value: image-registry.openshift-image-registry.svc:5000/test/buildahuser
taskRef:
kind: Pipeline
name: pipeline-buildah-as-user-1000
workspaces:
- name: shared-workspace (3)
volumeClaimTemplate:
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Mi
1 | In this example, we will use the git-clone ClusterTask to fetch the source containing a Dockerfile and then use that new buildah task to build it. |
2 | We are refering to our modified buildah Task |
3 | We are using a PVC, automatically created by the controller, to share data between the git-clone Task and our modified buildah task |
Known issues
This approach works relatively well with most Dockerfile
. However, there is some cases where a build will fail:
- Using the --mount=type=cache
will likely fail due to permissions issues, see here
- Using the --mount=type=secret
is bound to fail as well as it will try to mount something, and this requires additionnal capabilities that are not provided by our SCC (and that are closer to privileged capabilities).