How to Secure Containers with Cosign and Distroless Images

Container technology and the term “container images” are not new for many developers, SREs and DevOps engineers. But the need to have secure container images for the production deployment is something which we really need these days. We have seen the recent software supply chain attack on a SolarWinds commercial application by inserting a backdoor. As customers download the Trojan Horse installation packages from SolarWinds application, attackers can access those systems which are having the SolarWinds products running. In this post, we will discuss cosign and distroless container images that can help achieve your application containers more securely deploying and running in production.

What are Distroless Container Images?

Distroless Container Images are “language focused Docker images, minus the operating system”. This means that it only contains your application and its runtime dependencies, not other usual OS package managers, Linux shells or any such programs which we usually would expect in a standard Linux distribution. Distroless base images have lesser packages than their counterparts. This reduces the attack surface as there are fewer components which can be vulnerable and hence increases security.

Source: https://github.com/GoogleContainerTools/distroless

Benefits of distroless container images:

• Improves container security.
• Reduces container size.
• Reduces compliance scope.
• Improves container distribution performances.
• Cost-efficient by utilizing lesser resources.

Distroless container images for our application along with Cosign verification

Cosign is a tool developed by Google in collaboration with the Linux Foundation Project called sigstore to simplify the signing and verification of the container images. Google announced that the distroless images are now signed by Cosign. So this means that all users of distroless can verify that they are indeed using the base image they intended to, before kicking off image builds, making distroless images even more trustworthy.

General Cosign workflow

Why do we need Cosign along with distroless container images?

The need for Cosign is because even with the distroless images there is a chance of facing some security threats such as typosquatting attacks, or receiving a malicious image. If the distroless build process is compromised, it makes users vulnerable to accidentally using the malicious image instead of the actual distroless image.

Typosquatting is a kind of social engineering attack where attackers publish malicious packages to a registry with the hope of tricking users into installing them with similar names of the package or registry/domain.

How to build the application container images with Cosign and distroless images?

Verify the distroless container base image with Cosign verification.

Verification of distroless image gcr.io/distroless/static with Cosign public key shared by publisher Google. This should be the first step of the image building process.

$ cosign verify -key publisher-shared-cosign-pub.key gcr.io/distroless/static

Verification for gcr.io/distroless/static --The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures    were verified against the specified public key - Any certificates were verified against the Fulcio roots.{"critical":{"identity":{"docker-    reference":"gcr.io/distroless/static"},"image":{"docker-manifest-digest":"sha256:c9320b754c2fa2cd2dea50993195f104a24f4c7ebe6e0297c6ddb40ce3679e7d"},"type":"cosign   container image signature"},"optional":null}

Build application container images with Cosign verified distroless base images

There are a couple of approaches for building application images as the distroless container images.

• Use respective build tools to build the application from the source code. After that, by using the Docker directives COPY or ADD to build the Docker images with the build artifacts copied or added.
• Multi-stage Docker build.

I am following multiple scenarios here, where I am building a Docker container image for my Hello-World Go web app, which you can find here with static distroless image and standard base images.

1. App Image built with gcr.io/distroless/static as base image in a multistage build process.

   FROM golang:1.15 as builder
   COPY . /usr/local
   WORKDIR /usr/local/
   RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -tags netgo -ldflags '-w -extldflags "-static"' -o cmd/app
   
   FROM gcr.io/distroless/static
   USER nonroot:nonroot
   COPY --from=builder  --chown=nonroot:nonroot /usr/local/cmd/app /bin/app
   ENTRYPOINT ["/bin/app"]
   

2. App Image built with golang:1.15 in a single stage build process.

   FROM golang:1.15 as builder
   COPY . /usr/local
   WORKDIR /usr/local/
   RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o /bin/app
   ENTRYPOINT ["/bin/app"]
   

3. App Image built with ubuntu:21.04 as base image in a multi stage build process.

   FROM golang:1.15 as builderstage
   COPY . /usr/local
   WORKDIR /usr/local/
   RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o cmd/app
   
   FROM ubuntu:21.04
   COPY --from=builderstage /usr/local/cmd/app /bin/app
   ENTRYPOINT ["/bin/app"]
   

You can find the details of Docker scan reports in the next topic which identifies the significance of using the distroless base images for your application.

Container image scanning: distroless container images vs standard container images

I have used the Docker native scanning tool called Synk, which is now available in the newer versions of Docker package. You can use this utility by using the docker command: “docker scan image_name”.

Image scan report

Note: You can find the Synk scan full report from the references.

Sign the distroless application container image built using Cosign.

Generate the cosign key pairs

$ cosign generate-key-pair

Enter password for private key:
Enter again:
Private key written to cosign.key
Public key written to cosign.pub

We need to tag the locally built container image with registry details and push to the remote registry. Sign the remote container image with the generated private key.

$ cosign sign -key cosign.key jeswinkninan/distroless:v1

Enter password for private key:
Pushing signature to: index.docker.io/jeswinkninan/distroless:sha256-41fd2ec0997d91c5df7c7d58d0a2433a5744119d79a803123541cdd2b0e93f08.sig

Verify the application container image with the Cosign public key generated

$ cosign verify -key cosign.pub jeswinkninan/distroless:v1

Verification for jeswinkninan/distroless:v1 --
The following checks were performed on each of these signatures:
- The cosign claims were validated
- The signatures were verified against the specified public key
- Any certificates were verified against the Fulcio roots.
{"critical":{"identity":{"docker-reference":"index.docker.io/jeswinkninan/distroless"},"image":{"docker-manifest-     digest":"sha256:41fd2ec0997d91c5df7c7d58d0a2433a5744119d79a803123541cdd2b0e93f08"},"type":"cosign container image signature"},"optional":null}

Key points

• Distroless are language focused Docker images, minus the operating system.
• Lesser the packages, less the chance of having vulnerabilities. Distroless images will be having only the necessary packages.
• Distroless images are now Cosigned.
• Verify every distroless image with the Cosign public key before each build.
• Container image scanning is a must have in your CICD build system.
• Cosign the internal application image built from base distroless image and verify before using it in production to ensure the image trust.

We hope you enjoyed reading this blog post. If you love hearing us from more, connect with us over Twitter or Linkedin

References

• Making the Internet more secure one signed container at a time
• What is typosquatting and how typosquatting attacks are responsible for malicious modules in npm
• Generate a keypair
• A Vulnerability in SolarWinds Serv-U Could Allow for Remote Code Execution
• Full Image Scan Report