How to choose the best container base image? What does “best” mean in this context? This blog post will not try to determine the best base image. We will pick just one of the aspects: security. We will have a look at how you can give your container base images a head start.
Do you want a secure image? Search no further, you will not find it. Nothing is secure.
The only truly secure system is one that is powered off, cast in a block of concrete and sealed in a lead-lined room with armed guards – and even then I have my doubts.
— Gene Spafford
Now that we know what to expect, let’s have a look at what we can do. As the above quote isn’t an option, we will use hardened base images.
There are already some articles about hardened base images. This post is a fresh-up and will also add a new aspect to the discussion.
Overview of container base images
You have a nearly endless list of images you can choose from. In no specific order, just a short list of some which I’m aware of:
- debian slim
- Red Hat Universal Base Image 8 Minimal
- scratch (Ok, this is not an image, but we need this later.)
So, what should you choose?
How to measure security for container base images
As we want to have a look at the security of some possible base images, how do we measure security?
For ease of discussion, we will only go with the CVE (Common Vulnerabilities and Exposures) count. That said, there are other considerations:
- Zero-day vulnerabilities
- A known CVE does not automatically mean you use the vulnerable code path.
- Is there a fix for the CVE?
- Does your scanner know about a CVE?
- Is a known vulnerability exploitable?
- What is the process for patch releases for the chosen image?
CVE Overview of base images
For this overview, we will stick to Trivy to scan the images. The results may vary depending on your scanner. I ordered them by the total number of CVEs:
|Image name||Scan results|
|debian:buster-slim||Total: 109 (UNKNOWN: 0, LOW: 80, MED: 10, HIGH: 17, CRIT: 2)|
|ubi8/ubi-minimal||Total: 105 (UNKNOWN: 0, LOW: 41, MED: 58, HIGH: 3, CRIT: 3)|
|ubuntu:20.04||Total: 36 (UNKNOWN: 0, LOW: 27, MED: 7, HIGH: 2, CRIT: 0)|
|ubi8/ubi-micro||Total: 25 (UNKNOWN: 0, LOW: 10, MED: 12, HIGH: 0, CRIT: 3)|
|distroless/base-debian10||Total: 25 (UNKNOWN: 0, LOW: 17, MED: 4, HIGH: 3, CRIT: 1)|
|alpine:3.14||Total: 1 (UNKNOWN: 1, LOW: 0, MED: 0, HIGH: 0, CRIT: 0)|
|distroless/static-debian10||Total: 0 (UNKNOWN: 0, LOW: 0, MED: 0, HIGH: 0, CRIT: 0)|
|scratch||Total: 0 (UNKNOWN: 0, LOW: 0, MED: 0, HIGH: 0, CRIT: 0)|
The above numbers are only a snapshot at the time of this writing. But they give an impression of what you can expect.
Hardened container base images
So, what about the distroless images? Why do some show CVEs for distro-related packages? The distroless images aren’t without any packages of a distribution. They only contain much fewer packages than the distribution image itself. Especially, packages that would allow further compromise or privilege escalation once someone compromised the container.
“Distroless” images contain only your application and its runtime dependencies. They do not contain package managers, shells or any other programs you would expect to find in a standard Linux distribution.
So perhaps, “lessdistro” would be a better match, but it isn’t that catchy of a name.
The best options would be to use scratch or static-debian10 images. Basically, nothing is in them, so no CVEs and no updates. But these images are only useable when you have a statically compiled binary to execute. It gets much more tricky when you try to run, for example, a Java program inside a scratch image. There are ways to do so, e.g. with GraalVM, but this doesn’t work in every case and isn’t a one-size-fits-all solution.
Alpine also has a very low CVE count. It will also work in most cases. It only gets problematic when you depend on libc, which it replaces with muslc. Which is still a problem to fix for the OpenJDK. Perhaps this changes with Java 16 and AdoptOpenJDK.
Although the scratch or static-debian10 image would be the best solution, this also is a compromise. Perhaps you have to use one of the other images to get your app running inside a container.
Security and hardened base images
This article states that containers do not contain.
And as mentioned above, there is more to consider than just CVEs. When running your containers, there are other security measures you should implement:
- Do not run containers as root.
- Limit the capabilities of the container.
- Apply seccomp and AppArmor or SELinux.
- Network Policies
So, hardened images are only one part of your security posture, but one which isn’t to neglect.
CVE count over time
In case you couldn’t opt for a scratch or distroless static image, the initial CVE count of your base image is just that, “initial”. As time goes by, your CVE count will go up because more vulnerabilities will be known. You have to keep your images up to date. As long as you do not regularly rebuild your images with an updated base image and deploy them, your CVE count in production will go up.
But what about third party software you use? These images mostly get built when the project releases a new version of the actual software. Let’s assume you use a well-matured open-source software that only releases a bug fix now and then. Between these releases, the project will not release an update to the image. So, if you want to keep your CVE count low, you have two options:
- Build the image regularly on your own.
- Support the project with the needed time/money/infrastructure, to deliver this service to the whole community.
I don’t think this is the project’s responsibility. In my opinion, the container is something like a software package for a distribution (think .dep or .rpm files). For distributions, such packages mostly have their own maintainers, which aren’t necessarily developers of the packaged software.
The base image you select plays its part in your container security measures. But just one part. Furthermore, the CVE count only shows a snapshot. To keep your CVE count on the selected base images low, you need a process to keep them up to date in production. Everything else doesn’t count.
And as always, it is a compromise between security, usability, and effort to get your software running inside a container. You could secure your container in a way as mentioned in the quote at the beginning, but that cannot be the solution.