The Three Pillars For Container Security

When you are surfing the web, you can find countless approaches on how to secure containers. There are plenty best practices available, but this abundance of information can often be misleading and confusing.

This dedicated guide aims to cut through the noise and present a focused perspective on what I believe are the three most critical pillars of container security. These pillars are essential for anyone in the security field to understand and implement when securing containers. This guide assumes that you already have a basic understanding of what Docker and containers are.

So, let’s get started.

TL;DR of the Three Pillars

💡 Securing Docker Images, Securing Kubernetes, Runtime Security.

1️⃣ Securing Docker Images

Securing your Dockerfile is the first process you should follow.

Some Basics: Understanding the Docker Build Process

A Docker build process involves creating a Docker image from a Dockerfile. A Dockerfile is essentially a blueprint containing instructions that define the environment and the steps required to build the application.

During the build process, Docker reads these instructions, executes them, and packages the results into an image. This image can then be run as a container, ensuring a consistent and reproducible environment for applications.

💡 A great explanation of the build process can be found here: https://iximiuz.com/en/posts/you-need-containers-to-build-an-image/

Dockerfile Security

From what I have seen so far, the three most important aspects of securing Dockerfiles are:

Pulling a secure base image.
Securing additional dependencies and packages.
Ensuring containers do not run as root.

Base Image

The first layer in a Dockerfile is the base image. This is the foundation upon which all other layers are built. The base image typically comes from an external registry.

It is important to check the current vulnerability state of an image in the external registry. For instance, Docker Hub provides information on the existing vulnerabilities in an image.

In the following example, you can see an node:10.0.0 image sourced from Docker Hub. The image shows a significant number of vulnerabilities and was last pushed six years ago.
This indicates that the image is outdated and vulnerable. Therefore, it is advisable to upgrade to a newer version or change the image to a different tag to ensure better security.
Node

Securing additional dependencies and packages.

Since a Dockerfile is built using layers and starting with a base image, developers will often need to add dependencies and packages on top of that. These dependencies and packages are crucial for the functionality of the application.

It is essential to ensure that all dependencies and packages are up-to-date and secure.

💡 For more information on securing dependencies and packages, feel free to read my previous blog: https://benbenhemo.com/post/securing-dependencies/

Ensuring containers do not run as root

By default, containers run as root. Since containers might share the same host, a container running as root will have access to the root of the host node. This means it can potentially access other containers on the same node, posing significant security risks.

To mitigate these risks, ensure that your Dockerfile is configured to run containers as a non-root user.

# Use an official base image
FROM python:3.9-slim

# Create a non-root user and set permissions
RUN useradd -m myuser

# Set the user to the newly created non-root user
USER myuser

Container Scanning in CI/CD

Implementing and configuring security scanners as part of the CI/CD pipeline is a topic for another day, but since it is a major component of securing Docker images, it should be mentioned in this blog.

Container scanning in CI/CD pipelines ensures that any vulnerabilities in your Docker images are detected early in the development cycle. This approach helps maintain the security and integrity of your applications by detecting and addressing potential vulnerabilities before they reach production.

The Trivy documentation is a great resource and can help you gain a more in-depth understanding of the concept: Trivy Documentation

2️⃣ Securing Kubernetes

once your Dockerfile is secure, the next important layer is securing your Kubernetes cluster.

Kubernetes, an open-source project, is an important tool for deploying, scaling, and managing containerized applications. Its flexibility and power make it a key component in modern infrastructure, but this also means that securing Kubernetes effectively is essential to protect your applications and data from potential threats.

https://kubernetes.io/docs/home/

https://anthonyspiteri.net/wp-content/uploads/2019/07/k8severywhere.jpg

Securing The Control Plane

The Kubernetes API server is the heart of your cluster’s control plane, managing all interactions within the cluster. Given its critical role, securing the API server is extremely important. Here are the key steps to ensure its security:

Implement Role-Based Access Control (RBAC): Role-Based Access Control (RBAC) is essential for limiting access to the Kubernetes API based on the principle of least privilege. RBAC allows you to define precise roles and permissions for users and applications, ensuring that only authorized entities can interact with the API.
Secure API Communication with TLS: Ensure that the API server is only accessible over a secure, encrypted connection using TLS. TLS encryption protects the data transmitted between the API server and its clients, preventing unauthorized access and tampering.
Regularly Auditing API Server Logs: Monitor and audit API server logs to detect unusual or unauthorized activities. Implement logging and monitoring tools to provide visibility into API usage and potential security incidents.

You can also check: https://kubernetes.io/docs/concepts/security/controlling-access/

Pod Security

Pods are groups of containers running on the same host, sharing a network stack and other resources. It’s essential to configure your pods to limit their permissions and capabilities, minimizing potential attack surfaces and preventing lateral movement. This can be achieved by:

Setting Pod Security Admission: Define a set of conditions that a pod must meet to be accepted into the system. These conditions control aspects such as privilege escalation, host network access, and volume usage.
- More details can be found here: https://kubernetes.io/docs/concepts/security/pod-security-admission/
Using Security Contexts: Security contexts allow you to define privileges and access control settings for a pod or container. For example, you can prevent containers from running as root by setting the runAsNonRoot option. https://kubernetes.io/docs/concepts/security/pod-security-standards/

Host Security

Host security is a critical yet often overlooked aspect of securing Kubernetes. The host machines, whether they are bare-metal or virtual machines (VMs), must be properly secured to protect the entire Kubernetes environment.

Restrict Access: Limit access to your host machines by using dedicated VPNs and strict access controls. This helps prevent unauthorized users from accessing the underlying infrastructure.
Minimize Attack Surface: Only install the necessary components required for running Kubernetes, such as the Kubernetes code, its dependencies (like Docker), and essential supporting features (logging or security tools). This is often referred to as running a “thin OS.” Container-specific distributions like CoreOS Container Linux, RancherOS, or Red Hat Atomic can help minimize the installed code on your hosts and include features like a read-only root filesystem.
Use Hardened OS Distributions: Consider using container-specific OS distributions or general-purpose Linux distributions that are stripped of unnecessary libraries and tools. This reduces potential vulnerabilities by limiting the software running on the host.

💡 Discussing Kubernetes security can be an extensive and varied topic. For a comprehensive and informative guide, I recommend reading the book Kubernetes Security by Liz Rice. It’s an excellent resource that covers a wide range of important topics in depth.

3️⃣ Runtime Security

If you work in the security industry, you’ve probably heard the phrase “Runtime Security.” Although runtime security is not entirely new, it has gained significant adoption in recent years, with companies like Wiz, Upwind Security, and Sweet Security that utilizing this technology. But what exactly is runtime security, and how does it work?

Understanding Runtime Security

Runtime security refers to the protection of applications while they are running, in that simple but that complex too. It involves monitoring and securing the application in real time to detect and mitigate threats as they occur. This is crucial because even if your application is secure at the build and deploy stages, vulnerabilities can still be exploited at runtime.

eBPF: The Engine Behind Runtime Security

eBPF (short for Extended Berkeley Packet Filter) is a powerful Linux kernel technology that enables efficient and dynamic tracing of various system events such as network packets, function calls, and system events. Originally, BPF (Berkeley Packet Filter) was developed in the early 1990s to filter network packets in a performant way, primarily in network monitoring tools like tcpdump.

Kernel vs. User Space and How eBPF Leverages It

The Linux kernel is the core software layer that sits between applications and the hardware they run on. Applications operate in a restricted environment called user space, where they can’t directly access hardware. Instead, they make system calls to the kernel, which handles these requests.

eBPF operates in the kernel space, allowing it to monitor and interact with system events in real time. What makes eBPF powerful is that it can do this without modifying the kernel itself, maintaining system stability and security. This capability enables eBPF to provide deep visibility and control, making it a vital tool for real-time security monitoring.

https://www.oreilly.com/api/v2/epubs/9781098135119/files/assets/lebp_0101.png

eBPF technology allows companies and products to monitor and extract crucial insights and data in real time during application runtime. Since every network call and system call originates from the kernel, eBPF operates close to the kernel level, providing deep visibility into system behavior without affecting the actual kernel operations.

A great book about eBPF: https://isovalent.com/books/learning-ebpf/
I believe the best way to learn about runtime security is by getting hands-on experience with it. I highly recommend exploring the Falco project, an open-source security tool that leverages runtime security concepts. https://falco.org/

So, I hope I’ve provided a short and precise summary of the three key aspects to focus on when securing your containers. From the first step of building a secure container with a well-written Dockerfile, scanning it during the build process, securing the orchestration platform, to implementing runtime security to monitor and prevent threats in real time. I hope you find this helpful and that it guides you in securing your containerized environments.