How to Harden K8S

My only impression of those spy movies is the scene when agents from NSA eavesdrop on communications outside the United States. But it is not the only job that the National Security Agency does. Apart from that, the NSA also contributes to the cybersecurity community.

NSA made the original SELinux, Linux’s optional mandatory access control (MAC) architecture. Also, they wrote guidelines on securing video conferencing, testing, and collaboration tools while everyone is working remotely. Recently, NSA updated the Kubernetes Hardening Guide, and thus I would like to share these great resources with you and other best practices on K8S security.

Background

The first version of the hardening guide was released in August 2021, authored by the National Security Agency (NSA). The guidance was also co-written by the Cybersecurity and Infrastructure Security Agency (CISA) to create users’ awareness of critical threats and configurations to minimize risk.

In March 2022, NSA updated its Kubernetes Hardening Guide, which is more valuable today. For example, the NCC Group made several comments on the 1.0 version.

NCC saw that the first version’s information about Kubernetes authentication was “largely incorrect” because it stated that Kubernetes doesn’t provide an authentication method by default (Fact: K8S natively supports both token and certification authentication.)

Why Should We Protect K8S? Cloud-Native Survey 2021

According to the 2021 Cloud-Native Survey by Cloud Native Computing Foundation (CNCF):

• 96% of organizations now use or are evaluating Kubernetes.
• 5.6 million developers are using Kubernetes, which equals around one-third worldwide.

Now, of that vast number of adoption, how many do you think are securing Kubernetes properly? Not much. Notably, Kubernetes does not offer “secure-enough” by default configuration. As Red Hat pointed out:

94% of those surveyed admitted they had experienced a Kubernetes and container environments security incident in the last 12 months.

Human errors or misconfiguration is a severe problem. Adversaries know, as well as we do, that we’re now shifting to containers and Kubernetes, and the change is fast. As the NSA indicates, Kubernetes clusters are a prime target for cyberattacks, including:

The NSA and CISA joint announcement during the release of the hardening guide stated:

The security of applications running in Kubernetes and their third-party dependencies relies on the trustworthiness of the developers and the defense of the development infrastructure. A malicious container or application from a third party could provide cyber actors with a foothold in the cluster.

Presently, data theft is the number one target, followed by hijacking Kubernetes clusters for crypto-mining. As a result, if leaving the Kubernetes installation by default, you probably give hackers access to your data and computing power inside the cluster.

Vulnerabilities — Inside Out

The Kubernetes project, like any large open-source project, has been subject to several security vulnerabilities. Therefore, it is crucial to be aware of new vulnerabilities and have a planned procedure to patch Kubernetes once the updates are released quickly.

Here is the list of known vulnerabilities found in K8S:

The NSA and CISA also make a particular point regarding supply chain threats, including software and hardware dependencies that are possibly be compromised in the supply chain before deployment — even the tiniest mistake can break the system.

One notable example is the Log4j vulnerability that caused a significant breakdown recently. Log4j is, in fact, not a component in K8S but Java, which is widely adopted in applications. As you can see, the focus is not only on the vulnerabilities of K8S itself but on all components inside the cluster.

Nothing Easy Around Kubernetes

I know those are basics, but it doesn’t mean they are easy to do. For example, patching in Kubernetes environments is challenging. Besides Kubernetes, numerous other programs run with it in a real-world application. If it is simple, there would not be a place for using K8S. It is also valid for securing it.

Let’s take a look at root access. We all know it isn't good to run applications as root. Yet, many Kubernetes container services run as root privilege by default. As a result, the applications that run within them inherit as root. It is because:

The root user (UID 0) is the default user inside a container. If you don’t specify a non-root user, the container runs as root. Since the majority of container images in public repositories do not swap out the root user, this situation drags on.

The key to hardening guidance is standard best practices. Still, the report also presents an in-depth look at using standard security mitigation measures in complex environments, often in the cloud.

NSA’s Recommendations

Let’s go through the basics first! According to the hardening guide, some steps should be taken in a production K8S environment.

• Scan for vulnerabilities or misconfigurations in containers and Pods.
• Apply least privilege principles in containers and Pods.
• Employ strong authentication and authorization to limit user access and reduce the attack surface.
• Use network segmentation to reduce the exposure if compromised.
• Use firewalls to limit excessive network access.
• Use encryption to protect confidentiality.
• Review regularly all Kubernetes settings and use vulnerability scans to account for the acceptable security risks.
• Finally, capture and monitor audit logs to alert potential malicious activity.

These are not all but a starting point for gaining some confidence when putting your containerized applications into production.

K8S Security Best Practices

Apart from the NSA K8S hardening guide, another helpful resource from NIST is the Application Container Security Guide or SP 800–190. This document outlines the container-related security concerns and offers recommendations accordingly.

0# Containers Security

Implementing security practices in the CI/CD pipeline and the entire build, store and deploy process of container images would include:

• securely storing the container images,
• vulnerability scanning for container images and;
• managing/ monitoring the container runtime security.

But if we consider K8S alone, we need something more specific. At the same time, the NSA’s guide is a great starting point. And if you follow through with the guide and build your K8S cluster, it will be fair enough and confident enough to put it in production. Below are some other areas that need to highlight.

1# Role-Based Access Control (RBAC)

RBAC is the frontline of defense in Kubernetes. It lets the user control who can access the API and what permissions they have. Although RBAC is enabled by default, it may not be enabled if you upgrade from an old version of Kubernetes. Also, the RBAC settings should be checked and reviewed regularly to ensure no unnecessary permissions are granted.

Simply having RBAC enabled is not good enough. There are authorization policies that should be careful with and used properly. Always use RBAC with the least privilege principle.

That means using it to limit users and groups to only the actions and tasks they require. Confirm that Kubernetes service and other user accounts have the minimal privileges needed. Moreover, we need to make sure not to give cluster-wide permissions and not give anyone cluster-admin privileges unless necessary.

(For more details, please refer to the official Kubernetes RBAC documentation.)

2# Encryption Keys (Kubernetes Secrets)

There are different forms of encryption keys, such as a password, a token, or an SSH key. Kubernetes secrets support the initial communication securely between artifacts and pods.

For example, when a pod starts, it will require to access its secrets. Also, whenever a service account is created, a Kubernetes secret holding its authorization token is generated on the fly. In addition, Kubernetes supports encryption at rest. This encryption offers an additional level of protection when backups are not encrypted, or an attacker gets read access to etcd.

According to the official document, we should also make sure SSL/TLS is applied in the communication between:

1. from the API server to the kubelets, and
2. users and the API server.

Lastly, it is recommended to have a short lifetime for a secret or credential to reduce the time an attacker uses them when they are lost. To achieve that, set short lifetimes on certificates and automate key rotation.

3# API is the New Endpoint

Despite what I mentioned in #1 and #2, access control and transport security of Kubernetes API, Kubernetes API endpoint is another area we should be careful of. Administrators can configure the API endpoint of a Kubernetes cluster as part of a private or public subnet.

It is better to configure the Kubernetes API endpoint as a private endpoint by default. Thus, the master inaccessible from the public internet as the API endpoint inside the control plane has a private IP address. This is essential for fully private clusters that don’t have or expose any public IPs. As a result, there would be no ingress/egress of traffic to and from the internet.

(For more information, please refer to the official documentation.)

4# OS, Nods, and Pods Security

To harden and secure K8S, we cannot ignore the basics — OS, Nodes, and Pods Security. First, a K8S node is a worker node running in an OS that can be a virtual or physical machine. The services running on a node include:

• the container runtime,
• kubelet and
• kube-proxy.

To keep the focus on K8S, the Center for Internet Security (CIS) Kubernetes benchmark is a well-known best practice guide. For sure, all the security practices on a server, including vulnerability management, anti-malware, EDR, and patch management need to be in place for a node.

In addition to workload security, there is a security risk regarding internal communications. It is recommended that a gateway should be configured for access outside the cluster if needed. Network port access on nodes should be managed via network access control lists. Limiting Secure Shell (SSH) access to the nodes should also be considered.

On the other hand, a pod is a group of one or more containers that run on nodes. Initially, we need to use network policies to control the communication for pods within a cluster. To achieve that, we need the network plugin implements network policies, and using them may require a network driver that supports policies.

The pod-level network security posture can be strengthened by a combination of network policies and access control list down to host levels. Also, the pod security context of K8S supports the mapping between privilege and access-control settings for containers and pods. (For the complete list of security contexts, please refer to the official guide #securitycontext.

Nevertheless, the core of pod security should be Pod security policies that let a user control the runtime execution properties of the pods, i.e., the power of running containers as privileged containers leveraging the host file system, network, and ports.

Final Words

Your ultimate security goal is to push it as challenging as possible for hackers to gain access to your systems; secure K8S is the same. And even if that happens, you should be good enough to detect abnormal activity and take action to prevent it from spreading.

Securing Kubernetes should be a full-time job. The NSA mentions that there are third-party security programs that can help. But, of course, these also come with security concerns. Security is vital for containers, and it’s something you need to take into account and manage when working with Kubernetes.

Reference:

• https://www.cvedetails.com/product/34016/Kubernetes-Kubernetes.html
• https://www.nsa.gov/Press-Room/News-Highlights/Article/Article/2716980/nsa-cisa-release-kubernetes-hardening-guidance/
• https://kubernetes.io/blog/2016/08/security-best-practices-kubernetes-deployment/

----

Thank you for reading. May InfoSec be with you🖖.