Microservice architectures and cloud-native applications have been increasingly trending in recent years. More and more organizations are dividing their huge monolithic applications into smaller services and packaging them as containers. However, deploying and managing hundreds of microservices is not straightforward and requires special tooling. Kubernetes is today the de facto container orchestration platform to run scalable and reliable applications. With its easy-to-use API and developer-friendly characteristics, Kubernetes has become an indispensable part of the cloud ecosystem.
The paradigm shift in software architecture and operations has also inspired significant security changes. With microservice applications and containers deployed to cloud platforms such as Kubernetes, cloud-native security principles are required. This is because, unlike traditional data centers, the cloud has unclear security perimeters, leading to new challenges. The most common cloud security problems and risks are explained in depth in our article “What is cloud security?”
This guide will focus on the Kubernetes part of cloud security, discussing the attack surface, security lifecycle and best practices.
Kubernetes Attack Surface
A Kubernetes cluster consists of control plane components and nodes as diagrammed in Figure 1.
Figure 1: Kubernetes Components (Source: Kubernetes Docs)
The control plane is the brain of Kubernetes clusters, where definitions and the state of all Kubernetes resources are managed and stored. It is a critical vector for attackers. If a hacker gains access to your control plane components, they can deploy containers, read secrets or even delete the whole cluster.
The nodes are where the actual workload of the Kubernetes workload runs. All your applications and libraries are deployed as containers and run on the nodes, meaning they, plus the application layer, need protection as well. If the operating system of the node has vulnerabilities, it can lead to unwanted access. Similarly, if any dependency or library in your application containers has vulnerabilities, attackers can exploit these to access your application and data.
The third and last vector is where the Kubernetes clusters run: the cloud. Access to cloud services such as user management or a managed Kubernetes service could grant them privileges equal to a Kubernetes operator.
In the following section, we will discuss every step in the lifecycle of an application in a Kubernetes cluster to protect the attack surface.
What is the Kubernetes Security Lifecycle?
The Kubernetes security lifecycle covers the complete journey of an application, starting from the source code to a running deployment in Kubernetes. Each step has its specific vulnerability issues and requires great care.
Before the Kubernetes Cluster
Kubernetes runs applications packaged as containers. This means you need to create container images with an application executable and dependencies. With the latest trends in software development, the build and release of applications and container images are done in pipelines like Jenkins, GitHub Actions or GitLab CI/CD to deliver faster with less effort.
In addition, image scanning is a container security process handled as part of CI/CD pipelines for the early detection of security issues. It should include the following steps:
- Check your operating system files and configuration, software packages, libraries and binaries
- Analyze Dockerfile for security flaws such as exposed ports or privileged access
There are popular open-source and commercially available tools for container image scanning, such as Anchore Engine, CoreOS/Clair, OpenSCAP and Falcon Image Assessment. These tools also have a rich set of integration points to be used as part of CI/CD pipelines.
In short, there are two things you need to protect before starting your Kubernetes journey:
- CI/CD pipelines
- Container images
Inside the Kubernetes Cluster
Kubernetes is a complex system with a layered architecture and well-defined APIs. It is designed as an extensible platform with multiple extension points. These allow you to develop new networking plugins to support custom networking infrastructure or create Kubernetes operators to transform human expertise into code. However, this extendibility makes the whole cluster vulnerable at multiple locations.
There are five important places that you need to ensure to achieve security inside a Kubernetes cluster: the Kubernetes control plane, access to the Kubernetes API, networking, nodes, and the container and runtime.
Kubernetes Control Plane
The Kubernetes control plane consists of the core services that keep the whole cluster up and running. The specification and state of the Kubernetes resources are managed by control plane components and stored in etcd, an open-source distributed key/value database. Therefore, etcd should only be accessible via the Kubernetes API with correct permissions. It is suggested to limit access to etcd with the following restrictions:
- Firewalls such as iptables and netfilters for etcd instances
- Authenticated access between the Kubernetes API and etcd
The Kubernetes control plane handles the incoming requests by evaluating them in multiple components. It is also possible to intervene in this process using webhooks and check the incoming requests. There is already a long list of available admission controllers in upstream Kubernetes that you can configure and use. In addition, you can write your own admission controllers and implement custom security rules.
Every action in the Kubernetes control plane is recorded as part of audit logs with the date, time, action, initiator and state. You can configure an audit policy in the cluster to define which events are logged. You can then store the logs in the filesystem or send them to external systems via webhooks.
Access to the Kubernetes API
The Kubernetes API is the interface of the control plane for external users, making authentication and authorization crucial parts of security.
For authentication, Kubernetes has the concept of service accounts and external users. Service accounts are created like other Kubernetes resources and thus have limited configuration and user-related information. In order to authenticate real users, you need to connect with LDAP services or cloud identity services. Kubernetes provides OpenID Connect (OIDC) to use as an external authentication provider. This lets you use LDAP installation within your company and make everyone log in to Kubernetes with their work user accounts. It is highly suggested to secure your Kubernetes cluster access by authentication with an OIDC provider.
For authorization, Kubernetes offers role-based access control (RBAC) out of the box. You can define access control specifications with actions such as read, update, and delete on every Kubernetes object and namespace. You need to design the hierarchy of users, user groups and access levels with great care and create corresponding RBAC definitions in Kubernetes clusters.
Pods running in Kubernetes clusters can easily connect to other pods with Kubernetes networking capabilities. To create secure communication within the cluster, you should configure the network policies; these are Kubernetes-native specifications to create firewalls with pods and namespaces. In addition, you can deploy third-party networking plugins like Calico or Weaveworks and a service mesh like Istio. If you install any networking plugin, it is vital to configure all security options and keep them up to date.
Kubernetes nodes run the actual Kubernetes workload — your software as containerized applications. In every Kubernetes node, there is an agent called a kubelet to communicate between the control plane and the container engine on the node. Therefore, you need to secure the two sides of the kubelet:
- Communication between the Kubernetes API and the kubelet, which should connect to the Kubernetes API using certificates with a limited cluster role.
- Communication between the kubelet and the container engine. The kubelet and the container engine, such as Docker daemon, run on the node operating system, so it is highly suggested to use a minimal host operating system. It is also critical to have the latest system and security patches installed.
In cloud environments, nodes are assumed to be ephemeral, as they can be created and deleted on demand. This means it’s vital to not leave essential data on the node filesystem. In addition, it is suggested to use resource requests and limits to keep nodes healthy with enough capacity.
The Container and Runtime
The last step in a containerized application lifecycle is deploying it as a pod and running it as a container in a Kubernetes node. Even if you have scanned your container images, secured your Kubernetes API and configured the kubelet correctly, some vulnerabilities can come up during runtime:
- Zero-day vulnerabilities
- Privilege escalations
- Malware (e.g., cryptoming)
Runtime security in Kubernetes means protecting containers against active threats while they are running. There are open-source tools like seccomp, AppArmor or SELinux that focus on privilege escalations and limiting access to the binaries on Linux systems, but they are not sufficient for runtime protection from malware.
If you’re looking for a cloud-native and modern solution, you can check out CrowdStrike Cloud Workload Protection, a security isolation solution that can work across multiple cloud providers and secures both nodes and containers running on the nodes.
What are some Kubernetes Security Best Practices?
Kubernetes is an extensible platform with various security vulnerabilities and unclear security perimeters. Given this, there are some widely accepted best practices you should apply to keep your clusters and access safe:
- Third-party authentication: Integrate an external authentication provider and use already-defined user groups for authorization to access your Kubernetes API.
- RBAC: Ensure that RBAC is enabled and configured correctly, as a slight change in RBAC rules can make your clusters available to the world.
- Audit logging: Ensure that audit logging is enabled and available, even if the cluster is deleted.
- Node isolation and network traffic monitoring: Make sure you isolate the network of the nodes and monitor the traffic throughout the cluster.
- Kubelet configuration: Ensure kubelet has only the minimum required access levels and configuration options.
You can also read the “Kubernetes Hardening Guide” by the NSA, a detailed guide on threats to Kubernetes environments.
Kubernetes is a complex platform with an active community and an ever-changing environment, with new plugins and infrastructure extensions. Therefore, it comes with many risks in terms of creating a secure production-ready cluster.
Fortunately, Kubernetes is highly flexible and can work with other tools via extension points and the Kubernetes API. So, if you configure and use cloud-native modern tools, you can securely run your applications in Kubernetes in a reliable and scalable way.