Extended Berkeley Packet Filter, or eBPF, is a fascinating part of the Linux kernel that has seen rapid growth and improvement over the last few years. Originally designed for high-speed packet filtering, it is quickly becoming the backbone of networking and security tooling within the cloud-native landscape.

This blog, from one of the CrowdStrike Falcon® OverWatch™ team’s senior security researchers, takes a deep dive into some important security implications for eBPF users. It explores how eBPF is used in Linux and cloud environments, including how it can be abused by attackers to tamper with security tools and introduce backdoors into an environment.

Extending Berkeley: The Fundamentals

eBPF enables auditing and filtering of high-volume events, such as network packets or system calls, without the security or stability overhead of a custom kernel module. This is accomplished using small, lightweight programs that are written in a constrained language and are checked at compile- and runtime for security and performance. eBPF can be broken down into three components:

1. eBPF Programs: These are the special programs that get loaded into the kernel. They are attached to specific “hook points” in the kernel (such as when a network packet is received, or a syscall function is entered), and can inspect and make decisions about the action. eBPF programs are run in the context of the calling user-mode process, so it is possible to see what process is performing an action.
2. User-mode Controllers: These are user-mode programs that send the eBPF program to the kernel to be loaded. They also receive data back from the kernel programs, such as log messages or actions taken.
3. eBPF Maps: These provide the main communication channel between the user-mode and the kernel programs. Maps take the form of either a fixed-size array, circular buffer or key-value dictionary, and can be read and updated by both user-mode and kernel-mode programs.

On the Use of Maps

In order to ensure the eBPF programs are safe, small and stable, the kernel puts strict limits on how complex they can be. For this reason, the programs typically send information down to a user-mode program to handle the complex actions such as data aggregation, logging to a remote server, etc. This communication is done via eBPF maps.

Additionally, dynamic configuration is also often stored in maps to enable alterations without having to unload and reload the eBPF program.

An example could be an eBPF program designed to allow or block connections based on source IP. In this example, the list of IPs to allow could be stored in an eBPF map, so a user-mode controller can update the list of IPs without having to unload the kernel program, which could lead to connections being missed while reloading.

Tampering Trouble

Almost all eBPF-related actions require the CAP_SYS_ADMIN or CAP_BPF capabilities. In practical terms, this typically means running as root either on the host or inside a privileged container.

With these capabilities, however, it is possible to read and write to any eBPF map, including accessing maps created on the host from within a container. This means that any security tool using eBPF is susceptible to a privileged attacker tampering with their configuration, which could degrade the tool’s ability to detect or prevent future malicious activity.

Seeing Cilium

Cilium is a tool that provides network observability and security for Kubernetes clusters. In August 2020, Google announced Cilium would power its new dataplane in its managed Kubernetes environment. Cilum uses eBPF for most of its audit and security features, deploying an eBPF program for each group of containers it is protecting based upon configurable policy.

A privileged attacker can tamper with Cilium’s maps and make their own eBPF program run in place of Cilium’s programs. This new program could be used to circumvent Cilum’s network restrictions or serve as a backdoor into the cluster. Additionally, this attack can be done using the “bpf” system call, bypassing any Cilium or Kubernetes auditing. Detecting the tampered data would require knowledge of eBPF and Cilium to manually inspect and decode the map’s data to deduce the alterations.

The Difficulty of Defense

Protecting maps against tampering is less trivial than it may appear, and is something few tools currently do. eBPF is designed to allow anyone with privileges to read and alter maps, and due to the complexity restraints on the kernel programs, data is unlikely to be encrypted.

If a tool wishes to protect its own maps, a possible solution would be to first load an eBPF program to trace the “bpf” syscall and audit which maps are being accessed by which process. This could work for detection, but prevention is more difficult, as the kernel program would need to know what processes to allow, possibly creating a chicken-and-egg situation if it attempts to use a map to store the allowlisted IDs.

Creating a systemwide monitor using eBPF is even harder. In order to read or write to a map, two bpf syscalls are made:

1. A call to get a process-specific handle to a map, passing in a global Map ID.
2. A call to read or write to a map, passing in the process-specific handle.

This two-step process means that if the monitor is started after a process has done Step 1 and acquired a handle, it is not possible to say what map Step 2 is referring to, as the handle is only relevant to that process. A monitor would need to keep track of process+handle pairings so it can say which map is actually accessed.

Additionally, processes can fork (creating new processes with the same handles), duplicate handles (create a new handle that refers to the map) or even send the handle to another process, all of which make it harder for a monitor to audit the system as a whole. Finally, map alterations can be made by the kernel eBPF programs, which don’t use the syscall but instead access maps directly using a memory pointer. Some solutions or alternatives to a syscall-monitoring eBPF program include:

• Loop Over Every Map and Key Periodically: A monitor could just loop over every map, key and value periodically, keeping track of the previous items. This would miss short-lived changes that happen in between polls, and requires storing a copy of every map, key and value in between iterations. Additionally, it wouldn’t tell you what process made a change, only that a change has occurred.
• Use a Kernel Module: A non-eBPF kernel module could help link the handles to Map IDs, even if you miss the initial lookup. But this is only an option in places you can write and deploy a kernel module, which is unlikely in managed cloud environments — the very place where eBPF is likely to be used for security.
• Use KProbes on internal Kernel Functions: On top of network connections and syscalls, eBPF programs can also be attached to any kernel function using KProbes, enabling auditing of the parameters passed into it. A number of internal functions (such as bpf_map_update_value) contain enough information to provide accurate monitoring of what process is accessing what map. However, KProbes can be very fragile, as it relies on internal functions of the kernel not changing from version to version, which in eBPF’s case has happened a lot over the last couple of years.

Conclusion

EBPF is one of the most interesting and rapidly changing parts of the Linux kernel, and its use in Linux and cloud-native security is rapidly growing.

This blog highlights the importance of being aware of the implications of using this new technology in security tools, and provides guidance for developers looking to protect their eBPF maps. I have open-sourced a proof-of-concept eBPF monitor to demonstrate one method of monitoring using KProbes, which you can find here: https://github.com/CrowdStrike/bpfmon-example. 

Additional Resources

• Read about the Linux protection capabilities of the powerful, cloud-native CrowdStrike Falcon® platform. 
• Learn about recent intrusion trends, adversary tactics and highlights of notable intrusions identified by the CrowdStrike Falcon® OverWatch team in the 2020 Threat Hunting Report.
• Learn about the benefits of proactive managed threat hunting provided by the Falcon OverWatch human threat detection engine. 
• Get a full-featured free trial of CrowdStrike Falcon® Prevent™ and learn how true next-gen AV performs against today’s most sophisticated threats.