sanecz

Livepatching the Linux kernel : through the eyes of an SRE

sanecz — Sun, 18 Oct 2026 00:00:00 +0000

Introduction

Kernel live patching, or KLP, is a feature merged in the kernel 4.0 to apply fixes to the running kernel without requiring a reboot.

This might come in handy when you need to patch a bug or a security issue and cannot reboot the machine due to critical applications running on it, or need to minimize the downtime. KLP allows in-place patching of various functions in the kernel under some conditions, produces no service interruption (if patched properly) and is really fast to apply, but might require (a lot of) extra work.

In this article, we will first write a livepatch by hand for Januscape, let kpatch-build do the job for Pwnescape and then explore cumulative and atomic replaces with a Zapscape and Pwnescape.

Disclaimer: livepatches shown in this article were written and tested on my own test machines built for specific kernel versions. They are here to illustrate how livepatching works, not to be loaded on your production servers. If you do it anyway, keep in mind it may crash your server as our environments and kernel builds are not identical. Use at your own risk, I’m not responsible for your hypervisors (I already have mine to deal with).

Livepatch

How does it work ?

Livepatch relies on ftrace. When the kernel is built with function tracing enabled, the compiler adds a call to __fentry__ at the beginning of every function (-pg flag/-mfentry on x86_64). At boot, ftrace replaces all of these calls with a nop instruction, so they cost almost nothing when tracing is not used.

All the functions that can be traced are listed in /sys/kernel/tracing/available_filter_functions.

When a function is livepatched, KLP registers an ftrace handler on it, and the nop at the beginning of the old function becomes a call to this handler again. The handler then redirects the execution to the new function. The old function is still there and callers still call it, but it never runs its own code anymore.

Requirements

For KLP to work, you’ll need a kernel built with at least the following configuration: CONFIG_MODULES to enable loadable module support, CONFIG_LIVEPATCH/CONFIG_HAVE_LIVEPATCH, CONFIG_KALLSYMS, CONFIG_KALLSYMS_ALL, CONFIG_DYNAMIC_FTRACE_WITH_REGS or CONFIG_DYNAMIC_FTRACE_WITH_ARGS, with a kernel version >= 4.0.

CONFIG_HAVE_RELIABLE_STACKTRACE provides reliable stack traces used by KLP to known when a task can be switched to the new function. Without it, the patching transition might never complete.

CONFIG_TRIM_UNUSED_KSYMS should be disabled.

if CONFIG_X86_KERNEL_IBT is enabled, on Intel 11th gen+ CPUs and unless the kernel is booted with ibt=off or ibt=warn, calling functions resolved through kallsyms_lookup_name may trigger a control-protection exception and crash the host. Also kpatch-build does not support this option

The exact requirements depend on the architecture and kernel version. This is only a checklist for a x86_64 configuration used in this article, not a universal recipe.

grep -E 'CONFIG_(LIVEPATCH|HAVE_LIVEPATCH|MODULES|KALLSYMS|KALLSYMS_ALL|TRIM_UNUSED_KSYMS|DYNAMIC_FTRACE_WITH_(REGS|ARGS)|UNWINDER_ORC|HAVE_RELIABLE_STACKTRACE|X86_KERNEL_IBT)[= ]' /boot/config-$(uname -r)

Limitations

Only traced functions can be patched. This avoids issues like trying to patch functions implemented for ftrace itself or low-level architecture code. Tracing can be disabled per file by removing the function profiling flag from the compilation:

arch/x86/kernel/Makefile:CFLAGS_REMOVE_kvmclock.o = -pg
arch/x86/kernel/Makefile:CFLAGS_REMOVE_ftrace.o = -pg
arch/x86/kernel/Makefile:CFLAGS_REMOVE_early_printk.o = -pg
arch/x86/kernel/Makefile:CFLAGS_REMOVE_head64.o = -pg

arch/arm64/kernel/Makefile:CFLAGS_REMOVE_ftrace.o = $(CC_FLAGS_FTRACE)
arch/arm64/kernel/Makefile:CFLAGS_REMOVE_insn.o = $(CC_FLAGS_FTRACE)
arch/arm64/kernel/Makefile:CFLAGS_REMOVE_return_address.o = $(CC_FLAGS_FTRACE)

Functions using the notrace macro cannot be patched either as they won’t be available to ftrace, same for the inline functions because the kernel redefines inline and adds the notrace macro:

#define inline inline __gnu_inline __inline_maybe_unused notrace

Patching functions inlined by the compiler requires more tricks: the function doesn’t exist as a standalone function anymore and its code is copied into each caller, so patching it means patching all of its callers instead.

The livepatch replaces the original function with your new one, so you cannot call the original function from your patch. The call would go through its ftrace hook, then be redirected to the new function, creating an infinite recursion. You’ll have to copy the original code and modify it.

It only replace code, not data. If a fix changes a structure (adding a new field, bigger array etc etc), it can’t be patched as-is, the existing objects were allocated with the old structure layout, and the rest of the kernel keep using it. There is some workaround using shadow variables (klp_shadow_alloc(), klp_shadow_get()). It attaches extra data to an existing object stored on the side of the structure instead of inside of it. It does work but the livepatch is considerably harder to write.

Tracing tools are also affected. Since the original function never runs its own code anymore, kprobes placed inside its body (systemtap, perf probe, bpftrace kprobe probes with an offset) stop firing once it’s patched. A probe on the function entry shares the same ftrace call site as livepatch, it may still fire before the redirection, but it only tells you the old function was called, not that its code ran. Probes have to target the new function instead, those living in the livepatch module.

kretprobes can conflict with livepatch, as both rely on an ftrace handler that modifies the return address. Only one of them can acquire the handler: “the first user wins” and the other one is rejected.

Writing a livepatch by hand

Anatomy of a livepatch

A livepatch is just a kernel module with three structures that define the behavior of the patch, and your replacement code. Those nested structures define what to patch, where to patch and the initialization. This base livepatch comes from samples/livepatch/livepatch-sample.c.

Mandatory includes:

#include  /* THIS_MODULE, MODULE_LICENSE, MODULE_INFO, module_init, module_exit */
#include  /* klp_func, klp_object, klp_patch, klp_enable_patch */

Your new function, where KLP redirects callers to in place of the original:

#include 
static int livepatch_cmdline_proc_show(struct seq_file *m, void *v)
{
	seq_printf(m, "%s\n", "this has been live patched");
	return 0;
}

struct klp_func: tells KLP which function to replace with the new one. Since funcs[] is an array, it can contain multiple functions to be replaced.

static struct klp_func funcs[] = {
    { .old_name = "cmdline_proc_show",
      .new_func = livepatch_cmdline_proc_show, },
    { }   /* NULL terminator — required */
};

struct klp_object: tells KLP where the functions live.

static struct klp_object objs[] = {
    { .name = NULL, .funcs = funcs, }, /*  .name = NULL means vmlinux */
    { }   /* NULL terminator */
};

struct klp_patch: top-level structure, ties your struct klp_object to this livepatch module.

static struct klp_patch patch = {
    .mod = THIS_MODULE,
    .objs = objs,
};

And finally, the init/exit and MODULE_INFO, otherwise it wouldn’t be a working kernel module. livepatch_init() will call klp_enable_patch() to start the patching transition.

static int livepatch_init(void) { return klp_enable_patch(&patch); }
static void livepatch_exit(void) {}
module_init(livepatch_init);
module_exit(livepatch_exit);
MODULE_LICENSE("GPL");
MODULE_INFO(livepatch, "Y");

klp_enable_patch() will return before the patching is finished. There is a transition period where KLP switches tasks from the old function to our newly patched function, only when the task is no longer inside the old function. The same process applies when removing the patch: we first need to unpatch, wait for the transition, then unload the module.

To build this module, we will use the following Makefile:

obj-m := livepatch-cmdline.o
# KDIR will point to the build tree of the running kernel, you might need to install linux-headers-$(uname -r)
KDIR := /lib/modules/$(shell uname -r)/build
all:
	$(MAKE) -C $(KDIR) M=$(PWD) modules
clean:
	$(MAKE) -C $(KDIR) M=$(PWD) clean

Then run make, which will produce livepatch-cmdline.ko, then load the module:

# make
make -C /lib/modules/6.2.0-39-generic/build M=/home/lisa modules
make[1]: Entering directory '/usr/src/linux-headers-6.2.0-39-generic'
  CC [M]  /home/lisa/livepatch-cmdline.o
  MODPOST /home/lisa/Module.symvers
  CC [M]  /home/lisa/livepatch-cmdline.mod.o
  LD [M]  /home/lisa/livepatch-cmdline.ko
  BTF [M] /home/lisa/livepatch-cmdline.ko
Skipping BTF generation for /home/lisa/livepatch-cmdline.ko due to unavailability of vmlinux
make[1]: Leaving directory '/usr/src/linux-headers-6.2.0-39-generic'
# insmod livepatch-cmdline.ko

With dmesg we can follow the status of the patching transition:

# dmesg
[...]
[79191.929967] livepatch: 'livepatch_cmdline': starting patching transition
[79192.818083] livepatch: 'livepatch_cmdline': patching complete
# cat /proc/cmdline
this have been livepatched

livepatch-cmdline.ko becomes livepatch_cmdline because the kernel replaces the dashes in module names with underscores.

Loading the patch is not enough to be sure that the livepatch took effect. Looking at /sys/kernel/livepatch/ will give more data about the state of the livepatch:

# ls /sys/kernel/livepatch/livepatch_cmdline/
enabled     force       transition  vmlinux/
# cat /sys/kernel/livepatch/livepatch_cmdline/enabled
1
# cat /sys/kernel/livepatch/livepatch_cmdline/vmlinux/patched
1
# ls  /sys/kernel/livepatch/livepatch_cmdline/vmlinux/
cmdline_proc_show,1  patched

livepatch_cmdline/enabled returning 1 means the livepatch is active, then vmlinux is our patched object and the function patched is cmdline_proc_show.

If /sys/kernel/livepatch/livepatch_cmdline/transition stays at 1, it means that the transition isn’t done yet. Either a task is still sleeping inside the old function and will be switched once it returns, or it’s a long-running task that never leaves the old function. The blocking tasks can be found with /proc//patch_state, which shows 0 while a task is still unpatched. The kernel automatically sends them a fake signal to try to make them leave the old function, which might help some tasks but not those that never leave the old function. In that case, the transition might never complete.

Disabling the livepatch:

# echo 0 >/sys/kernel/livepatch/livepatch_cmdline/enabled
# rmmod livepatch-cmdline
# dmesg
[...]
[79397.945201] livepatch: 'livepatch_cmdline': starting unpatching transition
[79399.862745] livepatch: 'livepatch_cmdline': unpatching complete

The directory /sys/kernel/livepatch/livepatch_cmdline also contains a force file. Writing 1 to it marks all the remaining task as patched to end a stuck transition. If you force it, a task may still be running the old code, and the module can no longer be removed afterwards. This totally bypass the task-consistency checks and other alternatives should really be considered before using the force file.

Livepatching Januscape

Januscape is the second of three KVM escapes reported by V4Bel. According to V4bel, it is the first KVM/x86 vulnerability that can trigger either on Intel or AMD architectures. This exploit, correctly executed, might allow an attacker to execute code on the host and the available POC cause denial of service by triggering a kernel panic on the host.

To write the Januscape livepatch, we will need to implement two fixes from two separate commits: 81ccda30b4e8 and 0cb2af2ea66ad.

The first commit 81ccda30b4e8 patches kvm_mmu_get_child_sp:

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 9368a71336fe4..c13b80fe3125b 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -2459,13 +2459,15 @@ static struct kvm_mmu_page *kvm_mmu_get_child_sp(struct kvm_vcpu *vcpu,
 						 u64 *sptep, gfn_t gfn,
 						 bool direct, unsigned int access)
 {
-	union kvm_mmu_page_role role;
+	union kvm_mmu_page_role role = kvm_mmu_child_role(sptep, direct, access);

-	if (is_shadow_present_pte(*sptep) && !is_large_pte(*sptep) &&
-	    spte_to_child_sp(*sptep) && spte_to_child_sp(*sptep)->gfn == gfn)
+	if (is_shadow_present_pte(*sptep) &&
+	    !is_large_pte(*sptep) &&
+	    spte_to_child_sp(*sptep) &&
+	    spte_to_child_sp(*sptep)->gfn == gfn &&
+	    spte_to_child_sp(*sptep)->role.word == role.word)
 		return ERR_PTR(-EEXIST);

-	role = kvm_mmu_child_role(sptep, direct, access);
 	return kvm_mmu_get_shadow_page(vcpu, gfn, role);
 }

Before writing any code, we need to ensure that the function we want to replace actually exists and can be traced by ftrace:

# grep -E ' kvm_mmu_get_child_sp' /proc/kallsyms
ffffffffc10c1900 t kvm_mmu_get_child_sp	[kvm]
# grep -E '^kvm_mmu_get_child_sp ' /sys/kernel/tracing/available_filter_functions
kvm_mmu_get_child_sp [kvm]

Note that KLP doesn’t care whether kvm_mmu_get_child_sp is exported: the livepatch core finds the function to replace by looking it up in kallsyms itself. However, if the compiler inlined the function into some of its callers, those callers would have to be patched too. It can be checked by looking at the disassembly of the module.

We will start with a naive implementation of this function in our livepatch. The struct klp_object will now contain the kvm module to patch instead of NULL, as we are not patching vmlinux functions anymore.

Click here to display full code

Demystified: the IPMI System Event Log

sanecz — Tue, 24 Dec 2024 22:40:26 +0000

Introduction

IPMI is a useful tool for monitoring server hardware, and one of the features we’ll cover here is the System Event Log (SEL), which helps track system health. The SEL records events such as hardware failures (e.g., ECC errors, power supply failures), boot events, and sensor state changes.

In this article, we’ll look at the differences between the ipmitool sel list and ipmitool sel elist commands, and how each presents SEL data in different formats. We’ll also explore raw IPMI commands for accessing log entries directly and explain how to decode the raw output with some examples.

ipmitool sel list vs ipmitool sel elist

The ipmitool sel list command is used to list SEL entries. It provides an overview and summary of each entry. The output is concise and includes several pieces of information such as:

Record ID: Identifier for each log entry
Timestamp: Time when the event was logged
Sensor Type / Sensor number: Sensor that generated the event
Sensor state: Value interpreted based on the current event type
Event Direction: Assertion/Deassertion event

e.g:

575 | 12/03/2024 | 11:27:10 | Module / Board #0x43 | Upper Critical going high | Asserted

Here, the information is pretty succint, some module on the board reached a critical value. But what is happening ?

Sometimes…

On the other hand, ipmitool sel elist includes a little more details about the events but it may not apply for every entries. This might a little more information about OEM events. Even it’s not possible to decode everything, it can give a a human readable hint about the sensor that triggered the event.

This includes:

Decoded sensor number: Full name of the sensor that generated the event from SDR (Sensor Data Record)
Context: depends of the event, may contains the reading of the sensor that triggered the event, the threshold, the severity etc..

e.g

575 | 12/03/2024 | 11:27:10 | Module / Board PWR_GB1_TOT_HSC | Upper Critical going high | Asserted | Reading 5865 > Threshold 5865 Watts

Now it is clear that the power sensor reached the critical threshold value but also the current reading of the sensor and the threshold value are 5865 Watts.

ipmitool raw and the SEL

The ipmitool raw command lets you send raw IPMI commands directly to the BMC and see the responses. We’ll use this to retrieve logs stored in the BMC’s non-volatile memory, but first, we need to understand how these commands are structured.

Querying the SEL with raw commands

When sending a command to the BMC, you need two key pieces: the NetFn and the CMD.

The NetFn (Network Function) is a command category that groups related operations, so the BMC knows what kind of task to perform. This tells the BMC which category of commands you’re using. There is a NetFn for sensors, power settings, storage and many more ! We’ll use the storage NetFn (code 0x0A) because it’s responsible for managing system logs, sensor data, and hardware inventory.

Each NetFn has specific sub functions called commands (CMD). For the netfn storage, these include:

Write FRU Data (0x12): Writes hardware information to the FRU memory.
Set SEL Time (0x49): Updates the timestamp for the logs.
Get SEL Entry (0x43): Reads a specific log entry.

To read a log entry, we combine the Storage NetFn (0x0A) with the Get SEL Entry command (0x43). This tells the BMC to fetch data from the system event log stored in its memory.

Here’s what the raw command looks like:

ipmitool raw 0x0a 0x43

The code for the availables netfn and commands are in the Intelligent Platform Management Interface Specification ¹:

Network Function Codes: Table 5 (page 67)
Sub-functions: Appendix H (page 615)

Using the IPMI specification table for the GET SEL Entry, we can retrieve any log entry by specifying its record ID. For example, let’s say we want to fetch the entire record for the entry with record ID 71h.

Here’s how we construct the command:

0x0A: Storage NetFn.
0x43: Get SEL Entry command.

With the arguments:

0x00 0x00: Reservation ID, 0x00 0x00 = no reservation is required.
0x00 0x71: Record ID (71h) that we want to retrieve.
0x00: Offset field, read from the beginning of the record.
0xff: Retrieve the entire record.

Intelligent Platform Management Interface Specification page 424

Which gives:

ipmitool raw 0x0a 0x43 0x00 0x00 0x71 0x00 0x00 0xff

Decoding the raw output

After running the command, we get the following raw output:

 72 00 71 00 02 7d 48 31 67 20 00 04 09 ff 6f 00 00 00

Using the response data from the table get SEL entry, we can extract some fields.

Note: the completion code is not returned by ipmitool.

[0:1] is 72 00, the next record ID, that is 72h
[2:N] is 71 00 02 7d 48 31 67 20 00 04 09 ff 6f 00 00 00, as this is the record data, we need another documentation to decode it.

Intelligent Platform Management Interface Specification page 424

Then we could use the System Event Log (SEL) Troubleshooting Guide from Intel ² to decode furthermore.

System Event Log (SEL) Troubleshooting Guide page 16

The event data can only be decoded after decoding the record type, sensor type, and event type. Both the record type and event type determine which decoding table to use. Additionally, they can indicate if the record is an OEM event, in which case you’ll need the manufacturer’s documentation to decode it.

For example, in our case, the record type is 02h, which indicates a system event (not OEM), so it should be easier to decode. The event type is 6Fh, and the event data is 00 00 00.

System Event Log (SEL) Troubleshooting Guide page 40

A case of OEM SEL record

Let’s start with this SEL entry, which doesn’t seem too helpful at first glance:

49b | 12/04/2024 | 12:00:13 | OEM record dc | 001647 | 00747a031678

Query the BMC for the full details using the command:

ipmitool raw 0x0a 0x43 0x00 0x00 0x9b 0x04 0x00 0xff

We get the following output:

9c 04 9b 04 dc 4d 44 50 67 47 16 00 00 74 7a 03 16 78

Now let’s break it down step by step:

9c 04: LSB First. Next SEL record ID: 49Ch.
9b 04: Bytes[1,2] LSB First. Current record ID: 49Bh.
dc: Byte[3] Record type: OEM Timestamped.

Then, we decode the timestamp:

4d 44 50 67: Byte[4-7]: LSB First, which is 6750444Dh or 1733313613 in decimal, converted to date: 04/12/2024 12:00:13.

As this is an DCh OEM record, we’ll use the table 3 (SEL Troubleshooting page 18) to decode the next part:

47 16 00: Bytes[8-10]: LSB First. This is the Manufacturer ID. When decoded, 0x1647 corresponds to NVIDIA (ID 5703). The enterprise can be found on the IANA Enterprise Numbers registry from the manufacturer ID.

Since this is an OEM record, full decoding of bytes 11 to 16 requires the manufacturer’s documentation. Or, you know, just kindly ask them directly — here it seems that some RAM decided to give up and crash the server! ;)

IPv6 connectivity issue : a journey into radvd and socket options

sanecz — Tue, 02 Apr 2024 09:00:00 +0000

Context

Shadow is a cloud computing service specialized in Gaming. To provide isolated machines to users, they rely heavily on virtual machines.

Each virtual machine has dual-stack connectivity with both IPv4 and IPv6. Using IPv4 the network configuration is pushed via DHCP providing both addressing and default route informations. Using IPv6, we need to use the Router Advertisement mechanism.

Recently, we changed our networking stack on the hypervisor, whereas the virtual interfaces for each VMs were created beforehand and persistent we moved to transient virtual interfaces creating them and removing each times a virtual machine is started and stopped.

After a few weeks we noticed virtual machines could randomly lose their IPv6 connectivity to the Internet. While disabling the IPv6 connectivity allowed reliable access to the VM to be restored, we dug to find out what could be the cause of this defect.

Analysis of the IPv6 connectivity in the VM

After a short analysis we noticed that the Windows routing table no longer had an IPv6 route but instead, only a link-local route.

> route print
[...]
IPv6 Route Table
Active Routes:
If	Metric	Network Destination		Gateway
1	331	::1/128				On-link
4	271	fd12:0:0:a01::a/128		On-link
4	271	fe80::/64			On-link
4	271	fe80::d756:522a:7e8b:339c/128	On-link
1	331	ff00::/8			On-link
4	271	ff00::/8			On-link
Persistent Routes: None

A correct configuration would be:

> route print
[...]
IPv6 Route Table
Active Routes:
If	Metric	Network Destination		Gateway
13 	271 	::/0				fe80::fc42:c6ff:fe7e:6388
1 	331 	::1/128				On-link
13 	271 	fd12:0:0:a08::a/128		On-linka
13 	271	fe80::/64			On-link
13	271 	fe80::6b86:ba8a:cc33:f514/128	On-link
1	331	ff00::/8			On-link
13 	271	ff00::/8			On-link
Persistent Routes: None

The VM does not know where to send its IPv6 packets despite having IPv6 connectivity.

Once the VM initialized its network interface, it sends a RS (Router Sollicitation) request which aims to discover available IPv6 routers. This is radvd’s time to shine, as it’s in charge of answering this RS by a RA (Router Advertisement). The VM will then know that this router will handle its traffic (thus adding the ::/0 route via fe80::fc42:c6ff:fe7e). These mechanisms are part of the Neighbor Discovery Protocol.

This router advertisement is never sent when radvd encounters a precise bug… And tcpdump made pretty clear that this router advertisement went missing.

Understanding the missing Router Advertisment

To receive Router Solicitation and send Router Advertisement, radvd configures its socket to join the multicast group of the virtual machine network interface. This is done by using the setsockopt syscall. setsockopt is a syscall allowing to specify options on a socket.

For example, one can use this syscall to force the reuse of an address (with SO_REUSE_ADDR), change the size of the buffer allocated for receiving data on this socket (with SO_RCVBUF). In our case, it is used to join a multicast group (with IPV6_ADD_MEMBERSHIP).

Using strace we were able to trace these calls, their options and their return value. Below, a curated output of the strace attached to the radvd ¹ process.

$ strace -p 
[...]
setsockopt(3, SOL_IPV6, IPV6_ADD_MEMBERSHIP, {inet_pton(AF_INET6, "ff02::2", &ipv6mr_multiaddr), ipv6mr_interface=if_nametoindex("tap2")}, 20) = -1 ENOMEM (Cannot allocate memory)
[...]
setsockopt(3, SOL_IPV6, IPV6_ADD_MEMBERSHIP, {inet_pton(AF_INET6, "ff02::2", &ipv6mr_multiaddr), ipv6mr_interface=if_nametoindex("tap4")}, 20) = -1 ENOMEM (Cannot allocate memory)
[...]
setsockopt(3, SOL_IPV6, IPV6_ADD_MEMBERSHIP, {inet_pton(AF_INET6, "ff02::2", &ipv6mr_multiaddr), ipv6mr_interface=if_nametoindex("tap3")}, 20) = -1 ENOMEM (Cannot allocate memory)
[...]
setsockopt(3, SOL_IPV6, IPV6_ADD_MEMBERSHIP, {inet_pton(AF_INET6, "ff02::2", &ipv6mr_multiaddr), ipv6mr_interface=if_nametoindex("tap1")}, 20) = -1 ENOMEM (Cannot allocate memory)
[...]

Its argument in this context are :

3: the target socket file descriptor, that will always be the same
SOL_IPV6: IPv6 protocol
IPV6_ADD_MEMBERSHIP: States that we want to join a multicast group
{inet_pton(AF_INET6, "ff02::2", &ipv6mr_multiaddr), ipv6mr_interface=if_nametoindex("tap1")}: structure that contains options about the multicast group we aim to join
- net_pton(AF_INET6, "ff02::2", &ipv6mr_multiaddr): converts the ff02::2 IP address from a string to a binary sequence
- ipv6mr_interface=if_nametoindex("tap1"): target interface index, here tap1
20: the structure’s length

We basically try to join an IPv6 multicast group using tap1. This however results in an ENOMEM error. That means we run out of memory, but why? The server’s memory is absolutely not a concern, there’s even a lot of it that’s free!

So, it will be necessary to analyze in depth to determine exactly where this ENOMEM is being returned.

Digging a little deeper

We used the ftrace tool, which can track the execution of various kernel-side functions that compose the setsockopt syscall. ftrace or function tracer is an internal tracer in the Linux kernel that allows analysis, debugging, and examination of kernel activities via the tracefs filesystem.

$ cd /sys/kernel/tracing
# Request ftrace to trace only the radv process
$ echo  > set_ftrace_pid
# Trace only function calls
$ echo function_graph > current_tracer

To reproduce the issue, we create a network interface in tap mode and launch a VM attempting to boot over network, which will generate a router advertisement from radvd (and thus trigger this setsockopt() call)

$ ip tuntap add mode tap tap1
$ ip link set tap1 up
$ ip addr add fd12:0:0:a08::$1/64 dev tap1
$ QEMU="qemu-system-x86_64 -boot n -net nic -net tap,ifname=tap$1,script=no,downscript=no -nographic"

We now stop ftrace:

$ echo nop > /sys/kernel/tracing/current_tracer

Trace has been shrunk for the sake of clarity

$ cat /sys/kernel/tracing/trace
 60) ! 304.905 us  |  } /* syscall_trace_enter.constprop.0 */
 60)               |  __x64_sys_setsockopt() {
 60)               |    __sys_setsockopt() {
 60)               |      sockfd_lookup_light() {
 60)               |        __fdget() {
 60)   0.376 us    |          __fget_light();
 60)   0.995 us    |        }
 60)   1.719 us    |      }
[..]
 60)               |      sock_common_setsockopt() {
 60)               |        rawv6_setsockopt() {
 60)               |          ipv6_setsockopt() {
 60)               |            do_ipv6_setsockopt() {
 60)               |              rtnl_lock() {
 60)               |                mutex_lock() {
 60)   0.308 us    |                  __cond_resched();
 60)   0.866 us    |                }
 60)   1.440 us    |              }
 60)               |              sockopt_lock_sock() {
 60)   0.285 us    |                __cond_resched();
 60)               |                _raw_spin_lock_bh() {
 60)   0.321 us    |                  preempt_count_add();
 60)   0.998 us    |                }
 60)               |                _raw_spin_unlock_bh() {
 60)               |                  __local_bh_enable_ip() {
 60)   0.278 us    |                    preempt_count_sub();
 60)   0.860 us    |                  }
 60)   1.400 us    |                }
 60)   4.289 us    |              }
 60)               |              ipv6_sock_mc_join() {
 60)               |                __ipv6_sock_mc_join() {
 60)               |                  rtnl_is_locked() {
 60)   0.286 us    |                    mutex_is_locked();
 60)   0.864 us    |                  }
 60)   0.375 us    |                  sock_kmalloc();
 60)   6.348 us    |                }
 60)   7.243 us    |              }
 60)               |              sockopt_release_sock() {
 60)               |                release_sock() {
 60)               |                  _raw_spin_lock_bh() {
 60)   0.318 us    |                    preempt_count_add();
 60)   1.033 us    |                  }
 60)               |                  _raw_spin_unlock_bh() {
 60)               |                    __local_bh_enable_ip() {
 60)   0.307 us    |                      preempt_count_sub();
 60)   0.878 us    |                    }
 60)   1.407 us    |                  }
 60)   3.493 us    |                }
 60)   4.086 us    |              }
[..]
 60) + 22.281 us   |            }
 60) + 23.195 us   |          }
 60) + 23.953 us   |        }
 60) + 24.821 us   |      }
 60)   0.302 us    |      kfree();
 60) + 32.012 us   |    }
 60) + 32.633 us   |  }

Making use of the setsockopt syscall involves calling:

sock_common_setsockopt
rawv6_setsockopt
do_ipv6_setsockopt
sockopt_lock_sock
ipv6_sock_mc_join
__ipv6_sock_mc_join
sock_kmalloc
sockopt_release_sock

The ipv6_sock_mc_join (“ipv6 socket multicast join”) function catched our attention here since its role is to add the interface to the multicast group. It also calls sock_kmalloc.

(ipv6_sock_mc_join is a mere wrapper of __ipv6_sock_mc_join that just adds an argument).

static int __ipv6_sock_mc_join(struct sock *sk, int ifindex,
			       const struct in6_addr *addr, unsigned int mode)
{
	struct net_device *dev = NULL;
	struct ipv6_mc_socklist *mc_lst;
	struct ipv6_pinfo *np = inet6_sk(sk);
	struct net *net = sock_net(sk);
	int err;

	ASSERT_RTNL();

	if (!ipv6_addr_is_multicast(addr))
		return -EINVAL;

	for_each_pmc_socklock(np, sk, mc_lst) {
		if ((ifindex == 0 || mc_lst->ifindex == ifindex) &&
		    ipv6_addr_equal(&mc_lst->addr, addr))
			return -EADDRINUSE;
	}
[...]

struct ipv6_mc_socklist structure represents a multicast group member, as seen earlier in the trace: {inet_pton(AF_INET6, "ff02::2", &ipv6mr_multiaddr), ipv6mr_interface=if_nametoindex("tap1")}

addr: multicast group address
ifindex: index of the interface we want to join the multicast group with
sfmode: set to MCAST_EXCLUDE (this is the result of the use of the ipv6_sock_mc_join wrapper, that’s the mode argument of __ipv6_sock_mc_join)
rcu: Read-Copy-Update, a mechanism ensuring the reading of valid data in the context of data read much more frequently than written, concurrently (see: https://www.kernel.org/doc/html/next/RCU/whatisRCU.html)

struct ipv6_mc_socklist {
	struct in6_addr		addr;
	int			ifindex;
	unsigned int		sfmode;		/* MCAST_{INCLUDE,EXCLUDE} */
	struct ipv6_mc_socklist __rcu *next;
	struct ip6_sf_socklist	__rcu *sflist;
	struct rcu_head		rcu;
};

This struct is 56 bytes long on a 6.1 kernel.

gdb /usr/lib/debug/lib/modules/6.1.0-18-amd64/vmlinux
GNU gdb (Debian 13.1-3) 13.1
(gdb) print sizeof(struct ipv6_mc_socklist)
$1 = 56
(gdb)

After a few checks the sock_kmalloc function is then called, with the following arguments :

sk: current socket structure
sizeof(struct ipv6_mc_socklist): added structure length
GFP_KERNEL: Get Free Page flag, controls memory allocator behavior (see: https://www.kernel.org/doc/html/next/core-api/memory-allocation.html)

[__ipv6_sock_mc_join continued]
	mc_lst = sock_kmalloc(sk, sizeof(struct ipv6_mc_socklist), GFP_KERNEL);

	if (!mc_lst)
		return -ENOMEM; <- here we go

[...]

sock_kmalloc seems to yield NULL since what we get from ipv6_sock_mc_join is an ENOMEM and no other function in the code has been executed (see ftrace previous output)

/*
 * Allocate a memory block from the socket's option memory buffer.
 */
void *sock_kmalloc(struct sock *sk, int size, gfp_t priority)
{
	int optmem_max = READ_ONCE(sock_net(sk)->core.sysctl_optmem_max);

	if ((unsigned int)size <= optmem_max &&
	    atomic_read(&sk->sk_omem_alloc) + size < optmem_max) {
		void *mem;
		/* First do the add, to avoid the race if kmalloc
		 * might sleep.
		 */
		atomic_add(size, &sk->sk_omem_alloc);
		mem = kmalloc(size, priority);
		if (mem)
			return mem;
		atomic_sub(size, &sk->sk_omem_alloc);
	}
	return NULL;
}

sk_omem_alloc (socket option/other memory alloc) is an atomic_t variable type (that’s an int32_t) that holds the grand total of every memory allocation that were made on this socket for the option buffer (see atomic_add(size, &sk->sk_omem_alloc);)

optmem_max is the net.core.optmem_max sysctl value. That’s the maximal memory size that can be allocated for a socket’s option buffer. This is set to 20480 on our hypervisors.

$ sysctl net.core.optmem_max
net.core.optmem_max = 20480

Basically there is a maximum size for the option buffer of a socket, which is set by the net.core.optmem_max sysctl setting. The struct ipv6_mc_socklist structure that stores multicast group member interfaces data is 56 bytes long. In our setup, it takes 365 group members to entirely fill the option buffer of that socket, taking 204440 bytes. The 366th member will result in an ENOMEM as 20486 bytes would be required to store this new member alongside the existing ones.

In theory, this buffer should not be entierly filled, however radvd does not remove any interface from the multicast group when an interface is removed (tap VM interface are dynamically created and removed on-the-fly when a VM boots). The more connections there are to this server, the less space is available in the “option” buffer. This can only be fixed via a service restart (we can also increase the value of net.core.optmem_max but the issue will come back if the value is not high enough, and this is a very temporary fix)

The fix

We saw earlier that we can add members to an IPv6 multicast group by using IPV6_ADD_MEMBERSHIP. Shouldn’t we remove them altogether? There’s a setsockopt option named IPV6_DROP_MEMBERSHIP whose goal is to remove an interface from a multicast group. That’s the sole principle of the cleanup_iface function within radvd’s code:

int cleanup_iface(int sock, struct Interface *iface)
{
	/* leave the allrouters multicast group */
	cleanup_allrouters_membership(sock, iface);
	return 0;
}

Where cleanup_allrouters_membership is:

int cleanup_allrouters_membership(int sock, struct Interface *iface)
{
	struct ipv6_mreq mreq;

	memset(&mreq, 0, sizeof(mreq));
	mreq.ipv6mr_interface = iface->props.if_index;

	/* ipv6-allrouters: ff02::2 */
	mreq.ipv6mr_multiaddr.s6_addr32[0] = htonl(0xFF020000);
	mreq.ipv6mr_multiaddr.s6_addr32[3] = htonl(0x2);
	setsockopt(sock, SOL_IPV6, IPV6_DROP_MEMBERSHIP, &mreq, sizeof(mreq));
	return 0;
}

cleanup_iface only gets called at the end of the program’s execution. Fix would be to call cleanup_iface for each interface deletion event:

	if (nh->nlmsg_type == RTM_DELLINK) {
		dlog(LOG_INFO, 4, "netlink: %s removed, cleaning up", iface->props.name);
		cleanup_iface(icmp_sock, iface);
	}

A patch has been submitted to radvd.

Thanks to @xdbob for the patch, Claire for the network debugging and @cedricmaunoury for the help during the bug search!

We did not have any logs due to radvd logging by default to stdout and then daemonized itself, closing those fd ↩︎

Retrieving the temperature of an NVIDIA GPU from a GTX 1080 on Linux, without nouveau or nvidia_drm

sanecz — Fri, 03 Mar 2023 09:00:00 -0700

I have several NVIDIA GPUs (~25,000) in passthrough on VMs, but I need to monitor their temperature from the hypervisor (qemu). There’s no nvidia_smi available, no nouveau or nvidia_drm driver loaded, only vfio-pci and qemu here.

So, I dug into it to retrieve their temperatures from the hypervisor!

All tests will be conducted on a GTX 1080 (GP104). This technique works on the Pascal and Ampere families. The same logic could possibly be applied to other families; however, I’m unable to test it.

How do we start ?

The envytools website provides a starting point for research and offers some leads, including the ranges of MMIO (memory mapped i/o) registers as well as the description of PCI BARs (base register addresses). Particularly, this includes the PTHERM register. The other lead is the kernel itself, hoping that the nouveau driver can retrieve the GPU temperature (spoiler: yes). It’s just a matter of digging into the code to better understand what might be happening. In any case, a more specific idea is emerging: reading from the GPU memory, somewhere.

Let’s get some reading

Snippet of code to retrieve the GPU temperature: source: linux/drivers/gpu/drm/nouveau/nvkm/subdev/therm/gp100.c

static int
gp100_temp_get(struct nvkm_therm *therm)
{
        struct nvkm_device *device = therm->subdev.device;
        struct nvkm_subdev *subdev = &therm->subdev;
        u32 tsensor = nvkm_rd32(device, 0x020460);
        u32 inttemp = (tsensor & 0x0001fff8);

        /* device SHADOWed */
        if (tsensor & 0x40000000)
                nvkm_trace(subdev, "reading temperature from SHADOWed sensor\n");

        /* device valid */
        if (tsensor & 0x20000000)
                return (inttemp >> 8);
        else
                return -ENODEV;
}

source: linux/drivers/gpu/drm/nouveau/include/nvkm/core/device.h

// Structure reduced for readability
struct nvkm_device {
    [...]
    struct device *dev;
    const char *name;
    [...]
    void __iomem *pri;
    [...]
    struct list_head subdev;
    [...]
};
[...]

#define nvkm_rd32(d,a) ioread32_native((d)->pri + (a))

The sensor value is retrieved using the nvkm_rd32 macro, which reads a 32-bit register at the offset device->pri + 0x020460. Masks are then applied to the sensor value to obtain only the temperature. All of this needs to be kept in mind.

source: linux/drivers/gpu/drm/nouveau/nkvm/engine/device/base.c

    [...]
    mmio_base = device->func->resource_addr(device, 0);
    mmio_size = device->func->resource_size(device, 0);
    [...]
    device->pri = ioremap(mmio_base, mmio_size);

ioremap will map the MMIO to a virtual memory space and return a pointer to the beginning of this space corresponding to the start of the GPU’s physical memory at the requested address. Hence the use of ioread32 since it is not recommended to directly dereference these addresses.

What do mmio_base and mmio_size correspond to? According to the envy documentation, the memory space of the MMIO registers is located on BAR 0 and is 16M in size. It is a 32-bit memory addressing area that is non-prefetchable.

So, mmio_base corresponds to the starting address of BAR 0, and mmio_size is 16M. This can also be verified using lspci.

# lspci -d 10de: -s .0 -vv
02:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1080] (rev a1) (prog-if 00 [VGA controller])
Subsystem: ZOTAC International (MCO) Ltd. GP104 [GeForce GTX 1080]
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Interrupt: pin A routed to IRQ 26
NUMA node: 0
IOMMU group: 45
//   BAR 0         mmio_base                                mmio_size
//     v               v                                       v
Region 0: Memory at c6000000 (32-bit, non-prefetchable) [size=16M]
Region 1: Memory at 27fe0000000 (64-bit, prefetchable) [size=256M]
Region 3: Memory at 27ff0000000 (64-bit, prefetchable) [size=32M]
Region 5: I/O ports at 6000 [size=128]
Expansion ROM at c7000000 [disabled] [size=512K]
[...]
Kernel driver in use: vfio-pci

In practice

Now we need to write a small piece of code that allows accessing my GPU via vfio, and read a value from the sensor at the previously retrieved offset, then apply the masks to extract the GPU temperature.

The code is quite basic and comes from the kernel documentation about vfio. In this example, I want to access the device 02:00.0 which is located in the IOMMU group 45 (seen in the above lspci output).

#include 
#include 
#include 
#include 
#include 
#include 
#include 

int main() {
  int container, group, device, i;
 struct vfio_group_status group_status =
   { .argsz = sizeof(group_status) };
 struct vfio_iommu_type1_info iommu_info = { .argsz = sizeof(iommu_info) };
 struct vfio_iommu_type1_dma_map dma_map = { .argsz = sizeof(dma_map) };
 struct vfio_device_info device_info = { .argsz = sizeof(device_info) };

 /* Create a new container */
 container = open("/dev/vfio/vfio", O_RDWR);

 if (ioctl(container, VFIO_GET_API_VERSION) != VFIO_API_VERSION)
   return -1;
 /* Unknown API version */

 if (!ioctl(container, VFIO_CHECK_EXTENSION, VFIO_TYPE1_IOMMU))
   return -1;
 /* Doesn't support the IOMMU driver we want. */

 /* Open the group */
 group = open("/dev/vfio/45", O_RDONLY);
 if (group < 0)
   return -1;

 /* Test the group is viable and available */
 ioctl(group, VFIO_GROUP_GET_STATUS, &group_status);

 if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE))
   /* Group is not viable (ie, not all devices bound for vfio) */
   return -1;

 /* Add the group to the container */
 ioctl(group, VFIO_GROUP_SET_CONTAINER, &container);

 /* Enable the IOMMU model we want */
 ioctl(container, VFIO_SET_IOMMU, VFIO_TYPE1_IOMMU);

 /* Get addition IOMMU info */
 ioctl(container, VFIO_IOMMU_GET_INFO, &iommu_info);

 /* Allocate some space and setup a DMA mapping */
 dma_map.vaddr = mmap(0, 1024 * 1024, PROT_READ | PROT_WRITE,
		      MAP_PRIVATE | MAP_ANONYMOUS, 0, 0);
 dma_map.size = 1024 * 1024;
 dma_map.iova = 0; /* 1MB starting at 0x0 from device view */
 dma_map.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE;

 ioctl(container, VFIO_IOMMU_MAP_DMA, &dma_map);

 /* Get a file descriptor for the device */
 device = ioctl(group, VFIO_GROUP_GET_DEVICE_FD, "0000:02:00.0");

 /* Test and setup the device */
 ioctl(device, VFIO_DEVICE_GET_INFO, &device_info);

 /* Working only on BAR 0 */
 struct vfio_region_info regs = {
   .argsz = sizeof(struct vfio_region_info),
   .index  = 0
 };

 ioctl(device, VFIO_DEVICE_GET_REGION_INFO, &regs);

 uint8_t* ptr = mmap(0, regs.size, PROT_READ, MAP_SHARED, device, 0);

 /* Stolen from you know where ;) */
 uint32_t tsensor = *(uint32_t*)(ptr + 0x020460);
 uint32_t inttemp = (tsensor & 0x0001fff8);

 if (tsensor & 0x40000000)
   printf("shadowed sensor\n");

 if (tsensor & 0x20000000)
   printf("temp %d\n", inttemp >> 8);

 /* Gratuitous device reset and go... */
 ioctl(device, VFIO_DEVICE_RESET);
 munmap(ptr, regs.size);

 return 0;
}

# gcc vfio.c && ./a.out
temp 28

Here we can see that the GPU is at 28°C (they are water-cooled). Unfortunately, the last problem is that if the GPU is already in use, for example via qemu, the open() call on the IOMMU group will return that it is already in use. This time there is another solution, which is to directly access memory with qemu monitor, using the xp command.

I start a VM using this GPU, and using the base address retrieved from BAR0 (with pci info), we will access it by adding the offset of the register we previously retreived containing the temperature:

(qemu) info pci
[...]
  Bus  2, device   0, function 0:
    VGA controller: PCI device 10de:1b80
      PCI subsystem 19da:1425
      IRQ 0, pin A
      BAR0: 32 bit memory at 0xc1000000 [0xc1ffffff].
      BAR1: 64 bit prefetchable memory at 0x1000000000 [0x100fffffff].
      BAR3: 64 bit prefetchable memory at 0x1010000000 [0x1011ffffff].
      BAR5: I/O at 0xb000 [0xb07f].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0007fffe].
[..]
# We read a 32-bit register at position 0xc1020460
(qemu) xp /1w (0xc1000000 + 0x020460)
00000000c1020460: 0x20001ca0

# We apply the shifts retrieved from the code in nouveau
In [1]: (0x20001ca0 & 0x0001fff8) >> 8
Out[1]: 28 <-- la temperature

Conclusion

With all this, it is possible to monitor the temperature of these GPUs very simply from the host, using these two techniques, provided that they do not interfere with the VM startup.

However, it’s not over yet; on some models, there are still other sensors to retrieve… It was a pretty fun little project; maybe I’ll try it on other families of NVIDIA GPUs if I can get my hands on them, or even maybe some AMD GPUs.