~blog

Livepatching the Linux kernel : through the eyes of an SRE

Posted at — Oct 18, 2026

# Introduction

Kernel live patching, or KLP, is a feature merged in the kernel 4.0 to apply fixes to the running kernel without requiring a reboot.

This might come in handy when you need to patch a bug or a security issue and cannot reboot the machine due to critical applications running on it, or need to minimize the downtime. KLP allows in-place patching of various functions in the kernel under some conditions, produces no service interruption (if patched properly) and is really fast to apply, but might require (a lot of) extra work.

In this article, we will first write a livepatch by hand for Januscape, let kpatch-build do the job for Pwnescape and then explore cumulative and atomic replaces with a Zapscape and Pwnescape.

Disclaimer: livepatches shown in this article were written and tested on my own test machines built for specific kernel versions. They are here to illustrate how livepatching works, not to be loaded on your production servers. If you do it anyway, keep in mind it may crash your server as our environments and kernel builds are not identical.
Use at your own risk, I’m not responsible for your hypervisors (I already have mine to deal with).

# Livepatch

## How does it work ?

Livepatch relies on ftrace. When the kernel is built with function tracing enabled, the compiler adds a call to __fentry__ at the beginning of every function (-pg flag/-mfentry on x86_64). At boot, ftrace replaces all of these calls with a nop instruction, so they cost almost nothing when tracing is not used.

All the functions that can be traced are listed in /sys/kernel/tracing/available_filter_functions.

When a function is livepatched, KLP registers an ftrace handler on it, and the nop at the beginning of the old function becomes a call to this handler again. The handler then redirects the execution to the new function. The old function is still there and callers still call it, but it never runs its own code anymore.

## Requirements

For KLP to work, you’ll need a kernel built with at least the following configuration: CONFIG_MODULES to enable loadable module support, CONFIG_LIVEPATCH/CONFIG_HAVE_LIVEPATCH, CONFIG_KALLSYMS, CONFIG_KALLSYMS_ALL, CONFIG_DYNAMIC_FTRACE_WITH_REGS or CONFIG_DYNAMIC_FTRACE_WITH_ARGS, with a kernel version >= 4.0.

CONFIG_HAVE_RELIABLE_STACKTRACE provides reliable stack traces used by KLP to known when a task can be switched to the new function. Without it, the patching transition might never complete.

CONFIG_TRIM_UNUSED_KSYMS should be disabled.

if CONFIG_X86_KERNEL_IBT is enabled, on Intel 11th gen+ CPUs and unless the kernel is booted with ibt=off or ibt=warn, calling functions resolved through kallsyms_lookup_name may trigger a control-protection exception and crash the host. Also kpatch-build does not support this option

The exact requirements depend on the architecture and kernel version. This is only a checklist for a x86_64 configuration used in this article, not a universal recipe.

grep -E 'CONFIG_(LIVEPATCH|HAVE_LIVEPATCH|MODULES|KALLSYMS|KALLSYMS_ALL|TRIM_UNUSED_KSYMS|DYNAMIC_FTRACE_WITH_(REGS|ARGS)|UNWINDER_ORC|HAVE_RELIABLE_STACKTRACE|X86_KERNEL_IBT)[= ]' /boot/config-$(uname -r)

## Limitations

Only traced functions can be patched. This avoids issues like trying to patch functions implemented for ftrace itself or low-level architecture code. Tracing can be disabled per file by removing the function profiling flag from the compilation:

arch/x86/kernel/Makefile:CFLAGS_REMOVE_kvmclock.o = -pg
arch/x86/kernel/Makefile:CFLAGS_REMOVE_ftrace.o = -pg
arch/x86/kernel/Makefile:CFLAGS_REMOVE_early_printk.o = -pg
arch/x86/kernel/Makefile:CFLAGS_REMOVE_head64.o = -pg

arch/arm64/kernel/Makefile:CFLAGS_REMOVE_ftrace.o = $(CC_FLAGS_FTRACE)
arch/arm64/kernel/Makefile:CFLAGS_REMOVE_insn.o = $(CC_FLAGS_FTRACE)
arch/arm64/kernel/Makefile:CFLAGS_REMOVE_return_address.o = $(CC_FLAGS_FTRACE)

Functions using the notrace macro cannot be patched either as they won’t be available to ftrace, same for the inline functions because the kernel redefines inline and adds the notrace macro:

#define inline inline __gnu_inline __inline_maybe_unused notrace

Patching functions inlined by the compiler requires more tricks: the function doesn’t exist as a standalone function anymore and its code is copied into each caller, so patching it means patching all of its callers instead.

The livepatch replaces the original function with your new one, so you cannot call the original function from your patch. The call would go through its ftrace hook, then be redirected to the new function, creating an infinite recursion. You’ll have to copy the original code and modify it.

It only replace code, not data. If a fix changes a structure (adding a new field, bigger array etc etc), it can’t be patched as-is, the existing objects were allocated with the old structure layout, and the rest of the kernel keep using it. There is some workaround using shadow variables (klp_shadow_alloc(), klp_shadow_get()). It attaches extra data to an existing object stored on the side of the structure instead of inside of it. It does work but the livepatch is considerably harder to write.

Tracing tools are also affected. Since the original function never runs its own code anymore, kprobes placed inside its body (systemtap, perf probe, bpftrace kprobe probes with an offset) stop firing once it’s patched. A probe on the function entry shares the same ftrace call site as livepatch, it may still fire before the redirection, but it only tells you the old function was called, not that its code ran. Probes have to target the new function instead, those living in the livepatch module.

kretprobes can conflict with livepatch, as both rely on an ftrace handler that modifies the return address. Only one of them can acquire the handler: “the first user wins” and the other one is rejected.

# Writing a livepatch by hand

## Anatomy of a livepatch

A livepatch is just a kernel module with three structures that define the behavior of the patch, and your replacement code. Those nested structures define what to patch, where to patch and the initialization. This base livepatch comes from samples/livepatch/livepatch-sample.c.

Mandatory includes:

#include <linux/module.h> /* THIS_MODULE, MODULE_LICENSE, MODULE_INFO, module_init, module_exit */
#include <linux/livepatch.h> /* klp_func, klp_object, klp_patch, klp_enable_patch */

Your new function, where KLP redirects callers to in place of the original:

#include <linux/seq_file.h>
static int livepatch_cmdline_proc_show(struct seq_file *m, void *v)
{
	seq_printf(m, "%s\n", "this has been live patched");
	return 0;
}

struct klp_func: tells KLP which function to replace with the new one. Since funcs[] is an array, it can contain multiple functions to be replaced.

static struct klp_func funcs[] = {
    { .old_name = "cmdline_proc_show",
      .new_func = livepatch_cmdline_proc_show, },
    { }   /* NULL terminator — required */
};

struct klp_object: tells KLP where the functions live.

static struct klp_object objs[] = {
    { .name = NULL, .funcs = funcs, }, /*  .name = NULL means vmlinux */
    { }   /* NULL terminator */
};

struct klp_patch: top-level structure, ties your struct klp_object to this livepatch module.

static struct klp_patch patch = {
    .mod = THIS_MODULE,
    .objs = objs,
};

And finally, the init/exit and MODULE_INFO, otherwise it wouldn’t be a working kernel module. livepatch_init() will call klp_enable_patch() to start the patching transition.

static int livepatch_init(void) { return klp_enable_patch(&patch); }
static void livepatch_exit(void) {}
module_init(livepatch_init);
module_exit(livepatch_exit);
MODULE_LICENSE("GPL");
MODULE_INFO(livepatch, "Y");

klp_enable_patch() will return before the patching is finished. There is a transition period where KLP switches tasks from the old function to our newly patched function, only when the task is no longer inside the old function. The same process applies when removing the patch: we first need to unpatch, wait for the transition, then unload the module.

To build this module, we will use the following Makefile:

obj-m := livepatch-cmdline.o
# KDIR will point to the build tree of the running kernel, you might need to install linux-headers-$(uname -r)
KDIR := /lib/modules/$(shell uname -r)/build
all:
	$(MAKE) -C $(KDIR) M=$(PWD) modules
clean:
	$(MAKE) -C $(KDIR) M=$(PWD) clean

Then run make, which will produce livepatch-cmdline.ko, then load the module:

# make
make -C /lib/modules/6.2.0-39-generic/build M=/home/lisa modules
make[1]: Entering directory '/usr/src/linux-headers-6.2.0-39-generic'
  CC [M]  /home/lisa/livepatch-cmdline.o
  MODPOST /home/lisa/Module.symvers
  CC [M]  /home/lisa/livepatch-cmdline.mod.o
  LD [M]  /home/lisa/livepatch-cmdline.ko
  BTF [M] /home/lisa/livepatch-cmdline.ko
Skipping BTF generation for /home/lisa/livepatch-cmdline.ko due to unavailability of vmlinux
make[1]: Leaving directory '/usr/src/linux-headers-6.2.0-39-generic'
# insmod livepatch-cmdline.ko

With dmesg we can follow the status of the patching transition:

# dmesg
[...]
[79191.929967] livepatch: 'livepatch_cmdline': starting patching transition
[79192.818083] livepatch: 'livepatch_cmdline': patching complete
# cat /proc/cmdline
this have been livepatched

livepatch-cmdline.ko becomes livepatch_cmdline because the kernel replaces the dashes in module names with underscores.

Loading the patch is not enough to be sure that the livepatch took effect. Looking at /sys/kernel/livepatch/ will give more data about the state of the livepatch:

# ls /sys/kernel/livepatch/livepatch_cmdline/
enabled     force       transition  vmlinux/
# cat /sys/kernel/livepatch/livepatch_cmdline/enabled
1
# cat /sys/kernel/livepatch/livepatch_cmdline/vmlinux/patched
1
# ls  /sys/kernel/livepatch/livepatch_cmdline/vmlinux/
cmdline_proc_show,1  patched

livepatch_cmdline/enabled returning 1 means the livepatch is active, then vmlinux is our patched object and the function patched is cmdline_proc_show.

If /sys/kernel/livepatch/livepatch_cmdline/transition stays at 1, it means that the transition isn’t done yet. Either a task is still sleeping inside the old function and will be switched once it returns, or it’s a long-running task that never leaves the old function. The blocking tasks can be found with /proc/<pid>/patch_state, which shows 0 while a task is still unpatched. The kernel automatically sends them a fake signal to try to make them leave the old function, which might help some tasks but not those that never leave the old function. In that case, the transition might never complete.

Disabling the livepatch:

# echo 0 >/sys/kernel/livepatch/livepatch_cmdline/enabled
# rmmod livepatch-cmdline
# dmesg
[...]
[79397.945201] livepatch: 'livepatch_cmdline': starting unpatching transition
[79399.862745] livepatch: 'livepatch_cmdline': unpatching complete

The directory /sys/kernel/livepatch/livepatch_cmdline also contains a force file. Writing 1 to it marks all the remaining task as patched to end a stuck transition. If you force it, a task may still be running the old code, and the module can no longer be removed afterwards. This totally bypass the task-consistency checks and other alternatives should really be considered before using the force file.

## Livepatching Januscape

Januscape is the second of three KVM escapes reported by V4Bel. According to V4bel, it is the first KVM/x86 vulnerability that can trigger either on Intel or AMD architectures. This exploit, correctly executed, might allow an attacker to execute code on the host and the available POC cause denial of service by triggering a kernel panic on the host.

To write the Januscape livepatch, we will need to implement two fixes from two separate commits: 81ccda30b4e8 and 0cb2af2ea66ad.

The first commit 81ccda30b4e8 patches kvm_mmu_get_child_sp:

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 9368a71336fe4..c13b80fe3125b 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -2459,13 +2459,15 @@ static struct kvm_mmu_page *kvm_mmu_get_child_sp(struct kvm_vcpu *vcpu,
 						 u64 *sptep, gfn_t gfn,
 						 bool direct, unsigned int access)
 {
-	union kvm_mmu_page_role role;
+	union kvm_mmu_page_role role = kvm_mmu_child_role(sptep, direct, access);

-	if (is_shadow_present_pte(*sptep) && !is_large_pte(*sptep) &&
-	    spte_to_child_sp(*sptep) && spte_to_child_sp(*sptep)->gfn == gfn)
+	if (is_shadow_present_pte(*sptep) &&
+	    !is_large_pte(*sptep) &&
+	    spte_to_child_sp(*sptep) &&
+	    spte_to_child_sp(*sptep)->gfn == gfn &&
+	    spte_to_child_sp(*sptep)->role.word == role.word)
 		return ERR_PTR(-EEXIST);

-	role = kvm_mmu_child_role(sptep, direct, access);
 	return kvm_mmu_get_shadow_page(vcpu, gfn, role);
 }

Before writing any code, we need to ensure that the function we want to replace actually exists and can be traced by ftrace:

# grep -E ' kvm_mmu_get_child_sp' /proc/kallsyms
ffffffffc10c1900 t kvm_mmu_get_child_sp	[kvm]
# grep -E '^kvm_mmu_get_child_sp ' /sys/kernel/tracing/available_filter_functions
kvm_mmu_get_child_sp [kvm]

Note that KLP doesn’t care whether kvm_mmu_get_child_sp is exported: the livepatch core finds the function to replace by looking it up in kallsyms itself.
However, if the compiler inlined the function into some of its callers, those callers would have to be patched too. It can be checked by looking at the disassembly of the module.

We will start with a naive implementation of this function in our livepatch. The struct klp_object will now contain the kvm module to patch instead of NULL, as we are not patching vmlinux functions anymore.

Click here to display full code

#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include <linux/module.h>
#include <linux/kernel.h> /* pr_info, pr_err */
#include <linux/livepatch.h>
#include <linux/kvm_host.h> /* struct kvm_vcpu */

static struct kvm_mmu_page *livepatch_kvm_mmu_get_child_sp(struct kvm_vcpu *vcpu,
                          u64 *sptep, gfn_t gfn,
                          bool direct, unsigned int access)
{
    union kvm_mmu_page_role role = kvm_mmu_child_role(sptep, direct, access);

    if (is_shadow_present_pte(*sptep) &&
        !is_large_pte(*sptep) &&
        spte_to_child_sp(*sptep) &&
        spte_to_child_sp(*sptep)->gfn == gfn &&
        spte_to_child_sp(*sptep)->role.word == role.word)
        return ERR_PTR(-EEXIST);
    return kvm_mmu_get_shadow_page(vcpu, gfn, role);
}

static struct klp_func funcs[] = {
    { .old_name = "kvm_mmu_get_child_sp", .new_func = livepatch_kvm_mmu_get_child_sp, },
    { }
};
static struct klp_object objs[] = {
    { .name = "kvm", .funcs = funcs, },
    { }
};
static struct klp_patch patch = { .mod = THIS_MODULE, .objs = objs, };

static int livepatch_init(void) {
       int ret;
       ret = klp_enable_patch(&patch);
       if (ret) { pr_err("klp_enable_patch failed: %d\n", ret); return ret; }
       pr_info("CVE-2026-53359 patch applied\n");
       return 0;
}
static void livepatch_exit(void) {}
module_init(livepatch_init);
module_exit(livepatch_exit);
MODULE_DESCRIPTION("CVE-2026-53359 (Januscape) livepatch");
MODULE_LICENSE("GPL");
MODULE_INFO(livepatch, "Y");

Unfortunately, if we try to build this module, the compiler will throw a bunch of errors as none of the functions like kvm_mmu_child_role are declared anywhere.

[...]
/home/lisa/livepatch.c: In function ‘livepatch_kvm_mmu_get_child_sp’:
/home/lisa/livepatch.c:11:36: error: implicit declaration of function ‘kvm_mmu_child_role’ [-Werror=implicit-function-declaration]
   11 |     union kvm_mmu_page_role role = kvm_mmu_child_role(sptep, direct, access);
      |                                    ^~~~~~~~~~~~~~~~~~
/home/lisa/livepatch.c:11:36: error: invalid initializer
/home/lisa/livepatch.c:13:9: error: implicit declaration of function ‘is_shadow_present_pte’ [-Werror=implicit-function-declaration]
   13 |     if (is_shadow_present_pte(*sptep) &&
      |         ^~~~~~~~~~~~~~~~~~~~~
/home/lisa/livepatch.c:14:10: error: implicit declaration of function ‘is_large_pte’ [-Werror=implicit-function-declaration]
   14 |         !is_large_pte(*sptep) &&
      |          ^~~~~~~~~~~~
/home/lisa/livepatch.c:15:9: error: implicit declaration of function ‘spte_to_child_sp’ [-Werror=implicit-function-declaration]
   15 |         spte_to_child_sp(*sptep) &&
      |         ^~~~~~~~~~~~~~~~
/home/lisa/livepatch.c:16:33: error: invalid type argument of ‘->’ (have ‘int’)
   16 |         spte_to_child_sp(*sptep)->gfn == gfn &&
[...]

The functions is_shadow_present_pte, is_large_pte, spte_to_child_sp and the kvm_mmu_page_role union are declared in arch/x86/kvm/mmu/mmu_internal.h and arch/x86/kvm/mmu/spte.h. To be able to use those headers, we also need to specify the include path. Note that those headers are subsystem internals and are not available via linux-headers.

Also, the source tree must match the running kernel exactly. Same version, same config. Otherwise some structures might have a different layout, the livepatch will still build, but read and write at the wrong fields once loaded.

We also need to declare a few symbols/macros that are used inside those headers (but declared in mmu.h).
Makefile:

ccflags-y := -I/usr/src/linux-source-6.2.0/arch/x86/kvm/mmu

livepatch.c:

extern bool enable_mmio_caching;

#include "mmu_internal.h"   /* arch/x86/kvm/mmu/mmu_internal.h */
#include "spte.h"           /* arch/x86/kvm/mmu/spte.h */

The functions kvm_mmu_get_shadow_page and kvm_mmu_child_role are static and not declared in any header, so they are not exported. We need to find those symbols elsewhere, for example inside the kernel symbol table: /proc/kallsyms.

# grep -E 'kvm_mmu_child_role|kvm_mmu_get_shadow_page' /proc/kallsyms
ffffffffc10b91d0 t kvm_mmu_child_role	[kvm]
ffffffffc10c0950 t __kvm_mmu_get_shadow_page	[kvm]
ffffffffc10b91c0 t __pfx_kvm_mmu_child_role	[kvm]
ffffffffc10c0940 t __pfx___kvm_mmu_get_shadow_page	[kvm]

kvm_mmu_child_role has the symbol type “t”, which means it’s in the text section but not exported, so we cannot call it directly: the module loader only resolves exported symbols. We also have __kvm_mmu_get_shadow_page instead of kvm_mmu_get_shadow_page, but we will get to this later.

If a symbol does not appear, it means it’s not resolvable, and it has probably been inlined and/or optimized away.

KLP actually has a proper way to do this: livepatch relocations. Instead of looking up the symbols ourselves, the livepatch module contains special relocation sections listing the non-exported symbols it needs, and the livepatch core resolves them itself when the patch is enabled, and again each time the target module is loaded. Unfortunately, these sections have a specific format that is not really meant to be written by hand, so we’ll go with the kallsyms workaround for now (spoiler: kpatch-build generates them for us).

Using kallsyms_lookup_name, we are able to resolve a symbol name to a memory address. Unfortunately, it has been unexported since kernel 5.7. The common workaround is to plant a kprobe on the symbol kallsyms_lookup_name (which is still present in the kernel) to get the address of this function.

Click here to display full code

#include <linux/kprobes.h>
[...]
typedef unsigned long (*kallsyms_lookup_name_t)(const char *name);
// defines the kprobe we want to plant
static struct kprobe kp = { .symbol_name = "kallsyms_lookup_name" };

[..]
static int livepatch_init(void)
{
	kallsyms_lookup_name_t my_kallsyms;
	int ret;
	// register the probe
	ret = register_kprobe(&kp);
	if (ret < 0) { pr_err("kprobe failed: %d\n", ret); return ret; }
	// retrieve the address of kallsyms_lookup_name
	my_kallsyms = (kallsyms_lookup_name_t)kp.addr;
	// release the probe
	unregister_kprobe(&kp);
	[...]
}

As kvm_mmu_child_role is a symbol present in the kernel, we will declare a function pointer with exactly the same signature as the original function, and store the address of kvm_mmu_child_role retrieved by my_kallsyms inside fn_child_role. Now we can replace the call to kvm_mmu_child_role inside livepatch_kvm_mmu_get_child_sp by fn_child_role.

Click here to display full code

[...]
typedef unsigned long (*kallsyms_lookup_name_t)(const char *name);
static struct kprobe kp = { .symbol_name = "kallsyms_lookup_name" };
static union kvm_mmu_page_role (*fn_child_role)(u64 *sptep, bool direct, unsigned int access);

static struct kvm_mmu_page *livepatch_kvm_mmu_get_child_sp(struct kvm_vcpu *vcpu,
                          u64 *sptep, gfn_t gfn,
                          bool direct, unsigned int access)
{
    union kvm_mmu_page_role role = fn_child_role(sptep, direct, access);
    [..]
}

[..]


static int livepatch_init(void)
{
        [...]
	fn_child_role = (void *)my_kallsyms("kvm_mmu_child_role");
	if (!fn_child_role) {
	    pr_err("can't resolve kvm_mmu_child_role\n");
	    return -ENOENT;
	}
	[...]
}

The last function to find is kvm_mmu_get_shadow_page. We can’t resolve the symbol as seen earlier, which means we will need to find a workaround.
Looking at the code of kvm_mmu_get_shadow_page, it is a wrapper around __kvm_mmu_get_shadow_page, which is available through kallsyms.

Click here to display full code

static struct kvm_mmu_page *kvm_mmu_get_shadow_page(struct kvm_vcpu *vcpu,
                                                    gfn_t gfn,
                                                    union kvm_mmu_page_role role)
{
        struct shadow_page_caches caches = {
                .page_header_cache = &vcpu->arch.mmu_page_header_cache,
                .shadow_page_cache = &vcpu->arch.mmu_shadow_page_cache,
                .shadowed_info_cache = &vcpu->arch.mmu_shadowed_info_cache,
        };

        return __kvm_mmu_get_shadow_page(vcpu->kvm, vcpu, &caches, gfn, role);
}

This function is not too complicated, which allows us to copy the code of kvm_mmu_get_shadow_page into our livepatch function.
Like kvm_mmu_child_role, the symbol for __kvm_mmu_get_shadow_page is available via kallsyms, so we will retrieve it the same way.

Click here to display full code

// first declare the struct
struct shadow_page_caches {
    struct kvm_mmu_memory_cache *page_header_cache;
    struct kvm_mmu_memory_cache *shadow_page_cache;
    struct kvm_mmu_memory_cache *shadowed_info_cache;
};
// prototype the __kvm_mmu_get_shadow_page
static struct kvm_mmu_page *(*fn_get_shadow_page)(struct kvm *kvm, struct kvm_vcpu *vcpu, struct shadow_page_caches *caches,  gfn_t gfn, union kvm_mmu_page_role role);

[..]
static struct kvm_mmu_page *livepatch_kvm_mmu_get_child_sp(struct kvm_vcpu *vcpu,
                          u64 *sptep, gfn_t gfn,
                          bool direct, unsigned int access)
{
    union kvm_mmu_page_role role = fn_child_role(sptep, direct, access);
    //  arch/x86/kvm/mmu/mmu.c:2226-2230
    struct shadow_page_caches caches = {
        .page_header_cache   = &vcpu->arch.mmu_page_header_cache,
        .shadow_page_cache   = &vcpu->arch.mmu_shadow_page_cache,
        .shadowed_info_cache = &vcpu->arch.mmu_shadowed_info_cache,
    };

    if (is_shadow_present_pte(*sptep) &&
        !is_large_pte(*sptep) &&
        spte_to_child_sp(*sptep) &&
        spte_to_child_sp(*sptep)->gfn == gfn &&
        spte_to_child_sp(*sptep)->role.word == role.word)
        return ERR_PTR(-EEXIST);

    // __kvm_mmu_get_shadow_page instead of kvm_mmu_get_shadow_page
    return fn_get_shadow_page(vcpu->kvm, vcpu, &caches, gfn, role);
}
[..]
static int livepatch_init(void)
{
        [...]
	// retrieve __kvm_mmu_get_shadow_page via my_kallsyms
	fn_get_shadow_page = (void *)my_kallsyms("__kvm_mmu_get_shadow_page");
    	if (!fn_get_shadow_page) { pr_err("can't resolve __kvm_mmu_get_shadow_page\n"); return -ENOENT; }

	[...]
}

Patching kvm_mmu_get_child_sp alone is not enough. The complete fix for Januscape is composed of two patches, and the second fix resides in __link_shadow_page. We must also check that __link_shadow_page can be patched:

# grep -E ' __link_shadow_page\s' /proc/kallsyms
ffffffffc10bf960 t __link_shadow_page	[kvm]
# grep -E '^__link_shadow_page ' /sys/kernel/tracing/available_filter_functions
__link_shadow_page [kvm]

Then the second patch (0cb2af2ea66ad) should be implemented:

@@ -2544,13 +2534,16 @@ static void __link_shadow_page(struct kvm *kvm,

 	BUILD_BUG_ON(VMX_EPT_WRITABLE_MASK != PT_WRITABLE_MASK);

-	/*
-	 * If an SPTE is present already, it must be a leaf and therefore
-	 * a large one.  Drop it, and flush the TLB if needed, before
-	 * installing sp.
-	 */
-	if (is_shadow_present_pte(*sptep))
-		drop_large_spte(kvm, sptep, flush);
+	if (is_shadow_present_pte(*sptep)) {
+		struct kvm_mmu_page *parent_sp;
+		LIST_HEAD(invalid_list);
+
+		parent_sp = sptep_to_sp(sptep);
+		WARN_ON_ONCE(parent_sp->role.level == PG_LEVEL_4K);
+
+		mmu_page_zap_pte(kvm, parent_sp, sptep, &invalid_list);
+		kvm_mmu_remote_flush_or_zap(kvm, &invalid_list, true);
+	}

So we need to add a second klp_func for __link_shadow_page and copy the whole function, since we cannot call the original:

Click here to display full code

[...]
static struct klp_func funcs[] = {
    { .old_name = "kvm_mmu_get_child_sp", .new_func = livepatch_kvm_mmu_get_child_sp, },
    { .old_name = "__link_shadow_page", .new_func = livepatch__link_shadow_page, },
    { }
};
[...]
static void livepatch__link_shadow_page(struct kvm *kvm,
			       struct kvm_mmu_memory_cache *cache, u64 *sptep,
			       struct kvm_mmu_page *sp, bool flush)
{
	u64 spte;

	BUILD_BUG_ON(VMX_EPT_WRITABLE_MASK != PT_WRITABLE_MASK);

	if (is_shadow_present_pte(*sptep)) {
		struct kvm_mmu_page *parent_sp;
		LIST_HEAD(invalid_list);

		parent_sp = sptep_to_sp(sptep);
		WARN_ON_ONCE(parent_sp->role.level == PG_LEVEL_4K);

		mmu_page_zap_pte(kvm, parent_sp, sptep, &invalid_list);
		kvm_mmu_remote_flush_or_zap(kvm, &invalid_list, true);
	}

	spte = make_nonleaf_spte(sp->spt, sp_ad_disabled(sp));

	mmu_spte_set(sptep, spte);

	mmu_page_add_parent_pte(cache, sp, sptep);

	if (sp->unsync_children || sp->unsync)
		mark_unsync(sptep);
}
[...]

We now have multiple symbols that we need to resolve or find a workaround for: BUILD_BUG_ON, is_shadow_present_pte, LIST_HEAD, sptep_to_sp, WARN_ON_ONCE, PG_LEVEL_4K, mmu_page_zap_pte, kvm_mmu_remote_flush_or_zap, make_nonleaf_spte, sp_ad_disabled, mmu_spte_set, mmu_page_add_parent_pte, mark_unsync.

BUILD_BUG_ON will be dropped, it is a macro only used for compile-time assertions to check masks.
LIST_HEAD/WARN_ON_ONCE are generic kernel macros and PG_LEVEL_4K is an enum constant.

Searching for symbols in /proc/kallsyms:

# grep -E 'is_shadow_present_pte|sptep_to_sp|mmu_page_zap_pte|kvm_mmu_remote_flush_or_zap|make_nonleaf_spte|sp_ad_disabled|mmu_spte_set|mmu_page_add_parent_pte|mark_unsync' /proc/kallsyms
ffffffffc10b92b0 t mark_unsync	[kvm]
ffffffffc10ba7e0 t mmu_page_zap_pte	[kvm]
ffffffffc10d7f47 t mmu_page_zap_pte.cold	[kvm]
ffffffffc10b92a0 t __pfx_mark_unsync	[kvm]
ffffffffc10ba7d0 t __pfx_mmu_page_zap_pte	[kvm]
ffffffffc10c98d0 t __pfx_make_nonleaf_spte	[kvm]
ffffffffc10c98e0 t make_nonleaf_spte	[kvm]

Search for the remaining functions in the kernel source tree:

grep --include='*.h' -rE 'is_shadow_present_pte|sptep_to_sp|mmu_page_zap_pte|kvm_mmu_remote_flush_or_zap|make_nonleaf_spte|sp_ad_disabled|mmu_spte_set|mmu_page_add_parent_pte|mark_unsync' /usr/src/linux-source-6.2.0/arch/x86
/usr/src/linux-source-6.2.0/arch/x86/kvm/mmu/spte.h:static inline struct kvm_mmu_page *sptep_to_sp(u64 *sptep)
/usr/src/linux-source-6.2.0/arch/x86/kvm/mmu/spte.h:static inline bool is_shadow_present_pte(u64 pte)
/usr/src/linux-source-6.2.0/arch/x86/kvm/mmu/spte.h:static inline bool sp_ad_disabled(struct kvm_mmu_page *sp)
/usr/src/linux-source-6.2.0/arch/x86/kvm/mmu/spte.h:u64 make_nonleaf_spte(u64 *child_pt, bool ad_disabled);

(Probably a real IDE instead of me running a bare emacs and grep would have been more efficient for reading code.)

We will retrieve mark_unsync, mmu_page_zap_pte and make_nonleaf_spte via kallsyms.

sptep_to_sp, is_shadow_present_pte and sp_ad_disabled are static inline functions inside spte.h (included earlier for the first patch).
mmu_page_zap_pte.cold is unlikely branch code split out by the compiler. As we do resolve the mmu_page_zap_pte symbol, we just ignore this part.

Click here to display full code

[...]
static int (*fn_mmu_page_zap_pte)(struct kvm *, struct kvm_mmu_page *, u64 *, struct list_head *);
static u64 (*fn_make_nonleaf_spte)(u64 *child_pt, bool ad_disabled);
static void (*fn_mark_unsync)(u64 *);
[...]

static void livepatch__link_shadow_page(struct kvm *kvm,
			       struct kvm_mmu_memory_cache *cache, u64 *sptep,
			       struct kvm_mmu_page *sp, bool flush)
{
	u64 spte;

	if (is_shadow_present_pte(*sptep)) {
[...]

		fn_mmu_page_zap_pte(kvm, parent_sp, sptep, &invalid_list);
		kvm_mmu_remote_flush_or_zap(kvm, &invalid_list, true);
	}

	spte = fn_make_nonleaf_spte(sp->spt, sp_ad_disabled(sp));

[...]

	if (sp->unsync_children || sp->unsync)
		fn_mark_unsync(sptep);
}

static int livepatch_init(void)
{
    [...]
    fn_mmu_page_zap_pte = (void *)my_kallsyms("mmu_page_zap_pte");
    if (!fn_mmu_page_zap_pte) { pr_err("can't resolve mmu_page_zap_pte\n"); return -ENOENT; }

    fn_make_nonleaf_spte = (void *)my_kallsyms("make_nonleaf_spte");
    if (!fn_make_nonleaf_spte) { pr_err("can't resolve make_nonleaf_spte\n"); return -ENOENT; }

    fn_mark_unsync = (void *)my_kallsyms("mark_unsync");
    if (!fn_mark_unsync) { pr_err("can't resolve mark_unsync\n"); return -ENOENT; }
    [...]
}

kvm_mmu_remote_flush_or_zap, mmu_spte_set and mmu_page_add_parent_pte were inlined away and have no symbol. These functions have to be rebuilt from the source code.

The original mmu_spte_set is a wrapper around __set_spte():

static void mmu_spte_set(u64 *sptep, u64 new_spte)
{
	WARN_ON(is_shadow_present_pte(*sptep));
	__set_spte(sptep, new_spte);
}
[...]
#ifdef CONFIG_X86_64
static void __set_spte(u64 *sptep, u64 spte)
{
	WRITE_ONCE(*sptep, spte);
}
[...]

This allows us to skip this function and directly implement the WARN_ON + WRITE_ONCE.

mmu_page_add_parent_pte is also a wrapper around pte_list_add, which is available via kallsyms and retrieved the same way as earlier.

Click here to display full code

[...]
static void livepatch__link_shadow_page(struct kvm *kvm,
			       struct kvm_mmu_memory_cache *cache, u64 *sptep,
			       struct kvm_mmu_page *sp, bool flush)
{
	u64 spte;

	if (is_shadow_present_pte(*sptep)) {
		struct kvm_mmu_page *parent_sp;
		LIST_HEAD(invalid_list);

		parent_sp = sptep_to_sp(sptep);
		WARN_ON_ONCE(parent_sp->role.level == PG_LEVEL_4K);

		fn_mmu_page_zap_pte(kvm, parent_sp, sptep, &invalid_list);
		kvm_mmu_remote_flush_or_zap(kvm, &invalid_list, true);
	}

	spte = fn_make_nonleaf_spte(sp->spt, sp_ad_disabled(sp));

	WARN_ON(is_shadow_present_pte(*sptep));
	WRITE_ONCE(*sptep, spte);

	fn_pte_list_add(cache, sptep, &sp->parent_ptes);

	if (sp->unsync_children || sp->unsync)
		fn_mark_unsync(sptep);
}

static int livepatch_init(void)
{
    [...]
    fn_pte_list_add = (void *)my_kallsyms("pte_list_add");
    if (!fn_pte_list_add) { pr_err("can't resolve pte_list_add\n"); return -ENOENT; }
    [...]
}

The last function to rebuild is kvm_mmu_remote_flush_or_zap, defined as:

static bool kvm_mmu_remote_flush_or_zap(struct kvm *kvm,
					struct list_head *invalid_list,
					bool remote_flush)
{
	if (!remote_flush && list_empty(invalid_list))
		return false;

	if (!list_empty(invalid_list))
		kvm_mmu_commit_zap_page(kvm, invalid_list);
	else
		kvm_flush_remote_tlbs(kvm);
	return true;
}

Both symbols resolve, but kvm_mmu_commit_zap_page.part.0 might be fragile to import: the function was split and partially inlined by the compiler. This might change (function not split, more parts, etc.) between compiler versions, optimizations and surrounding code:

# grep -E 'kvm_mmu_commit_zap_page|kvm_flush_remote_tlbs' /proc/kallsyms
ffffffffc10b6630 t kvm_mmu_commit_zap_page.part.0	[kvm]
ffffffffc10b6620 t __pfx_kvm_mmu_commit_zap_page.part.0	[kvm]
ffffffffc10bfcc0 t __pfx_kvm_flush_remote_tlbs_with_address	[kvm]
ffffffffc1059040 t kvm_flush_remote_tlbs	[kvm]
ffffffffc10bfcd0 t kvm_flush_remote_tlbs_with_address	[kvm]
ffffffffc1059030 t __pfx_kvm_flush_remote_tlbs	[kvm]

Here, the compiler moved the empty list check of kvm_mmu_commit_zap_page into its callers, so calling the .part.0 directly is only safe because we check that invalid_list is not empty before. It is not even guaranteed to take the same arguments as the original function. In this case we also need to check its disassembly before calling it with the original prototype.

Finally, putting all the changes together, including the rebuild of kvm_mmu_remote_flush_or_zap results in a livepatch including functions from subsystem internal headers, symbols from kallsyms and some small rewrites. Full code can be found on Github.

# insmod ./livepatch.ko
# dmesg -t | tail -n4 
livepatch: enabling patch 'livepatch'
livepatch: 'livepatch': starting patching transition
livepatch: CVE-2026-53359 patch applied
livepatch: 'livepatch': patching complete

Here the livepatch is complete, but some issues remains. The function pointers are resolved once, when the livepatch is loaded. If we unload and reload the kvm module, those pointers will still point to the old module’s addresses and the next call will crash the host. This could be worked around with register_module_notifier, by resolving the symbols again when kvm is loaded. Also if kvm is not loaded yet, the livepatch_init() would fails with -ENOENT.

kpatch-build avoid this issue by letting the livepatch core resolve those symbols itself every time the module is loaded.

# Generating a livepatch with kpatch-build

## Building a patch for Pwnescape

Pwnescape is the latest known CVE at this time of writing, that allows a guest-host escape, found by pwn.ai. This one is a little different than the Januscape/Zapscape/ITscape trilogy, instead of exploiting KVM, it is using a deterministic TOCTOU vulnerability inside the kernel networking stack leading to a KVM escape. Full write-up can be found here.

(Dear security guys, SRE speaking there: stop releasing big CVE on Thursday night/Friday OR during public holiday, pretty please, we do not enjoy this), img source

Unlike a livepatch written by hand, kpatch-build rebuilds the kernel with and without the fix and compares the result, so it needs the full source code of the running kernel, its config (with CONFIG_DEBUG_INFO) and the vmlinux with debug symbols.

You have to retrieve the debug symbols and the source code for your kernel:

# apt-get install linux-image-unsigned-$(uname -r)-dbgsym
# apt-get source linux-image-unsigned-$(uname -r)

Then install and build kpatch-build:

# git clone https://github.com/dynup/kpatch.git
# cd kpatch && make && make install

One naive approche to create the livepatch is to retreive the patch from the commit 6785011f8b16 and start kpatch-build:

# kpatch-build -s ~/linux-src -c /boot/config-$(uname -r)   -v /usr/lib/debug/boot/vmlinux-$(uname -r) -n livepatch-pwnescape pwnescape.patch
Using source directory at /home/lisa/linux-src
Testing patch file(s)
Reading special section data
Building original source
Building patched source
Extracting new and modified ELF sections
[...]
LD [M] net/openvswitch/vport-gre.ko
LD [M] net/vmw_vsock/vsock.ko
LD [M] net/netfilter/nfnetlink_queue.ko
LD [M] net/netfilter/xt_CHECKSUM.ko
LD [M] net/bridge/netfilter/nf_conntrack_bridge.ko
net/xfrm/xfrm_output.o: changed function: xfrm_output
net/ipv4/ip_output.o: changed function: ip_do_fragment
ERROR: drivers/net/ethernet/chelsio/cxgb4/t4_hw.o: reference to static local variable t4_decode.7 in CSWTCH.896 was removed
create-diff-object: unreconcilable difference
[...]

But kpatch failed on this run. Functions were correctly found but then failed on a Chelsio network driver that has nothing to do with our patch. The upstream code adds inside the include/net/ip.h the function ip_check_csum_start(), so every file including this header was rebuilt and compared.

The upstream patch is made to be merged into the kernel not used as a livepatch. You should understand how the patch works and what it is patching then rewrite appropiartly the patch.

Here we have just inlined the checks from ip_check_csum_start() in each patched function so ip.h stays untouched.

Click here to display full code

diff --git a/net/bridge/netfilter/nf_conntrack_bridge.c b/net/bridge/netfilter/nf_conntrack_bridge.c
--- a/net/bridge/netfilter/nf_conntrack_bridge.c
+++ b/net/bridge/netfilter/nf_conntrack_bridge.c
@@ -40,9 +40,14 @@ static int nf_br_ip_fragment(struct net *net, struct sock *sk,
 	int err = 0;

 	/* for offloaded checksums cleanup checksum before fragmentation */
-	if (skb->ip_summed == CHECKSUM_PARTIAL &&
-	    (err = skb_checksum_help(skb)))
-		goto blackhole;
+	if (skb->ip_summed == CHECKSUM_PARTIAL) {
+		if (unlikely(skb->csum_start < skb->network_header + ip_hdrlen(skb)))
+			err = -EINVAL;
+		if (!err)
+			err = skb_checksum_help(skb);
+		if (err)
+			goto blackhole;
+	}

 	iph = ip_hdr(skb);

diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
--- a/net/ipv4/ip_output.c
+++ b/net/ipv4/ip_output.c
@@ -779,9 +779,14 @@ int ip_do_fragment(struct net *net, struct sock *sk, struct sk_buff *skb,
 	int err = 0;

 	/* for offloaded checksums cleanup checksum before fragmentation */
-	if (skb->ip_summed == CHECKSUM_PARTIAL &&
-	    (err = skb_checksum_help(skb)))
-		goto fail;
+	if (skb->ip_summed == CHECKSUM_PARTIAL) {
+		if (unlikely(skb->csum_start < skb->network_header + ip_hdrlen(skb)))
+			err = -EINVAL;
+		if (!err)
+			err = skb_checksum_help(skb);
+		if (err)
+			goto fail;
+	}

 	/*
 	 *	Point into the IP datagram header.
diff --git a/net/netfilter/nfnetlink_queue.c b/net/netfilter/nfnetlink_queue.c
--- a/net/netfilter/nfnetlink_queue.c
+++ b/net/netfilter/nfnetlink_queue.c
@@ -653,6 +653,13 @@ nla_put_failure:

 static int nf_queue_checksum_help(struct sk_buff *entskb)
 {
+	if (entskb->protocol == htons(ETH_P_IP)) {
+		if (!pskb_network_may_pull(entskb, sizeof(struct iphdr)) ||
+		    unlikely(entskb->csum_start <
+			     entskb->network_header + ip_hdr(entskb)->ihl * 4))
+			return -EINVAL;
+	}
+
 	if (skb_csum_is_sctp(entskb))
 		return skb_crc32c_csum_help(entskb);

diff --git a/net/netfilter/xt_CHECKSUM.c b/net/netfilter/xt_CHECKSUM.c
--- a/net/netfilter/xt_CHECKSUM.c
+++ b/net/netfilter/xt_CHECKSUM.c
@@ -25,8 +25,12 @@ MODULE_ALIAS("ip6t_CHECKSUM");
 static unsigned int
 checksum_tg(struct sk_buff *skb, const struct xt_action_param *par)
 {
-	if (skb->ip_summed == CHECKSUM_PARTIAL && !skb_is_gso(skb))
+	if (skb->ip_summed == CHECKSUM_PARTIAL && !skb_is_gso(skb)) {
+		if (xt_family(par) == NFPROTO_IPV4 &&
+		    unlikely(skb->csum_start < skb->network_header + ip_hdr(skb)->ihl * 4))
+			return NF_DROP;
 		skb_checksum_help(skb);
+	}

 	return XT_CONTINUE;
 }
diff --git a/net/xfrm/xfrm_output.c b/net/xfrm/xfrm_output.c
--- a/net/xfrm/xfrm_output.c
+++ b/net/xfrm/xfrm_output.c
@@ -810,16 +810,23 @@ int xfrm_output(struct sock *sk, struct sk_buff *skb)
 	}

 	if (skb->ip_summed == CHECKSUM_PARTIAL) {
-		err = skb_checksum_help(skb);
-		if (err) {
-			XFRM_INC_STATS(net, LINUX_MIB_XFRMOUTERROR);
-			kfree_skb(skb);
-			return err;
+		if (skb->protocol == htons(ETH_P_IP)) {
+			if (unlikely(skb->csum_start < skb->network_header + ip_hdrlen(skb))) {
+				err = -EINVAL;
+				goto error;
+			}
 		}
+		err = skb_checksum_help(skb);
+		if (err)
+			goto error;
 	}

 out:
 	return xfrm_output2(net, sk, skb);
+error:
+	XFRM_INC_STATS(net, LINUX_MIB_XFRMOUTERROR);
+	kfree_skb(skb);
+	return err;
 }
 EXPORT_SYMBOL_GPL(xfrm_output);

Re-run kpatch-build. As there is fewer functions touched, the whole build with kpatch-build is much faster, it has to deal only with the files we actually patched.

# kpatch-build -s ~/linux-src -c /boot/config-$(uname -r) -v /usr/lib/debug/boot/vmlinux-$(uname -r) -n livepatch-pwnescape pwnescape-fixed.patch
Using source directory at /home/lisa/linux-src
Testing patch file(s)
Reading special section data
Building original source
Building patched source
Extracting new and modified ELF sections
net/xfrm/xfrm_output.o: changed function: xfrm_output
net/bridge/netfilter/nf_conntrack_bridge.o: changed function: nf_ct_bridge_post
net/ipv4/ip_output.o: changed function: ip_do_fragment
net/netfilter/nfnetlink_queue.o: changed function: __nfqnl_enqueue_packet
net/netfilter/xt_CHECKSUM.o: changed function: checksum_tg
Patched objects: net/bridge/netfilter/nf_conntrack_bridge.ko net/netfilter/nfnetlink_queue.ko net/netfilter/xt_CHECKSUM.ko vmlinux
Building patch module: livepatch-pwnescape.ko
SUCCESS

Original code was fixing functions nf_br_ip_fragment and nf_queue_checksum_help but they don’t appears in the build output. The compiler have inlined them into their callers, so kpatch patched nf_ct_bridge_post and __nfqnl_enqueue_packet instead. This is exactly what we had to check manually for the januscape manual patch.

livepatch-pwnescape.ko can now be insmod.

Alternativetly, kpatch load can also be used.

# kpatch load livepatch-pwnescape.ko
loading patch module: livepatch-pwnescape.ko
waiting (up to 15 seconds) for patch transition to complete...
transition complete (2 seconds)
# dmesg | tail -n3
[247032.855125] livepatch: enabling patch 'livepatch_pwnescape'
[247032.869226] livepatch: 'livepatch_pwnescape': starting patching transition
[247034.753344] livepatch: 'livepatch_pwnescape': patching complete
# kpatch list
Loaded patch modules:
livepatch_pwnescape [enabled]

Installed patch modules:
# ls -R /sys/kernel/livepatch/livepatch_pwnescape/
/sys/kernel/livepatch/livepatch_pwnescape/:
enabled  force  nf_conntrack_bridge  nfnetlink_queue  replace  transition  vmlinux xt_CHECKSUM

/sys/kernel/livepatch/livepatch_pwnescape/nf_conntrack_bridge:
nf_ct_bridge_post,1  patched

/sys/kernel/livepatch/livepatch_pwnescape/nf_conntrack_bridge/nf_ct_bridge_post,1:

/sys/kernel/livepatch/livepatch_pwnescape/nfnetlink_queue:
__nfqnl_enqueue_packet,1  patched
[...]

A livepatch lives only in memory, if the server reboots on the same kernel, the fix is gone and the vulnerability is back. kpatch install livepatch-pwnescape.ko will store the module so kpatch loads it again at boot. Since the livepatch is built for one specific kernel, it will be only loaded on that kernel. Once you reboot on a different kernel, there is no patch to load. Also don’t forget to enable the kpatch service (systemctl enable kpatch).

No official POC was released yet for Pwnescape, only the security patch. Using a POC by PeronGH, the OOB reads reproduce on an unpatched kernel. With the livepatch loaded, the skb is rejected before fragmentation and no fragment is captured at all.

# Cumulative patches and atomic replace

KVM vulnerabilities went wild recently and kept coming. Zapscape (also by V4Bel) which appeared one month after Januscape (which was one month after ITscape), was also a guest-to-host escape, allowing an attacker to execute commands with the root privileges on the host.

Let's say after Januscape, you rebooted your whole fleet on a kernel including the fix. A few weeks later, Zapscape comes out. Without livepatching, either the production still runs an unpatched kernel or another emergency reboot campaign has to be made. Another month later, Pwnescape comes out. There is another livepatch or another livemigration + reboot campaign.

Once you livepatched Zapscape, you could load a second livepatch on top of the first one. But a cleaner way can also be to build a single cumulative livepatch containing both Zapscape and Pwnescape, which replaces the first Zapscape livepatch.

Cumulative patches are about the content. A single livepatch that contains every fix needed on the running kernel.
Atomic replaces are the mechanism used to load it: the new livepatch replaces all the previous one in a single transition, there is no moment where the server runs without any fix. Once the transition is complete, the previous livepatches are disabled and all functions they patched that are not in the new livepatch goes back to the original code.

On a hand-written livepatch, atomic replace is a single field in struct klp_patch. When this field is unset, the default value is false. The livepatch would be stacked on top of the previous ones. Setting it to true enables atomic replace:

static struct klp_patch patch = {
    .mod = THIS_MODULE,
    .objs = objs,
    .replace = true, /* replaces all the previously loaded livepatches */
};

kpatch-build builds atomic replace livepatches by default.

This is the current state of the test server:

# kpatch list
Loaded patch modules:
livepatch_zapscape [enabled]

The zapscape patch was built the same way with kpatch-build, from the upstream commit 2abd5287f083.

Now, the goal is to add the Pwnescape patch. Januscape is already fixed on my test kernel, so it’s not needed but Zapscape is. If we load the previously built Pwnescape patch, Zapscape vulnerability come back without any warning:

# kpatch load livepatch-pwnescape.ko
loading patch module: livepatch-pwnescape.ko
waiting (up to 15 seconds) for patch transition to complete...
transition complete (3 seconds)
# kpatch list
Loaded patch modules:
livepatch_pwnescape [enabled]

livepatch_zapscape is silently replaced and from now zapscape is not fixed anymore. And it gets worse if we need to roll back:

# kpatch unload livepatch_pwnescape
disabling patch module: livepatch_pwnescape
waiting (up to 15 seconds) for patch transition to complete...
transition complete (2 seconds)
unloading patch module: livepatch_pwnescape
# kpatch list
Loaded patch modules:

Installed patch modules:

The zapscape livepatch does not come back automatically. A replaced livepatch stays disabled even if the module is still loaded. The server now runs with no fix at all, zapscape livepatch has to be loaded again manually:

# lsmod  | grep patch
livepatch_zapscape     36864  0
# kpatch load livepatch-zapscape.ko
loading patch module: livepatch-zapscape.ko
waiting (up to 15 seconds) for patch transition to complete...
transition complete (2 seconds)

This is why a livepatch built with atomic replace should be cumulative.

kpatch-build accepts multiple patch files, so a single livepatch can be built to contains both of the fixes:

# kpatch-build -s ~/linux-src -c /boot/config-$(uname -r) -v /usr/lib/debug/boot/vmlinux-$(uname -r) -n livepatch-cumulative zapscape.patch pwnescape-fixed.patch
Using source directory at /home/lisa/linux-src
Testing patch file(s)
Reading special section data
Building original source
Building patched source
Extracting new and modified ELF sections
net/xfrm/xfrm_output.o: changed function: xfrm_output
arch/x86/kvm/mmu/mmu.o: changed function: paging32_page_fault
arch/x86/kvm/mmu/mmu.o: changed function: paging64_page_fault
arch/x86/kvm/mmu/mmu.o: changed function: ept_page_fault
arch/x86/kvm/mmu/mmu.o: changed function: direct_page_fault
net/ipv4/ip_output.o: changed function: ip_do_fragment
net/bridge/netfilter/nf_conntrack_bridge.o: changed function: nf_ct_bridge_post
net/netfilter/nfnetlink_queue.o: changed function: __nfqnl_enqueue_packet
net/netfilter/xt_CHECKSUM.o: changed function: checksum_tg
Patched objects: arch/x86/kvm/kvm.ko net/bridge/netfilter/nf_conntrack_bridge.ko net/netfilter/nfnetlink_queue.ko net/netfilter/xt_CHECKSUM.ko vmlinux
Building patch module: livepatch-cumulative.ko
SUCCESS

Then loading it on a server already patched:

# kpatch list
Loaded patch modules:
livepatch_zapscape [enabled]
# kpatch load livepatch-cumulative.ko
loading patch module: livepatch-cumulative.ko
waiting (up to 15 seconds) for patch transition to complete...
transition complete (2 seconds)
# kpatch list
Loaded patch modules:
livepatch_cumulative [enabled]
# cat /sys/kernel/livepatch/livepatch_cumulative/replace
1

livepatch_zapscape is out of the list, it was also automatically replaced and disabled. Its module is still loaded though, but can be now removed

# lsmod | grep patch
livepatch_cumulative		61440  1
livepatch_zapscape		36864  0
# rmmod livepatch_zapscape

New livepatch covers both fixes, on vmlinux, kvm and all other modules involved:

# ls  /sys/kernel/livepatch/livepatch_cumulative/vmlinux/
ip_do_fragment,1  patched  xfrm_output,1
# ls /sys/kernel/livepatch/livepatch_cumulative/kvm
direct_page_fault,1  ept_page_fault,1  paging32_page_fault,1  paging64_page_fault,1  patched
# ls  /sys/kernel/livepatch/livepatch_cumulative/nfnetlink_queue/
__nfqnl_enqueue_packet,1  patched
[...]
# cat /sys/kernel/livepatch/livepatch_cumulative/kvm/patched
1

Patches can also be loaded side by side, using --non-replace:

# kpatch-build --non-replace -s ~/linux-src -c /boot/config-$(uname -r) -v /usr/lib/debug/boot/vmlinux-$(uname -r) -n livepatch-pwnescape pwnescape-fixed.patch
[...]

# kpatch list
Loaded patch modules:
livepatch_zapscape [enabled]

# kpatch load livepatch-pwnescape.ko
loading patch module: livepatch-pwnescape.ko
waiting (up to 15 seconds) for patch transition to complete...
transition complete (2 seconds)

# kpatch list
Loaded patch modules:
livepatch_pwnescape [enabled]
livepatch_zapscape [enabled]

Both patches are now enabled side by side. It’s simple here because they don’t touch the same functions. Each CVE adds another module to track and they have to be disabled in reverse order to be removed.

# So, should you livepatch?

In this blog, we talked about Januscape, Zapscape and PwnEscape, which fortunately were cases of fixes being livepatchable. Those functions were hookable and transitioned cleanly: short-lived, they return immediately. It was also isolated to small and specific portions of the kernel code.

If the problem was more complex, affecting multiple functions, data structures etc, it can’t be done. There are no miracles that can replace a good ol’ pssh reboot < /dev/null.

Honestly, the manual livepatch remains fragile: resolving a split symbol through .part.0, whole function copies and hand-made helpers that were inlined away. This ties the livepatch to this exact kernel version or with some luck to some newer versions if the functions copied do not change. Any change in the compiler, the symbol names, the surrounding code (e.g. kvm mmu management) or a function that suddenly becomes inlined, and this module won’t load or the kernel may panic. If it doesn’t fail loudly, it may corrupt quietly in the exact fault path we were trying to protect.

Home-made livepatches might help you buy time until you are able to reboot your servers to a newer kernel version. But it still requires good knowledge of Linux and C. Paid alternatives that deliver livepatch exists and might suit your needs for critical production servers.

The hardest part of livepatching is not only writing the patch itself, but making sure the replacement is correct, thoroughly tested, compatible with the running kernel, safe to transition to and, most of all, maintainable on a fleet of servers.

So, would you livepatch ?

Thanks to my coworkers for putting up with the monkey I am in production, especially when I’m tinkering with kpatch and kexec 💜

Maybe in a future article, we will talk about klp-build, which became the in-tree sucessor of kpatch-build since kernel version 6.19.