Dynamic / Live kernel patching

I have been using Ksplice on Ubuntu / Ubuntu Server since 2011 after Oracle acquired Ksplice. It’s great technology and tool to minimize downtime and maximize server uptime, especially to those servers running in the cloud. Works like a charm. On Ubuntu, it even offers support for LTS releases running the latest HWE kernels, it just rocks!

Ksplice has been running since I spun off my Digital Ocean (KVM powered) VPS, its uptime is close to 300 days (298 days as of now). The Ubuntu 12.04 LTS instance is running the original release kernel 3.2 (DO currently does NOT offer kernel upgrade…), it’s been taken care of and updated to the latest. See below :-D

uptrack-show | grep "\[" | wc -l

shows that 317 patches have been applied.

NOTE: Ksplice is free for Ubuntu / Ubuntu Server and Fedora.

Now, Ksplice is no longer the only player in Dynamic / Live Kernel Patching.

On 31 January, 2014 SUSE announced their live kernel patching project called kGraft.

Not long after, on 26 February, 2014 Red Hat announced their dynamic / live kernel patching project kpatch.

Wow! All of a sudden we have 3 players.

Looks like both SUSE and Red Hat are pushing hard trying to make their own dynamic/live kernel patching code into the upstream Linux kernel. Who will win? Up in the air, too early to say.

Playing with kpatch on Ubuntu 14.04 LTS

I’ve seen kGraft demo video by its main developer Jiri Slaby, looks pretty cool. But due to the lack of documentations, it’s hard to get started and try it with my own hands.

kpatch source code is hosted on GitHub, the README contains enough information to get started, time to give it a shot!

To be quick, use a Ubuntu Vagrant image or Opscode bento image to start with.

OS Version: Ubuntu 14.04 (Trusty) LTS

Install dependencies

apt-get install build-essentials libelf-dev

# Packages required to compile Linux Kernel
apt-get build-dep linux

# Compiler cache
apt-get install ccache

# Optional: kernel source code
apt-get install linux-source

Install kernel debug symbols (dbgsym kernel)

# Add ddebs repository
codename=$(lsb_release -sc)
sudo tee /etc/apt/sources.list.d/ddebs.list << EOF
deb http://ddebs.ubuntu.com/ ${codename} main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-security main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-proposed main restricted universe multiverse
EOF

# add APT key
wget -Nq http://ddebs.ubuntu.com/dbgsym-release-key.asc -O- | sudo apt-key add -
apt-get update && apt-get install linux-image-$(uname -r)-dbgsym

NOTE: dbgsym kernel (with debug symbols) is as huge as 401M, but it is indeed needed!

1. If dbgsym kernel is NOT installed, you’ll get “ERROR: kernel-debuginfo not installed” when using kpatch-build to build a patch module.
2. Create symbolic link for the debug symoble kernel image
   ln -s /usr/lib/debug/boot/vmlinux-$(uname -r) /usr/lib/debug/lib/modules/$(uname -r)/vmlinux

The following lines in kpatch-build indicates a symbolic link needs to be created

[[ -z $VMLINUX ]] && VMLINUX=/usr/lib/debug/lib/modules/${ARCHVERSION}/vmlinux
[[ -e "$VMLINUX" ]] || die "kernel-debuginfo not installed"

OK, ready to cook!

Create a simple PoC patch

cat meminfo.patch
--- linux-3.13.0/fs/proc/meminfo.c 2014-01-20 02:40:07.000000000 +0000
+++ linux-kpatch/fs/proc/meminfo.c 2014-05-20 10:39:05.451122874 +0000
@@ -95,7 +95,7 @@
"Committed_AS: %8lu kB\n"
"VmallocTotal: %8lu kB\n"
"VmallocUsed: %8lu kB\n"
- "VmallocChunk: %8lu kB\n"
+ "VMALLOCCHUNK: %8lu kB\n"
#ifdef CONFIG_MEMORY_FAILURE
"HardwareCorrupted: %5lu kB\n"
#endif

Build the patch module

kpatch-build -d /path/to/meminfo.patch
DEBUG mode enabled
Using cache at /root/.kpatch/src
Testing patch file
checking file fs/proc/meminfo.c
Building original kernel
Building patched kernel
Detecting changed objects
Rebuilding changed objects
Extracting new and modified ELF sections
meminfo.o: changed function: meminfo_proc_show
Building patch module: kpatch-meminfo.ko
SUCCESS

NOTE: Above is the 2nd build, that’s why it used the cache. If it is the 1st build, kpatch will trigger download of the kernel source code and other required packages, extract it to $HOME/.kpatch. Look like below.

See output

Done! kpatch-meminfo.ko will be output to the current working directory.

Apply the patch to the running kernel

kpatch load module.ko

Unfortunately kpatch assumes insmod and rmmod reside in /usr/sbin which is NOT the case on Debian and Ubuntu. By looking at the code, I was able to figure out how to run it manually.

See the screenshot

Or replace all /usr/sbin/insmod with sed -i 's/\/usr\/sbin\/insmod/insmod/g' kpatch and sed -i 's/\/usr\/sbin\/rmmod/rmmod/g' kpatch via sed. The script needs some more work to support more Linux distributions later.

Now kpatch works on Ubuntu ;-D