Raspberry Pi - Compiling a Module for the 64-bit Kernel

Recently I’ve been working on a network router using the Compute Module 4 and a custom base board with a Realtek RTL8111H PCIe Ethernet controller. Raspberry Pi OS doesn’t come pre-installed with the r8168 or r8169 driver needed for the RTL8111, but getting it compiled and installed is pretty easy. Or at least until I found out that it’s possible switch to a 64-bit kernel by adding arm_64bit=1 to /boot/config.txt! The kernel is 64-bit, but all of the user space programs are still 32-bit. This is fine since all of the routing hard work stays within the kernel anyway. The full 64-bit Raspberry Pi OS is still in development so I wanted to stick with the 32-bit version for now.

UPDATE August 2022:
Raspberry Pi OS includes the r8169 driver since the 2022-01-28 release so now there's no need to build the driver yourself.

I’ve never really touched Linux kernel stuff before. The first hurdle was that the raspbian repository doesn’t have a cross-compiler for 64-bit ARM (aarch64) or headers for the 64-bit kernel. However, the kernel source which includes the headers can be downloaded from the Raspberry Pi GitHub repo. After more googling I found that the best way of getting an aarch64 compiler would be download the ubuntu arm64 base from http://cdimage.ubuntu.com/ubuntu-base/releases/ and follow the steps described here and here.

I was still getting errors though. Mainly about a missing modules.lds file, but copying that file from the headers of the 32-bit kernel (/lib/modules/5.10.17-v7l+/build/scripts/) seemed to fix it. It’s just a small text file that doesn’t seem to be architecture specific.

The module finally compiled! But now I was getting Exec format errors when attempting to load it. modinfo r8168 seemed to say that everything was correct; compiled for my kernel version and as aarch64.

$modinfo r8168
[cut]
name:           r8168
[cut]
vermagic:       5.10.17-v8 SMP preempt mod_unload modversions aarch64
[cut]

$sudo modprobe r8168
modprobe: ERROR: could not insert 'r8168': Exec format error

After more hours of futile attempts I finally came across this commit in the RPi kernel repo which mentions:

...you might have prepared the minimal setups for external modules by ‘make defconfig && make modules_preapre’...

-*masahir0y*

Building an external module is exactly what I’ve been trying to do, so the make modules_prepare command looks promising! I ran the new command in the kernel source directory, recompiled the r8168 module and finally modprobe didn’t return an error! lsmod | grep r8168 shows that the driver has been loaded and a new network interface (eth1) appears in ip link show!

$ip link show
1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether dc:a6:32:ff:7d:f7 brd ff:ff:ff:ff:ff:ff
3: eth1: mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 48:4d:7e:9f:e5:10 brd ff:ff:ff:ff:ff:ff

So, here’s the full set of commands needed to compile a 64-bit kernel module in 32-bit Raspberry Pi OS running a 64-bit kernel. There will probably be some warnings about missing symbols, but it doesn’t seem to be a problem.

OS: Raspberry Pi OS 10 (buster) Lite 2021-05-07 (32-bit)
Kernel: 5.10.17-v8+ (64-bit)
Module: r8168 v8.049.01

Grab the r8168 driver source from Realteks website.

mkdir ubuntu64
cd ubuntu64

wget http://cdimage.ubuntu.com/ubuntu-base/releases/21.04/release/ubuntu-base-21.04-base-arm64.tar.gz
tar -xf ubuntu-base-21.04-base-arm64.tar.gz

wget https://github.com/raspberrypi/linux/archive/refs/tags/raspberrypi-kernel_1.20210527-1.tar.gz
tar -xf raspberrypi-kernel_1.20210527-1.tar.gz

mv ../r8168-8.049.01.tar.bz2 .
tar -xf r8168-8.049.01.tar.bz2


cat /etc/resolv.conf >etc/resolv.conf
sudo mount -t tmpfs tmp tmp

sudo chroot .

apt update
apt install locales-all wget build-essential bc bison flex libssl-dev kmod

cd linux-raspberrypi-kernel_1.20210527-1/
export KCFLAGS="-march=armv8-a+crc -mtune=cortex-a72"
make -j4 bcm2711_defconfig
make -j4 modules_prepare

cd ../r8168-8.049.01
make -j4 -C ../linux-raspberrypi-kernel_1.20210527-1 M=$(pwd)/src modules

exit

sudo umount tmp


sudo mkdir /lib/modules/$(uname -r)/kernel/drivers/net/ethernet/realtek/
sudo cp r8168-8.049.01/src/r8168.ko /lib/modules/$(uname -r)/kernel/drivers/net/ethernet/realtek/
sudo depmod $(uname -r)
sudo modprobe r8168