Recently I’ve been working on a network router using the Compute Module 4 and a custom base board with a Realtek RTL8111H PCIe Ethernet controller. Raspberry Pi OS doesn’t come pre-installed with the r8168 or r8169 driver needed for the RTL8111, but getting it compiled and installed is pretty easy. Or at least until I found out that it’s possible switch to a 64-bit kernel by adding
/boot/config.txt! The kernel is 64-bit, but all of the user space programs are still 32-bit. This is fine since all of the routing hard work stays within the kernel anyway. The full 64-bit Raspberry Pi OS is still in development so I wanted to stick with the 32-bit version for now.
I’ve never really touched Linux kernel stuff before. The first hurdle was that the raspbian repository doesn’t have a cross-compiler for 64-bit ARM (aarch64) or headers for the 64-bit kernel. However, the kernel source which includes the headers can be downloaded from the Raspberry Pi GitHub repo. After more googling I found that the best way of getting an aarch64 compiler would be download the ubuntu arm64 base from http://cdimage.ubuntu.com/ubuntu-base/releases/ and follow the steps described here and here.
I was still getting errors though. Mainly about a missing modules.lds file, but copying that file from the headers of the 32-bit kernel (
/lib/modules/5.10.17-v7l+/build/scripts/) seemed to fix it. It’s just a small text file that doesn’t seem to be architecture specific.
The module finally compiled! But now I was getting Exec format errors when attempting to load it.
modinfo r8168 seemed to say that everything was correct; compiled for my kernel version and as aarch64.
1 2 3 4 5 6 7 8 9 $modinfo r8168 [cut] name: r8168 [cut] vermagic: 5.10.17-v8 SMP preempt mod_unload modversions aarch64 [cut] $sudo modprobe r8168 modprobe: ERROR: could not insert 'r8168': Exec format error
After more hours of futile attempts I finally came across this commit in the RPi kernel repo which mentions:
…you might have prepared the minimal setups for external modules by ‘make defconfig && make modules_preapre’… -masahir0y
Building an external module is exactly what I’ve been trying to do, so the
make modules_prepare command looks promising! I ran the new command in the kernel source directory, recompiled the r8168 module and finally modprobe didn’t return an error!
lsmod | grep r8168 shows that the driver has been loaded and a new network interface (eth1) appears in
ip link show!
1 2 3 4 5 6 7 $ip link show 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth0: mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether dc:a6:32:ff:7d:f7 brd ff:ff:ff:ff:ff:ff 3: eth1: mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether 48:4d:7e:9f:e5:10 brd ff:ff:ff:ff:ff:ff
So, here’s the full set of commands needed to compile a 64-bit kernel module in 32-bit Raspberry Pi OS running a 64-bit kernel. There will probably be some warnings about missing symbols, but it doesn’t seem to be a problem.
OS: Raspberry Pi OS 10 (buster) Lite 2021-05-07 (32-bit)
Kernel: 5.10.17-v8+ (64-bit)
Module: r8168 v8.049.01
Grab the r8168 driver source from Realteks website.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 mkdir ubuntu64 cd ubuntu64 wget http://cdimage.ubuntu.com/ubuntu-base/releases/21.04/release/ubuntu-base-21.04-base-arm64.tar.gz tar -xf ubuntu-base-21.04-base-arm64.tar.gz wget https://github.com/raspberrypi/linux/archive/refs/tags/raspberrypi-kernel_1.20210527-1.tar.gz tar -xf raspberrypi-kernel_1.20210527-1.tar.gz mv ../r8168-8.049.01.tar.bz2 . tar -xf r8168-8.049.01.tar.bz2 cat /etc/resolv.conf >etc/resolv.conf sudo mount -t tmpfs tmp tmp sudo chroot . apt update apt install locales-all wget build-essential bc bison flex libssl-dev kmod cd linux-raspberrypi-kernel_1.20210527-1/ export KCFLAGS="-march=armv8-a+crc -mtune=cortex-a72" make -j4 bcm2711_defconfig make -j4 modules_prepare cd ../r8168-8.049.01 make -j4 -C ../linux-raspberrypi-kernel_1.20210527-1 M=$(pwd)/src modules exit sudo umount tmp sudo mkdir /lib/modules/$(uname -r)/kernel/drivers/net/ethernet/realtek/ sudo cp r8168-8.049.01/src/r8168.ko /lib/modules/$(uname -r)/kernel/drivers/net/ethernet/realtek/ sudo depmod $(uname -r) sudo modprobe r8168