Raspberry Pi – Compiling a Module for the 64-bit Kernel

Recently I’ve been working on a network router using the Compute Module 4 and a custom base board with a Realtek RTL8111H PCIe Ethernet controller. Raspberry Pi OS doesn’t come pre-installed with the r8168 or r8169 driver needed for the RTL8111, but getting it compiled and installed is pretty easy. Or at least until I found out that it’s possible switch to a 64-bit kernel by adding arm_64bit=1 to /boot/config.txt! The kernel is 64-bit, but all of the user space programs are still 32-bit. This is fine since all of the routing hard work stays within the kernel anyway. The full 64-bit Raspberry Pi OS is still in development so I wanted to stick with the 32-bit version for now.

I’ve never really touched Linux kernel stuff before. The first hurdle was that the raspbian repository doesn’t have a cross-compiler for 64-bit ARM (aarch64) or headers for the 64-bit kernel. However, the kernel source which includes the headers can be downloaded from the Raspberry Pi GitHub repo. After more googling I found that the best way of getting an aarch64 compiler would be download the ubuntu arm64 base from http://cdimage.ubuntu.com/ubuntu-base/releases/ and follow the steps described here and here.

I was still getting errors though. Mainly about a missing modules.lds file, but copying that file from the headers of the 32-bit kernel (/lib/modules/5.10.17-v7l+/build/scripts/) seemed to fix it. It’s just a small text file that doesn’t seem to be architecture specific.

The module finally compiled! But now I was getting Exec format errors when attempting to load it. modinfo r8168 seemed to say that everything was correct; compiled for my kernel version and as aarch64.

$modinfo r8168
[cut]
name:           r8168
[cut]
vermagic:       5.10.17-v8 SMP preempt mod_unload modversions aarch64
[cut]

$sudo modprobe r8168
modprobe: ERROR: could not insert 'r8168': Exec format error

After more hours of futile attempts I finally came across this commit in the RPi kernel repo which mentions:

…you might have prepared the minimal setups for external modules by ‘make defconfig && make modules_preapre’…

masahir0y

Building an external module is exactly what I’ve been trying to do, so the make modules_prepare command looks promising! I ran the new command in the kernel source directory, recompiled the r8168 module and finally modprobe didn’t return an error! lsmod | grep r8168 shows that the driver has been loaded and a new network interface (eth1) appears in ip link show!

$ip link show
1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether dc:a6:32:ff:7d:f7 brd ff:ff:ff:ff:ff:ff
3: eth1: mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 48:4d:7e:9f:e5:10 brd ff:ff:ff:ff:ff:ff

So, here’s the full set of commands needed to compile a 64-bit kernel module in 32-bit Raspberry Pi OS running a 64-bit kernel. There will probably be some warnings about missing symbols, but it doesn’t seem to be a problem.

OS: Raspberry Pi OS 10 (buster) Lite 2021-05-07 (32-bit)
Kernel: 5.10.17-v8+ (64-bit)
Module: r8168 v8.049.01

Grab the r8168 driver source from Realteks website.

mkdir ubuntu64
cd ubuntu64

wget http://cdimage.ubuntu.com/ubuntu-base/releases/21.04/release/ubuntu-base-21.04-base-arm64.tar.gz
tar -xf ubuntu-base-21.04-base-arm64.tar.gz

wget https://github.com/raspberrypi/linux/archive/refs/tags/raspberrypi-kernel_1.20210527-1.tar.gz
tar -xf raspberrypi-kernel_1.20210527-1.tar.gz

mv ../r8168-8.049.01.tar.bz2 .
tar -xf r8168-8.049.01.tar.bz2


cat /etc/resolv.conf >etc/resolv.conf
sudo mount -t tmpfs tmp tmp

sudo chroot .

apt update
apt install locales-all wget build-essential bc bison flex libssl-dev kmod

cd linux-raspberrypi-kernel_1.20210527-1/
export KCFLAGS="-march=armv8-a+crc -mtune=cortex-a72"
make -j4 bcm2711_defconfig
make -j4 modules_prepare

cd ../r8168-8.049.01
make -j4 -C ../linux-raspberrypi-kernel_1.20210527-1 M=$(pwd)/src modules

exit

sudo umount tmp


sudo mkdir /lib/modules/$(uname -r)/kernel/drivers/net/ethernet/realtek/
sudo cp r8168-8.049.01/src/r8168.ko /lib/modules/$(uname -r)/kernel/drivers/net/ethernet/realtek/
sudo depmod $(uname -r)
sudo modprobe r8168

9 comments

1 ping

Skip to comment form

    • Luca Di Gregorio on August 17, 2021 at 4:18 pm
    • Reply

    I’m testing the full set of commands in a Raspberry Pi Zero, I don’t have yet a Compute Module 4.

    I’m connected as the user ‘pi’ . The ‘sudo chroot .’ command fails:

    pi@raspberry:~/ubuntu64 $ sudo chroot .
    chroot: failed to run command ‘/bin/bash’: Exec format error

    I’ve not added, in /boot/config.txt, the line ‘arm_64bit=1’, because it is not in the full set of commands.

    Any idea?
    Thanks in advance, regards

    1. The Pi Zero CPU is only 32-bit, but the ubuntu base used here has 64-bit binaries which require a Pi with a 64-bit CPU, so a Pi 3 or 4. You could try downloading the armhf version of the ubuntu base instead of arm64, but to compile the module for the 64-bit kernel you’ll need a cross-compiler and things could start to get messy. It’s probably best to wait for your CM4.

        • Luca Di Gregorio on August 17, 2021 at 6:04 pm
        • Reply

        Thank yo very much. Ok, I’ll wait my cm4. Just the last question for you: is it mandatory to set, in order to compile the driver, ‘arm_64bit=1’ in /boot/config.txt? If yes: after compiling the driver, can ‘arm_64bit=1’ be removed from /boot/config.txt?

        1. arm_64bit=1 tells the Pi to load the 64-bit kernel. Without it none of the 64-bit stuff will work.

            • Luca Di Gregorio on August 25, 2021 at 3:08 pm

            Hi, I finally got my CM4, the expansion board is by dfrobot https://www.dfrobot.com/product-2242.html

            I added arm_64bit=1 to /boot/config.txt

            I ran the full set of commands (I downloaded the linux kernel with git clone https://github.com/raspberrypi/linux.git instead of the .tar.gz, and I dowloaded a newer version of the driver: r8168-8.049.02.tar.bz2)

            Everything was fine, no errors in compiling.

            I rebooted and:

            $ dmesg | grep -i 81
            [ 0.000000] percpu: Embedded 32 pages/cpu s91224 r8192 d31656 u131072
            [ 0.000000] pcpu-alloc: s91224 r8192 d31656 u131072 alloc=32*4096
            ...
            [ 1.256121] pci 0000:01:00.0: [10ec:8168] type 00 class 0x020000
            ...
            [ 3.723882] r8168: no symbol version for module_layout
            [ 3.723937] r8168: loading out-of-tree module taints kernel.
            [ 3.777171] r8168 Gigabit Ethernet driver 8.049.02-NAPI loaded
            [ 3.778003] r8168 0000:01:00.0: enabling device (0000 -> 0002)
            [ 3.778061] r8168 0000:01:00.0: region #1 not an MMIO resource, aborting
            ...

            ~ $ lspci
            00:00.0 PCI bridge: Broadcom Limited Device 2711 (rev 20)
            01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)

            $ lsmod | grep r8168
            r8168 475136 0

            $ modinfo r8168
            filename: /lib/modules/5.10.52-v8+/kernel/drivers/net/ethernet/realtek/r8168.ko
            version: 8.049.02-NAPI
            license: GPL
            description: RealTek RTL-8168 Gigabit Ethernet driver
            author: Realtek and the Linux r8168 crew
            srcversion: 2F9162CFFE44026BC72A4F3
            alias: pci:v00001186d00004300sv00001186sd00004B10bc*sc*i*
            alias: pci:v000010ECd00002600sv*sd*bc*sc*i*
            alias: pci:v000010ECd00002502sv*sd*bc*sc*i*
            alias: pci:v000010ECd00008161sv*sd*bc*sc*i*
            alias: pci:v000010ECd00008168sv*sd*bc*sc*i*
            depends:
            name: r8168
            vermagic: 5.10.60-v8 SMP preempt mod_unload modversions aarch64
            parm: speed_mode:force phy operation. Deprecated by ethtool (8). (uint)
            parm: duplex_mode:force phy operation. Deprecated by ethtool (8). (uint)
            parm: autoneg_mode:force phy operation. Deprecated by ethtool (8). (uint)
            parm: advertising_mode:force phy operation. Deprecated by ethtool (8). (uint)
            parm: dynamic_aspm:int
            parm: aspm:Enable ASPM. (int)
            parm: s5wol:Enable Shutdown Wake On Lan. (int)
            parm: s5_keep_curr_mac:Enable Shutdown Keep Current MAC Address. (int)
            parm: rx_copybreak:Copy breakpoint for copy-only-tiny-frames (int)
            parm: use_dac:Enable PCI DAC. Unsafe on 32 bit PCI slot. (int)
            parm: timer_count:Timer Interrupt Interval. (int)
            parm: eee_enable:Enable Energy Efficient Ethernet. (int)
            parm: hwoptimize:Enable HW optimization function. (ulong)
            parm: s0_magic_packet:Enable S0 Magic Packet. (int)
            parm: dynamic_aspm_packet_threshold:Dynamic ASPM packet threshold. (int)
            parm: debug:Debug verbosity level (0=none, ..., 16=all) (int)

            But eth1 is not shown:

            $ ip link show
            1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
            link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
            2: eth0: mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
            link/ether e4:5f:01:2a:19:f9 brd ff:ff:ff:ff:ff:ff
            3: wlan0: mtu 1500 qdisc pfifo_fast state UP mode DORMANT group default qlen 1000
            link/ether e4:5f:01:2a:19:fa brd ff:ff:ff:ff:ff:ff

            Do you have any idea? Thanks a lot again

          1. Hmm, this would the main error [ 3.778061] r8168 0000:01:00.0: region #1 not an MMIO resource, aborting.
            I don’t think the changes in the newer driver version would cause that error. Try using the headers for your kernel version (5.10.52) rather than the latest git, which looks to be 5.10.60. If that doesn’t work try using an older Rasperry Pi OS with an older kernel version, like the same version as what I used here.

    • Luca Di Gregorio on August 25, 2021 at 7:30 pm
    • Reply

    I froze the kernel to the 5.10.17-v8+ with:

    sudo apt-mark hold libraspberrypi-bin libraspberrypi-dev libraspberrypi-doc libraspberrypi0
    sudo apt-mark hold raspberrypi-bootloader raspberrypi-kernel raspberrypi-kernel-headers

    and now I can see the eth1 interface, with the r8168-8.049.02.tar.bz2 (new version of the driver)

    However, I would like to know how to download headers for a specific kernel version.
    In fact, the newer kernel that I was trying with, is: 5.10.52-v8+ #1441 SMP PREEMPT Tue Aug 3 18:14:03 BST 2021 aarch64 GNU/Linux

    But I couldn’t download the headers with:

    wget https://github.com/raspberrypi/linux/archive/refs/tags/raspberrypi-kernel_1.20210803-1.tar.gz

    ERROR 404: Not Found.

    Do you know how to download it?
    Thanks

        • Luca Di Gregorio on August 26, 2021 at 5:07 am
        • Reply

        I unfroze the kernel and updated it:

        sudo apt-mark unhold libraspberrypi-bin libraspberrypi-dev libraspberrypi-doc libraspberrypi0 raspberrypi-bootloader raspberrypi-kernel
        sudo apt-get update
        sudo apt-get upgrade -y

        $ uname -a
        Linux dfrobot 5.10.52-v8+ #1441 SMP PREEMPT Tue Aug 3 18:14:03 BST 2021 aarch64 GNU/Linux

        I downloaded the 20210805 kernel:

        wget https://github.com/raspberrypi/linux/archive/refs/tags/1.20210805.tar.gz

        I went again through the full set of commands and recompiled the driver.

        It works:

        $ ip link show
        1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        2: eth0: mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
        link/ether e4:5f:01:2a:19:f9 brd ff:ff:ff:ff:ff:ff
        3: eth1: mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
        link/ether e2:ff:a2:0f:9e:7d brd ff:ff:ff:ff:ff:ff
        4: wlan0: mtu 1500 qdisc pfifo_fast state UP mode DORMANT group default qlen 1000
        link/ether e4:5f:01:2a:19:fa brd ff:ff:ff:ff:ff:ff

        $ modinfo r8168
        filename: /lib/modules/5.10.52-v8+/kernel/drivers/net/ethernet/realtek/r8168.ko
        version: 8.049.02-NAPI
        license: GPL
        description: RealTek RTL-8168 Gigabit Ethernet driver
        author: Realtek and the Linux r8168 crew
        srcversion: 2F9162CFFE44026BC72A4F3
        alias: pci:v00001186d00004300sv00001186sd00004B10bc*sc*i*
        alias: pci:v000010ECd00002600sv*sd*bc*sc*i*
        alias: pci:v000010ECd00002502sv*sd*bc*sc*i*
        alias: pci:v000010ECd00008161sv*sd*bc*sc*i*
        alias: pci:v000010ECd00008168sv*sd*bc*sc*i*
        depends:
        name: r8168
        vermagic: 5.10.52-v8 SMP preempt mod_unload modversions aarch64
        parm: speed_mode:force phy operation. Deprecated by ethtool (8). (uint)
        parm: duplex_mode:force phy operation. Deprecated by ethtool (8). (uint)
        parm: autoneg_mode:force phy operation. Deprecated by ethtool (8). (uint)
        parm: advertising_mode:force phy operation. Deprecated by ethtool (8). (uint)
        parm: dynamic_aspm:int
        parm: aspm:Enable ASPM. (int)
        parm: s5wol:Enable Shutdown Wake On Lan. (int)
        parm: s5_keep_curr_mac:Enable Shutdown Keep Current MAC Address. (int)
        parm: rx_copybreak:Copy breakpoint for copy-only-tiny-frames (int)
        parm: use_dac:Enable PCI DAC. Unsafe on 32 bit PCI slot. (int)
        parm: timer_count:Timer Interrupt Interval. (int)
        parm: eee_enable:Enable Energy Efficient Ethernet. (int)
        parm: hwoptimize:Enable HW optimization function. (ulong)
        parm: s0_magic_packet:Enable S0 Magic Packet. (int)
        parm: dynamic_aspm_packet_threshold:Dynamic ASPM packet threshold. (int)
        parm: debug:Debug verbosity level (0=none, ..., 16=all) (int)

        Thanks again!

  1. […] at Zak’s Electronics Blog is working on a network router using the Raspberry Pi Compute Module 4 and a custom base board with […]

Leave a Reply

Your email address will not be published.

Are you human? *