Building the Xenomai kernel
After ensuring that I could build the RPi kernel that still boots the Rasbian distribution that I downloaded (since I did not changing any kernel config significantly except CONFIG_DEBUG_INFO I expected this to work), I started changing the kernel config for Xenomai. On x86, I build the kernel and a light weight rootfs built with Buildroot. This time, I don't need the rootfs, but it's still easier to have Buildroot build Xenomai.
export $HOME/rpi/tools/arm-bcm2708/arm-rpi-4.9.3-linux-gnueabihf/bin:$HOME
export CROSS_COMPILE=arm-linux-gnueabihf-
To build a kernel that is just like the current kernel, I copy the kernel config from the known good kernel. This method has the advantage that you can just keep reusing the kernel modules that came with your install of Rasbian.
pi@raspberrypi$ sudo modprobe configs
pi@raspberrypi$ gunzip -c /proc/config.gz > ~/.config
Once I copy the .config file above to the Linux repo on the development host, I can bootstrap the linux repo (rpi-4.9.y branch to match the Rasbian image downloaded from the Raspberry site) by applying the .configs file saved above, like this (the make targets are as recommended by the above RPi document):
parallels@ubuntu:~/rpi/linux$ cp ../BAK/.config .
parallels@ubuntu:~/rpi/linux$ ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make oldconfig
Save this config as a defconfig by running this command:
parallels@ubuntu:~/rpi/linux$ ARCH=arm make savedefconfig
parallels@ubuntu:~/rpi/linux$ mv defconfig ../BAK/rpi_linux_defconfig
Cross-compiling the RPi kernel with Buildroot
Before building the Xenomai kernel, I practiced building a regular kernel that can still boot the Rasbian image downloadable from raspberrypi.org--following the steps outlined on the raspberrypi.org. One tip is to NOT use the toolchain recommended on the webpage; gcc version 4.8 has a bug that errors out during a kernel module build later. I found that the other toolchain, also available from the RPi tools repo works:export $HOME/rpi/tools/arm-bcm2708/arm-rpi-4.9.3-linux-gnueabihf/bin:$HOME
export CROSS_COMPILE=arm-linux-gnueabihf-
To build a kernel that is just like the current kernel, I copy the kernel config from the known good kernel. This method has the advantage that you can just keep reusing the kernel modules that came with your install of Rasbian.
pi@raspberrypi$ sudo modprobe configs
pi@raspberrypi$ gunzip -c /proc/config.gz > ~/.config
Once I copy the .config file above to the Linux repo on the development host, I can bootstrap the linux repo (rpi-4.9.y branch to match the Rasbian image downloaded from the Raspberry site) by applying the .configs file saved above, like this (the make targets are as recommended by the above RPi document):
parallels@ubuntu:~/rpi/linux$ cp ../BAK/.config .
parallels@ubuntu:~/rpi/linux$ ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make oldconfig
Save this config as a defconfig by running this command:
parallels@ubuntu:~/rpi/linux$ ARCH=arm make savedefconfig
parallels@ubuntu:~/rpi/linux$ mv defconfig ../BAK/rpi_linux_defconfig
To build just the kernel (no need to build the kernel modules if the config has not changed), just the zImage target is sufficient.
ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make zImage
The resulting binary arch/arm/boot/zImage can be copied to the SD card, and the config.txt can point to it:
kernel=zImage
Xenomai kernel patch is available as a i-pipe patch download from the download area. Since my RPi kernel version is 4.9.41, I found the patch that is as close to it as possible: ipipe-core-4.9.38-arm-3.patch
I then ran the script that Xenomai guys refer to in their install guide:
parallels@ubuntu:~/rpi/xenomai-3.0.6$ scripts/prepare-kernel.sh --linux=../linux.xeno --ipipe ../ipipe-core-4.9.38-arm-3.patch --arch=arm
Unfortunately, this fail because of the difference between the mainline kernel version 4.9.38 and the RPi kernel 4.9.41--despite being only 3 versions apart. I worked around by:
parallels@ubuntu:~/rpi/linux.xeno$ ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make zImage module
The built modules must be copied to the SD card's ext4 partition, pointing INSTALL_MOD_PATH at wherever the ext4 partition is mounted at on your dev host (when the SD card is inserted into the dev host).
sudo make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- INSTALL_MOD_PATH=<mount point> modules_install
The new kernel is still named zImage, and copied to the boot partition of the SD card. RPi bootloader's config.txt points at this zImage:
kernel=zImage
There is a tense moment when the modified SD card is inserted into the RPi and I turn on the RPi. I seem to have been lucky: this kernel boots up with the Rasbian image, and the Xenomai kernel patch ran during the kernel bootup:
pi@raspberrypi:~ $ dmesg | grep -Ei 'xeno|pipe'
[ 0.000000] I-pipe, 19.200 MHz clocksource, wrap in 960767920505705 ms
[ 0.000000] clocksource: ipipe_tsc: mask: 0xffffffffffffffff max_cycles: 0x46d987e47, max_idle_ns: 440795202767 ns
[ 0.001474] Interrupt pipeline (release #3)
[ 0.172468] clocksource: Switched to clocksource ipipe_tsc
[ 0.278993] [Xenomai] scheduling class idle registered.
[ 0.279039] [Xenomai] scheduling class rt registered.
[ 0.279289] I-pipe: head domain Xenomai registered.
[ 0.284432] [Xenomai] Cobalt v3.0.6 (Stellar Parallax)
Since Xenomai is a way to run real-time code in userspace, it makes sense that I need to build userspace libs.
parallels@ubuntu:~/rpi/xenomai-3.0.6$ scripts/bootstrap
parallels@ubuntu:~/rpi/xenomai-3.0.6$ ./configure --with-core=cobalt --enable-smp --host=arm-linux-gnueabihf CFLAGS="-mcpu=cortex-a7 -mfpu=neon-vfpv4 -mfloat-abi=hard" LDFLAGS="-mcpu=cortex-a7 -mfpu=neon-vfpv4 -mfloat-abi=hard"
I tried mcpu=cortex-a53, but that seemed to be incompatible with enable-smp option. Maybe gcc-4.9 doesn't know about cortex-a53. The --host argument should be the same as the CROSS_COMPILE environment variable, but without the trailing dash. This is consistent with the gcc option you can see when run gcc on Raspberry natively (gcc -v). I suspect that I can use the more modern version of the FPU, but Buildroot seems to use VFP4 as well (as of Buildroot version 2017.10), so I am not going to be aggressive about it. Keep the build in a separate folder.
parallels@ubuntu:~/rpi/xenomai-3.0.6$ make DESTDIR=~/build install
The build output will then be neatly packaged in ~/build folder.
parallels@ubuntu:~/rpi/xenomai-3.0.6$ ls ~/build
dev usr
$ cd ~/build
$ tar czf xeno-userspace.tar.gz *
$ cd usr/xenomai/include/
$ tar czf ../../../xeno-headers.tar.gz *
$ cd ../lib
$ tar czf ../../../xeno-libs.tar.gz *
These xenomai userspace lib has to be extracted to the root folder /:
pi@raspberrypi:~ $ cd /
pi@raspberrypi:/ $ sudo tar xzf ~/xeno-userspace.tar.gz
This will put the Xenomai device mount points in /dev folder, and the userspace headers and libs in /usr/xenomai folder. The lib folder has to be appended to the /etc/ld.so.conf.d:
pi@raspberrypi:/ $ echo "/usr/xenomai/lib/" | sudo tee /etc/ld.so.conf.d/xenomai.conf
pi@raspberrypi:/ $ sudo ldconfig -v
pi@raspberrypi:/ $ sync
pi@raspberrypi:/ $ sudo reboot
ldconfig modifies the dynamic linker's bindings to include Xenomai dylibs. sync to flush to the disk is an overkill in my opinion, but I thought a harmles neat trick.
pi@raspberrypi:/ $ sudo /usr/xenomai/bin/latency
Note that running Xenomai requires sudo privilege because the Xenomai devices are accessible only to the root user by default (I ignored the build/config option to expose the Xenomai device to a designated group, because I think I have to call mlockall as sudo anyway). Here are my typical results (tabulated ever second) on RPi3:
RTH|----lat min|----lat avg|----lat max|-overrun|---msw|---lat best|--lat worst
RTD| 1.310| 2.229| 8.602| 0| 0| -1.166| 32.842RTD| 1.252| 2.234| 10.471| 0| 0| -1.166| 32.842
I stressed the system by scp'ing 9 GB file (loading the Ethernet--USB really--and the filesystem and the SD card driver), and installing beefy APT packages like emacs. You can see that the worst case latency during the whole duration is greater than 2 orders of magnitude of the average latency.
RTT| 00:29:04 (periodic user-mode task, 1000 us period, priority 99)
RTH|----lat min|----lat avg|----lat max|-overrun|---msw|---lat best|--lat worst
RTD| 0.793| 2.277| 14.126| 0| 0| -1.961| 418.934
Therefore, it appears that while Xenomai is certainly much better than stock Linux, it still shouldn't be trusted to run hardware loops faster than 100~200 Hz (using my rule of thumb that the jitter should be < 1% of the sampling period).
Dec 20 17:40:49 raspberrypi kernel: [ 9091.384209] brcmfmac: brcmf_sdio_readframes: RXHEADER FAILED: -110
Dec 20 17:40:49 raspberrypi kernel: [ 9091.384232] brcmfmac: brcmf_sdio_rxfail: abort command, terminate frame, send NAK
Patching the RPi kernel with Xenomai Cobalt co-kernel
Before patching the currently working Linux kernel, I copy the kernel to another folder : ~/rpi/linux.xeno.Xenomai kernel patch is available as a i-pipe patch download from the download area. Since my RPi kernel version is 4.9.41, I found the patch that is as close to it as possible: ipipe-core-4.9.38-arm-3.patch
I then ran the script that Xenomai guys refer to in their install guide:
parallels@ubuntu:~/rpi/xenomai-3.0.6$ scripts/prepare-kernel.sh --linux=../linux.xeno --ipipe ../ipipe-core-4.9.38-arm-3.patch --arch=arm
Unfortunately, this fail because of the difference between the mainline kernel version 4.9.38 and the RPi kernel 4.9.41--despite being only 3 versions apart. I worked around by:
- deleting the patches that fail from the *.patch file
- applying the reduced patch
- manually applying the patch to the files that failed the automatic patch
- saving the resulting diff as a new patch file (git diff > ../ipipe-core-4.9.41-arm-3.patch)
Configuring the kernel for Cobalt
On recommendation from the Xenomai folks, I disabled the following kernel features:- CONFIG_CPU_FREQ
- CONFIG_CPU_IDLE
- CONFIG_KGDB
Build the kernel
Given the massive changes to the kernel, I can no longer get away with just building the kernel proper: I have to also build the kernel modules.parallels@ubuntu:~/rpi/linux.xeno$ ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make zImage module
The built modules must be copied to the SD card's ext4 partition, pointing INSTALL_MOD_PATH at wherever the ext4 partition is mounted at on your dev host (when the SD card is inserted into the dev host).
sudo make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- INSTALL_MOD_PATH=<mount point> modules_install
The new kernel is still named zImage, and copied to the boot partition of the SD card. RPi bootloader's config.txt points at this zImage:
kernel=zImage
There is a tense moment when the modified SD card is inserted into the RPi and I turn on the RPi. I seem to have been lucky: this kernel boots up with the Rasbian image, and the Xenomai kernel patch ran during the kernel bootup:
pi@raspberrypi:~ $ dmesg | grep -Ei 'xeno|pipe'
[ 0.000000] I-pipe, 19.200 MHz clocksource, wrap in 960767920505705 ms
[ 0.000000] clocksource: ipipe_tsc: mask: 0xffffffffffffffff max_cycles: 0x46d987e47, max_idle_ns: 440795202767 ns
[ 0.001474] Interrupt pipeline (release #3)
[ 0.172468] clocksource: Switched to clocksource ipipe_tsc
[ 0.278993] [Xenomai] scheduling class idle registered.
[ 0.279039] [Xenomai] scheduling class rt registered.
[ 0.279289] I-pipe: head domain Xenomai registered.
[ 0.284432] [Xenomai] Cobalt v3.0.6 (Stellar Parallax)
Since Xenomai is a way to run real-time code in userspace, it makes sense that I need to build userspace libs.
Building the Xenomai userspace
In the same scripts/ folder where I ran the prepare-kernel.sh above, there is "bootstrap" script, which generates the "configure" script.parallels@ubuntu:~/rpi/xenomai-3.0.6$ scripts/bootstrap
parallels@ubuntu:~/rpi/xenomai-3.0.6$ ./configure --with-core=cobalt --enable-smp --host=arm-linux-gnueabihf CFLAGS="-mcpu=cortex-a7 -mfpu=neon-vfpv4 -mfloat-abi=hard" LDFLAGS="-mcpu=cortex-a7 -mfpu=neon-vfpv4 -mfloat-abi=hard"
I tried mcpu=cortex-a53, but that seemed to be incompatible with enable-smp option. Maybe gcc-4.9 doesn't know about cortex-a53. The --host argument should be the same as the CROSS_COMPILE environment variable, but without the trailing dash. This is consistent with the gcc option you can see when run gcc on Raspberry natively (gcc -v). I suspect that I can use the more modern version of the FPU, but Buildroot seems to use VFP4 as well (as of Buildroot version 2017.10), so I am not going to be aggressive about it. Keep the build in a separate folder.
parallels@ubuntu:~/rpi/xenomai-3.0.6$ make DESTDIR=~/build install
The build output will then be neatly packaged in ~/build folder.
parallels@ubuntu:~/rpi/xenomai-3.0.6$ ls ~/build
dev usr
Deploying the Xenomai userspace
I created 3 separate tarballs in this folder (are the headers and the lib tarballs actually used?):$ cd ~/build
$ tar czf xeno-userspace.tar.gz *
$ cd usr/xenomai/include/
$ tar czf ../../../xeno-headers.tar.gz *
$ cd ../lib
$ tar czf ../../../xeno-libs.tar.gz *
These xenomai userspace lib has to be extracted to the root folder /:
pi@raspberrypi:~ $ cd /
pi@raspberrypi:/ $ sudo tar xzf ~/xeno-userspace.tar.gz
This will put the Xenomai device mount points in /dev folder, and the userspace headers and libs in /usr/xenomai folder. The lib folder has to be appended to the /etc/ld.so.conf.d:
pi@raspberrypi:/ $ echo "/usr/xenomai/lib/" | sudo tee /etc/ld.so.conf.d/xenomai.conf
pi@raspberrypi:/ $ sudo ldconfig -v
pi@raspberrypi:/ $ sync
pi@raspberrypi:/ $ sudo reboot
ldconfig modifies the dynamic linker's bindings to include Xenomai dylibs. sync to flush to the disk is an overkill in my opinion, but I thought a harmles neat trick.
Testing the Xenomai userspace
The latency test exercises the Xenomai libs:pi@raspberrypi:/ $ sudo /usr/xenomai/bin/latency
Note that running Xenomai requires sudo privilege because the Xenomai devices are accessible only to the root user by default (I ignored the build/config option to expose the Xenomai device to a designated group, because I think I have to call mlockall as sudo anyway). Here are my typical results (tabulated ever second) on RPi3:
RTH|----lat min|----lat avg|----lat max|-overrun|---msw|---lat best|--lat worst
RTD| 1.310| 2.229| 8.602| 0| 0| -1.166| 32.842RTD| 1.252| 2.234| 10.471| 0| 0| -1.166| 32.842
I stressed the system by scp'ing 9 GB file (loading the Ethernet--USB really--and the filesystem and the SD card driver), and installing beefy APT packages like emacs. You can see that the worst case latency during the whole duration is greater than 2 orders of magnitude of the average latency.
RTT| 00:29:04 (periodic user-mode task, 1000 us period, priority 99)
RTH|----lat min|----lat avg|----lat max|-overrun|---msw|---lat best|--lat worst
RTD| 0.793| 2.277| 14.126| 0| 0| -1.961| 418.934
Therefore, it appears that while Xenomai is certainly much better than stock Linux, it still shouldn't be trusted to run hardware loops faster than 100~200 Hz (using my rule of thumb that the jitter should be < 1% of the sampling period).
Show stopper: random crash after a few hours
I noticed system freezes when I left the latency test running overnight.Dec 20 17:40:49 raspberrypi kernel: [ 9091.384209] brcmfmac: brcmf_sdio_readframes: RXHEADER FAILED: -110
Dec 20 17:40:49 raspberrypi kernel: [ 9091.384232] brcmfmac: brcmf_sdio_rxfail: abort command, terminate frame, send NAK
Disabled wireless in general (bad idea for robust operation in real-world noisy environment) in /boot/config.txt.
dtoverlay=pi3-disable-bt
dtoverlay=pi3-disable-wifi
These steps do not seem to solve the problem. I still think it has to do with some device drivers expecting power management or CPU frequency adjustment, which I turned off at the kernel level above, but the root cause is unclear.
