SMP Linux on a SINGLE core in a dual core CPU

In recent posts on buildroot generated root file system and NFS RFS, I got a uClinux running on the Zedboard (rev c), which has dual ARM Cortex A9 cores in the Zynq 7020 CPU package.  Looking at /proc/cpuinfo confirms that Linux kernel is running on both cores.  But I want to use a core on a multi-core platform for hard real-time SW, so in this blog entry I will study running uClinux on a single core.

Perhaps a dozen years ago or so, the whole computer industry made the difficult transition to multi-core HW and SW.  Back then, I had to configure SMP explictly; now, I have to explicitly configure to NOT use all the cores in the HW!  Thesedays, I am not even sure if Linux is even tested on a single core in a multi-core platform, so the 1st thing I try is adding the "maxcpus=1" u-boot bootarg, like this:

setenv bootargs 'console=ttyPS0,115200 root=/dev/nfs ip=192.168.1.9 nfsroot=192.168.1.2:/export/root/zed rw earlyprintk maxcpus=1' boot

Unfortunately, the kernel froze:

SMP: Total of 1 processors activated. CPU: All CPU(s) started in SVC mode. ... io scheduler noop registered io scheduler deadline registered io scheduler cfq registered (default)

I googled around for "Zynq maxcpus=1 hang" and found other people having similar but not quite the same problems in this forum, and slept on the problem.  But when I woke up and tried it again, the kernel started up without a problem and gave me a login prompt!  And it kept working without a problem since.  The number of CPUs is verified with this command

$ cat /proc/cpuinfo

I was thinking about letting Linux boot all CPUs and then powering off the last CPU with this kind of command, but I guess I won't need it for now.

$ echo 0 >> /sys/devices/system/cpu/cpu1/online

Minimal Zedboard rootfs produced by buildroot

In my previous attempts at creating a functional GUI platform on Linaro Ubuntu desktop distribution, I learned that there is just too much going on in a large Linux distribution to really understand a problem and troubleshoot it.  Therefore, this is a bottom-up approach to creating a root file system.

Get Buildroot

Packages required by Buildroot (for xconfig--the QT4 GUI; GTK GUI option requires GTK2+ dev packages, which I don't want to bother with):

sudo apt-get install \ build-essential gawk bison flex gettext texinfo patch gzip bzip2 perl tar wget cpio python unzip rsync libqt4-dev

Get the buildroot source.

$ git clone git://git.buildroot.net/buildroot

And then switch over to the latest stable branch (which is 2014.08 right now)

$ git checkout 2014.08 $ git pull . 2014.08

Configure Buildroot

Spencer Gilliland already checked in a config for Zedboard, so I compared it with what I did in the "Ubuntu on Zedboard" tutorial.

BR2_arm=y BR2_cortex_a9=y BR2_ARM_ENABLE_NEON=y BR2_KERNEL_HEADERS_VERSION=y BR2_DEFAULT_KERNEL_VERSION="3.8" BR2_PACKAGE_HOST_LINUX_HEADERS_CUSTOM_3_8=y BR2_TARGET_GENERIC_GETTY_PORT="ttyPS0"	From the Avnet "Ubuntu on Zedboard" tutorial I went through, the kernel from ADI is version 3.15.  ttyPS0 matches the console device used in "Ubuntu on Zedboard" tutorial.
BR2_LINUX_KERNEL=y BR2_LINUX_KERNEL_CUSTOM_GIT=y BR2_LINUX_KERNEL_CUSTOM_REPO_URL= "git://github.com/Xilinx/linux-xlnx.git" BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION= "xilinx-v2014.1" BR2_LINUX_KERNEL_DEFCONFIG="xilinx_zynq" BR2_LINUX_KERNEL_UIMAGE_LOADADDR="0x8000"	Note the CUSTOM_GIT; I need the kernel from ADI https://github.com/analogdevicesinc/linux.git xcomm_zynq branch.  DEFCONFIG target is zynq_xcomm_adv7511.  uImage is made with mkimage produced from U-boot (then soft lined to /usr/bin/mkimage).  Load addr is the same.
BR2_LINUX_KERNEL_DTS_SUPPORT=y BR2_LINUX_KERNEL_INTREE_DTS_NAME="zynq-zed"	In "Ubuntu on Zedboard" tutorial, the DTB file produced was zynq-zed-adv7511.dtb, was renamed to devicetree.dtb in the SD card BOOT partition.
BR2_TARGET_ROOTFS_CPIO=y BR2_TARGET_ROOTFS_CPIO_GZIP=y BR2_TARGET_ROOTFS_CPIO_UIMAGE=y	Root file system
BR2_TARGET_UBOOT=y BR2_TARGET_UBOOT_BOARDNAME="zynq_zed" BR2_TARGET_UBOOT_CUSTOM_GIT=y BR2_TARGET_UBOOT_CUSTOM_REPO_URL= "git://github.com/Xilinx/u-boot-xlnx.git" BR2_TARGET_UBOOT_CUSTOM_REPO_VERSION= "xilinx-v2014.1" BR2_TARGET_UBOOT_FORMAT_IMG=y BR2_TARGET_UBOOT_SPL=y BR2_TARGET_UBOOT_SPL_NAME="boot.bin"	The git repository is the same as "Ubuntu on Zedboard" tutorial, but this branch is newer than the 2013.4 used in "Ubuntu on Zedboard" tutorial.  Config target is the same.  It would be nice if there is a way to specify the default bootargs (like maxcpus=1 that I will need to specify eventually). Buildroot should NOT create the boot.bin for Zynq, because it has no way to insert the bitstream file into boot.bin.

Note that the toolchain was NOT specified above.

Fire up the config tool

$ make xconfig

Load the above config in the xconfig GUI (menu --> File --> Load).  Going through the items, I saw some strange choices:

• Target options
• Target ABI: shouldn't it be EABIhf rather than EABI (default)?  No unless the toolchain actually uses EABIhf as well, which the XSDK (Sourcery Codebench) toolchain doesn't.  This blog (dated 2010) discussed the feature to my satisfaction; soft float still uses the FP HW, but the arguments are passed through the integer registers rather than special registers for FP.
• Shouldn't the floating point strategy be NEON, rather than Soft float, which (according to the above blog) speeds up the floating point register use in function calls?  The risk is not in rebuilding the toolchain in buildroot, but whether all the SW I need have been ported to EABIhf.  I decided not to take the risk, but only in the 2nd pass.

So for the 1st pass, here are the options I am using that are different than configs/zedboard_defconfig:

• Build options
• Compile cache disabled.  This may be helpful later when the packages are debugged on my system.
• gcc optimization level: I changed from "size" to "level 3", because there is plenty of storage, but the CPU is relatively slow.
• Toolchain: I tried to use Xilinx SDK as the external tool, but it did not work out, so left it at Buildroot toolchain
• Kernel headers: I tried to specify Linux version 3.15.0, but this failed, so I just specified Linux 3.16.x kernel headers and cross my finger.
• C library: uClibc
• System configuration
• System hostname: zed
• System banner: "Welcome to Zedboard!"
• Root password: I changed from the empty (no password) to something else
• Check "install timezone info"
• default local time: America/Los_Angeles
• Path to users tables: /mnt/work/zynq/BR2/ROOTFS_USERS_TABLES, which looks like this:
  

  foo -1 wheel -1 =<password> /home/foo /bin/sh - Foo user

• Kernel
• Change kernel version from "Custom Git repository" to "Local directory", because I want to use my own defconfig (named zynq_xcomm_adv7511_nfs, for NFS root filesystem) which cannot be checked into the original git source (ADI github account).  For now, I am keeping a clone of the ADI Zynq kernel at /mnt/work/zynq/kernel.
• Load address (unchanged): 0x8000
• Device tree support
• DTS source file name: zynq-zed-adv7511
• Target packages
• Shell and utilities: screen, sudo, vi (note that vi will bin in /bin rather than /usr/bin, so that you have to make a symbolic link for visudo command later)
• Filesystem images
• Uncheck "cpio the root filesystem (for use as an initial RAM filesystem), because I want to NFS export the root filesystem for now.
• When I also unchecked "tar the root filesystem", I did not get ANY root file system (output/target LOOKS like RFS, but in that folder, there is a file THIS_IS_NOT_YOUR_ROOT_FILESYSTEM).
• Bootloaders: U-Boot
• U-Boot board name: zynq_zed
• Custom repository version: xilinx-v2013.4
• U-Boot binary format: u-boot.elf
• Uncheck "U-Boot SPL support", because I am using Xilinx FSBL (necessary for AMP).
• Environment image: I should "saveenv" on u-boot shell, and then copy that text file to the host, and use this feature.

Build

Save the changes (menu --> File --> Save As) to .config file, then kick off make.  The build went smoothly until building u-boot, then just stopped without any error.  I CD'ed into the u-boot folder and just ran make there to finish building u-boot, then came back to the top of the buildroot and retried make again, which went through this time.  The build products are in output/images folder:

$ file output/images/* output/images/rootfs.tar:           POSIX tar archive (GNU) output/images/u-boot:               ELF 32-bit LSB  shared object, ARM, EABI5 version 1 (SYSV), statically linked, not stripped output/images/uImage:               u-boot legacy uImage, Linux-3.15.0, Linux/ARM, OS Kernel Image (Not compressed), 3124128 bytes, Wed Sep 24 19:50:31 2014, Load Address: 0x00008000, Entry Point: 0x00008000, Header CRC: 0xB6EDEAD5, Data CRC: 0xC213B552 output/images/zynq-zed-adv7511.dtb: data

Recall in "Ubuntu on Zedboard" tutorial, I renamed a few files when copying to the SD card:

• u-boot to u-boot.elf, for the Xilinx FSBL to find it in the boot partition
• zynq-zed-adv7511.dtb to devicetree.dtb , because that is the filename u-boot is looks for

Buildroot imperfection induced chores

It looks like Buildroot ignored my request for u-boot.elf, and I will have to write a post-build script to rename the zynq-zed-adv7511.dtb to devicetree.dtb, and untar the rootfs.tar in the /export/root folder.  For the first time, do it manually.

Recall setting up the NFS export on the development host in another blog entry.  Let's copy the boot.bin from XSDK (generated for the "Ubuntu on Zedboard" tutorial, u-boot.elf, uImage, and devicetree.dtb to an SD card's FAT32 BOOT partition first, and then wipe out the ext4 rootfs partition I used to use for the root file system, to make sure I don't accidentally use that root file system.

The rootfs.tar generated by buildroot was extracted to the NFS export (as root, not just with sudo), and when I booted the Zedboard, I was able to log in finally as the user I created in ROOTFS_USERS_TABLES, which looks like this:

foo -1 wheel -1 =<clear passwd> /home/foo /bin/sh - Foo user

Add group wheel to sudoers

The group "wheel" is a choice made to easily allow "foo" into the sudoers file, which already has a COMMENTED OUT entry:

## Uncomment to allow members of group wheel to execute any command # %wheel ALL=(ALL) ALL                                                           ## Same thing without a password                                    # %wheel ALL=(ALL) NOPASSWD: ALL

All I have to do is uncomment either of the above lines for user foo to get sudo, but there doesn't seem to be a sudoers template file in <BR2>/system/skeleton, so I went straight to the template in to sudo package: <BR2>/output/target/etc/sudoers, which CAN be erased when you blow away output/build.

Add DNS server

If I specify "ip=dhcp" to the kernel boot args, the target SHOULD get a name server through DHCP, as in this dmesg snippet:

IP-Config: Got DHCP answer from 192.168.1.1, my address is 192.168.1.9
IP-Config: Complete:
     device=eth0, hwaddr=00:0a:35:00:01:22, ipaddr=192.168.1.9, mask=255.255.255.0, gw=192.168.1.1
     host=192.168.1.9, domain=, nis-domain=(none)
     bootserver=0.0.0.0, rootserver=192.168.1.2, rootpath=
     nameserver0=192.168.1.1

Unfortunately, the nameserver0 value does not seem to make its way to the /etc/resolv.conf, because Buildroot creates the /etc/resolv.conf as a softlink to /tmp/resolv.conf, which does NOT exist, even after booting.  I can work around this by changing /exports/root/zed/etc/resolv.conf softlink into a file containing a single line:

nameserver 192.168.1.1

I tried to change the file in <BR2>/output/target/skeleton/etc, but that was overwritten for some reason.

Once the DNS works, I can run ntpdate on the target to correct the time, like this:

# ntpdate ntp.ubuntu.com

 7 Dec 22:50:05 ntpdate[798]: step time server 91.189.94.4 offset 1417992459.172424 sec

I can probably put this command in cron (or dcron, if using busybox).  TODO.

Boot the target

When I pop in the SD card into the Zedboard's SD card slot, with the boot mode pin set to 5'b01100, the targets boots up all the way to user shell in about 20 seconds:

... VFS: Mounted root (nfs filesystem) on device 0:12. devtmpfs: mounted Freeing unused kernel memory: 204K (c0612000 - c0645000) Starting logging: OK Initializing random number generator... random: dd urandom read with 32 bits of entropy available done. Starting network... ip: RTNETLINK answers: File exists Welcome to Zeboard! zed login: foo Password:  $

Serving up root file system over NFS on an Ubuntu development host

Until now, I've been writing the Zedboard root file system on SD card (and dealing with the vagaries of the SD card and card readers).  After a mixed success with the downloaded Zedboard HDMI and Linux tutorials, I've decided to bite the bullet and study the low level details of HDMI display on Zedboard.  Trial and error is inevitable; for fast turnaround, writing the root file system to the SD card is just too inconvenient.  I will keep the root file system on the development host and just serve them up over NFS.

Fixed IP address for the Ubuntu development host

I have been running with default networking settings on my Ubuntu PC since installing 14.04.1 LTS, which means I am using DHCP at the moment.

1. I changed my home router to use 192.168.1.10~ for the DHCP address, so I can assign the single digits as fixed addresses.
2. Added following to /etc/network/interfaces file

   iface eth0 inet static address 192.168.1.2 netmask 255.255.255.0 gateway 192.168.1.1 dns-nameservers 75.75.75.75 8.8.8.8

3. Restarted the host for the setting to change (because the following command, which used to work on Ubuntu as of 8 LTS or so did not work any more on Ubuntu 14 LTS)
   

   sudo /etc/init.d/networking restart

4. Check
   

   henry@Zotac64:~$ ifconfig eth0; ping -c 1 google.com eth0      Link encap:Ethernet  HWaddr 00:01:2e:2f:25:26             inet addr:192.168.1.2  Bcast:192.168.1.255  Mask:255.255.255.0 ... PING google.com (74.125.239.97) 56(84) bytes of data. 64 bytes from nuq05s01-in-f1.1e100.net (74.125.239.97): icmp_seq=1 ttl=55 time=13.2 ms

NFS server on the Ubuntu host

Followed Ubuntu help page: https://help.ubuntu.com/community/SettingUpNFSHowTo.  On the server end:

1. Install Ubuntu package
   

   $ sudo apt-get install nfs-kernel-server

2. Confirm that NFS security is NOT applied in /etc/default/nfs-kernel-server

   # Do you want to start the svcgssd daemon? It is only required for Kerberos # exports. Valid alternatives are "yes" and "no"; the default is "no". NEED_SVCGSSD=""

3. Confirm that NFS ID mapping uses nobody:nogroup in /etc/idmapd.conf

   [Mapping] Nobody-User = nobody Nobody-Group = nogroup

4. And that these IDs are already registered in /etc/passwd and /etc/groups.  This way, the target does not need to have the same user and group as the host.
   

   $ grep nogroup /etc/group nogroup:x:65534: $ grep nobody /etc/passwd nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin

5.  Create the folders to export.
   

   $ sudo mkdir -p /export/root $ sudo chmod -R 777 /export

6. Export the folders in /etc/exports to all hosts in 192.168.[0.1|2|3].0 subnet.  Note the shorter subnet mask length (by 2 bits)
   

   /export       192.168.0.0/22(rw,fsid=0,insecure,no_subtree_check,async) /export/root  192.168.0.0/22(rw,no_root_squash,no_subtree_check)

7. Restart the server
   

   sudo service nfs-kernel-server restart

NFS enabled kernel

In the kernel folder, see Documentation/filesystems/nfs/nfsroot.txt.  This is what I did after reading it:

In (x|g|menu)config window, navigate to File systems --> check Network File Systems --> check "NFS client support NFS version 4" because the host's nfs-kernel-server supports NFS 4.  (a bit tricky: you have to explicitly CHECK--meaning, built-in rather than as kernel module--"NFS client support").  Once you check it, the "Root file system on NFS" appears, and you can check it.  The additional CONFIG entries for required for NFS root file system in .config file are shown below.  Apparently, the first line must replace the #CONFIG_NETWORK_FILESYSTEMS not set (some kind of override rule)

CONFIG_NETWORK_FILESYSTEMS=y CONFIG_NFS_FS=y CONFIG_NFS_V2=y CONFIG_NFS_V3=y # CONFIG_NFS_V3_ACL is not set CONFIG_NFS_V4=y # CONFIG_NFS_SWAP is not set # CONFIG_NFS_V4_1 is not set CONFIG_ROOT_NFS=y CONFIG_NFS_USE_KERNEL_DNS=y # CONFIG_NFSD is not set CONFIG_NFS_COMMON=y

These changes can be appended to the defconfig used for the kernel config used for the kernel build (I used buildroot).

Target configuration

I explored how the Zedboard (Xilinx Zynq target) boots in another post, but to recap briefly:

1. BootROM in the chip memory determines the boot mode (IO pins on Zynq) and loads the boot image from the boot device (in my case, SD card so far) into OCM (On Chip Memory), then starts running the FSBL.  Therefore, there is no programming involved here.
2. FSBL (Xilinx SDK supplied first stage boot loader) loads the bitstream (also in the boot image) and programs the FPGA, then loads all ELF files in the boot partition (which contains the boot image) into respective load address in DRAM, then starts the 1st application.  In this case, there is only 1 ELF file: U-Boot, which is the SSBL (second stage boot loader for Linux).
3. U-Boot determines the boot mode (in this case SD), and should execute the corresponding boot script (in this case sdboot).
1. Regardless of the boot mode, a boot script should load the Linux kernel into memory.
2. Optionally but usually for ARM, device tree binary is also loaded into a different memory address and passed to the kernel for use during boot.
3. Also optionally, a initramfs (of the root file system) can be loaded into the memory.  Since the subject of this blog is serving RFS over NFS, this is clearly unnecessary.
4. Linux kernel starts running the init process (PID 0), which launches all child processes.

I built U-Boot within Buildroot as well (so Buildroot for everything except the FSBL: SSBL, kernel, and root filesystem).  Unfortunately, I'd have to write some scripts to customize the default load scripts (in this case sdboot) within Buildroot.  But before automation, let's configure this manually.  By default, U-Boot waits 3 seconds before starting the bootup.  I halt it, and entered the following commands to set U-Boot environment variables:

setenv ipaddr 192.168.1.9 setenv serverip 192.168.1.2 setenv sdboot 'if mmcinfo; then run uenvboot; echo Copying Linux from SD to RAM...RTF in ext4 && fatload mmc 0 0x3000000 ${kernel_image} && fatload mmc 0 0x2A00000 ${devicetree_image} && bootm 0x3000000 - 0x2A00000; fi' setenv bootargs 'console=ttyPS0,115200 ip=192.168.1.9:192.168.1.2:192.168.1.1:255.255.255.0 root=/dev/nfs nfsroot=192.168.1.2:/export/root/zed rw earlyprintk' saveenv

Let's go over the above, in slow motion.  As discussed in the 1st section, the NFS server's static IP address is 192.168.1.2 in my home network.  I picked the static IP address of 192.168.1.9 for the target.  The sdboot command is straight from the "Ubuntu on Zedboard" tutorial.  The console device has to be in DTB, and earlyprintk is a good debugging option.  The NFS root is specified as /dev/nfs, with the other options trailing it.  I INTEND TO write files into the root file system, so the "rw" option for the file system is important.  Since I use static IP address for the target, I specify the gateway and netmask in the "ip" kernel boot parameter (the syntax is ip=<target IP>:<server IP>:<gateway IP>:<netmask>:<hostname>:<eth device>).  I wish I can also specify the DNS name server in the same line, but that can only be done in /etc/resolv.conf, which will require modifying the Buildroot filesystem skeleton.  I could specify DNS server ip in the U-Boot environment, but that will not get passed to the kernel.

A workaround is to reserve a static IP address for the target on the DHCP server (I use Netgear), and simply use DHCP in the bootarg:

setenv bootargs 'console=ttyPS0,115200 ip=dhcp root=/dev/nfs nfsroot=192.168.1.2:/export/root/zed rw earlyprintk'

Either way, if I let U-Boot start now, without the root file system actually copied to /export/root/zed, then the kernel will start but soon panic.  This leads us to...

Root file system

The root file system should be extracted out to the export folder; symbolic link is probably not going to work (TODO try out symbolic link).

~/work/zynq/buildroot/output/images$ mkdir /export/root/zed ~/work/zynq/buildroot/output/images$ sudo su [sudo] password for henry:  # tar -C /export/root/zed -xf /mnt/work/zynq/buildroot/output/images/rootfs.tar

Extracting the tar as user root (rather than as regular user using sudo) and the no_root_squash nfs-kernel-server option were the keys to success, after many tries.