Multi-monitor Buildroot x64 target

I am working on an application SW that displays to multiple monitors (somewhere between 2 to 4).  Eventually, I want to drive the multi-monitor display from an SoC, but working out the SW architecture with a multi-monitor GPU plugged into a PCIe slot of a modern PC is an excellent way to understand and derisk problems.  Before diving into a multi-monitor GPU, I can experiment with software supported multi-monitor setup in virtualbox.

NFS booting the virtual multi-monitor x64 target from a virtual Ubuntu server

In a previous blog entry, I PXE-booted a Buildroot x64 target (Dell Optiplex 755) from an Ubuntu server.  If I run both the target and the server as Virtualbox guests, I can use the Virtualbox internal network, which is completely a software network stack.  This is convenient when studying the kernel and software on a laptop, while away from the desktop server.

Setup virtualbox target

Created a virtualbox x64 guest with these settings:

• General, Basic
• Name: Target
• Type: Linux
• Version: Other Linux (64 bit)
• System
• Base memory: 512 MB
• Boot order: Network ONLY
• Display
• Video memory: 64 MB
• Monitor count: 4
• Enable 3D acceleration
• No storage, no audio, no serial port, no USB, no shared folder
• Network:
• 1st Adapter: NAT
• 2nd adapter: internal network "intnet", which should MATCH the name of the 2nd adapter's internal network for the server.

Cross-compiling the target on the virtual Ubuntu server

WARNING: Buildroot must be extracted and then built on a filesystem that supports softlinks (so don't do this on an NTFS!).

After getting the latest stable Buildroot as shown in a previous blog entry, I configure Buildroot with the following options.

• Target options: x86_64, corei2 (core7 selects SSE4 features, which Virtualbox does NOT support yet)
• Build options:
• enable ccache, but change the cache location to within the Buildroot directory (to avoid saving to the virtual HDD, and keep all work in the Vbox shared folder): $(TOPDIR)/.buildroot-ccache
• Optimization level 3
• Toolchain
• glibc
• Enable C++ support, necessary for Qt5
• Build cross gdb for the host: does NOT build on the vbox server for some reason.  Besides, the native gdb should just work
• Register toolchain within Eclipse Buildroot plug-in
• System configuration
• /dev management: eudev
• Kernel
• Using a custom--as supposed to in-tree--(def)config file: I need to add a couple of options for the NFS rootfs. I changed x86_64_defconfig from the kernel.org to create /home/henry/x64/BR2/kernel.config
• Target packages
• Debugging
• gdb: only the gdbserver
• Graphics
• mesa3d
• Gallium swrast (software OpenGL) driver
• OpenGL EGL
• OpenGL ES
• Qt5
• Approve free license
• Compile and install examples
• gui module
• widgets
• OpenGL: OpenGL ES 2.0 and opengl module
• linuxfb support
• eglfs support
• Default platform: linuxfb
• GIF, JPEG, PNG support
• Hardware handling
• lshw (does NOT even build!)
• pciutils (lspci)
• Networking applications
• openssh: necessary to connect to the target from gdb on the server

Kernel config for NFS rootfs

The x86_64_defconfig alsready has a few options I needed, so I just added the following:

CONFIG_NETWORK_FILESYSTEMS=y

CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_E1000E=y

The stock x86_64_defconfig has CONFIG_E1000 but does NOT contain CONFIG_E1000E, so this is just a safety measure.

Build and deploy the binaries

After Buildroot make runs at the top level, its output/images will contain bzImage.  Copy this to the tftp download on the Virtualbox host.  On Windows, this is in C:\Users\<your uid>\.VirtualBox\TFTP folder, as shown here:

The rootfs images need to be extracted to the NFS export on the server:

~/o755/buildroot$ sudo tar -C /export/root/o755/ -xf ~/o755/buildroot/output/images/rootfs.tar

Setup the Virtualbox NFS server

Create a virtualbox x64 guest, with at least 2 CPUs, as much memory as possible, and the network with 2 adapters:

• 1st adapter: NAT: for general Internet access (I am writing this blog on the host)
• 2nd adapter: internal network: to communicate with the target.  Assign a static IP address 192.168.2.1 in Ubuntu Unity network settings, so that the target can refer to the NFS server with a static IP address (see below).

Virtualbox NAT network already has a PXE enabled DHCP server.  On Windows, the folder C:\Users\<your account>\.VirtualBox\TFTP plays the role of /var/lib/tftpboot (by default) for the Ubuntu hpa-tftp server.  So the following files should be put into that folder:

• bzImage: compressed Linux kernel built by Buildroot, in its output/images (see above)
• Target.pxe: I copied this file from Ubuntu desktop's /usr/lib/syslinux/pxelinux.0.  VirtualBox DHCP server has a rule for mapping the PXE image for each virtual box by its <name>.pxe.  Since my target's name is "Target", the PXE binary should be named Target.pxe.
• menu.c32: This is the PXE menu program, copied verbatim from Ubuntu desktop's /usr/lib/syslinux/ folder.
• pxelinux.cfg/ folder, which will contain the PXE menu entry
• default: the catch-all menu.  PXE has lots of rules for matching the menu by the target's IP address, netmask, etc.  But default is the final fallback.  This file should point to the NFS rootfs the virtual server will host (see below).  My "default" file therefore looks like this:

DEFAULT menu.c32

PROMPT 0

MENU TITLE PXE Boot Menu

TIMEOUT 50 #This means 5 seconds

LABEL buildroot

 MENU LABEL buildroot kernel

 kernel o755Image

 append ip=192.168.2.3:192.168.2.1:192.168.2.1:255.255.255.0:o755:eth1:off root=/dev/nfs nfsroot=192.168.2.1:/export/root/o755 rw earlyprintk

The "ip" (used to be called nfsaddr) syntax is ip=<client-ip>:<server-ip>:<gw-ip>:<netmask>:<hostname>:<device>:<autoconf>

Setup the NFS server

The NFS server should get a static IP address.  I prefer to use the Ubuntu Unity tool, as you can see below.

Install the NFS server:

$ sudo apt-get install nfs-kernel-server

I configured the NFS server in /etc/exports, to serve any target in the "intnet":

/export         192.168.2.0/24(rw,fsid=0,insecure,no_subtree_check,async)

/export/root    192.168.2.0/24(rw,no_root_squash,no_subtree_check)

Remember to restart the NFS server after saving this file.  The root file system Buildroot generated should be expanded to the /export/root folder just mentioned, like this:

$ sudo mkdir /export/root/o755

$ sudo tar -C /export/root/o755/ -xf ~/o755/buildroot/output/images/rootfs.tar

nVidia Quadro multi-monitor video card

I bought an nVidia Quadro NVS 420 (Dell PN K722J)--I wanted an nVidia card because IMHO nVidia has the best Linux driver support--from eBay.  As you can see below, there was even a driver update earlier this year.

Now to the fine prints, from the driver README file: X is required, and glibc >= 2.0 is required.  The X server requirement is a deal-breaker for me: I want to keep the embedded distribution small.  Maybe a better alternative is to pick up an open source driver from the noveau project.  Quadro NVS 420 is supported under the NV50 Tesla family, code name NV98 (G98), as you can see on this page.

Buildroot config for Qt5, OpenCV, and nVidia Quadro 420 GPU

Since I am an embedded SW engineer, I treat even the PCs like targets (rather than desktops).  In a previous blog, I demonstrated an NFS booted Buildroot distribution for this Intel Core2 Duo Dell PC.  Except for updating Buildroot to the latest stable release (2015.02), I'll pick up from where I left off.

$ cd buildroot
$ git checkout 2015.02
$ git pull . 2015.02

The only difference in the Buildroot config between the virtual target and this real target is the Gallium nouveau driver (which supports all nVidia cards), under Target packages --> Graphics libraries and applications --> mesa3d.

To pick up the nouveau device driver, I added CONFIG_DRM_NOVEAU=y to the kernel defconfig file.

Also, I can connect over serial to the real target, by adding console=ttyS0,115200 to the kernel parameters in the pxelinux.cfg/default's "append" line.  During boot, the GPU is probed:

...
[    1.198462] nouveau  [  DEVICE][0000:03:00.0] BOOT0  : 0x298c00a2
[    1.204534] nouveau  [  DEVICE][0000:03:00.0] Chipset: G98 (NV98)
[    1.210605] nouveau  [  DEVICE][0000:03:00.0] Family : NV50
[    1.216169] nouveau  [   VBIOS][0000:03:00.0] checking PRAMIN for image...
[    1.233709] Console: switching to colour frame buffer device 160x64
[    1.245164] i915 0000:00:02.0: fb0: inteldrmfb frame buffer device
[    1.251332] i915 0000:00:02.0: registered panic notifier
[    1.307274] nouveau  [   VBIOS][0000:03:00.0] ... appears to be valid
[    1.313692] nouveau  [   VBIOS][0000:03:00.0] using image from PRAMIN
[    1.320231] nouveau  [   VBIOS][0000:03:00.0] BIT signature found
[    1.326303] nouveau  [   VBIOS][0000:03:00.0] version 62.98.6f.00.07
[    1.352876] nouveau 0000:03:00.0: irq 27 for MSI/MSI-X
[    1.352886] nouveau  [     PMC][0000:03:00.0] MSI interrupts enabled
[    1.359245] nouveau  [     PFB][0000:03:00.0] RAM type: GDDR3
[    1.364969] nouveau  [     PFB][0000:03:00.0] RAM size: 256 MiB
[    1.370868] nouveau  [     PFB][0000:03:00.0]    ZCOMP: 960 tags
[    1.378477] nouveau  [    VOLT][0000:03:00.0] GPU voltage: 1110000uv
[    1.915015] tsc: Refined TSC clocksource calibration: 2992.481 MHz
[    2.024020] nouveau  [  PTHERM][0000:03:00.0] FAN control: PWM
[    2.029844] nouveau  [  PTHERM][0000:03:00.0] fan management: automatic
[    2.036484] nouveau  [  PTHERM][0000:03:00.0] internal sensor: yes
[    2.062678] nouveau  [     CLK][0000:03:00.0] 03: core 169 MHz shader 358 MHz memory 100 MHz
[    2.071087] nouveau  [     CLK][0000:03:00.0] 0f: core 550 MHz shader 1400 MHz memory 700 MHz
[    2.079645] nouveau  [     CLK][0000:03:00.0] --: core 550 MHz shader 1400 MHz memory 702 MHz
[    2.088296] [TTM] Zone  kernel: Available graphics memory: 1001774 kiB
[    2.094802] [TTM] Initializing pool allocator
[    2.099147] [TTM] Initializing DMA pool allocator
[    2.103841] nouveau  [     DRM] VRAM: 256 MiB
[    2.108181] nouveau  [     DRM] GART: 1048576 MiB
[    2.112869] nouveau  [     DRM] TMDS table version 2.0
[    2.117987] nouveau  [     DRM] DCB version 4.0
[    2.122500] nouveau  [     DRM] DCB outp 00: 02000386 0f220010
[    2.128312] nouveau  [     DRM] DCB outp 01: 02000302 00020010
[    2.134124] nouveau  [     DRM] DCB outp 02: 040113a6 0f220010
[    2.139935] nouveau  [     DRM] DCB outp 03: 04011312 00020010
[    2.145747] nouveau  [     DRM] DCB conn 00: 00005046
[    2.150791] nouveau  [     DRM] DCB conn 01: 00006146
[    2.182206] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
[    2.188798] [drm] Driver supports precise vblank timestamp query.
[    2.247926] nouveau  [     DRM] MM: using M2MF for buffer copies

[    2.287545] nouveau 0000:03:00.0: No connectors reported connected with modes

[    2.294654] [drm] Cannot find any crtc or sizes - going 1024x768

[    2.302510] nouveau  [     DRM] allocated 1024x768 fb: 0x60000, bo ffff88007a126c00

[    2.310181] fbcon: nouveaufb (fb1) is primary device

[    2.310182] fbcon: Remapping primary device, fb1, to tty 1-63

[    2.453168] nouveau 0000:03:00.0: fb1: nouveaufb frame buffer device

[    2.459500] [drm] Initialized nouveau 1.2.1 20120801 for 0000:03:00.0 on minor 1

The GPU shows up as another framebuffer device /dev/fb1 (/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:00.0/0000:03:00.0/graphics/fb1).  The multiple folder level corresponds to the PCI switches built into the card (apparently), as can be seen from the lspci output:

01:00.0 PCI bridge: NVIDIA Corporation NF200 PCIe 2.0 switch for Quadro Plex S4 / Tesla S870 / Tesla S1070 / Tesla S2050 (rev a3)
02:00.0 PCI bridge: NVIDIA Corporation NF200 PCIe 2.0 switch for Quadro Plex S4 / Tesla S870 / Tesla S1070 / Tesla S2050 (rev a3)
02:02.0 PCI bridge: NVIDIA Corporation NF200 PCIe 2.0 switch for Quadro Plex S4 / Tesla S870 / Tesla S1070 / Tesla S2050 (rev a3)
03:00.0 VGA compatible controller: NVIDIA Corporation G98 [Quadro NVS 420] (rev a1)

I can see Qt GUI drawing to fb1 by running Qt examples like these:

/usr/lib/qt/examples/opengl/2dpainting/2dpainting -platform linuxfb:fb=/dev/fb1

Of course, OpenGL display doesn't work against the linuxfb platform.  But there seems to be a problem with the mesa3d nouveau platform I compiled, because gbm cycles through a few Gallium drivers EXCEPT the one I actually built: nouveau_dri, which is even in the search path (/usr/lib/dri).

# ./application -platform eglfs
gbm: failed to open any driver (search paths /usr/lib/dri)
gbm: Last dlopen error: /usr/lib/dri/i915_dri.so: cannot open shared object file: No such file or directory
...
Could not initialize egl display

I realized that eglfs platform is querying the framebuffer property from /dev/fb0 even though I set the environment variable QT_QPA_EGLFS_FB to /dev/fb1.  So I removed the Intel GPU from the picture by commenting out the Intel GPU drivers from the kernel config.  And now I see this message:

# ./openglwindow -platform eglfs
Unable to query physical screen size, defaulting to 100 dpi.
To override, set QT_QPA_EGLFS_PHYSICAL_WIDTH and QT_QPA_EGLFS_PHYSICAL_HEIGHT (in millimeters).
EGL Error : Could not create the egl surface: error = 0x300b
Aborted

The error (EGL_BAD_NATIVE_WINDOW) is logged in src/plugins/platforms/eglfs/qeglfswindow.cpp, QEglFSWindow::resetSurface(), but thrown in eglCreateWindowSurface:

    if (!rx::IsValidEGLNativeWindowType(win))
    {
        recordError(egl::Error(EGL_BAD_NATIVE_WINDOW));
        return EGL_NO_SURFACE;
    }

This begs the question: what is a Qt native window?  I decided that I still do NOT know the low level graphics SW stack, and just stick to either linuxfb or directfb.

directfb: the lowest level display abstraction in userspace

DirectFB (Direct Frame Buffer) is a software library with a small memory footprint that provides graphics acceleration, input device handling and abstraction layer, and integrated windowing system with support for translucent windows and multiple display layers on top of the Linux framebuffer without requiring any kernel modifications.  DirectFB allows applications to talk directly to video hardware through a direct API, speeding up and simplifying graphic operations.

But directfb does not support Quadro (noveau) drivers?  See directfb/gfxdrivers/nvidia/nvidia.c  Platform independent examples src is in buildroot/output/build/directfb-examples-1.6.0/srcles

Each GPU detected by DRM is referred as a DRM device, and a device file /dev/dri/cardX (where X is a sequential number) is created to interface with it, as in this example for the NVS420 card in a PC:

# ls /dev/dri
card0       controlD64  renderD128

Note that on Zedboard which lacks a GPU, Lars Clausen (of ADI)'s adv7511 DRM driver does not offer the renderD128 file:

# ls /dev/dri

card0       controlD64

User space programs that want to talk to the GPU must open the file and use ioctl calls to communicate with DRM. Different ioctls correspond to different functions of the DRM API.  A library called libdrm was created to facilitate the interface of user space programs with the DRM subsystem, as shown here:

This library is merely a wrapper that provides a function written in C for every ioctl of the DRM API, as well as constants, structures and other helper elements.  DRM consists of two parts: a generic "DRM core" and a specific one ("DRM Driver") for each type of supported hardware.  DRM driver, on the other hand, implements the hardware-dependent part of the API, specific to the type of GPU it supports; it should provide the implementation to the remainder ioctls not covered by DRM core, but it may also extend the API offering additional ioctls with extra functionality, for which extra userspace library is offered.  For the nVidia card, we therefore see:

# ls /usr/lib/libdrm*

/usr/lib/libdrm.so.2.4.0   /usr/lib/libdrm_nouveau.so.2.0.0

But strangely, Zedboard has enumerated a non-existent card (Adreno)?

# ls -Lh /usr/lib/libdrm*

/usr/lib/libdrm.so.2.4.0   /usr/lib/libdrm_freedreno.so.1.0.0

GEM (graphics executaion manager) manages graphics buffers.  Through GEM, a user space program can create, handle and destroy memory objects living in the GPU's video memory.  Confusingly, there are mesa3d userspace drivers that use the kernel drivers, as you can see in the AMD example below:

As the demand for better graphics increased, hardware manufacturers created a way to decrease the amount of CPU time required to fill the framebuffer. This is commonly called "graphics accelerating".  Common graphics drawing commands (many of them geometric) are sent to the graphics accelerator in their raw form. The accelerator then rasterizes the results of the command to the framebuffer.

Debugging a full screen directfb application

The df_fire example did NOT run against the nouveau driver, so let's debug into it.  Following relevant build variables are in the Makefile:

CFLAGS = -Wall -O3 -pipe -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -pipe -O3  -Werror-implicit-function-declaration

CPPFLAGS = -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64

DIRECTFB_CFLAGS = -D_REENTRANT -I/home/henry/work/o755/buildroot/output/host/usr/x86_64-buildroot-linux-gnu/sysroot/usr/include/directfb  

DIRECTFB_LIBS = -ldirectfb -lfusion -L/home/henry/work/o755/buildroot/output/host/usr/x86_64-buildroot-linux-gnu/sysroot/usr/lib -ldirect -lpthread  

AM_CFLAGS = -D_REENTRANT -I/home/henry/work/o755/buildroot/output/host/usr/x86_64-buildroot-linux-gnu/sysroot/usr/include/directfb   -D_GNU_SOURCE

LIBADDS = \

        -ldirectfb -lfusion -L/home/henry/work/o755/buildroot/output/host/usr/x86_64-buildroot-linux-gnu/sysroot/usr/lib -ldirect -lpthread  

AM_CPPFLAGS = \

        -DDATADIR=\"${datarootdir}/directfb-examples\" \

        -DFONT=\"$(fontsdatadir)/decker.ttf\"

Trying the Quadro NVS 420 nVidia card (model G98) on Ubuntu desktop

I could not get Ubuntu to see the DLP 3010 evaluation module, so I checked whether the nVidia proprietary drivers could The list of driver updates can be queried:

henry@o755:~$ sudo ubuntu-drivers devices

== /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:00.0/0000:03:00.0 ==

model    : G98 [Quadro NVS 420]

modalias : pci:v000010DEd000006F8sv000010DEsd0000057Ebc03sc00i00

vendor   : NVIDIA Corporation

driver   : nvidia-304-updates - distro non-free

driver   : xserver-xorg-video-nouveau - distro free builtin

driver   : nvidia-331 - distro non-free recommended

driver   : nvidia-331-updates - distro non-free

driver   : nvidia-304 - distro non-free

driver   : nvidia-173 - distro non-free

An easy way to install the proprietary device drivers is through System Settings (unity-control-center) --> System --> Software & Updates --> Additional Drivers.  Even after installing the nVidia proprietary driver, the 2nd monitor would not enumerate; I had to run the nVidia Xserver settings tool, and explicitly detect the display for the Xserver settings to update.  I think this means that nouveau driver cannot drive NVS420 card in multi-monitor mode.

Indeed, the computer only sees 1 framebuffer device:

$ ls -lhg /sys/class/graphics/
total 0
lrwxrwxrwx 1 root 0 Apr 25 12:06 fb0 -> ../../devices/pci0000:00/0000:00:02.0/graphics/fb0
lrwxrwxrwx 1 root 0 Apr 25 11:50 fbcon -> ../../devices/virtual/graphics/fbcon

As this nouveau multi-monitor setup explanation shows, it is the X server that lays out the multi-monitor on the desktop.  Multiple monitors has to be organized as 1 logical screen, because an X application can only display to 1 screen (i.e. application cannot choose to run on multiple screens, nor change from 1 screen to another dynamically).