Dec 23, 2014

Zedboard for data center SW/HW development

In my previous blog entries (this and this), I started to explore using Zedboard as a instrument controller platform.  Zedboard is also a fine option for early stage prototype for an ARM server running special applications with custom FPGA logic within a data center.  To be sure, data centers are currently overwhelmingly filled with x64, and I explored how to integrate custom data processing HW into an x64 server with a PCIe slot in a blog entry a few years ago.  Zedboard does NOT have a PCIe slot (Avnet MiniITX does have a PCIe Gen2 x4) but NOT having to deal with PCIe when debugging the FPGA logic is actually productivity booster!  Once the logic is proven, I can use Xillybus to cross the PCIe barrier.  So in this article, I explore using Buildroot to run a minimal Linux distribution and a few data-center-ish applications on Zedboard (rev c).

Modify Buildroot config

  • Toolchain
    • C library: glibc 2.19, because data center applications will expect glibc for the most part, and Ubuntu 14.04 LTS (which most data center applications WILL support) is on glibc 2.19 (run "ldd --version" to check).
    • binutils 2.24 (Ubuntu 14.04 LTS is on 2.24, according to this)
    • gcc 4.8.x
    • Build cross gdb for the host: gdb 7.7.x (check "gdb -v" on the host)
    • Register toolchain without Eclipse Buildroot plugin, because I want the Buildroot Eclipse CDT integration.
  • System configuration
    • /dev management: Dynamic using eudev (servers run udev, and this is the closest thing Buildroot offers)
    • Unchecked "Install timezone info"
  • Target packages
    • Debugging, profiling, and benchmarking
      • gdb: just the gdbserver
      • oprofile
      • trace-cmd
      • NO valgrind: valgrind is slow even on the host; on a slow target like on Zynq, it will probably be too slow
    • Graphic libraries and applications: turn off everything
    • Libraries
      • Graphics: turn off everything
      • Hardware handling: turn off everything
      • JSON/XML: turn off everything
    • Networking
      • Uncheck dropbear (does not support FTP) and check openssh

Build the Buildroot and change rootfs as necessary to work around buildroot bugs

See my previous blog entry.  Also, setup Buildroot eclipse integration by first installing eclipse CDT

sudo apt-get install eclipse eclipes-cdt

Do NOT run above command, because Ubuntu eclipes package is severely down-revved for this reason, which is a problem because Buildroot integration does not seem to install on Juno (Eclipse 3.8).  So instead, install maven first, and then download the latest Eclipse for C/C++ from here, and then unzip to your home directory, so that you can ~/eclipse/eclipse

Now for the Buildroot integration: according to this this recipe, Buildroot Eclipse 3.x integration should point http://buildroot.org/downloads/eclipse/luna.  I thought I should try to keep the different Buildroot folders (for ARM vs. x86 for example) separate by putting the Eclipse workspace/ folder within the ~/work/zed folder, but as you can see below, the Buildroot Eclipse plugin discovers all Buildroot roots anyway (how??):

Boot the target

See this and this.

Write userspace applications running on Buildroot

See this blog entry.

Debug userspace application from Eclipse CDT

Eclipse CDT can debug userspace application project over ssh connection.  It can also do that through other dstore and supposedly that is easier, but Buildroot out of the box does not support dstore, so it's just easier to do it through ssh.  A new connection can be created in "Remote System Explorer" perspective.  Here, I chose the ssh connection kind, which works through sftp (to upload files) and the ssh shell, as you can see here:
I am using the root user for ease of development.  Every time I create a new rootfs, I will have to keep deleting ~/.ssh/known_hosts, but it's not THAT annoying.  Once I establish that I can upload files to /root folder, I can define a new debug configuration.

Setup a remote connection

The Buildroot CDT integration already added a configuration template that uses the correct cross-debugger.  The important thing is to specify the correct connection, and where to put the executable, as you can see below.

When I start the debug session, Buildroot will upload the debug executable to the designated location through sftp, and then start gdbserver (gdbserver :2345 /root/ocm) on the target through the ssh shell, and then make a TCP connection through ssh to gdbserver.

Dec 21, 2014

Debugging Atari 2600 S-Video Mod

Why am I trying to mod Atari 2600?

My friend Rich had 2 Atari 2600 (4 switch model, circa 1980) in his garage (which I call "museum") that we stumbled upon (I forget why we were messing around in his garage).  He said he doesn't play it any more because he threw away the analog TV.  Apparently, Atari 2600 output its video over VHF channel 3 or 4 (there is a toggle button on the console), and he--later than the rest of us--has  finally thrown out his VHF TV.  Because I like taking thing apart, I offered to "fix it" for him WITHOUT knowing anything about Atari 2600 or VHF; I don't even play video games.

Dorking with an S-Video generator circuit

After some Googling, I ordered the parts listed on this website (except FMS6400, which is obsolete; I should have ordered FMS6410 from Digikey, but instead I ordered FMS6410BCS-ND, which is pin compatible with FSM6400, but in hind sight was a mistake, as it is too small to solder well), and started bread-boarding.

When I connected the S-Video cable to the TV, of course things don't work.  So this post is about the debugging my breadboard.  The circuit (should) roughly works like this:
  1. Multi-channel buffer to mix the outputs from Atari TIA chip pins 5, 7, and 8 into the base junction of a transistor.
  2. Addition of transisot emitter, TIA 6 buffered output, and TIA 9 are sent to the CIN (chroma?) of FMS6400.
  3. Transistor's emitter output drives YIN (luma?) of FMS6400.  
  4. FMS6400's YOUT and COUT are sent to S-Video, and optionally, the CYOUT is sent to the composite video cable (the yellow RCA cable).
  5. TIA12 and TIA13 should be the stereo audio output, which can be connected straight to (with a 1.8 K pull-up resistor) to the RCA audio cables.
After checking the signals and concluding that they are semi-alive going into the FMS6400 chip, I hypothesized that I might have the wrong S-Video generation chip.  This was a bit of a leap, since audio does NOT go through FMS6400 and yet, I was not hearing anything.  The gamble paid off: after getting a new FMS6400 alternative (FMS6410BCS-ND) and carefully checking all traces, I started seeing blurred image of the "Space Invader" and heard some noise.  It looked like the following:

I was stuck on this until I tried the same breadboard on another Atari 2600, which happened to be of type B.  I actually did NOT know there were different types of Atari until I started to look for other mods besides the S-Video, and landed upon this page.  Picture looks beautiful now, especially on Rich's big projector (my iPhone4 camera takes a poor picture in low light)!

Overall, the project cost me only ~$50 in parts, but A LOT of time understanding analog circuits.  But as Master Card would say, the whole experience was priceless.  Merry Christmas, Rich!

Appendix: S-Video

After checking that I have 5 V from the Atari motherboard (tapped from TIA20, VDD), I checked whether I have signals on TIA pins 2, 5, 7, 8, 6, 9.  Get the data sheet on 8201 C010444D-03?  "Longhorn engineer" actually used 8222 C010444D-03 in his screenshots.  The pinout for C0104444D - NTSC is at atarimax.com:
  • TIA2: SYNC
  • Luma
    • TIA5: LUM1
    • TIA7: LUM2
    • TIA8: LUM0
  • Colors
    • TIA6: BLK
    • TIA9: COL
  • Audio
    • TIA12: AUDI
    • TIA13: AUDO
  • Of potential interest
    • TIA11: OSC
    • Chip selects: TIA21~TIA24: nCS3, nCS2, CS1, CS0
On a (cheapo) logic analyzer, I see buffer output on TIA2, TIA5, TIA8, TIA6 during the power-up, like this.

This would mean SYNC, LUM1, LUM0, and BLK are active, but LUM2 and COL are inactive. TIA6 and TIA9 together make up the chroma, so the remaining question is whether missing LUM2 and COL is OK for forming a valid S-video image.  I guess I'll find out soon enough.

The transistor's base is the mixed (with heuristically decided resistor values) the buffered BLK, LUM1, and LUM2 (which seems to be low all the time, so what's the point?).  If I don't pull the base up to 5 V, the collected signals get filtered out, as you can see below.

With the pull-up of the transistor base to 5 V, and the voltage dividers filled out, the transistor base and emitter lines look line this (yellow: base, green: base).
Note that the signal amplitude (even the emitter output) is almost completely dominated by the SYNC (yellow below):

It's possible that the resistors from the buffer out to the base for the different channels have to be modified later for the best color.

Appendix: Audio

I cannot hear anything on the TV either.  Most of the audio signal happens on TIA13 (AUDO), but sometimes, there is a large signal on TIA12 (AUDI) as well,

Dec 13, 2014

PXE booting x86 kernel from Ubuntu host

By itself, building my own Linux kernel and root file system only marginally boosted my understanding of the Linux kernel.  But since modifying the kernel features, and if necessary, kernel sources is a good way to understand the Linux kernel, rolling my own Linux distribution with Buildroot is critical.  For an ARM target, my recent blog entries (thisthis, and this) documented downloading the Linux kernel from a development host over TFTP, and then exporting the rootfs over NFS.  I was writing this as part of my blog entry on ways to study the Linux kernel, but it got too big, so I am breaking out just the part about building an x86 Buildroot distribution and booting that over PXE. For an x86 target, I went through the same exercise a few years ago while studying sample loop jitter on Linux, but I documented it in Google docs--which seems to be less accessible for people than blogger.  So I am going to repeat the exercise again using a Dell Optiplex 755 tower (model DCNE)  with a 16x PCIe slot.  So this section is about Buildroot for my Optiplex 755, which packs Core2 duo CPU, a Gbit Ethernet.  As PXE works the same for x86 vs. x64 targets, you might see parts of the steps illustrated for my older 32-bit laptop; don't be confused if you see the name change.

The overall interaction between the target and the development host follows the the x86 laptop example I went through in my Google docs blog entry from a few years ago.

Get Buildroot

The required packages (for xconfig--the QT4 GUI; GTK GUI option requires GTK2+ dev packages, which I don't want to bother with) on Ubuntu 14.04 LTS are:
$ sudo apt-get install git build-essential gawk bison flex gettext texinfo libqt4-dev

Get the buildroot source.
$ git clone git://git.buildroot.net/buildroot

And then switch over to the latest stable branch (which is 2014.08 right now)
$ cd buildroot
$ git checkout 2014.08
$ git pull . 2014.08

Configure Buildroot for the target CPU and glibc

Unlike a Zedboard (ARM based) which has a reference Buildroot config in (<BR2>/board/avnet/zedboard), there does not seem to be a checked in config for T40.  But this old blog entry seems like a good place to start.  Fire up xconfig to configure Buildroot.
$ make xconfig

These are the configs I changed from the Buildroot default:
  • Target option: x86_64/core2
  • Build options
    • Enable compiler cache
    • gcc optimization level: O3 (rather than for size)
  • Toolchain
    • glibc 2.19 (to mimic the Ubuntu 14.04 environment, which Cuda supports).
      • But stop and think: why didn't I just use the gcc I got with build-essential package?  Because Buildroot manual says: "We also do not support using the distribution toolchain (i.e. the gcc/binutils/C library installed by your distribution) as the toolchain to build software for the target. This is because your distribution toolchain is not a "pure" toolchain (i.e. only with the C/C++ library), so we cannot import it properly into the Buildroot build environment. So even if you are building a system for a x86 or x86_64 target, you have to generate a cross-compilation toolchain with Buildroot or crosstool-NG."
    • Enable C++ suport
    • Build cross gdb for the host
    • Purge unwanted locales
      • Locales to keep: en_US
    • Register toolchain without Eclipse Buildroot plugin, because I want the Buildroot Eclipse CDT integration.
  • System configuration
    • hostname: o755
    • /dev management: Dynamic using eudev, to more closely match Ubuntu
    • Root password: I changed from the empty (no password) to something else
    • Check "install timezone info"
      • default local time: America/Los_Angeles
    • Path to users tables: /mnt/work/o755/BR2/ROOTFS_USERS_TABLES, which looks like this:
      foo -1 wheel -1 =<password> /home/foo /bin/sh - Foo user
  • Kernel: Enable "Linux Kernel" to show kernel options
    • Kernel version: Local directory: /mnt/work/zed/kernel.  This is the kernel from ADI git tree, for Zedboard.  Why don't I just use the stock kernel that Buildroot points to?  Further down, I discuss the kernel configs necessary for rootfs over NFS.  The typical defconfig used for x86 is "i386" and ia64 is "generic".  My defconfig will have to be used on the "generic", but there is no Buildroot hook to copy a defconfig from elsewhere into the downloaded kernel.  So it's easier just to use a kernel that already has the defconfig I want.
    • Defconfig name: x86_64_nfs (which is in <kernel>/arch/x86/configs; see below for my defconfig)
  • Target packages
    • Debugging profiling and benchmarking
      • strace
      • trace-cmd
    • Networking applications
      • dropbear: do NOT optimize for size (I want speed instead)
      • ethtool: used extensively in Linux Kernel Networking by Rami Rosen.
    • Shell and utilities
      • file
      • logrotate
      • screen
      • sudo
Note that I did NOT configure the bootloader; the PC architecture specifies its own bootloader: it used to be the BIOS, but now the industry has transitioned to UEFI (Unified Enhanced Firmware Interface?).  From my 

Configure the kernel for NFS rootfs and kgdb

<kernel>/arch/x86/configs/x86_64_defconfig is already quite extensive.  I appended the following configs to create x86_64_nfs_defconfig for the NFS rootfs feature:

CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_ROOT_NFS=y
CONFIG_NFS_USE_KERNEL_DNS=y
# CONFIG_NFSD is not set
CONFIG_NFS_COMMON=y
CONFIG_E1000E=y

Note that I am overriding the CONFIG_NFSD=m set earlier in the file.  CONFIG_E1000E is technically NOT an NFS rootfs config, but the kernel needs a device driver statically compiled in, to get IP communication before the the rootfs is available.

I added the following configs for the ftrace feature, which I discussed in a past blog entry:

CONFIG_KPROBES_ON_FTRACE=y
CONFIG_HAVE_KPROBES_ON_FTRACE=y
# CONFIG_PSTORE_FTRACE is not set
CONFIG_DYNAMIC_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_FTRACE=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_FTRACE_MCOUNT_RECORD=y
# CONFIG_FTRACE_STARTUP_TEST is not set

I also added the following configs for the kgdb feature:

CONFIG_KGDB=y
CONFIG_DEBUG_INFO=y

Optional but recommended (makes sense for me):
  • CONFIG_FRAME_POINTER=y -- save frame information, to allow gdb to more accurately construct stack trace
  • CONFIG_KGDB_SERIAL_CONSOLE=y -- kgdb over Ethernet is not in mainline, so stick with the tried and true.  This also allow kgdbwait and early debugging
  • # CONFIG_DEBUG_RODATA is not set -- This option marks certain regions of the kernel's memory space as RO.  I did NOT have this option to begin with, so I will leave this alone.  If my processor did NOT have HW breakpoint, I should turn this off.  Zynq has 5 breakpoints and 1 watchpoint registers--which I found kgdb cannot use for some reason.
  • # CONFIG_CC_OPTIMIZE_FOR_SIZE is not set -- I have plenty of disk space but the processor is relatively slow.  Besides, I find it confusing if the compiler optimizes some code away when debugging.

Buildroot (everything)

"make" at the BR2 folder builds everything (by I debug my configuration mistakes a few times).  After "make" goes through, the compressed kernel image and the tarred rootfs are in output/images:

henry@Zotac64:~/work/o755/buildroot$ ls -gh output/images/
total 25M
-rw-rw-r-- 1 henry 5.4M Nov 28 18:07 bzImage
-rw-rw-r-- 1 henry  19M Nov 28 18:19 rootfs.tar

We need to export the rootfs to /export/root later.

Setting up the PXE DHCP server on the development host

A development host needs a fixed IP address for the target to download the kernel.  How will the host and the target get such fixed IP address?  One way is to reserve fixed IP address on the DHCP server.

Dorking with the DHCP server: probably not an option for most hobbysts

I like to use DHCP because the target can access the Internet even without my having to setup a route on the host.  At home, I have the convenience of setting aside a static IP addresses through the admin console (I use Netgear), as you can see below, where my host IP address is 192.168.1.2:
I use this method in the past, in a setting: just tell the IT department, and they will take care of it (They are happy to do it!  They don't like people like me setting up my own router within the company, and it gives them a chance to give you a ticket, and "get paid"--one way or another).  Alas, when using PXE, you need an enhanced version of the DHCP server, which is NOT what the cheapo home router/DHCP server runs.  I am kicking myself because I used to have a Lynksys WRT router, on which you can run a Linux network server, but gave that away when I upgraded my house to wireless.

Alternatively, get a 2nd NIC and a cross Ethernet cable

With a PCI or USB Ethernet adapter like the Lynksys USB300M shown below (paid $1 PLUS $6 shipping!), and a cross cable, I can have a subnet (consisting of 2 nodes) even without a switch.
Cisco-Linksys USB Ethernet Adapter - Used See DescriptionThe only thing to watch out for is the Linux device driver support.  I bought USB300M because my previous company used it on Linux.  When I plug this into the host, it shows up in "ifconfig" as eth1.  The interface should be statically configured, like this:
This interface becomes active only when there is another machine on the other end.  Is there a way to change this behavior?

The DHCP server is installed on Ubuntu with this command:

$ sudo apt-get install isc-dhcp-server

isc-dhcp-server config

I can add this to the INTERFACES line in /etc/default/isc-dhcp-server:

INTERFACES="eth1"

Note that in the introduction picture, my target also included a virtualbox machine, which I am no longer using.  The latest version of the isc-dhcp-server does seems to have a problem starting if I specify "allow booting" and "allow bootp" in the isc-dhcp-server file.  I do not understand why, but when I move those 2 lines to the dhcpd.conf, the isc-dhcpd-server started.

Next, reserve the IP address for all targets I expect to serve over PXE in /etc/dhcp/dhcpd.conf:

subnet 192.168.2.0 netmask 255.255.255.0 {
  range 192.168.2.10 192.168.2.19; # dynamic will not be used
  option routers 192.168.2.1;
  next-server 192.168.2.1; # point targets at the TFTP server (me)
  filename "pxelinux.0";
  allow booting;
  allow bootp;
  option domain-name-servers 192.168.1.1;
  default-lease-time 600;
  max-lease-time 7200;
  ddns-update-style none;

  host o755 {
    hardware ethernet 00:1A:A0:CB:54:FF;
    fixed-address 192.168.2.3;
    option root-path "192.168.2.1:/export/root/o755/" ;
  }
}

When I start the daemon to check for any errors, I see the expected output:

henry@Zotac64:~/work$ sudo dhcpd
Internet Systems Consortium DHCP Server 4.2.4
Copyright 2004-2012 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
WARNING: Host declarations are global.  They are not limited to the scope you declared them in.
Wrote 0 deleted host decls to leases file.
Wrote 0 new dynamic host decls to leases file.
Wrote 0 leases to leases file.
Listening on LPF/eth1/48:f8:b3:45:48:b0/192.168.2.0/24
Sending on   LPF/eth1/48:f8:b3:45:48:b0/192.168.2.0/24
...
Sending on   Socket/fallback/fallback-net

When the target obtains its IP address, it is also told to download the SSBL (2nd stage bootloader) (pxelinux.0) from "next-server" (over TFTP, implicit in the PXE protocol).

Running the ISC DHCP server in Ubuntu service framework

You can always restart the server like this example:

henry@Zotac64:~$ sudo service isc-dhcp-server start
isc-dhcp-server start/running, process 3358

The server will start on bootup automatically, but as discussed here, it errors out because the network is not ready.  Rather than put in a 10 second delay, I will just manually start things for now.

Putting PXE DHCP to a test

Let take the DHCP server out for a spin.  The target exchanges the DHCP messages with the server, as appearing in /var/log/syslog:

Zotac64 NetworkManager[881]: <info> (eth1): carrier now ON (device state 100)
Zotac64 kernel: [ 1937.698818] asix 2-6:1.0 eth1: link up, 100Mbps, full-duplex, lpa 0xC1E1
Zotac64 dhcpd: DHCPDISCOVER from 00:1a:a0:cb:54:ff via eth1
Zotac64 dhcpd: DHCPOFFER on 192.168.2.3 to 00:1a:a0:cb:54:ff via eth1
Zotac64 dhcpd: DHCPREQUEST for 192.168.2.3 (192.168.2.1) from 00:1a:a0:cb:54:ff via eth1
Zotac64 dhcpd: DHCPACK on 192.168.2.3 to 00:1a:a0:cb:54:ff via eth1
Zotac64 in.tftpd[3446]: tftp: client does not accept options

As expected, the target cannot locate the PXE configuration file (because I did NOT create one yet!), and boot fails.

Installing (and configuring) the TFTP server on the development host

Install the tftp server with this command:

sudo apt-get install tftpd-hpa

My only addition to the default TFTP config (/etc/default/tftpd-hpa) is this line (if I  leave IPv6 running; otherwise, I have to change the address to bind to, and specify --ipv4 to the server option):

RUN_DAEMON="yes"

Then restart the TFTP daemon:

sudo service tftpd-hpa start

The SSBL (pxelinux.0) and associated menu (menu.c32) should be COPIED (rather than soft-linked) from /usr/lib/syslinux folder into the /var/lib/tftpd folder).   And then the create the pxelinux.cfg folder, containing a single file called default for now:

henry@Zotac64:/var/lib/tftpboot$ cat pxelinux.cfg/default
DEFAULT menu.c32
PROMPT 0
MENU TITLE PXE Boot Menu
TIMEOUT 50 #This means 5 seconds apparently
LABEL buildroot
 MENU LABEL buildroot kernel
 kernel o755Image
 append ip=dhcp

The last line appears to be the kernel arg.

Sanity test tftp

Download tftp

sudo apt-get install tftp-hpa

Then try to get the file, first on the server itself

tftp localhost -v -m binary -c get pxelinux.0

If you don't receive this file (perhaps due to some corruption while installing something else), removing, and then reinstalling the tftpd-hpa may get you out of the jam (as it did for me).

Kernel downloads and boots

With this, the kernel (~/work/o755/buildroot/out/images/bzImage) I copied into /var/lib/tftpboo downloads and boots!  But since the kernel does not know to use NFS rootfs, it panics:

...
---[ end Kernel panic - not synching: VHS: Unable to mount root fs on unknown-block(2,0)

As in the ARM case, the kernel needs to be told to run NFS rootfs through the bootarg.

Serial console to the target

But if the kernel does not boot (hint: expect problems every step of the way), having a serial console is necessary for debugging.  I connected the target and host PC with a 9-pin null-modem (and a USB-serial cable), so that the target's /dev/ttyS0 (on the target side) appears as /dev/ttyUSB0 on the development host.

NFS rootfs kernel arg

In the pxelinux menu above, I coded the kernel image and boot arg in the the "default" config.  Let's make things a bit more interesting: specify the kernel image and option tailored to a target.  As explained here, the default config is the LAST config the pxelinux will check.  The first is the PXE GUID (printed on the console), and the 2nd is the (LOWER case, HYPHENATED) MAC address, which for this target is 00-1a-a0-cb-54-ff.  Another way is to tie the configuration to the target's IP address, which is 192.168.2.3 in this case, translating to C0A80203 (uppercase hex notation) as the filename in pxelinux.cfg, which now looks like this:

DEFAULT menu.c32
PROMPT 0
MENU TITLE PXE Boot Menu
TIMEOUT 20 #This means 2 seconds apparently
LABEL buildroot
 MENU LABEL buildroot kernel
 kernel o755Image
 append ip=dhcp root=/dev/nfs nfsroot=192.168.2.1:/export/root/o755,nfsver=3 rw earlyprintk

During the 2 second wait, you can halt the boot and review the argument that PXE picked up.  Recall that Buildroot built rootfs, but we did not yet export it to NFS, so the kernel will still panic when trying to mount the rootfs.  So let's do this now.

$ sudo mkdir /export/root/o755
$ sudo tar -C /export/root/o755/ -xf ~/work/o755/buildroot/output/images/rootfs.tar

The NFS server export option must specify the "no_root_squash" option as explained in my earlier blog entry.  As a sanity check, I could mount this on the same machine with a mount command:

$ mkdir ~/o755
$ sudo mount 192.168.2.1:/export/root/o755 ~/o755

Don't use localhost, because the /etc/export only allows 192.168.2.0/22 mask.  Alas, The target was still not initiating NFS mount.  An ARM target (zedboard) can mount NFS just fine (a folder right next to o755 using the same set of kernel arguments), and shows these printk messages:

IP-Config: Got DHCP answer from 192.168.1.1, my address is 192.168.1.9
...
VFS: Mounted root (nfs filesystem) on device 0:13.

For NFS mount to succeed, the Kernel has to get exchange DHCP, so the question is: why doesn't o755 target attempt DHCP?  When the target successfully boots Ubuntu 14.04 LTS (and can respond over the network), it probes e1000e successfully, as you can see from this dmesg output:

[    2.184242] e1000e: Intel(R) PRO/1000 Network Driver - 2.3.2-k
...
[    2.558382] e1000e 0000:00:19.0 eth0: Intel(R) PRO/1000 Network Connection
[    2.570813] e1000e 0000:00:19.0 eth0: MAC: 7, PHY: 6, PBA No: 1041FF-0FF

I then checked the kernel config: CONFIG_E1000E was missing in x86_64, so I pulled that into my defconfig based on x86_64.  But that got me past DHCP, but I still did not see

VFS: Mounted root (nfs filesystem) on device 0:13.

Instead, I see:

VFS: Unable to mount root fs via NFS, trying floppy
VFS: Cannot open root device "nfs" or unknown-block(2,0): error -6

In /var/log/syslog, there is no log after a successful DHCP exchange.  And there is no error message between DHCP and the VFS failure above.  What is actually failing is sys_mount() being called in do_mount_root().  After debugging with kgdb (see appendix below), and realizing that I don't know how to specify the TCP transport to nfsroot kernel argument (yes, I tried appending ",tcp" to the nfsroot argument; the problem seems to be with the nfsvers=4 rather than the transport, because nfsvers=3 worked with either TCP or UDP) I changed the nfsroot kernel option's nfsvers argument back to 3 to work around, and voila!

Welcome to o755
o755 login: 

Appendix: Debug the NFS mounting problem with kgdbwait

Unfortunately, the nfs file handle is still not being found.  To debug, I thought I would give the kgdb a shot (because unlike on ARM, I don't have a JTAG debugger for this PC).  Earlier, I already added the Kernel configs necessary for kgdb, so I just have to tell the kernel in the kernel arg, by APPENDING the following to the "append" line in the PXE menu:

kgdboc=ttyS0,115200 kgdbwait

On the VGA screen, I see that the kernel has setup ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) and ttyS1 at I/O 0xec98 (irq = 17, base_baud = 115200), right before blocking on connection from remote gdb, so I dutifully oblige by firing up the cross gdb (the host gdb probably works fine, but since I went through all the trouble of building the cross gdb, why not use it?):

henry@Zotac64:~/work/o755/buildroot$ output/host/usr/bin/x86_64-linux-gdb output/build/linux-custom/vmlinux

I am not sure whether I got the serial port right (ttyS0) on the target side, but on the host, there is no doubt about the tty device because I am using USB-serial adapter, and I see only 1 /dev/ttyUSB0. On Ubuntu, this device is only writable by the group "dialout", so I have to add myself to that group.

$ sudo apt-get remove modemmanager
$ sudo adduser henry  dialout
$ sudo chmod a+rw /dev/ttyUSB0

Then I can connect to the targe within gdb:

(gdb) set serial baud 115200
(gdb) target remote /dev/ttyUSB0
Remote debugging using /dev/ttyUSB0
kgdb_breakpoint () at kernel/debug/debug_core.c:1051
1051 wmb(); /* Sync point after breakpoint */

(gdb) hb nfs_validate_text_mount_data
(gdb) c

I wonder whether I am really using the HW breakpoint on this target.  I traced the failure to fs/nfs/super.c:nfs_fs_mount(), due to error -22, or EINVAL, because of the transport protocol disagreement:

nfs_validate_transport_protocol(args);
if (args->nfs_server.protocol == XPRT_TRANSPORT_UDP)
goto out_invalid_transport_udp;

When I stared at the code for a while, I realized that Linux NFSv4 wants to use TCP as the transport.