Noise cancelling with dual omni directional microphones

Good bye Steve, I'll miss you

The Jawbone Noise-Assassin product lines (ERA, etc.) were category-creating.  When I got to Jawbone in September 2015, the many of the original people who worked on it had already left, including the former chief scientist Greg Burnett.  But the remaining people gave me some clues to learning the technology on my own.  One of them was Steve Forestieri, the director of the DSP group.  Just a few weeks ago week, he had promised me to teach the noise cancelling algorithm, but he had a heart attack on the same day, and passed away a week later.  I had seen Steve struggle with work related stress for the last couple of months, and correlate it with his heart attack.  The irony was that he had a heart attack on the same day he seemed to come to a resolution with what was frustrating him, and decided to move on with the rest of his life.  I am saddened by his passing, and am reminded again of the song Enjoy yourself (It's later than you think):

Enjoy yourself, it's later than you think
Enjoy yourself, while you're still in the pink
The years go by, as quickly as a wink
Enjoy yourself, enjoy yourself, it's later than you think

Well, learning new things is how I enjoy myself, so here goes nothing.

DOMA (dual omni-directional microphone array)

The seminal technology behind Jawbone's Noise Assassin algorithm is described in Greg Burnett's patent (USPN 8731211B2 and USPN 8837746B2).  Even though they are well written, I learn better when I can rewrite it in my own words.

Geometry

Suppose I have 2 mics O1 and O2 that have essentially the same frequency response (getting that same frequency response is the subject of the DOMA calibration patent), spaced 2 x d0 apart and sampling sound source S that is ds away from the mid-point of the mic pair, as shown below.

The geometry of 2 mics in relation to the S is:

d1 = SQRT[ds² – 2 ds d0 cosθ + d0²] 

d2 = SQRT[ds² + 2 ds d0 cosθ + d0²]

If S is a point source, the sound amplitude attenuates like R^2.  On the ground, the ground reflection reduces the attenuation to something less than R^2, but still, it's probably not 1/R as the patent claims.  Let's just continue using the 1/R and see if 1/R vs. 1/R^2 even matters in the end.  O2(t) then will experience a sound wave that is attenuated by β = (d1/d2) of O1(t), and delayed by (d2 - d1) / c in wavelengths, where c is the sound wave speed.  In a discrete sampled system with Fs sampling frequency, that delay is

γ (gamma) = (d2 - d1) Fs / c in samples (not unnecessarily an integer)

As a comparison, at Fs = 48 kHz, c / Fs = 7.3 mm.  So the z transform of O1 and O2 can be related:

O₂(z) = β z^-γ O₁(z)

Let's visualize the quantity β and γ for d0 = 10 mm, and ds = [10 cm 30 cm 1m].

For sampling frequency Fs = 16 kHz, if the sound source is 10 cm (ds = 0.1 m) and directly in front of O1 (θ = 0),  β = 0.8182 and γ = 0.9275.  Let's call these B and G, to indicate they are constants.

It makes sense that as the sound source becomes more distant, the sound source "sounds" the same to the 2 mics.  ds = 100 d0 is a practical infinite distance in this scheme.

Gradient mic pair

Now imagine a linear combination of arbitrary delay of O1/O2, as shown below.

Let's form V2 as a transfer function of what the speech mic O1 experiences.

V₂(z) = O₂(z) – B z^-G O₁(z) = β z^-γ O₁(z) – B z^-G O₁(z) = (β z^-γ – B z^-G) O₁(z)

V2 has a speech null

I can evaluate the transfer function V2(z) / O1(z) by letting z = exp(jω):  (β e^-jωγ – B e^-jωG), whose magnitude and phase looks like this for sound source within +/- 20% of the expected distance (Ds = 10 cm):

Here, I took the frequency up to the Nyquist limit; black = 10 Hz, red = 100 Hz, green = 1 kHz, blue = 4 kHz, cyan = 8 kHz.  Even though the penalty for being wrong about the true distance of the speech source from the mic array gets larger for high frequency (solid line is ideal, dashed and dotted are +/- 20%), there is still a null for angle +/- 30 degrees: the relatively small space where V2 picks up NO sound from O1: this is apparently called "null" among "those skilled in the art".  But it does pick up OTHER sound--which I will call N2 here (for noise emitted by V2).

V1 picks up speech AND noise N1

Now consider another arrangement of O1 and O2, using the same constants B and G as above

V₁(z) = z^-G O₁(z) – B O₂(z)  = z^-G O₁(z) – B β z^-γ O₁(z) =  (z^-G – B β z^-γ) O₁(z)

I first thought the design goal is to directly cancel out noise (S relatively far away from the mouth) by designing V1 ~ V2 for noise.  BUT as you can see below, that is not true in general (gets worse as frequency goes up).

Fundamentally, V1 is picking up the difference of the sound heard by O1 and O2, which you can see by staring at the transfer function above when we are "reasonably close" to the assumed values B and G.

V₁(z)/O₁(z) =  z^-G – B β z^-γ ~ z^-G (1 – B β) ~ z^-G (1 – B²)

What the above plot tells me is that V2 is NOT emitting the same nose as V1.  This is one of the keys to understanding the Jawbone noise cancelling.

For speech near the assumed distance, the |V1| is nearly constant for +/- 20 degrees within the design angle to the sound source, as you can see on the left plot below.

When the sound source is relatively far away from the array, O1 and O2 hear roughly the same thing.  For the purpose of rejecting distant sound, let's call this sound N and plug into O1 and O2 in the equations for V1 and V2.

V₁(z) = z^-G O₁(z) – B O₂(z)  ~ z^-G N(z) – B N(z) = (z^-G – B) N(z)

V₂(z) = O₂(z) – B z^-G O₁(z) ~ N(z) – B z^-G N(z) = (1 – B z^-G) N(z) 

=> H(z) = V₂(z) / V₁(z) ~ (1 – B z^-G) / (z^-G – B)

Again, I evaluate the transfer function with z = exp(jω) to get the frequency response.

V₂(z) / V₁(z) ~ (1 – B e^-jωG) / (e^-jωG – B)

So contrary to my initial expectation, DOMA cannot cancel noise; it merely transforms the speech and (far away) noise into speech and 2 related noise.

V₁ = S + N1

V₂ = N2

To kill N1 in V₁, we turn to adaptive filter.

LMS (least mean squared) adaptive filter

When there is no speech (S = 0), N1 and N2 are related: they are both driven by the same noise (IF there is 1 dominant noise source from the mic array--which is true in most cases).  An adaptive filter then should yield an estimate for N1 such that the residual noise Nresidual is the minimum obtainable.

There are whole books written on adaptive filters, so LMS can potentially be replaced with other adaptive filters.  Regardless of the adaptive algorithm used, the adaption must STOP when S is active, to avoid the filter adapting to the speech and therefore canceling it.  That is what the Jawbone VAD (voice activity detection) patent (USPN 8467543) covers.

But wait, there is more; a LOT more!

This is just the core of the noise cancelling covered by the Jawbone patent.  If you have ever worked on a real product, you know that things get complicated quickly.  And a lot of that is the "magic sauce" or "dark art" trade secret.  I may add more material as I learn about noise cancelling, but from this point on, I am going to be careful about what I write here.

QPN DPP FW on Dialog DA14580

I ported the QPN-QK (QPC-nano: a bare metal, hard real-time C framework) for the Dialog's BLE chip DA14580 basic dev kit.  DA14580 has 84 KB ROM, but most of that is taken up by the bootloader and the Dialog's SDK (including the BLE functionality).  The OTP (on-time programmable) memory is only 32 KB, which is the code size limit of the Keil uVision free version.  The OTP is NOT executable; the intention is for the bootloader (in the ROM) to copy the code from OTP to (volatile) SRAM on boot or wakeup from deep sleep, during which time the volatile (vs. retained) SRAM content is lost.  The bootloader can also boot from UART, SPI flash, or I2C EEPROM, but in all cases, the code is first written to the SRAM before the real execution starts, so 32 KB of OTP is the practical code size limit.

DA1450x SDK (v 5.0.2) blinky example uVision 5 project

The SDK uses a boot_vectors.s (in peripheral_examples/shared/startup/) containing the vector table and Reset_Handler.

Reset_Handler   PROC
                EXPORT  Reset_Handler             [WEAK]
                IMPORT  __main
                IMPORT  SystemInit

                LDR     R0, =SystemInit
                BLX     R0
                LDR     R0, =__main
                BX      R0
                ENDP

SystemInit() is a CPU/board specific function, supplied in system_ARMCM0.c (right next to the voot_vectors.s).  For the peripheral example, it uses the internal 16 MHz crystal oscillator.

void SystemInit (void)
{
  SetWord16(TIMER0_CTRL_REG,0x6);      // stop timer
  NVIC_DisableIRQ(SWTIM_IRQn);     // disable software timer interrupt
   
  if ((GetWord16(CLK_CTRL_REG) & RUNNING_AT_XTAL16M) == 0)
  {
    while( (GetWord16(SYS_STAT_REG) & XTAL16_SETTLED) == 0 );
    SetBits16(CLK_CTRL_REG , SYS_CLK_SEL ,0);
    while( (GetWord16(CLK_CTRL_REG) & RUNNING_AT_XTAL16M) == 0 ); // wait for actual switch
  }
}

__main() is not supplied in the SDK, but it's probably just a C runtime init and then a jump to the application main.

Memory map: use case 23

The boot_vectors.s also hardcodes the stack and the heap size:

Stack_Size      EQU     0x00000200

Heap_Size       EQU     0x00000100

The scatter file (equivalent to the GNU ld file) also hard codes the stack size.

LR_IROM1 0x20000000 0x00009800  {
...
    RW_IRAM1 (0x20009800 - 0x200) UNINIT 0x200 {         ; Stack
        .ANY (STACK)
    }
}

The system memory starting and ending address is for the memory map case 23, among the memory map choices listed in DA1258x SDK reference manual (Dialog document UM-B-051).  The blinky example puts everything (code, data, bss) in the 38 KB non-retention RAMP.

Debugger init

When debugging, the debugger ini file should match this starting address, as in this example:

RESET
E long 0x50000012 = 0xa4
E long 0x50003308 = 0x2e
LOAD %L
SP = _RDWORD (0x20000000)
$ = _RDWORD (0x20000004)

Let's briefly examine the system register settings 0x00A4 and 0x002E above.  SYS_CTRL_REG is configured for:

• DEBUGGER_ENABLE
• PAD_LATCH_EN
• RET_SYSRAM: in development mode, system is NOT actually powered off (so that SysRAM is retained), and .data is NOT copied to SysRAM when the uC wakes up.

GP_CONTROL_REG = 0x2E = 0x17 << 1 + 1 specifies:

• EM_MAP = 0x17 = case 23
• BLE_WAKEUP_REQ: BLE wakes up

System init

To understand the system_init() called by main()--what kind of initialization does the example do?--, I need the Registers section of the DA14580 datasheet.

void system_init(void)
{
    SetWord16(CLK_AMBA_REG, 0x00);                 // set clocks (hclk and pclk ) 16MHz
    SetWord16(SET_FREEZE_REG,FRZ_WDOG);            // stop watch dog   
    SetBits16(SYS_CTRL_REG,PAD_LATCH_EN,1);        // open pads
    SetBits16(SYS_CTRL_REG,DEBUGGER_ENABLE,1);     // open debugger
    SetBits16(PMU_CTRL_REG, PERIPH_SLEEP,0);
}

Annotation of the above code:

• APB (peripheral bus) and AHB (high speed bus) runs at the same frequency as the source (16 MHz above).
• Freeze the watchdog HW.
• Latch is transparent, pad can be retained (control signal is NOT retained)??
• Retain SWD IO/CLK pins for debugging (instead of using them as GPIO).
• Supply power to peripheral.  You can turn off the radio in the same PMU_CTRL_REG.

periph_init() repeats the above setup exactly anyway, so system_init() is technically unnecessary.  Once the entire peripheral block is powered up, the uC will get past this:

    while (!(GetWord16(SYS_STAT_REG) & PER_IS_UP));

Even though the peripheral block itself is now powered, clocks to individual HW (e.g. UART) must be turned on separately through the CLK_PER_REG.  The example blinks the GPIO LED, so that GPIO port is configured for output

    GPIO_ConfigurePin(LED_PORT, LED_PIN, OUTPUT, PID_GPIO, false);

Since the blinky example writes to the serial console, UART2 pins are configured.

    GPIO_ConfigurePin(UART2_GPIO_PORT, UART2_TX_PIN, OUTPUT, PID_UART2_TX, false);
    GPIO_ConfigurePin(UART2_GPIO_PORT, UART2_RX_PIN, INPUT, PID_UART2_RX, false);

    SetBits16(CLK_PER_REG, UART2_ENABLE, 1); // enable  clock for UART 2
    uart2_init(UART2_BAUDRATE, UART2_DATALENGTH);

Booting the FW from UART

During development, it is convenient to download and run (and debug) the FW from an IDE such as the Keil uVision.  But to use DA1458x as a slave uC in a production setting, a boot master needs to hold the FW and feed it to DA1458x during boot, according to the Dialog application note AN-B-001: DA1458x Booting from serial interface.  The boot code on the OTP checks all 1458x's serial peripherals sequentially, in the following steps:

DA1458x is acting as a slave device in steps 1 through 6, and as a master device for the rest of the steps.  Checking for an external master only happens once during reset, but the boot code loops through steps 7 and on 5 times before giving up and spinning, waiting for a SWD master intervention.  According to the above steps, the earliest step where an external UART master can boot the DA1458x is step 3, but for the fastest UART baud rate (8 bit, no parity in all cases) I must use step 4 (P0.2/3), as you can see in the following table:

The message protocol for transferring the FW image from the UART master to DA1458x is in the DA1458x connected to UART section of the same application note.  Dialog's own SmartSnippets GUI uses this protocol to download FW to DA1458x.

In a production device, where the boot device/pin is fixed, checking an external SPI master lengthens the boot latency.  Another Dialog document UM-B-012 DA14580/581/583 Creation of a secondary bootloader explains modifying the boot code to skip the unnecessary checks.

DPP (dining philosopher problem) solved with QPN on DA14580

The QK port for arm-cm discriminates itself for M0/M1 (which has no preemption feature) and M3/4/7.  I copied the example dpp_nucleo_l053r8 and modified only the bsp.c

In the CMSIS SystemInit(), I chose to use the internal 16 Mhz clock and power up the peripheral domain power.

void SystemInit (void)
{
    SetWord16(TIMER0_CTRL_REG,0x6);      // stop timer
    NVIC_DisableIRQ(SWTIM_IRQn);     // disable software timer interrupt
   
    if ((GetWord16(CLK_CTRL_REG) & RUNNING_AT_XTAL16M) == 0)
    {
        while( (GetWord16(SYS_STAT_REG) & XTAL16_SETTLED) == 0 );     // wait for XTAL16 settle
        SetBits16(CLK_CTRL_REG , SYS_CLK_SEL ,0);                     // switch to XTAL16
        while( (GetWord16(CLK_CTRL_REG) & RUNNING_AT_XTAL16M) == 0 ); // wait for actual switch
    }
    //SetWord16(GP_CONTROL_REG, 1 << 1 | 0);//Case 1 in DA1458x SDK ref Appendix A.1
    // system init
    SetWord16(CLK_AMBA_REG, 0x00);                 // set clocks (hclk and pclk ) 16MHz
    SetWord16(SET_FREEZE_REG,FRZ_WDOG);            // stop watch dog   
    SetBits16(SYS_CTRL_REG,PAD_LATCH_EN,1);        // open pads
    SetBits16(SYS_CTRL_REG,DEBUGGER_ENABLE,1);     // open debugger
    //SetBits16(PMU_CTRL_REG, PERIPH_SLEEP,0);     // exit peripheral power down
    // Power up peripherals' power domain
    //Q: Why set the PMU_CTRL_REG again with same value?
    SetBits16(PMU_CTRL_REG, PERIPH_SLEEP, 0);
}

The DA14580devkB board has only 1 user LED, so I used it for the philosopher[0] stat.

void BSP_displayPhilStat(uint8_t n, char const *stat) {
    if (stat[0] == 'h') {
        GPIO_SetActive( LED_PORT, LED_PIN);/* turn LED on  */
    }
    else {
        GPIO_SetInactive(LED_PORT, LED_PIN);/* turn LED off */
    }
}

The DPP example uses systick interrupt, but a typical low power FW will use a tick-less timer instead, rendering the current systick related code unnecessary in QF_onStartup():

void QF_onStartup(void) {
    /* set up the SysTick timer to fire at BSP_TICKS_PER_SEC rate */
    SetBits32(&SysTick->CTRL, SysTick_CTRL_ENABLE_Msk, 0);          // disable systick
    SetBits32(&SysTick->LOAD, SysTick_LOAD_RELOAD_Msk, BSP_TICKS_PER_SEC-1); // set systick timeout based on 1MHz clock
    SetBits32(&SysTick->VAL,  SysTick_VAL_CURRENT_Msk, 0);          // clear the Current Value Register and the COUNTFLAG to 0
    SetBits32(&SysTick->CTRL, SysTick_CTRL_TICKINT_Msk, 1); // generate interrupt
    SetBits32(&SysTick->CTRL, SysTick_CTRL_CLKSOURCE_Msk, 0);// use a reference clock (1 MHz)
    SetBits32(&SysTick->CTRL, SysTick_CTRL_ENABLE_Msk, 1);          // enable systick
    /* set priorities of ALL ISRs used in the system, see NOTE00
    *
    * !!!!!!!!!!!!!!!!!!!!!!!!!!!! CAUTION !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    * Assign a priority to EVERY ISR explicitly by calling NVIC_SetPriority().
    * DO NOT LEAVE THE ISR PRIORITIES AT THE DEFAULT VALUE!
    */
    NVIC_SetPriority(SysTick_IRQn,   SYSTICK_PRIO);
    /* ... */
    /* enable IRQs... */
}

The DPP project structure looks like this:

Besides the board specific code in bsp.c (discussed above),  DA14580devkB folder contains the board specific files.

The uVision project generates afx file (equivalent to ELF file from gnu toolchain), but to generate a binary file (for booting the target from another uC for example), I added a post-build step in the User tab of the project option:

fromelf --bin --output .\out\app_uart.bin .\out\dpp_qk.axf

Unlike the example blinky project, QPN projects do NOT use heap space, and I specified that as a #define into the assembler (which assembles the boot_vectors.s)

The app FW bin (app_uart.bin) produced with these changes is 6.5 KB (which should contain everything that will be in the RAM, except the .bss section).  When I download the binary from uVision to the target over USB J-Link connection (DA14580devkB feature), I see the philosopher hungry state go on and off on the green user LED--as expected.

Adding QS real-time software tracing to QPN

When Miro Samek created QPN from QPC, he left out QSpy tracing feature of the QPC to keep the memory footprint down.  The QSpy sources are nicely refactored out in QPC:

• include/
• qs.h
• source
• qs.c
• qs_fp.c
• qs_pkg.h
• qs_rx.c: receive and parse command from the QSpy viewer.  Unnecessary for passive tracing.
• qstamp.c: build date.

I had to define the QSignal and QEvtSize data type sizes in qs_port.h, because QPN hard codes those types and QS cannot figure it out on its own.

#define Q_SIGNAL_SIZE 1
#define QF_EVENT_SIZ_SIZE 1
#define QF_CRIT_ENTRY(dummy) QF_INT_DISABLE()
#define QF_CRIT_EXIT(dummy)  QF_INT_ENABLE()

Also, on Cortex M0, there is no preemption of the ISR, so it's OK to just lock out the whole interrupt.  To use Q_SPY, I have to define Q_SPY to the CPP.

Now the app FW bin size is 9.4 KB, and the total size on RAM < 12 KB.

Appendix: More memory for the application FW

Since I am not interested in using DA14580's BLE HW, I can grab all memory that was reserved for BLE in the blinky example.  Unfortunately, there is no memory map configuration WITHOUT any memory for the BLE.  The best I can do is use case 1.

The starting address is still 0x20000000, but the end address is now 0x2000C000.  Whether I actually retain the top 6 KB is up to me (it does NOT retain out of reset).  The linker script (scatter file) for this modified memory map reserves only 1 KB for the stack, and removes heap altogether.

LR_IROM1 0x20000000 0xC0000  {

    ER_IROM1 0x20000000 0xB000  {
        *.o (RESET, +First)
        *(InRoot$$Sections)
        .ANY (+RO)
        .ANY (+RW)
    }
    ER_IROM2 +0  {
     .ANY (+ZI)
    }

    RW_IRAM1 (0x2000C000 - 0x1000) UNINIT 0x1000 { ; Stack
        .ANY (STACK)
    }

}

The uC's memory map is changed during runtime by setting GP_CONTROL_REG (0x50003308) to (1 << 1) + 0 = 2; the 0 implies BLE_WAKEUP_REQ is turned off, like this:

    SetWord16(GP_CONTROL_REG, 1 << 1 | 0);

This should be done in SystemInit (which is called even before the C runtime initializes), as soon as the oscillator stabilizes.

Hard faulting as soon as I start!

SNR considerations in wearable bio-optical sensor

I recently looked into the low-level (AKA front-end--vs. statistical algorithmic, which sits at the backend) problems of optical (vs. electrical) sensing of human bio-signals.  In this blog entry, I want to share only the already publically accessible knowledge, and my original thoughts (which means the material in this entry is free of any copyright).

Multi-spectral sensing is based on different reflection (actually includes back scattering as well) behavior of the constituent material in the optical path of the sensor at different wavelength.  Beer-Lambert law describing the absorption phenomenon relates the optical path length, material concentration, and the measured light loss this way: A = ε C L = log10 (Light intensity IN / Light intensity OUT), where

• ε is the molar extinction coefficient
• C is the material concentration in the optical path
• L is the optical path length.

So if there is no absorption (Light intensity IN == Light intensity OUT), A = 0, and if there is completion loss, A ~ INF.  ε is an optical property of a material found in the optical path on the wrist, such as dermis, melanin, oxygenated/deoxygenated blood.  Since we know that the ratio of oxygenated/deoxygenated blood changes in a heart cycle, we want to observe that change by measuring the light returning back from the wrist.  A problem is that the light NOT lost is measured with integrating sphere which can pick up light going in all direction (isotropically, as illustrated above), while OUR sensor will be a relative small photo-diode array situated not far from the light source, leading to at least 2 problems:

1. We will pick up a lot of excitation light (rather than the light bouncing off Hemoglobin molecules in the skin)
2. We are picking up only a small fraction (well below 1% even in the best case) of the photons that are bouncing back from the Hemoglobin molecules.  A photo-diode array found in a tightly integrated SoC may be on the order of a 1 mm^2.  Since isotropic radiation loses strength as 1/radius^2, the signal bearing light landing on the photodiode rapidly goes down, while leakage light from the ambient (the baseline) goes up rapidly when the sensor is worn loosely around the wrist.

Right away, we know that the excitation light has to be pretty strong, to get reasonable photons landing back on the photo-diodes.  In fact, it has to be strong enough to reach into the dermis layer, where there is blood flow due to the blood pressure pulse.  The blood movement we CAN see in the epidermis layer moves (other than the movement of the wrist and the skin due to the heart beat itself) through diffusion.  The pictures below show that the excitation light should be FOCUSED at much deeper than 100 um—ideally at 200 um or deeper, as shown in the following images available at Wikipedia:

The light that reaches into the dermis layer will see primarily the epi/dermis tissue, melanosome (organelle that makes and stores melanin in the epidermis), and oxy/deoxy Hemoglobin molecules.  In the plot below, extinction (light loss) spectra of dermis (contains no melanin) and epidermis with 30 % volume fraction (for very dark skin case) of melanosome are shown.  The dermal tissue can be considered transparent compared to the melanin, which especially absorbs strongly in the shorter wavelengths according to the extinction spectra data I downloaded from http://www.OMLC.org

The visible light spectrum may be of help in understanding the difference between the oxygenated and deoxygenated Hemoglobin molecule: the large difference in the extinction coefficient in the deep blue, and to a lesser extent, near red) explains why the venous blood looks blue, and the arterial blood looks red.

The total absorbance is the sum of absorbance of all material found in the optical path; that is, Atot = SUM_i(Ai).  Because there is little blood in the epidermis (~0.1% by volume) and dermis (5~8% by volume), the light encounters mostly melanin, rather than the oxy/deoxy Hb molecules, so that the amount of light NOT absorbed/scattered (what survives the absorbance equation in the previous page) is dominated by the melanin amount.  Even for a lightly tanned Caucasian, with 10% melanosome volume fraction, the total light surviving the absorbance in the optical path looks virtually identical for both the oxygenated and deoxygenated Hb (dashed red and blue lines)—except in the deep blue region.

So even though melanin stops being a significant absorber at deep red and infra-red wavelengths, OBSERVABLE difference between the oxy/deoxy cases is quite small in this region.  The black line in the bottom of the above graph shows the DIFFERENCE between the 100% oxygenated and 100% deoxygenated case.  To see the difference spectrum in detail, we need to plot it on a log axis.

Other than showing again why the excitation light must be boosted for higher melanosome volume fraction case, this plot suggests where the wavelength peak for different photodiode band-pass filter should be, but even at the peak signal wavelengths, the minisuscule difference requires efforts to map the tiny differences into the available dynamic range of the measurement circuit.  Let's do a thought experiment.  Assuming:

• A Gaussian distribution of excitation light (say LED) source with 5% standard deviation of the mean (just an average voltage regulator spec), which is turned on and then off in step function manner
• Ambient (leakage) light of 20% of the intensity as the LED source, and 5% standard deviation of the mean intensity landing on the photodiode
• Photodiode conversion efficiency (photodiode output volt per lux) is 1 [V/lux]
• 5% of the excitation light reflects off the skin
• 100 PPM of the photons reaching INSIDE the skin (which I believe is quite generous)

The expected total output voltage from the photodiode is 8 bits larger than the expected voltage due to the signal.

If I were using a 12-bit ADC (as many bits as can be expected on an on-chip ADC), I would have only (12 - 8) = 4 bits left as the signal's dynamic range (averaging would gain back more bits).  How might we increase the dynamic range of the expected signal?

Idea 1: a photo-diode dedicated for ambient (leakage) light

What if we put not just 1 photodiode for the desired wavelength, but a PAIR of such photodiodes--one placed in a way to be shielded away from the light being reflected off the skin?  Since the photo-diodes will NOT be exactly the same (the band-pass filter and the photo-diodes are slightly different).  By measuring the output from the 2 photodiodes with the excitation light off, we should be able to estimate the leakage light in an unbiased manner.  We can gain back more than 2 bits in this example.  The ADC is measuring the sum of the reflection and true signal at this point.

Idea 2: subtract offset

Since the reflection still dominates in this example (by roughly 6 bits), what if we could programmatically set the ADC offset?  Observe the ADC output if we subtract 0.4 in the above example: we gain an additional ~3 bits, for a cumulative ~5 bits of information, so that even with a 12-bit on-chip ADC, we have ~9 bits of information (dynamic range of ~500).

Idea 3: estimate reflection and signal sum

To clarify idea 1, there is of course no way to KNOW the ambient light falling on the photodiode; the best I can do is build a HW that can estimate that unknown in an unbiased manner.  An averaging function is one such unbiased estimator, and one that can even be implemented in HW (like an FPGA) with some effort.  The moving average of the photodiode dedicated to measuring just the ambient light will rapidly approach the true value, and the moving average of the difference of the other photodiode (exposed to reflection and signal) and the unbiased estimate will also approach the true value (the bound drops by 1/SQRT(N), where N is the averaging window size), as shown below.

By oversampling (higher battery drain), I can build an estimator for the reflection and the signal with a few percent error.

Upper layer SW

At this point, the estimate is still dominated by the reflection light; oxygenated/deoxygenated Hb rides on top of the reflection at the period of the heart cycle.  In the frequency domain, the raw ADC signal will have peaks at the body and sensor movement frequency, and at the heart cycle frequency--which can coincide at times.  Discriminating the 2 cases therefore require algorithms that are too complicated to implement just in HW.

Mainline uClinux on STM32F429-discovery

In a previous blog, I explored the mainline U-Boot on the stm32f429-discovery eval board.  The mainline kernel now supports the same board, so I will study that uClinux (the Linux kernel without the MMU) on that board in this blog entry.

Mainline kernel's arch/arm/config/stm32_defconfig

Studying the kernel configs for the reference board should reveal what kernel features are available for the Cortex M4, so let's go through arch/arm/config/stm32_defconfig

CONFIG_NO_HZ_IDLE=y

Enable tickless idle system, required for any wearable device that can last more than a day on a small rechargeable battery.

CONFIG_HIGH_RES_TIMERS=y

I would guess this is only required for a high sampling frequency control system.  Why is this turned on?

CONFIG_LOG_BUF_SHIFT=16

Log buffer size.  64 KB seems reasonable.

CONFIG_BLK_DEV_INITRD=y

initramfs/initrd support.  Essential for a disk-less system.

CONFIG_CC_OPTIMIZE_FOR_SIZE=y

Choose "-Os" over "-O2", since the flash space is severely constrained (only 2 MB)

# CONFIG_UID16 is not set

Legacy 16-bit UID syscall wrappers.  Agree we do not need to support legacy syscall.

# CONFIG_BASE_FULL is not set

Disabling this option reduces the size of miscellaneous core kernel data structures. This saves memory on small machines, but may reduce performance.

# CONFIG_FUTEX is not set

Disabling this option will cause the kernel to be built without support for "fast userspace mutexes".  The resulting kernel may not run glibc-based applications correctly.  This is OK because I will use uclibc (smaller).

# CONFIG__EPOLL is not set

Do not support for epoll family of system calls.  "man epoll" yields:

The  epoll  API performs a similar task to poll(2): monitoring multiple file descriptors to see if I/O is possible on any of them.

I will need to turn this back on if any userspace libraries I need require it.

# CONFIG_SIGNALFD is not set

Needed if I want to receive signals on a file.

# CONFIG_EVENTFD is not set

Needed to receive events (kernel or userspace) on a file.

# CONFIG_AIO is not set

Disable POSiz async I/O (used by some high performance threaded applications) to save 7 KB from the kernel.

CONFIG_EMBEDDED=y

Need for embedded system so certain expert options are available for configuration; selects CONFIG_EXPERT.

# CONFIG_VM_EVENT_COUNTERS is not set

Turn off showing the event count in /proc/vmstat (to reduce the kernel size I guess).

# CONFIG_SLUB_DEBUG is not set

According to this discussion, we should be using the SLOB (drastically simpler and space efficient) as the SLAB allocator instead of the default SLUB allocator.  Anyway, Kconfig says:

SLUB has extensive debug support features. Disabling these can result in significant savings in code size. This also disables SLUB sysfs support. /sys/slab will not exist and there will be no support for cache validation etc. 

# CONFIG_LBDAF is not set

I do NOT need to support large (> 2 TB) block device.

# CONFIG_BLK_DEV_BSG is not set

No need for SCSI generic v4.

# CONFIG_IOSCHED_DEADLINE is not set

Exclude deadline-iosched.o

# CONFIG_IOSCHED_CFQ is not set

Excludes cfq-iosched.o

# CONFIG_MMU is not set

Cortex M CPUs have MMU.

CONFIG_ARM_SINGLE_ARMV7M=y

1 CPU ARMv7-M based platforms (Cortex-M0/M3/M4).  Selects:

• ARM_NVIC: ARM Cortex has NVIC (nested vectored interrupt controller) HW
• AUTO_ZRELADDR: ZRELADDR is the physical address where the decompressed kernel image will be placed. If AUTO_ZRELADDR is selected, the address will be determined at run-time by masking the current IP with 0xf8000000. This assumes the zImage being placed in the first 128MB from start of memory.
• CLKSRC_OF
• COMMON_CLK: single definition of struct clk, useful across many platforms
• CPU_V7M
• GENERIC_CLOCKEVENTS
• NO_IOPORT_MAP
• SPARSE_IRQ: Sparse irq numbering is useful for [if you] want to define a high CONFIG_NR_CPUS value but still want to have low kernel memory footprint on smaller machines. ( Sparse irqs can also be beneficial on NUMA boxes, as they spread out the interrupt descriptors in a more NUMA-friendly way. )
• USE_OF: FDT support

CONFIG_ARCH_STM32=y

There is no finer granularity for the STM32 flavors than this.  Selects:

• ARCH_HAS_RESET_CONTROLLER
• ARMV7M_SYSTICK: ARMv7m has systick HW
• CLKSRC_STM32
• RESET_CONTROLLER

CONFIG_SET_MEM_PARAM=y

This does NOT appear in any Kconfig, Makefile, or source?!

CONFIG_DRAM_BASE=0x90000000

Where the external SDRAM begins.

CONFIG_FLASH_MEM_BASE=0x08000000

The on-chip flash is at 0x08000000 for all STM32 variants.  The STM32Lxxx has another bank at 0x0, but let's not digress.

CONFIG_FLASH_SIZE=0x00200000

On-chip flash for the STM32F429 is 2 MB.

CONFIG_PREEMPT=y

Enable kernel preemption, which improves (but does NOT quite guarantee) real-time responsiveness.

# CONFIG_ATAGS is not set

No need for this traditional way of passing data to the kernel at boot time. Unnecessary if solely relying on the flattened device tree.

CONFIG_ZBOOT_ROM_TEXT=0x0

Kconfig:

The physical address at which the ROM-able zImage is to be placed in the target.  Platforms which normally make use of ROM-able zImage formats normally set this to a suitable value in their defconfig file.

Q: what is a ROM-able zImage?

ZBOOT_ROM lets you execute the zImage (compressed kernel image) directly from ROM/flash.

CONFIG_ZBOOT_ROM_BSS=0x0

 The base address of an area of read/write memory in the target for the ROM-able zImage which must be available while the decompressor is running. It must be large enough to hold the entire decompressed kernel (which is unlikely to be < 1 MB!) plus an additional 128 KiB.  Since ZBOOT_ROM_TEXT == ZBOOT_ROM_BSS, it means ZOOT_ROM (execute zImage directly from ROM/flash) is NOT selected.

CONFIG_XIP_KERNEL=y

Kconfig help:

Execute-In-Place allows the kernel to run from non-volatile storage directly addressable by the CPU, such as NOR flash. This saves RAM space since the text section of the kernel is not loaded from flash to RAM.  Read-write sections, such as the data section and stack, are still copied to RAM.  The XIP kernel is not compressed since it has to run directly from flash, so it will take more space to store it.  The flash address used to link the kernel object files, and for storing it, is configuration dependent. Therefore, if you say Y here, you must know the proper physical address where to store the kernel image depending on your own flash memory usage.  Also note that the make target becomes "make xipImage" rather than "make zImage" or "make Image".  The final kernel binary to put in ROM memory will be arch/arm/boot/xipImage.

Denx.de documentation:

• XIP conserves RAM at the expense of flash. This might be useful if you have a big flash memory and little RAM.

Q: Is the simplicity of running the kernel directly from the on-chip flash worth giving up the in-field upgrade capability? 

CONFIG_XIP_PHYS_ADDR=0x08008000

For !CONFIG_MMU, XIP_VIRT_ADDR is the same as CONFIG_XIP_PHYS_ADDR.  

This is NOT the uC's boot address, you can confirm in the STM32F4 reference manual Boot configuration section:

After this startup delay is over, the CPU fetches the top-of-stack value from address 0x0000 0000, then starts code execution from the boot memory starting from 0x0000 0004.  On the device, 0x0 ~ 0x1FFFFF is aliased to the boot memory (whether ROM, SRAM, or flash depending on the BOOT pins). 

So we are leaving 32 KB in the front--16 KB for the DTB at 0x08004000, and the first 16 KB (at the boot address 0x8000000) is meant for the bootloader (like U-Boot or afboot-stm32), which should pick up the address of the kernel entry function stextin the .head.text section (AKA __HEAD in head.nommu.S) at the CONFIG_XIP_PHYS_ADDR.  The address is "+ 1" to indicate thumb mode.  The linker script (arch/arm/kernel/vmlinux.lds.S) places the .head.text section at this address:

#ifdef CONFIG_XIP_KERNEL

. = XIP_VIRT_ADDR(CONFIG_XIP_PHYS_ADDR);

#else

. = PAGE_OFFSET + TEXT_OFFSET;

#endif

.head.text : {

_text = .;

HEAD_TEXT

}

CONFIG_BINFMT_FLAT=y

Picks up binfmt_flat.o

CONFIG_BINFMT_SHARED_FLAT=y

A kind of flat binary file.  I thought shared library is not viable on no-MMU, but when I checked the Buildroot target option, sure enough, I found "Shared binary" option under FLAT binary type.  But gcc does not even build for this option if NPTL threads is enabled.  Besides, the Buildroot arch/Config.in explains that BR2_BINFMT_FLAT_SHARED is no substitute for the "real" shared library:

        # Even though this really generates shared binaries, there is no libdl

        # and dlopen() cannot be used. So packages that require shared

        # libraries cannot be built. Therefore, we don't select

        # BR2_BINFMT_SUPPORTS_SHARED and therefore force BR2_STATIC_LIBS

Since I configured my toolchain without shared binary file format, the kernel support for it is pointless.  Still, it does raise a question on how one uses loads binary file

# CONFIG_COREDUMP is not set

Turns off kernel coredump support.  Is this the user space "core dump" I know about?

CONFIG_DEVTMPFS=y

Kconfig help:

This creates a tmpfs/ramfs filesystem instance early at bootup.  In this filesystem, the kernel driver core maintains device nodes with their default names and permissions for all registered devices with an assigned major/minor number.  Userspace can modify the filesystem content as needed, add symlinks, and apply needed permissions.  It provides a fully functional /dev directory, where usually udev runs on top, managing permissions and adding meaningful symlinks.  In very limited environments, it may provide a sufficient functional /dev without any further help. It also allows simple rescue systems, and reliably handles dynamic major/minor numbers.  Notice: if CONFIG_TMPFS isn't enabled, the simpler ramfs file system will be used instead.

CONFIG_DEVTMPFS_MOUNT=y

Kconfig help:

Automount devtmpfs at /dev, after the kernel mounted the rootfs.   This option does not affect initramfs based booting, here the devtmpfs filesystem always needs to be mounted manually after the rootfs is mounted. With this option enabled, it allows to bring up a system in rescue mode with init=/bin/sh, even when the /dev directory on the rootfs is completely empty.

# CONFIG_FW_LOADER is not set

Turns off the USERSPACE (vs. kernel) FW loading.  No loss of feature for me.

# CONFIG_BLK_DEV is not set

Turns off ALL block dev (even ramdisk or loop).

CONFIG_EEPROM_93CX6=y

EEPROM chipsets 93c46 and 93c66 driver.  Does stm32f429 even have an EEPROM?

# CONFIG_INPUT is not set

This board will be a headless system for now.

# CONFIG_SERIO is not set

Some input device (e.g. keyboard, PS2 mouse) uses serial I/O to communicate with the kernel.  Again, this is a headless system.

# CONFIG_VT is not set

VT = virtual terminal.  Again, this is a headless system.

# CONFIG_UNIX98_PTYS is not set

I won't run telnet or xterm into this system, so turning off PTY is OK.

# CONFIG_LEGACY_PTYS is not set

Ditto.

CONFIG_SERIAL_NONSTANDARD=y

Necessary for the STM32 serial port below.

# CONFIG_DEVKMEM is not set

No need for /dev/kmem to debug the kernel

CONFIG_SERIAL_STM32=y

This driver supports all industry standard baud rates (what might those be?).

CONFIG_SERIAL_STM32_CONSOLE=y

Here's the console on STM32!

# CONFIG_HW_RANDOM is not set

STM32F4 has a RNG HW, but perhaps older uC don't.  TODO: enable RNG support for the discovery board.

# CONFIG_HWMON is not set

Not going to care about the HW health for now.  Perhaps the on-chip temperature might be interesting later.

# CONFIG_USB_SUPPORT is not set

Let's not bother with USB for now.

CONFIG_NEW_LEDS=y

The discovery board has LEDs.

CONFIG_LEDS_CLASS=y

sysfs class in /sys/class/leds.  You'll need this to do anything useful with LEDs

CONFIG_LEDS_TRIGGERS=y

LED triggers allow kernel events to drive the LEDs and can be configured via sysfs, like blinking by a programmable timer, or one-shot blink, disk activity, or CPU activity, heartbeat, GPIO activity, or as a camera flash.

CONFIG_LEDS_TRIGGER_HEARTBEAT=y

LED flashes as a a hyperbolic function of the 1-minute load average.

# CONFIG_FILE_LOCKING is not set

Disable the standard file locking support (required for NFS and flock) to save 11 KB.

# CONFIG_DNOTIFY is not set

According to Kconfig file, what I am giving up is:

Dnotify is a directory-based per-fd file change notification system that uses signals to communicate events to user-space.  There exist superior alternatives.

# CONFIG_INOTIFY_USER is not set

According to Kconfig file, what I am giving up is:

Inotify allows monitoring of both files and directories via a single open fd.  Events are read from the file descriptor, which is also select()- and poll()-able.  Inotify fixes numerous shortcomings in dnotify and introduces several new features including multiple file events, one-shot support, and unmount notification. 

CONFIG_NLS=y

Native language support

CONFIG_PRINTK_TIME=y

Include time in printk.

CONFIG_DEBUG_INFO=y

"-g" compiler flag.  Essential for JTAG debugging, but might have to be sacrificed if I have only 1 MB flash.

# CONFIG_ENABLE_WARN_DEPRECATED is not set

We don't care to know, I guess:

#undef __deprecated

#undef __deprecated_for_modules

#define __deprecated

#define __deprecated_for_modules

# CONFIG_ENABLE_MUST_CHECK is not set

CPP just ignores "__must_check"

CONFIG_MAGIC_SYSRQ=y

Search for "Magic SysRq Key" in the Linux Kernel Development 3rd edition.

# CONFIG_SCHED_DEBUG is not set

I do NOT want to debug the scheduler.

# CONFIG_DEBUG_BUGVERBOSE is not set

Do NOT make the kernel's __BUG() macro verbose.

# CONFIG_FTRACE is not set

Ftrace is a kernel tracing feature.  Probably no room for this advanced feature?

CONFIG_CRC_ITU_T=y

Kconfig help:

This option is provided for the case where no in-kernel-tree modules require CRC ITU-T V.41 functions, but a module built outside the kernel tree does. Such modules that use library CRC ITU-T V.41 functions require M here.

CONFIG_CRC7=y

Same as above, except this is about the CRC7.

Building the stm32 defconfig kernel and rootfs in Buildroot

The 4.3.3 kernel with above defconfig builds nicely in latest mainline Buildroot (with the Cortex M4 addition to Buildroot I discussed in a previous blog entry).  My buildroot defconfig is:

BR2_arm=y

BR2_cortex_m4=y

BR2_ARM_FPU_VFPV4=y

BR2_ENABLE_DEBUG=y

BR2_TOOLCHAIN_BUILDROOT_LOCALE=y

# BR2_UCLIBC_INSTALL_UTILS is not set

BR2_BINUTILS_VERSION_2_25_X=y

BR2_GCC_VERSION_5_X=y

BR2_TOOLCHAIN_BUILDROOT_CXX=y

BR2_PACKAGE_HOST_ELF2FLT=y

BR2_PACKAGE_HOST_GDB=y

BR2_PACKAGE_HOST_GDB_TUI=y

BR2_PACKAGE_HOST_GDB_PYTHON=y

BR2_GDB_VERSION_7_10=y

BR2_ENABLE_LOCALE_PURGE=y

BR2_ENABLE_LOCALE_WHITELIST="en_US"

BR2_ECLIPSE_REGISTER=y

BR2_TARGET_GENERIC_HOSTNAME="uClinux"

BR2_TARGET_GENERIC_ISSUE="Welcome!"

BR2_ROOTFS_DEVICE_CREATION_STATIC=y

BR2_TARGET_GENERIC_ROOT_PASSWD="*******"

BR2_TARGET_GENERIC_GETTY_PORT="ttyPS0"

BR2_LINUX_KERNEL=y

BR2_LINUX_KERNEL_DEFCONFIG="stm32"

BR2_LINUX_KERNEL_IMAGE_TARGET_CUSTOM=y

BR2_LINUX_KERNEL_IMAGE_TARGET_NAME="xipImage"

BR2_LINUX_KERNEL_DTS_SUPPORT=y

BR2_LINUX_KERNEL_INTREE_DTS_NAME="stm32f429-disco"

BR2_TARGET_ROOTFS_INITRAMFS=y

# BR2_TARGET_ROOTFS_TAR is not set

Note that I left the thread library implementation at the default: NPTL, rather than "none" or "linuxthreads", as is common among other people working on "Linux on Cortex M", to preserve a sliver of hope of porting Qt5 (which current requires C++, NPTL toolchain, and shared lib) to a no MMU system.

[Update: this is a false hope.  uClibc currently requires MMU for NPTL, so the best I could do was Linux thread.  It's just that Buildroot did not expose this uClibc constraint properly, so I got the impression that I was building uClibc for NPTL, when in fact I was DISABLE threading altogether.]

I use the static device table because a wearable product will not have dynamic dev entries.  The devtempfs may still afford some convenience in device table population, so I may yet revert back to "dynamic with devtempfs".

The choice of root filesystem Buildroot offers is overwhelming, but this TI wiki page was of some help.

• axfs: advance XIP file system allowsfiles to be executed directly from flash/ROM (rather than being copied into RAM)
• cloop: compressed filesystem
• cramfs: compressed read-only filesystem
• ext2/3/4 root filesystem
• ext2 requires a block device (which does NOT account for the erase block behavior of the flash)
• initramfs linked into the kerne is the easiest option if:
• There is extra room where the kernel is stored (increases the kernel image size)
• There is extra RAM to hold the root filesystem that will be copied along with the kernel
• You don't need to persist changes to the filesystem (corollary of the above)
• j2ffs2 (journalling flash file system): mostly used for NOR flash.  TI wiki page quite helpful.
• romfs: small read-only filesystem
• squashfs
• ubifs: j2ffs2 successor (better scalability and NAND support)
• yaffs2: mostly for NAND flash

Since the only non-volatile storage on the stm32f429-discovery board is the on-chip flash the only viable option for a non-networked rootfs is the initramfs, which I've chosen in the Buildroot defconfig.

I deliberately did NOT build u-boot in Buildroot because the stm32f429-disco does NOT build with the eabihf toolchain, as I discussed in a previous blog entry.

After make <board>_defconfig and then "make", the resulting kernel (remember that debugging is turned on) is > 1 MB:

BRm4$ ls -glh output/images/

total 1.3M

-rw-r--r-- 1 henry  64K Jan  2 23:06 rootfs.cpio

-rw-r--r-- 1 henry 3.2K Jan  2 23:06 stm32f429-disco.dtb

-rwxr-xr-x 1 henry 1.1M Jan  2 23:06 xipImage

Since the xipImage is NOT an ELF file, I can't poke at it more.

BRm4$ file output/images/xipImage 

output/images/xipImage: data

But I CAN look at the vmlinux (uncompressed ELF file, which is used for source code debugging).

BRm4$ arm-linux-size output/build/linux-4.3.3/vmlinux

   text   data    bss    dec    hex filename

1082608  52900 100608 1236116 12dc94 output/build/linux-4.3.3/vmlinux

Even the text section is > 1 MB.  Running readelf on the same file shows that the text has been placed in the on-chip flash, and the kernel expects the .data and .bss sections at the external SDRAM (at 0x90000000):

Section Headers:

  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al

  [ 0]                   NULL            00000000 000000 000000 00      0   0  0

  [ 1] .head.text        PROGBITS        08008000 008000 000060 00  AX  0   0  4

  [ 2] .text             PROGBITS        08008060 008060 0b4ad0 00  AX  0   0 32

...

  [15] .data             PROGBITS        90008000 118000 00c9a0 00  WA  0   0 512

  [16] .bss              NOBITS          900149a0 1249a0 018900 00  WA  0   0 32

Note that the .data section is at DRAM_BASE + XIP kernel text offset.  You can also run "$(CROSS_COMPILE)readelf -s" to find where the kernel entry point stext was linked:

  Entry point address:               0x8008001

Describing the stm32f429-dicsovery board to the kernel

DTS is now the accepted way to describe the hardware to the kernel.  For this board, arch/arm/boot/dts/stm32f429-disco.dts is that file.  But since DTS inheritance is used extensively, this file inherits from increasingly non-board specific DTSI files: stm32f429.dtsi --> armv7-m.dtsi --> skeleton.dtsi.  The skeleton is literally just the skeleon, so let's start with the armv7-m.dtsi, which describes the interrupt controller and the systick hardware available on ALL ARM 7M architecture:

        nvic: nv-interrupt-controller  {

                compatible = "arm,armv7m-nvic";

                interrupt-controller;

                #interrupt-cells = <1>;

                reg = <0xe000e100 0xc00>;

        };

        systick: timer@e000e010 {

                compatible = "arm,armv7m-systick";

                reg = <0xe000e010 0x10>;

                status = "disabled";

        };

...

stm32f429.dtsi then supplies the ST specific hardware: clocks, timer, USART, random number generator, and RCC (reset and clock CONTROL), 

        clocks {

                clk_hse: clk-hse {

                        #clock-cells = <0>;

                        compatible = "fixed-clock";

                        clock-frequency = <0>;

                };

        };

        soc {

...

                usart1: serial@40011000 {

                        compatible = "st,stm32-usart", "st,stm32-uart";

                        reg = <0x40011000 0x400>;

                        interrupts = <37>;

                        clocks = <&rcc 0 164>;

                        status = "disabled";

                };

                rcc: rcc@40023810 {

                        #clock-cells = <2>;

                        compatible = "st,stm32f42xx-rcc", "st,stm32-rcc";

                        reg = <0x40023800 0x400>;

                        clocks = <&clk_hse>;

                };

                rng: rng@50060800 {

                        compatible = "st,stm32-rng";

                        reg = <0x50060800 0x400>;

                        interrupts = <80>;

                        clocks = <&rcc 0 38>;

                };

        };

The USART base address is found in the uC datasheet memory mapping table.  On the stm32f429-disco board, usart1 TX/RX are exposed on P1 header, as explained here.

Finally, the board specific DTS chooses the serial console, SDRAM address, external clock frequency, and the kernel boot arg:

/ {

        model = "STMicroelectronics STM32F429i-DISCO board";

        compatible = "st,stm32f429i-disco", "st,stm32f429";

        chosen {

                bootargs = "root=/dev/ram rdinit=/linuxrc";

                stdout-path = "serial0:115200n8";

        };

        memory {

                reg = <0x90000000 0x800000>;

        };

};

&clk_hse {

        clock-frequency = <8000000>;

};

&usart1 {

        status = "okay";

};

The DTB produced by Buildroot (see above) has to be written to the address the boot loader will tell the kernel: 0x08004000 in this case.  Since afboot-stm32 is essentially just a function call for me, I use openocd itself to write the DTB to the 2nd flash bank: 

BRm4$ sudo openocd -f /usr/local/share/openocd/scripts/board/stm32f429discovery.cfg \
-c "init" -c "reset init" \
-c "flash probe 0" -c "flash info 0" \
-c "flash write_image erase output/images/stm32f429-disco.dtb 0x08004000" \
-c "reset run" -c "shutdown"

Note that a NOR flash works by erasing a whole block (and therefore is NOT suitable for a random write usage).  Since the bootloader is at 0x08000000, writing to 0x08004000 would erase the boot loader if these 2 addresses were in the same flash block.  The blocks of the boot loader, the DTB, and the kernel was chosen to support independent update of these 3 binaries (that is why the kernel is at the 3rd flash block).

According to the openocd documentation, reading NOR flash works the same as reading memory, because NOR flash HW behaves like a RAM for reading.  So this command should print an endian flipped DTB header--one that matches the hexdump on the DTB. 

BRm4$ sudo openocd -f /usr/local/share/openocd/scripts/board/stm32f429discovery.cfg \
-c "init" -c "reset init" -c "mdw 0x08004000 4" -c "reset run" -c "shutdown"

JTAG debugging the stm32 defconfig kernel

I found a simple boot loader for the XIP kernel: afboot-stm32.  The code turns on all hardware on stm32f429, but since I turned off all peripheral init for the XIP kernel (was curious if the XIP kernel can initialize the external SDRAM by itself), the "bootloader" just jumps to the kernel entry location and consequently fits comfortably in the 16 KB NOR flash block at the uC boot address (0x08000000).  Since the kernel is on a different block, I can write the kernel image (in the vmlinux ELF produced right before the kernel makefile builds the xipImage) into the flash wit openocd (through the Eclipse CDT debugger for convenience), and debug the XIP kernel, with the following debug configuration for the ST-Link, where I turned off invoking the openocd within the debugger (because openocd requires sudo on Ubuntu--I suppose I can try to fix this with udev rules...):

Writing > 1 MB to the on-chip flash over 1800 kHz JTAG (requested 2000 kHz but got 1800 kHz instead) should only take 5 seconds but actually takes > 30 seconds (so the JTAG overhead to write to flash must be quite inefficient), but eventually, I hit the HW breakpoint set in arch/arm/mm/proc-v7m.S __v7m_setup (I set the break point there to debug why the kernel was jumping to 0x0 at some point).

I see that the vector_table (defined in entry-v7m.S) is expected to be somewhere on the SDRAM (0x9000e400).  That broke the initial SVCall, because the vector_table had not yet been copied to the SDRAM.

The vector table definition in the assembly file merely says to place it in the .data section:

.data
#if CONFIG_CPU_V7M_NUM_IRQ <= 112
.align 9
#else
.align 10
#endif

/*
 * Vector table (Natural alignment need to be ensured)
 */
ENTRY(vector_table)
.long 0 @ 0 - Reset stack pointer
.long __invalid_entry @ 1 - Reset
.long __invalid_entry @ 2 - NMI
.long __invalid_entry @ 3 - HardFault
.long __invalid_entry @ 4 - MemManage
.long __invalid_entry @ 5 - BusFault
.long __invalid_entry @ 6 - UsageFault
.long __invalid_entry @ 7 - Reserved
.long __invalid_entry @ 8 - Reserved
.long __invalid_entry @ 9 - Reserved
.long __invalid_entry @ 10 - Reserved
.long vector_swi @ 11 - SVCall
.long __invalid_entry @ 12 - Debug Monitor
.long __invalid_entry @ 13 - Reserved
.long __pendsv_entry @ 14 - PendSV
.long __invalid_entry @ 15 - SysTick
.rept CONFIG_CPU_V7M_NUM_IRQ
.long __irq_entry @ External Interrupts
.endr

ENTRY() macro merely aligns, and puts the symbol in global name space.

The .data section is placed ("linked") into the target memory in arch/arm/kernel/vmlinux.lds.S, specified by the PAGE_OFFSET

#ifdef CONFIG_XIP_KERNEL
__data_loc = ALIGN(4); /* location in binary */
. = PAGE_OFFSET + TEXT_OFFSET;
#else

...

#endif

.data : AT(__data_loc) {

_data = .; /* address in memory */

_sdata = .;

...

Remember the distinction between the data address within "binary" (kernel image) and "memory" where the kernel will run from (presumably the SDRAM).

The argument to AT() is the LMA (load address), not the virtual address.  Note that VMA (virtual address) is NOT specified here, and according to the LD manual:

If you do not provide address, the linker will set it based on region if present, or
otherwise based on the current value of the location counter.

The current location counter is in the SRAM territory in when .data is declared in vmlinux.lds.S.  So while __data_loc is in the flash, the .data section's VMA is in SRAM.  I think this is the bug: VMA should be in the flash (__data_loc), while the LMA should be the SRAM

Symbol table '.symtab' contains 21570 entries:
   Num:    Value  Size Type    Bind   Vis      Ndx Name
...
 18650: 08114914     0 NOTYPE  GLOBAL DEFAULT   14 __data_loc
...

We just saw that __data_loc was in the flash so why is .data in the SDRAM, according to "arm-linux-readelf -S vmlinux" as we saw above?

__data_loc used in arch/arm/head-common.S __mmap_switched_data:

.align 2
.type __mmap_switched_data, %object
__mmap_switched_data:
.long __data_loc @ r4
.long _sdata @ r5
.long __bss_start @ r6
.long _end @ r7
...

This seems to set R4 __load_loc.

I found CONFIG_PAGE_OFFSET in autoconf.h to be the same as the DRAM_BASE specified in the defconfig.  And TEXT_OFFSET was set as the -DTEXT_OFFSET=0x00008000 argument to the linker script generation, in turn specified by arch/arm/Makefile:

# Text offset. This list is sorted numerically by address in order to
# provide a means to avoid/resolve conflicts in multi-arch kernels.
textofs-y := 0x00008000
...
# The byte offset of the kernel image in RAM from the start of RAM.
TEXT_OFFSET := $(textofs-y)

Although reserving the 32 KB at the beginning of the SDRAM makes sense when the system runs on the SDRAM (like the laptop)--for the same reason we reserved the first 32 KB of the on-chip NOR flash as discussed above--this seems to be a waste of SDRAM.  [I'll try to make use of this 8 pages later.]

Putting all these together, I understand the ARM kernel boot code expects the memory that holds the data to have been started already.  Since I want to store the kernel image on the on-chip flash for now (i.e. the board does NOT have an external flash), it means I have to put the external SRAM boot-up code in the 16 KB, at 0x08000000.  But U-Boot text size is around 100 KB, even with debugging symbols removed (u-boot I built below has debugging symbols).

u-boot$ arm-linux-size u-boot
   text   data    bss    dec    hex filename
 126413   7384  40572 174369  2a921 u-boot

So I found a bare metal code called afboot-stm32 to bootstrap the SDRAM on the discovery board before starting the kernel.

Patch the binfmt_flag.c to add the thumb bit to the start address

In start_kernel() --> rest_init() --> kernel_thread --> do_fork, the init is forked off with a newly created kernel thread.  kernel_init runs the init, which in this case is in the initramfs.

if (ramdisk_execute_command) {
ret = run_init_process(ramdisk_execute_command);
if (!ret)
return 0;
pr_err("Failed to execute %s (error %d)\n",
      ramdisk_execute_command, ret);
}

But the flat binary produced by Buildroot has a bug on thumb: the start address of the executable is missing the "thumb bit".  A quick and dirty workaround is to force it in the kernel, in fs/binfmt_flat.c load_flat_binary(), like this:

start_addr = libinfo.lib_list[0].entry | 1;

But with the latest uClibc, this workaround might not be necessary any more.

Producing the STATIC_FLAT uClibc and busybox exe

At this point, the kernel was running the init program called "/linuxrc" (which happens to be just a softlink to busybox--as every executable on this build is).  More precisely arch/arm/kernel/vmlinux.lds.S places the COMPRESSED initramfs image at the end of the .init.data, as you can see here:

.init.data : {

#ifndef CONFIG_XIP_KERNEL

INIT_DATA

#endif

INIT_SETUP(16)

INIT_CALLS

CON_INITCALL

SECURITY_INITCALL

INIT_RAM_FS

}

This section is then copied to the SDRAM during in one of the kernel initcalls: do_initcalls() -->  populate_rootfs() --> unpack_to_rootfs(__initramfs_start, __initramfs_size).  INIT_RAM_FS macro above defines the __initramfs_start and initramfs_size used in the code (in my example 0x8118404 and 176319 respectively).  Next, the init executable ("/linuxrc" resolving to busybox) is located (AKA "loaded") in the rootfs, and relocated within the SDRAM (binfmt_flat.c), and "vforked" off.  The relocation behavior depends on the target file format.  Recall that the kernel is expecting BFLT format, but in the stm32 kernel config above "shared flat binary" feature was turned on.  I lost several days debugging the load failure because I turned on SHARED_FLAT target format in uClibc (vs. STATIC_FLAT or STATIC_FLAT_SEP_DATA).

The problem phenotype is that the busybox binary has a lot of GOT entries (which are 32 bits), and the MSB of those entries are 0x7F (which is certainly an invalid shared library ID allowed by the kernel--set to 3).  On a memory constrained system, sharing the executable's text section (cannot share data or bss) seems like a wonderful idea, and this feature seems to have been pioneered for blackfin.  But both the shared library and separate code/data features require gcc support (the blackfin specific gcc option is -mid-shared-library)--which has NOT yet been ported to the ARM.  While building the toolchain, using anything but the static flat binary format in the Buildroot config stops the gcc build (so you know).  But the Buildroot's toolchain exe format does not get propagated to the uClibc equivalent option, so for now, one must specify a custom uClibc defconfig.

Even after fixing this, /linuxrc did not run.  When I looked at the busybox executable produced by Buildroot in the output/build/busybox-<version> folder, I saw that it was only roughly 300 bytes: impossibly small.  After several days of after-work debugging, I learned that the final link step for busybox uses gcc rather than ld, but the LD_FLAGS were still being pulled in.  Normally, this is not a problem, but for the BFLT case, there are different flags for the "elf2flt" transformation:

• The gcc (when used as a linker) option is -Wl,-elf2flt
• The ld option is -elf2flt 

These CPPFLAGS and LDFLAGS workarounds (Buildroot patch for the nommu/flat) are necessary to skip the stripping step after the link (which is broken for the BFLT format).  Since gcc just passes the options starting with "-Wl," to the linker, the 2 options mean the same.  BUT, since the LD_FLAG included "-elf2flt", gcc interpreted this as the "entry point" ("-e" option) argument, and the link step failed silently, yielding an empty executable.  I worked around by replacing the "-elf2flt" to "-Wl,-elf2flt" in the makefiles, as shown in this example for the busybox/Makefile:

     "$(LDFLAGS) $(subst elf2flt,Wl$(comma)-elf2flt, $(EXTRA_LDFLAGS))" \

There were also a few patches I picked up from varcain repo.  With these changes, busybox builds and is > 500 KB, which makes more sense than the 300 B seen earlier.

BRm4/output/build/busybox-1.24.1$ arm-linux-size busybox_unstripped.gdb 

   text   data    bss    dec    hex filename

 381820 154816  19408 556044  87c0c busybox_unstripped.gdb

But I tried to pair this down a bit.  Dumping all network related commands and daemons saves nearly 100 KB.

BRm4/output/build/busybox-1.24.1$ arm-linux-size busybox_unstripped.gdb 

   text   data    bss    dec    hex filename

 318700 127232  18352 464284  7159c busybox_unstripped.gdb

Dumping commands for the storage media I will never use (like HDD, cdrom, floppy) knocks off another 50 KB.

BRm4/output/build/busybox-1.24.1$ arm-linux-size busybox_unstripped.gdb 

   text   data    bss    dec    hex filename

 282364 107256  18248 407868  6393c busybox_unstripped.gdb

I saved the resulting changed .config as BR2/package/busybox/stm32f429-disco.config, and made Buildroot use this as the busybox config.

As soon as I started running the right busybox exe ("/linuxrc"), I found that the init process crashed while running the very first shell script: /etc/init.d/rcS

Simplified init script for stm32

Debugging an init shell script was difficult (even with the shell script tracing option "set -x" added to the script), and I finally resorted to bisecting the script approach to hunt down the offending shell syntax:

for i in /etc/init.d/S??* ;do
     # Ignore dangling symlinks (if any).
     [ ! -f "$i" ] && continue

Firstly, the file pattern match did not even work: "S??*" did NOT get expanded out, but remained at "S??".  More immediately, the "test for file" syntax is what crashed busybox.  It looks like hush (the only shell available to the uClinux system for now) cannot parse legitimate sh scripts.  So I got rid of the fancy statements in the busybox provided rcS (which means I am not starting logging or urandom device driver).

Mount devtmpfs

mount: mounting devpts on /dev/pts failed: No such device

It's in varcain

[    0.880000] STM32 USART driver initialized
[    0.880000] 40011000.serial: ttyS0 at MMIO 0x40011000 (irq = 17, base_baud = 5625000) is a stm32-usart
[    1.130000] console [ttyS0] enabled
...
can't open /dev/ttyS0: No such file or directory

Being printed from getty open_tty() spawned from inittab (which launches init.d/rcS as well).

# Put a getty on the serial port
ttyS0::respawn:/sbin/getty -L  ttyS0 0 vt100 # GENERIC_SERIAL

These come from the Buildroot xconfig "getty" option.

Recall that we chose the devtmpfs in the kernel config.  It's supposed to populate the /dev, so why isn't it?  According to the kernel log, devtmpfs is getting initialized.

[    0.110000] devtmpfs: initialized

But I am not seeing "mounted".  I thought the kernel config specified devtmpfs mount, so I did not understand why it wasn't getting mounted, until I realized that the feature does NOT apply to initramfs rootfs.  So I mounted devtmpfs in rcS script:

mount -t devtmpfs devtmpfs /dev

As soon as I logged in as root, the system crashed after starting /bin/sh (with argument "-sh" for login) while running /etc/profile, which again has syntax that hush cannot handle (the bracket statement).  So I simplified it to this:

export PATH=/bin:/sbin:/usr/bin:/usr/sbin

export PS1='# '

export PAGER='/bin/more '
export EDITOR='/bin/vi'

# Source configuration files from /etc/profile.d
. /etc/profile.d/umask

And finally, I can see the shell!

/root # uname -a
[  266.740000] BINFMT_FLAT: Loading file: /bin/uname
[  266.740000] Mapping is 90200000, Entry point is dd1, data_start is 429a4
[  266.740000] Load /bin/uname: TEXT=90200040-902429a4 DATA=902429c0-902552c4 BSS=902552c4-902591bc
Linux (none) 4.4.0 #5 PREEMPT Sat Jan 30 19:25:21 PST 2016 armv7ml GNU/Linux
/root # 

According to top, it looks like I am using about half of the 8 MB SDRAM:

Mem: 3620K used, 4308K free, 0K shrd, 0K buff, 1556K cached

CPU:  88% usr  11% sys   0% nic   0% idle   0% io   0% irq   0% sirq

Load average: 0.00 0.00 0.00 1/19 29

  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND

   29    26 root     R      396   5% 100% top

   26     1 root     S      404   5%   0% -sh

    1     0 root     S      396   5%   0% {linuxrc} init

    2     0 root     SW       0   0%   0% [kthreadd]

    3     2 root     SW       0   0%   0% [ksoftirqd/0]

    4     2 root     SW       0   0%   0% [kworker/0:0]

    7     2 root     SW       0   0%   0% [rcu_preempt]

    8     2 root     SW       0   0%   0% [rcu_sched]

    9     2 root     SW       0   0%   0% [rcu_bh]

   10     2 root     SW       0   0%   0% [kdevtmpfs]

   11     2 root     SW<      0   0%   0% [writeback]

   12     2 root     SW<      0   0%   0% [bioset]

   13     2 root     SW<      0   0%   0% [kblockd]

   14     2 root     SW       0   0%   0% [kworker/0:1]

   15     2 root     SW       0   0%   0% [kswapd0]

   16     2 root     SW<      0   0%   0% [deferwq]

   17     2 root     SW       0   0%   0% [kworker/u2:1]

    5     2 root     SW<      0   0%   0% [kworker/0:0H]

    6     2 root     SW       0   0%   0% [kworker/u2:0]

Next step

Explore a sexy UI on a wearable band  

Appendix: kernel debugging techniques

I learned how to use the assembly view "instruction stepping mode" in Eclipse.

I also learned how to load busybox ELF file at the relocated text, data, and bss locations reported by the kernel.  To be pedantic, gdb cannot show me the symbols of the executable that has been relocated on the no-MMU system (the dynamic loader on MMU works auto-magically with the ELF file to pull this off), so I have to TELL gdb where busybox executable has been relocated.
Firstly, the BFLT loader in the kernel prints these value to the console only if a BLFT file has the kernel trace bit in the flag (the same flag that stores whether the file format is GOTPIC, RAM, etc).  This bit can be enabled by changing the "ktrace" counter in the elf2flt.c file, so I created a patch in Buildroot's package/elf2flt, to initialize the variable ktrace to 1 (instead of 0).

Then during kernel startup, I see flat file load addresses, like this example:

[  103.160000] BINFMT_FLAT: Loading file: /bin/sh
ogin[26]: root login on 'ttyS0'
[  103.170000] Mapping is 90600000, Entry point is dd1, data_start is 429a4
[  103.180000] Load /bin/sh: TEXT=90600040-906429a4 DATA=906429c0-906552c4 BSS=906552c4-906591bc

Then I fed those 3 addresses to gdb console, like this example:

add-symbol-file /mnt/work/band/uClinux/BRm4/output/build/busybox-1.24.1/busybox_unstripped.gdb 0x90600040 -s .data 0x906429c0 -s .bss 0x906552c4

The symbol can be removed as well:

remove-symbol-file /mnt/work/band/uClinux/BRm4/output/build/busybox-1.24.1/busybox_unstripped.gdb 

]