Coarse roll and pitch initialization using accelerometer in Unity

My humble dream

I am self-learning Unity3D game programming these days, to write an augmented reality peer-to-peer shooter game on a smartphone--just to play a game with my friend's 5-year old son in the living room.

The problem at hand

For a seamless integration of game objects on the screen, I need to track the phone's position and orientation within the room.  Since I don't know much about image processing beyond basic filtering and how to use OpenCV, I am looking into using relative 6 DOF tracking using the accelerometer and gyro available on every modern smartphones--despite the expected problems due to the high bias and noise of the MEMS sensors in a smartphone.  A solution discussed in many textbooks (I like Paul Groves the best so far) is to initialize the 6 DOF with some constraint (like when the IMU is stationary), and then integrate the angular rate and the acceleration sensed in the body frame of the phone.

All of this is elementary stuff for mechanical engineers.  But when I tried to apply this simple idea in Unity, I ran into complications: firstly Unity3D's left-handed coordinate frame.  At first, I tried to keep the navigation equations in RH system (as described in textbooks) and just convert to LH right before rendering, but the confusion did not go away.  So here's an attempt to re-derive the coarse attitude (but without the heading) initialization in Unity's LH system.  To motivate you to persevere through the trigonometry below, here is a screenshot illustrates the idea of putting the target plane (the ENEMY text) overlaid on top of the rear camera view.  As I tip and tilt my phone, my view of the target plane--which does NOT move or rotate in this demo--changes in the opposite angle of my current attitude estimate, which is displayed in the upper left corner of the video.

My Unity3D project is on my github repo: https://github.com/henrychoi/OpenCVUnity.  The red lines that flash on the screen when I move the phone are the optical flow vectors calculated using the OpenCV for Unity library ($95 on the Unity Asset store).  This is a bit of of a legacy: at first, I thought I might make a visual odometry based game, so I paid the money and tried OpenCV for Unity's optical flow demo--which works as you can see.  But when I studied how to calculate the current attitude from the optical flow vectors using inverse homology, I realized I did not understand the subject well enough.  If there is any interest in my project from other engineers, I will change out the WebcamTextureHelper script with just the plain WebcamTexture in Unity, and create a more recapitulatable project.

1st attempt: IMU frame as documented for the phone; wrong

When the device is (mostly) stationary, the accelerometer is just sensing gravity (reaction to gravity actually, but I'll explore that idea in a different blog entry).  In the diagram below, I've drawn an IMU (accel and gyro package) placed arbitrarily on a phone--in the RIGHT HANDED frame labeled 𝞵 (in navigation equations, the frame appears in superscripts and subscripts, so "IMU" is unwieldy), which is oriented also arbitrarily relative to the LEFT HANDED local tangent frame l (letter "ell").  Since I will play the game while holding my phone like I am taking a picture, I've drawn the LEFT HANDED body frame to be oriented the same way as the Unity's player frame when the phone is held that way: +Z away from the phone's rear facing camera, and +Y toward the top of the screen in a landscape mode, with the "Home" button on the right of the screen.

The gravity sensed by the IMU is g^𝞵_l𝞵: I've drawn the gravity in the -Z direction of the local tangent frame, and the gravity is resolved in 𝞵--specifically the x, y, z triad of the accelerometer.  As far as I know, the IMU is placed like this on all smartphones.  What are the x/y/z triad values as functions of the phone's roll and pitch?  If I define the roll and pitch to be zero when the body frame's Z axis is parallel to the l frame's -Z axis, and the body frame's X axis is parallel to the l frame's -Y axis, we can follow the roll and pitch as shown below.

Navigation equations can be confusing to anyone without spatial IQ of 200, so let's make a note that the blue line above is the physical components of the gravity, which will be SENSED by the accelerometer in 𝞵 frame (the blown) as the a vector.  After the roll and pitch, we can read off the components of gravity in each of the orthogonal axes of the IMU frame (brown above)--which may NOT be co-located with the body frame or even oriented exactly as drawn above (there are elaborate engineering requirement specs for the PCB placement of the IMU).  BUT if it were, the values would be as follows:

ax = -g cos𝟇 cos𝞱

ay = g sin𝟇

az = -g cos𝟇 sin𝞱

Let's not worry about the accel bias or the IMU mounting error relative to the body frame for now, and just solve the above over-constrained equations for the roll and pitch angles.

𝟇 = asin(ay/g)

𝞱 = atan2(-az, -ax) 

Note the negative sign into the arc tangent, necessary to land on the correct quadrant.

But when I tried initializing with these equations, the answer was clearly wrong: my spaceship jumped to a completely wrong orientation!

2nd attempt: Empirically determined IMU frame

I don't know if Unity will rotate this frame for a portrait mode, but it looks like this frame is just the same as the player frame, except for being RIGHT handed.

The last rotation looks confusing because of the choice of the rotation amount; it may help to imagine that you are looking at the b frame from below--but do that ONLY for the 3rd frame below.

The accel triad in the IMU frame (length of the blue arrows on the brown axes) then change to

ax = -g sin𝟇

ay = -g cos𝟇 cos𝞱

az = -g cos𝟇 sin𝞱

In which case the angles are:

𝟇 = asin(-ax/g)

𝞱 = atan2(-az, -ay) 

The sequence of rotation from the local tangent frame l to the current body frame attitude, expressed as LEFT to right (because that's how Unity applies the rotation--the opposite of almost all quaternion books I've read) quaternion multiplication is then

q_lb = Quaternion(0, 0, sin(𝞹/4), cos(𝞹/4)) // quarter turn about Z
    * Quaternion(sin(𝞹/4), 0, 0, cos(𝞹/4)) // quarter turn about X
    * Quaternion(0, 0, sin(𝟇/2), cos(𝟇/2))
    * Quaternion(sin(𝞱/2), 0, 0, cos(𝞱/2))

where I use the quaternion notation of the form [x, y, z, w] (rather than [w, x, y, z]).  Above algorithm has a singularity at roll angle = +/- 90 deg, because the y and z components of the acceleration are small (and ax is almost 1).

3rd attempt: yaw to avoid singularity at roll +/- 90 deg

To avoid singularity when roll is near +/- 90, I need to apply a yaw around Y (instead of the pitch around X as above).  I drew 2 pictures to convince myself that the same equation can handle both the +90 and the -90 case, as you can see below:

Note that the acceleration triad expressions are the same around positive and negative roll, so I don't have to come up with 2 distinct cases.  So how shall we solve the following accelerometer triad?

ax = -g sin𝟇 cos𝟁

ay = -g cos𝟇

az = g sin𝟇 sin𝟁

The angles are:

𝟇 = -sign(ax) acos(-ay/g)
𝟁 = asin2(az/(g sin𝟇))

The switch between the 1st and this alternative rotation scheme is whether the normalized |ax| is larger than some value, say 0.8.  The quaternion from the local frame to the body frame is then

q_lb = Quaternion(0, 0, sin(𝞹/4), cos(𝞹/4))
    * Quaternion(sin(𝞹/4), 0, 0, cos(𝞹/4))
    * Quaternion(0, 0, sin(𝟇/2), cos(𝟇/2))
    * Quaternion(0, sin(𝟁/2), 0, cos(𝟁/2))

When I tried this solution, I found a small jump in the quaternion values between the previous solution and this.  I struggled to for a day to understand this discontinuity, but could not.  Until I can figure this out, I decided to just spherically interpolate between the 2 solutions.  The behavior seemed reasonable on the Unity remote, and that is what you see in the introduction video.

Unity implementation

I tried to initialize my current attitude with Quaternion.Euler(x, y, z), whose definition of the rotation order miraculously matches the rotation order presented above.  But for some reason, it did not work (I am on Unity version 5.2.4f2), so I had to just brute-force multiply the quaternions in my PlayerController.FixedUpdate() method, as shown below:

Vector3 a_b = a_inb_lb_ens.Average.normalized;
float roll, pitch;
roll = Mathf.Asin (-a_b.x);
pitch = Mathf.Atan2 (-a_b.z, -a_b.y);
roll *= 0.5f; // Halve the Euler angles feed to quaternion construction
pitch *= 0.5f;

Quaternion q = // Apply the roll first, then pitch
    new Quaternion (0, 0, Mathf.Sin (roll), Mathf.Cos (roll))
    * new Quaternion (Mathf.Sin (pitch), 0, 0, Mathf.Cos (pitch));
const float THRESHOLD = 0.8f;

if (Mathf.Abs (a_b.x) < THRESHOLD) { // 1st case: cos(roll) reasonable value
    q_lb = Q_lb * q;
} else { // 2nd case, roll near +/- 90; ay << ax
    roll = -Mathf.Sign (a_b.x) * Mathf.Acos (-a_b.y);  
    float yaw = Mathf.Asin (a_b.z / Mathf.Sin (roll));

    roll *= 0.5f;
    yaw *= 0.5f;

    Quaternion q2 =// Apply the roll first, then yaw about Y
        new Quaternion (0, 0, Mathf.Sin (roll), Mathf.Cos (roll))
        * new Quaternion (0, Mathf.Sin (yaw), 0, Mathf.Cos (yaw));

    q_lb = Q_lb * Quaternion.Slerp (q, q2, Mathf.Abs (a_b.x) - THRESHOLD);
}
q_Ub = Q_Ul * q_lb; //Rotate the q_lb by Q_lU to go to Unity
myRigidbody.MoveRotation(q_Ub);

Q_lb and Q_Ub are constant quaternions relating respectively the attitude of the player WRT the local tangent frame, and the local tangent frame to Unity's inertial frame.
      static Quaternion Q_lb = new Quaternion (0, 0, Mathf.Sin (Mathf.PI / 4), Mathf.Cos (Mathf.PI / 4))
      * new Quaternion (Mathf.Sin (Mathf.PI / 4), 0, 0, Mathf.Cos (Mathf.PI / 4))

            ;

static Quaternion Q_Ul = //Will be used often, so calculate once
      new Quaternion(0, Mathf.Sin(-Mathf.PI/4), 0, Mathf.Cos(-Mathf.PI/4))
      * new Quaternion(Mathf.Sin(-Mathf.PI/4), 0, 0, Mathf.Cos(-Mathf.PI/4))
            ;

And here is the video of a short session, where I use the attitude initialization at 50 Hz (i.e. I am NOT yet integrating the INS input).

Conclusion: use the (left) hand, Luke--and sweat it out

Attitude initialization is the 1st step in an INS (inertial navigation system).  While the idea of using the side components of accelerometer triad to estimate the current tip and tilt is simple to imagine, actually pulling off a Unity game demo was far more difficult than I imagined at first, partly because Unity's left handed frame forced me to re-derive the gravity triad expressions.  In Groves' GNSS 2nd edition, this subject takes up just a single page (and the book and the appendices all combine to just shy of 1000 pages)!  Drawing clear pictures, and applying successive LEFT handed rotation step by step was the only way I could to come through the whole mess in one piece.  Although it was grueling, I am glad that I persevered through it; it was a character-building experience, and now I am on my way to integrating the gyro and accel into a 6 DOF solution.

ASR on a phone

My greatest frustration with Google Now is the latency.  I understand that the amount of computation and data required for modern ASR does not fit on a phone, hence the need to ship off the compressed voice data to the Google backend, and eat the latency penalty.  But if we want a Her-like speech interface, ASR will HAVE TO run on a phone-scale resource.  Since I have some experience with SW optimization, I looked for an ASR (automatic speech recognition) SW package I could play around with, and found Sirius: an open source IPA (intelligent personal assistant).

U. Michgan Sirius

Sirius bundles together the following open source projects to deliver IPA feature:

• ASR
• CMU Sphinx: widely used GMM (Gaussian mixture Model) ASR  SW.
• RWTH's RASR
• Kaldi: DNN (deep neural network) based ASR
• OpenEphyra: question and answer system, based on IBM's Watson
• SURF: image matching algorithm implemented using OpenCV

The GMM scores HMM state transitions by mapping an input feature vector into a multi-dimensional coordinate system and iteratively scores the features against the trained acoustic model.  DNN is defined by the number of hidden layers where scoring amounts to one forward pass through the network. In recent years, industry and academia have moved towards DNN over GMM due to its higher accuracy.  Text output from ASR is passed to the Q&A system, which uses 3core processes to extract textual information:

1. word stemming ("elected" --> "elect").  Porter stemming
2. regular expression matching ([#th] --> #)
3. part-of-speech tagging, using CRF (conditional random field)

QA service takes more time than ASR, and is more variable, primarily because of the time to select the most fitting answer.

Sirius Suite (C/C++) benchmark suite captures the bottlenecks:

• ASR: GMM/DNN scoring, rather than HMM
• GMM: nested loops that iteratively score the feature vector against training data (acoustic model, language model, dictionary).  Store entire data required for GMM in 2 GB.
• QA: all 3 core processes (above)
• Stemmer: check for multiple variables of a word (suffix, etc)

Building and running sirius-suite on Ubuntu 14

I downloaded sirius-suite-1.1 and sirius-caffe-1.0 (dependency), which requires cmake (to build yet another dependency: OpenCV).  I advise you run this before the official install process (the sirius developers missed these packages possibly because they are on Ubuntu 12):

$ sudo apt-get install cmake protobuf-compiler liblmdb-dev python-numpy-dev aptitude
$ sudo aptitude install libhdf5-serial-dev

Manually override the offered choice and accept an alternative, to resolve the dependencies for libhdf5-serial-dev.

Then follow the instruction for sirius-caffe link above to build both cafe and the sirius-suite.  The suite needs to be told where sirius-caffe libs are, so run the test command with LD_LIBRARY_PATH, like this:

$ LD_LIBRARY_PATH=/mnt/work/CL/sirius/caffe/distribute/lib make test
{
"kernel":"gaussian_mixture_model",
"abrv":"gmm",
"gmm": 13.183000
}

...

I don't know how to interpret the tests yet, so I keep moving to the individual kernel testing.  Even though the QA system is the tallest pole to whack for improved user experience, I am new to NLP (natural language processing), so I start where there is some physical signal processing, which is currently my expertise area.

GMM ASR scoring

In the gmm/ folder of the sirius suite, the build created only 2 folders: baseline and pthread, because I have not attached an ML605 FPGA (which I've had for a few years already) or a CUDA GPU.  My laptop actually has a CUDA GPU (Quadro K2100M), so I'll see if I can run the CUDA version too--but later.  This Sirius suite "kernel" comes with its (one) test data: gmm_data.txt, which is more than 100 MB untarred.  It looks like a time series of feature vector (N=29) time series, which looks like this (only the 1st vectors shown, because for each sample--5120--means and precs are matrices while weight and factor are vectors):

There is another vector: feature_vector (confusing!), which is just a bias for each features.
Without any idea of the underlying physical data (what was uttered), I'll try to understand the GMM code.  The score computation goes like this:

ln(Val2) = weight_f
+ \frac{1}{ln(1.0001)} \sum_f {prec_f (C_f - means_f)^2}
- factor_f
\\
ln\Delta = |score - ln(Val2)|
\\
ln(Highest) = \begin{cases}
score & score - ln(Val2) < 0\\
ln(Val2) & else
\end{cases}

add /usr/local/texlive/2015/bin/x86_64-linux to your PATH 

DNN (deep neural network) scoring

Machine learning is all the rage these days perhaps because the compute power available is finally allowing the learning algorithms to yield useful result.  Supposedly, DNN is 1 such example.  

ASR on Raspberry Pi 3

My Intel quad-core laptop, with 16 GB RAM does not have a mobile-device spec.  It would be great to run Sirius on an Android phone, but my phone is not open to running a stock Linux either.  Plus, I cannot add GPIO HW easily.  Except for somewhat stingy RAM size, the recently released Raspberry Pi 3 has a reasonable mobile-device spec:

• CPU: Broadcom BCM2837 (quad ARM Cortex-A53--64-bit ARMv8 architecture)
• Memory: 1 GB LP DDR2
• GPU: VideoCore IV

I prefer to roll my own Linux distribution with Buildroot.  The latest Buildroot even supports RPi 3 out of the box, so I will only need to modify the packages.

~/work/CL$ git clone git://git.buildroot.net/buildroot rproot
~/work/CL/rproot$ make raspberrypi3_defconfig
~/work/CL/rproot$ make xconfig

The Buildroot default configuration wants to create an SD card image, whereas I prefer to NSF mount the rootfs during development. 

Scented Death Star

My wife collects scented candles.  She doesn't actually use them; they just sit in my closet until I purge the house occasionally.  I had no expectation of this pattern changing, until recently, when I tried to think of a gift for my friend's 5-year old son.

On learning the boy's recent infatuation with Star Wars toys and noticing that the Death Star is missing from his collection, I looked for a decent scale model of the Death Star--preferably the half-constructed (but "fully operational")  The Return of the Jedi version as shown above--but I was shocked at the lameness of what I found on eBay.  The best looking model to be had for < $20 was just a rubber mold for a Death Star ice sphere (for Whiskey on Rocks Death Star as shown below), and I gave up the search.

A few days later, while reading the USB Complete, 4th Edition, I had an epiphany: why not melt the candles into the mold--and "kill 2 birds with 1 stone": get rid of my wife's stash of scented candles AND get a Death Star toy?  So I ordered the above "ice cube tray" from eBay, and waited a few weeks.  I don't understand how our Chinese brothers can make money selling something for like $3 including international shipping; but I wish we could on-shore value manufacturing for something like this back to the US.

Here's the 1st look at the mold on arrival.

The hemi-spheres are keyed, and the detail for the laser cannon firing dimple--the most important part of the model--seems acceptable.  Next, I grabbed one of the candles: Christmas Cookie scent--whatever--and melted it in a saucer pan filled with water, to separate the candle from the glass container.

After about 2 minutes, I could grab the wick to pull out the candle from the glass.

Then I cut out the candle into a roughly spherical shape, like so.

I closed up the mold, and small make-shift funnel on the pour hole indicated above.

I put the candle pieces and shavings back into the glass and reheated it in the saucer pan shown above, and then poured the melted wax into the funnel above, until the funnel overflowed.

It's a bit messy, but the wax comes off the surface with a utility knife.  To help wax better fill the nooks and crannies, I covered the pour hole, and tap it on the desk VIGOROUSLY while the wax is still in liquid state).  After 1 hour, open the mold.

There are few surface blemishes (hence the emphasis on vigorously for you), but a pretty good return for a $3 investment!

But I know are you thinking: "where's the laser cannon--the one that can destroy a planet in 1 shot?"

The laser cannon

I agree: a Death Star without the laser cannon is like fish & chips without the chips.  Fortunately, our Chinese manufacturers came through again: 5 mW laser pointer for $2 on eBay (batteries NOT included).

Do NOT shine the 5 mW laser pointer directly into the eye!

It's OD (outside diameter) is about 9/16", but the laser beam's width is only about 1/8".  I want to "fire" the laser from within the wax to diffract the laser beam, so I will use 2 different drill bits.

From the other end of the Death Star relative to the dimple, drill a hole wide enough to accommodate the laser pointer, but do NOT drill through (i.e. leave the dimple intact).  Now drill a 1/8" hole in the middle of the dimple, to let the laser light escape.

The Master Card moment

Insert the laser pointer into the larger hole, and shine THROUGH the 1/8" hole.  $3 for the mold, $2 for the laser pointer, $x for scented candle my wife got as a gift from someone.  The Death Star shining in your hand: priceless.  The laser beam actually lighting up a screen is extra credit.

If your Death Star looks cooler than this, please share a picture and your method.

USB HID on STM32F042

For most of my career, I've worked only on super expensive products; frankly I am tired of it.  Recently, I switched the industry and learned a lot about wireless/wearable consumer electronics at Jawbone.  I've been looking for a cheap USB-capable (phone to the uC) uC for a recently started hobby project, and decided to try the STM32L0 or STM32F0 line.  STM32L052xx have 64/32 KB flash and 8 KB SRAM, and available in 32-pin QFN package.  But STM32F042xx is even smaller, at 32/16 KB flash, 6 KB SRAM, and down to 20-pin (TSSOP) package.  I bought both the nucleo-l053r8 eval board and the nucleo-f042k6 eval board--rounding out my STM32 eval board collection (I now have an eval board for the STM32 L0, L1, L4, F0, F4, and F7 lines.  See my previous entries on running Linux on the STM32F4, and running the UBoot on STM32F7).

Update: after nearly 2 years of sitting on the problem, I found out why the device was not enumerating: it does enumerate if you run the release code (vs. debug).  But I still don't know the root cause.

Blinking the nucleo-32 user LED

Where ultra-low power consumption is not an overriding criteria, the F0 line is the more consumer/toy friendly option, so I begin my experiments with the nucleo-f042 board.
The big 3-color LED package right next to the micro-USB connector (at the top of the above picture) is the ST-Link indicator and is not accessible from the uC.  Of the 2 LEDs on the bottom of the above picture, the green LED (on the right) is the LD3 that the uC can control on the PB3 pin (pin D12 on the nucleo32 board), as you can see in the hardware block diagram.

To blink this LED from my GNU ARM Eclipse CDT environment (I could use the Keil uVision since the flash is only 32 KB--the max size supported by the free license of the Keil uVision--but I am more familiar with the Eclipse CDT environment), I created a new C project using the GNUARMEclipse plugin's STM32F0xx C/C++ project template, as shown below.

In the next screen, I chose the STM32F042 chip family, and the correct flash and RAM size (32 and 6, respectively).  The clock nominally an external clock source speed, but in this case it confusing, since this board does NOT have an external OSC, as you can see in the above block diagram.  Since I plan to use the 8 MHz internal (on-chip) RC clock source (multiplied 6x to 48 MHz) as you can see below, I entered 8000000 for the "Clock (Hz)" textbox.

Of the remaining options, I whack the "Use newlib nano", to make the most of the code space.

To debug the project from Eclipse through OpenOCD (you have to download/compile OpenOCD itself, since the Eclipse OpenOCD plugin does not seem to come with the executable itself), a few setups are required.  The Eclipse wide setup is available in Preferences --> Run/Debug --> OpenOCD.  Currently, it consists of just the executable name and path.  The debug configuration option for the project also needs to be told which OpenOCD config file to use for the board, as shown below.

After all this, the "blink" project does not blink at first, because the pin assignment is incorrect for the board.  In the auto-generated BlinkLed.h, I changed the port and pin to PB3, as you can see below.

// Port numbers: 0=A, 1=B, 2=C, 3=D, 4=E, 5=F, 6=G, ...
#define BLINK_PORT_NUMBER               (1)
#define BLINK_PIN_NUMBER                (3)

With the green LED now blinking away, I proceed convert this FW into a USB HID device.

Generating USB HID FW code in STM32CubeMx

The CubeMX GUI can assign pin functions and generate code stub for USB.  To get going, I just need to enable the USB FS 2.0 in the Pinout explorer.  I also want to test USB remote wakeup from a button, so I assigned PA0 to EXTI0, as you can see in the pinout view:

During development, a testpoint I can watch on the scope is super-helpful, so I assigned PA1 to GPIO output.  And I think the SWD semi-hosting is too slow, so I instead use UART2 to drive out SW tracing messages.  At 8 sample/bit oversampling, I can drive out 6 Mbaud from this peripheral, but my FT232R USB to serial translator chip can only go up to 3 Mbit/s anyway, so I constrain the peripheral to 3 Mbps, as shown below.

If you want to transmit an uint8_t type as a trace, you have to increase the word length to 8 bits (not counting parity).  If using the nucleo boards, the ST-Link VCOM port, using its VCOM port which shows up to the development Windows PC as a COMx device is a cleaner alternative--although I am not sure what maximum baud rate it supports.

I then generate a code through menu --> Project --> Generate code.  I have to copy (and replace if necessary; you can actually just create softlinks) some of these files to the blinky project I just ran in the last section, by dragging and dropping the items into the Eclipse project explorer.

• Inc/* --> include/
• Src/* --> src/.  Delete the existing Timer.c and Blink.c, since we will not be blinking the LED any more.
• Drivers/STM32F0xx_HAL_Driver/Inc/* (including the Legacy/ folder) --> system/include/stm32f0-stdperiph/.  Note that the HAL headers have "_hal_" in the name.
• Drivers/STM32F0xx_HAL_Driver/Src/* --> system/src/stm32f0-stdperiph/.  Delete the existing source files (those without "_hal_" in the name) since they are unnecessary now.
• Drivers/CMSIS/Device/ST/STM32F0xx/Include/*.h --> system/include/cmsis/
• Drivers\CMSIS\Device\ST\STM32F0xx\Source\Templates\system_stm32f0xx.c --> system/src/cmsis/
• Drivers\CMSIS\Device\ST\STM32F0xx\Source\Templates\gcc\startup_stm32f042x6.s --> system/src/cmsis/startup_stm32f042x6.S.  The extension change from ".s" to ".S" is required because Eclipse CDT expects assembly file extension to be ".S".
• Middlewares/ST/STM32_USB_Device_Library/Core/Inc/* --> include/
• Middlewares/ST/STM32_USB_Device_Library/Core/Src/* --> src/
• Middlewares/ST/STM32_USB_Device_Library/Class/CustomHID/Inc/* --> include/
• Middlewares/ST/STM32_USB_Device_Library/Class/CustomHID/Src/* --> src/

The USB middleware files are necessary only because I want to use the ST's USB middleware.  Initially, I tried to copy the folder Middlewares/ST/STM32_USB_Device_Library/ to the project root, but the GNUARMeclipse project template would not generate a recursive build rule for the STM32_USB_Device_Library folder I copied.  Figuring I will switch to a Makefile in the end, I just worked around by copying the sources individually to existing include/ and src/ folders.

The CPP requires the chip definition, so I added a new symbol "STM32F042x6" to  project properties --> C/C++ Build --> Settings --> Cross ARM C Compiler --> Preprocessor to ALL build configurations (Debug and Release).  With this change, the project builds again, and I can run the FW from the debugger.  But since I did not solder on USB connections, nothing interesting happens.  Soldering on a USB connector itself is easy enough; the most time consuming part are figuring out the D+/D- pins (3 and 2, which are in the middle of the USB standard type A connector; pin 1 is the Vbus, on the right hand side of the connector when you look straight at the cable) and shielding the D+/D- cables, for anything but the low speed USB devices.

I connected the D- cable to PA11 (nucleo board CN3.14) and the D+ cable to PA12 (CN3.5).  The Vbus cable is left unconnected until I will need to power the prototype from the Vbus.  Now I should be able to connect this FW to a host PC, but before I dive into the USB HID, let me insert SW tracing library into the project--because I expect things to NOT work the first time.

Marrying QPN with the STM32 HAL

The 32 KB flash and 6 KB SRAM leaves no room for high level SW such as an OS or a standard lib.  QPN is ideal for a moderately complex "multi-tasking" FW.  QPN has already been ported to CM0+ (on nucleo-l053r8), so let's start with a working example: $(QPN)/examples/arm-cm/dpp_nucleo-l053r8.  The example links against the nucleo-l053r8 board support files (I guess originally copied from the ST peripheral lib), but I am now generating all necessary files from the CubeMX, so I will just pick up the chip/board support from there.

I could not get CubeMX to change the code output folder (seems to be a bug), so I am living with the default output folder set when I created the CubeMX project.  I am going to build the project with a slightly modified version of the QPN DPP example Makefile.  The source and include paths should mirror the folders from which I copied manually in the first section above.

DRIVE := D:

QPN := $(DRIVE)/QP/qpn
# QP port used in this project
QP_PORT_DIR := $(QPN)/ports/arm-cm/qv/gnu

# I use CubeMX to generate the HAL and middleware files
CUBEMX_OUT := $(DRIVE)/uC/ST/play/nucleo-l042-hid
DEVICE := STM32F042x6
DEVICE_FAMILY := STM32F0xx
#Could not get this to work: DEVICE_FAMILY = $($(DEVICE):x%=xx)
HAL := $(CUBEMX_OUT)/Drivers/$(DEVICE_FAMILY)_HAL_Driver
CMSIS := $(CUBEMX_OUT)/Drivers/CMSIS

# list of all source directories used by this project
VPATH = $(QPN)/source $(QP_PORT_DIR) \
 $(CUBEMX_OUT)/Src $(HAL)/Src \
 $(CMSIS)/Device/ST/$(DEVICE_FAMILY)/Source/Templates \
 $(CUBEMX_OUT)/Middlewares/ST/STM32_USB_Device_Library/Core/Src \
 $(CUBEMX_OUT)/Middlewares/ST/STM32_USB_Device_Library/Class/CustomHID/Src

# list of all include directories needed by this project
INCLUDES  = -I. \
 -I$(QPN)/include -I$(QPN)/source -I$(QP_PORT_DIR) \
 -I$(CUBEMX_OUT)/Inc -I$(HAL)/Inc \
 -I$(CMSIS)/Include -I$(CMSIS)/Device/ST/$(DEVICE_FAMILY)/Include \
 -I$(CUBEMX_OUT)/Middlewares/ST/STM32_USB_Device_Library/Core/Inc \
 -I$(CUBEMX_OUT)/Middlewares/ST/STM32_USB_Device_Library/Class/CustomHID/Inc

C_SRCS := startup_stm32f042x6.c system_stm32f0xx.c \
 bsp.c main.c philo.c table.c \
 cubemx_main.c  stm32f0xx_it.c  usbd_conf.c  usbd_desc.c \
 usbd_core.c  usbd_ctlreq.c  usbd_ioreq.c \
 stm32f0xx_hal_msp.c  usb_device.c  usbd_custom_hid_if.c \
 usbd_customhid.c \
 stm32f0xx_hal.c stm32f0xx_hal_pcd.c stm32f0xx_hal_rtc.c \
 stm32f0xx_hal_cortex.c stm32f0xx_hal_pcd_ex.c stm32f0xx_hal_rtc_ex.c \
 stm32f0xx_hal_dma.c stm32f0xx_hal_pwr.c stm32f0xx_hal_tim.c \
 stm32f0xx_hal_flash.c stm32f0xx_hal_pwr_ex.c stm32f0xx_hal_tim_ex.c \
 stm32f0xx_hal_flash_ex.c  stm32f0xx_hal_rcc.c  stm32f0xx_hal_uart.c \
 stm32f0xx_hal_gpio.c  stm32f0xx_hal_rcc_ex.c  stm32f0xx_hal_uart_ex.c
 
QPN normally wants full control of the uC--running its tick update in the Systick_Handler, and handling the button press by directly checking the GPIO register.  But since I am trying to assess the benefit of the STM32 HAL, I changed the QPN bsp.c to use the HAL.

void HAL_SYSTICK_Callback(void) //HAL will call me back from Systick_Handler
{
    QF_tickXISR(0U); /* process time events for rate 0 */
}

void BSP_init(void) {
    BSP_randomSeed(1234U); /* seed the random number generator */
}

void BSP_displayPhilStat(uint8_t n, char const *stat) {
    if (stat[0] == 'h')
     HAL_GPIO_WritePin(GPIOA, GPIO_PIN_3, GPIO_PIN_SET);
    else
     HAL_GPIO_WritePin(GPIOA, GPIO_PIN_3, GPIO_PIN_RESET);
}

void QF_onStartup(void) {
    //CubeMX generates main(), which initializes every peripheral I configure
    //in CubeMX, but spins in a while loop.  I just rename the function and
    //delete the while(1) loop.
    extern void cubemx_main(); cubemx_main();
}

With these changes, the project builds cleanly, and generates the waveform for the philo[0] being hungry.

According to arm-none-eabi-size report, QPN and the DPP (dining philosopher) state machine application added less than 2 KB to the image size!

QP for QPN

QPN achieves its small footprint partly by leaving out
It does NOT support the QS tracing feature found in QPC, but I've ported it to run with QPN before, so I'll reuse my previous work here.  When using QS with QPN, the trickiest part is not colliding with the qpn header file.  I wrote the qs_port.h this way:

#define QS_TIME_SIZE     4
#define QS_OBJ_PTR_SIZE  4
#define QS_FUN_PTR_SIZE  4
#define Q_SIGNAL_SIZE 1
#define QF_EVENT_SIZ_SIZE 1

#ifndef QF_CRIT_ENTRY
#define QF_CRIT_ENTRY(dummy) QF_INT_DISABLE()
#endif
#ifndef QF_CRIT_EXIT
#define QF_CRIT_EXIT(dummy)  QF_INT_ENABLE()
#endif

#include <stdint.h>
#include "qpn.h"
typedef char char_t;
#include "qs.h"      /* QS platform-independent public interface */

Do NOT flush in QS_xxx_dict() function

There is a rather convoluted circular dependency between QS, QPN, and the ST HAL: normally, QS flushes the circular buffer after each dictionary item declaration.  Flushing requires the hardware setup, which happens in BSP_init(), which in this port relies on running the (modified) main() generated by CubeMx.  But the generated code enables the interrupts right away, so the Systick interrupt will trip in 1 ms (because the STM32 HAL hard-codes frequency to 1000 Hz) from roughly when I call BSP_init() in application main.  So if I put the flushing version of the QS dictionary declarations AFTER the HAL main(), there is a danger that the state machine initializations (happens inside QF_run) might not have completed by the time the Systick interrupt fires.  In a bare-metal ports of QPN, the interrupts enable is delayed until the state machines are initialized (in the QF_onStartup callback).  Since I want to leave the generated ST HAL code alone, I worked around this circular dependency by commenting out the flush at the end of the dictionary item declarations.

Now I am ready to instrument the CubeMX generated USB custom HID code.

STM32CubeMx generated custom HID stack

The USB processing seems to happen entire in the USB peripheral interrupt.

USB_IRQHandler(void)
--> HAL_PCD_IRQHandler(PCD_HandleTypeDef *hpcd)
    --> HAL_StatusTypeDef PCD_EP_ISR_Handler(PCD_HandleTypeDef *hpcd)
        --> HAL_PCD_DataOutStageCallback(PCD_HandleTypeDef *hpcd
                                       , uint8_t epnum)
            --> USBD_StatusTypeDef USBD_LL_DataOutStage(
                        USBD_HandleTypeDef *pdev, uint8_t epnum
                      , uint8_t *pdata)
                --> USBD_CUSTOM_HID.DataOut()
                    --> USBD_CUSTOM_HID_DataOut()
                        --> ((USBD_CUSTOM_HID_ItfTypeDef *)pdev->pUserData)
                                ->OutEvent(hhid->Report_buf[0]
                                         , hhid->Report_buf[1]);

The OutEvent function pointer is defined an also auto-generated interface:

typedef struct _USBD_CUSTOM_HID_Itf
{
  uint8_t                  *pReport;
  int8_t (* Init)          (void);
  int8_t (* DeInit)        (void);
  int8_t (* OutEvent)      (uint8_t, uint8_t );  
}USBD_CUSTOM_HID_ItfTypeDef;

The CubeMX generated OutEvent() does NOT do anything, as you can see here:

static int8_t CUSTOM_HID_OutEvent_FS  (uint8_t event_idx, uint8_t state)
{
  /* USER CODE BEGIN 6 */
  return (0);
  /* USER CODE END 6 */
}

The event_idx and state are merely the 1st 2 bytes of the HID report buffer, as you can see in how the lower level of the USB stack calls the OutEvent:

static uint8_t  USBD_CUSTOM_HID_DataOut (USBD_HandleTypeDef *pdev,
                              uint8_t epnum) {
  USBD_CUSTOM_HID_HandleTypeDef     *hhid = (USBD_CUSTOM_HID_HandleTypeDef*)pdev->pClassData; 
 
  ((USBD_CUSTOM_HID_ItfTypeDef *)pdev->pUserData)->OutEvent(
            hhid->Report_buf[0], hhid->Report_buf[1]);
   
  USBD_LL_PrepareReceive(pdev, CUSTOM_HID_EPOUT_ADDR , hhid->Report_buf,
                         USBD_CUSTOMHID_OUTREPORT_BUF_SIZE);
...

As it turns out, the Report_buf in the custom HID example is hard coded to only 2 bytes long in usbd_conf.h--short enough that I will overlook copying out the bytes from the USB register to the Report_buf  a byte at a time:

/*---------- -----------*/
#define USBD_MAX_NUM_INTERFACES     1
/*---------- -----------*/
#define USBD_MAX_NUM_CONFIGURATION     1
/*---------- -----------*/
#define USBD_MAX_STR_DESC_SIZ     512
/*---------- -----------*/
#define USBD_SUPPORT_USER_STRING     1
/*---------- -----------*/
#define USBD_DEBUG_LEVEL     0
/*---------- -----------*/
#define USBD_SELF_POWERED     1 // TODO change
/*---------- -----------*/
#define USBD_CUSTOMHID_OUTREPORT_BUF_SIZE     2
/*---------- -----------*/
#define USBD_CUSTOM_HID_REPORT_DESC_SIZE     2
/****************************************/
/* #define for FS and HS identification */
#define DEVICE_FS   0

* This device IS a FS device, so DEVICE_FS should be 1 above, but it does not seem to be used anyway.

At first glance, calling USBD_LL_PrepareReceive() AFTER the application has processed the Rerpot_buf seemed strange--until I saw the 1st invocation of USBD_LL_PrepareReceive() in USBD_CUSTOM_HID_Init(); PrepareReceive resets the EP's RX state, for asynchronous reception to happen in the future.

I also noticed that InEvent is missing in the interface, so USBD_CUSTOM_HID_DataOut() just returns without doing anything.  Clearly, if I want to an IN endpoint, I cannot use the auto-generated file verbatim.  But for now, let's put in some trace points into the generated code and watch the USB enumeration process.  To measure the interrupt latency through HAL, I assert the test point in the ISRs themselves (which are supplied in the stm32f0xx_it.c).  Auto-generated codes have designated user-code snippet areas that are preserved on code regeneration.  These are the places to put my testpoint codes:

/* USER CODE BEGIN 0 */
#include "bsp.h"
/* USER CODE END 0 */
...

void SysTick_Handler(void)
{
  /* USER CODE BEGIN SysTick_IRQn 0 */
  INT_TP(1);//measure on scope
  /* USER CODE END SysTick_IRQn 0 */
...

void USB_IRQHandler(void)
{
  /* USER CODE BEGIN USB_IRQn 0 */
  INT_TP(1);//measure on scope
  /* USER CODE END USB_IRQn 0 */
...

I will now sniff the USB packets in the Saleae Logic analyzer, with a copy of USB Complete, 4th Edition on one hand.  The book dedicates 3 separate CHAPTERS on HID, indicating how widely used the HID devices are.  Of particular importance is the HID report descriptor, which in this example is hard coded in usbd_custom_hid_if.c CUSTOM_HID_ReportDesc_FS:

__ALIGN_BEGIN static uint8_t CUSTOM_HID_ReportDesc_FS[USBD_CUSTOM_HID_REPORT_DESC_SIZE] __ALIGN_END =
{
  /* USER CODE BEGIN 0 */
  0x00,
  /* USER CODE END 0 */
  0xC0    /*     END_COLLECTION              */  
};

Reading about the END_COLLECTION tag, I realize that the above HID report never BEGAN a collection (application collection, to be specific, because all report items must be in an application collection--which begs the question of what the OTHER main item tags in a collection are for) with 0xA1, as demonstrated in USB Complete, 4th Edition Listing 11-2 of an HID descriptor.  Since this descriptor is technically not well-formed, I am not sure if the host application can actually use this custom HID device, and I enumeration seems to be failing on Windows as you can see in this screenshot of msinfo32:

I posted the question to ST forum, but meanwhile, let me try to work around this.

Switching to STM32F042F6P6

Just as I was about to attach the logic analyzer to the D+/D- pins of the STM32F042K6, the nucleo board suddenly stopped responding to the SWD.  I even tried to bypass the nucleo board's ST-Link and connect to the uC with an external ST-Link or J-Link, but the uC just would not respond.  Since I planned to migrate to the more production-like setup at some point, I ordered the smaller chip and an Aries TSSOP-20 adapter board from mouser, which lets me get at all 20 pins of the chip, as shown below.

In the above picture, I connect straight to the SWD pins from the Segger J-Link (SWD CLK and IO pins are pins 9 and 7 in the 20-pin JTAG connector on the J-Link), and I supply 3V from an external power supply.  But I later learned that it would be easier to buy a complete eval board that includes an LDO, and you always have to pay respects to people who not only create their own dev board, but also explain the design considerations.
Since I have a new uC, I created another CubeMX project for the new uC--and ran into another problem: CubeMX thinks STM32F042F6 does NOT have a USB and grays out the USB peripheral, as you can see below.

On looking at the CubeMX mcu template for the STM32F042Px, I found that the USB data lines are assigned only to PA11/12--which is distinctly missing in the above picture.  And the datasheet confirms that the USB pins are on PA11/12.  So I changed the PA9/10 pin assignment to PA11/12 in the SYS config, as you can see below, and magically, the USB peripheral becomes available again.

The existing project still builds fine for the new chip, because the device class (STM32F042x6) remains the same.  If I still use the UART2 for SW tracing and 2 GPIO for test points, I wind up using almost all available pins, as you can see below.

Note that I still want to leave the PB8 for the DFU (device firmware upgrade) feature--which I will explore later.  PF0/1 can potentially be used for additional GPIO since I do not use external oscillator (watch the BOM!).  I will write the code to handle the button press with PB1 GPIO_EXTI handler later.

Sniffing the USB HID enumeration in Logic Analyzer

Since I did not yet modify the CubeMX generated custom USB HID source--which includes the USB device description table in Src/usb_desc.c or the custom HID configuration descriptor in the usbd_custom_hid_if.c, this FW should enumerate without a problem as  VID=0x0483 and PID=0x5750, as hard coded in the beginning of the usbd_desc.c:

#define USBD_VID     1155
#define USBD_LANGID_STRING     1033
#define USBD_MANUFACTURER_STRING     "STMicroelectronics"
#define USBD_PID_FS     22352
#define USBD_PRODUCT_STRING_FS     "STM32 Custom Human interface"
#define USBD_SERIALNUMBER_STRING_FS     "00000000001A"
#define USBD_CONFIGURATION_STRING_FS     "Custom HID Config"
#define USBD_INTERFACE_STRING_FS     "Custom HID Interface"

These strings are arranged in an array that is later indexed by both the device and the host.

USBD_DescriptorsTypeDef FS_Desc =
{
  USBD_FS_DeviceDescriptor,
  USBD_FS_LangIDStrDescriptor,
  USBD_FS_ManufacturerStrDescriptor,
  USBD_FS_ProductStrDescriptor,
  USBD_FS_SerialStrDescriptor,
  USBD_FS_ConfigStrDescriptor,
  USBD_FS_InterfaceStrDescriptor,
};

When I connect the USB D+/D- pins to an old USB cable I sacrificed for the purpose and connect the "host" end (type A connector) to my laptop (burning out a USB 2.0 port in the trial-and-error stage; note to self for the future: ALWAYS use a cheapo hub to dork around with USB device!!), it tries to enumerate, as you can see in the properties discovered by the USBDeview program I downloaded to inspect all USB devices.

But I am curious what happened during enumeration, so in this section I will match up the Logic Analyzer's trace against my copy of USB Complete, 4th Edition.  You might want to skip to the next section if you don't care about how I teach myself the USB HID protocol.

I see the following packets on the wire:

1. Reset: 11 ms (> 10 ms required by the 2.0 spec) of D+/D- both pulled low.
2. 10 empty packets
1. SYNC
2. 8-bit PID.  PID[0:3] = ~PID[4:7] for error checking.  The 1st PID is SOF, which is followed by a 11-bit frame number (for FS).  Note that the first frame number is random.
3. CRC OK.  CRC is 16-bit for data, and 5-bit for address and EP
4. EOP: for FS, EP is D+ and D- both 0 for 2 bit widths.
3. SETUP packet.  Address = 0, endpoint = 0.  This is the step 8 in USB Complete, 4th Edition, Enumeration chapter.  8th byte of the device descriptor contains the maximum packet size supported by the EP0
4. DATA0
1. Direction: device -> host
2. Type: Standard
3. Recipient = Device
4. bRequest = 6 (GET_DESCRIPTOR)
5. wIndex = 0
6. wLength (11 bits) = 64
7. Data CRC (16-bits)
5. DATA1 (device --> host?).  The following information was hard coded in USBD_FS_DeviceDesc
1. bLength=18
2. bDescriptorType=1 (DEVICE)
3. bcdUSB = 0x200 (2.00)
4. bDeviceClass = 0 (deferred to interface descriptors)
5. bDeviceSubClass = 0
6. bDeviceProtocol = 0
7. bMaxPacketSize0 = 64.  Windows host requests 64 bytes but after receveiving just one packet
8. idVendor = 1155 (assigned to STM)
9. idProduct = 22352
10. bcdDevice = 0x200 (2.00)
11. iManufacturer = 1.
12. iProduct = 2
13. iSerialNumber = 3
14. bNumConfiguration = 1
6. ACK
7. IN, addr = 0, EP = 0
1. device NACKs this IN
2. Approximately 62 us later, host retries.
3. Device replies back with the same data as in DATA1 in step 5 above.
8. OUT, addr = 0, EP = 0.  There should be a data that follows an OUT packet, but the only data I see is an empty DATA1, followed by an ACK from the device.  Maybe this is just how the host and the device exchange ACK after an IN transaction?
9. The hub resets the device.  Reset lasted 11 ms.
10. 16 SOFs, with 1 ms interval go unanswered.
11. SETUP A=0, E=0
1. Followed by DATA0 (requestType = 0, Request=5, SET_ADDRESS=4)
2. And then an ACK
12. It's bizarre why the host would follow up immediately with an IN packet for A=0, E=0, and 10 more SOFs go by.  Maybe this is the "step 7 in the enumeration chapter of the USB Complete.
13. Another SETUP, this time A=4, E=0, followed by DATA = GET_DESCRIPTOR (wLength = 18).
1. The device responds with the same information as in step 5 above.
14. Host sends another GET_DESCRIPTOR, but this time for CONFIGURATION[0].
15. Host fires IN request, which the device NAKs.  Host retries after 45 us, and this time, the device returns the information hard coded in the auto-generated usbd_customhid.c USBD_CUSTOM_HID_CfgDesc:
1. the max power for  the configuration = 100 mA
2. Descriptor type = 0x04 (interface), interface number = 0, alternate setting = 0, interface class = 3 (HID), subclass = 0 (none), protocol = 0, iInterface = 0, bLength=9
3. DescriptorType = 0x21 (HID).   HID version = 0x0111 (1.11), country = not supported, bNumDescriptor = 1, HID report descriptor.
4. 1 IN interrupt report, and 1 OUT interrupt report. wMaxPacketsize = 2 bytes (I don't know why we use different defines than the USBD_CUSTOM_HID_REPORT_DESC_SIZE seen earlier; keeping track of yet another 2 defines CUSTOM_HID_EPOUT_SIZE and CUSTOM_HID_EPIN_SIZE just seems like a chore).  Polling interval both 20 ms.
16. OUT (A=4, EP=0), empty DATA
17. SETUP, GET_DESCRIPTOR for STRING descriptor, index 3, language = English
18. IN: answered by NAK
19. 80 us later, SOF, IN, with DATA this time, for the string descriptor at index (3+1) in the FS_Desc string array shown above.
20. Why this is followed immediately by an unnecessary OUT with empty DATA, I don't understand.
21. SETUP/GET_DESCRIPTOR for the string at string array index 0.  This is answered with IN, wLANGID=1033, which is at array index 1:SETUP/GET_DESCRIPTOR for string array index 2, wIndex=1033 (language = English).
22. SETUP/GET_DESCRIPTOR for string index 2, which is answered by the string at the array index 3 above (product)
23. SETUP-GET_DESCRIPTOR for DEVICE_QUALIFIER[0], which is answered by a STALL, which means the device does not support this request.
24. After 16 empty SOF/ACK exchange, the host seems to recover, and sends SETUP-GET_DESCRIPTOR for DEVICE[0].  The device answers with the same information as in step 5 above.
25. SETUP-GET_DESCRIPTOR for CONFIURATION[0].  As with all other GET_DESCRIPTOR, the host initiates IN transaction almost right away (14 us), and the device NAKs the first IN.  The content of the CONFIGURATION[0] is the same as in step 14 above.
26. CONFIGURATION[0] IN transaction is repeated AGAIN right away--why??
27. SETUP-SET_CONFIGURATION(0), which the device ACKs right away.  Note that this device only has 1 configuration
28. IN is NAKed, and then answered with an empty DATA
29. SETUP-HID SET_IDLE(duration = indefinite, report ID = 0).  This is to save BW by limiting the reporting frequency of an interrupt IN EP when the data hasn't changed since the last report.
30. IN-empty data
31. SETUP-GET_DESCRIPTOR[0] is answered by "Unknown main item (0); end collection".  The report descriptor is hard-coded in CUSTOM_HID_ReportDesc_FS discussed earlier.

Maybe the "end collection" tells the Windows host that there is no more to be learned, so it can loads the device driver.  But an ill-formed HID report descriptor (the 2 bytes shown in the previous section) prevents successful enumeration.

My own HID report descriptor

So I tried designing my own HID report descriptor using the USB-IF's "HID DT (descriptor tool)", like this:

__ALIGN_BEGIN static uint8_t CUSTOM_HID_ReportDesc_FS[USBD_CUSTOM_HID_REPORT_DESC_SIZE] __ALIGN_END =
{
  /* USER CODE BEGIN 0 */
  0x06, 0x00, 0xFF,//USAGE_PAGE (Vendor Defined Page 1)       // 3 B
  0x09, 0x00,      //USAGE (Undefined)                        // 5 B
  0xA1, 0x01,      //COLLECTION (Application)                 // 7 B
  0x75, 0x08,      //  REPORT_SIZE (8)                        // 9 B
  0x95, 0x01,      //  REPORT_COUNT (1)                       // 11B
  0x92, 0xA3, 0x01,//  OUTPUT (Cnst, Var, Abs, NPrf, Vol, Buf)// 14B
  /* USER CODE END 0 */
  0xC0    /*     END_COLLECTION              */
}

CubeMX unfortunately does not bracket the definition of USBD_CUSTOM_HID_REPORT_DESC_SIZE in "USER CODE" markers so that I have to keep changing the USBD_CUSTOM_HID_REPORT_DESC_SIZE every time I generate the code.  With this descriptor, the HID device enumerates, as you can see in the Device Manager --> HID view:

Let's see if I can talk to the device from the development PC.  But first, a little detour about USB suspend, because I noticed that the USB port gets powered off immediately after enumeration--I think that's how it is supposed to be.  Strangely, the keep alive messages (SYNC) do not cease on the wire?!

Bus powered device

In usbd_conf.h, the CubeMX hard codes the self powered setting, just like it hard codes the custom HID report description size.

#define USBD_SELF_POWERED     1
#define USBD_CUSTOM_HID_REPORT_DESC_SIZE 2 //should be 15; see above

Both hard-coded settings are inappropriate for me, and the fact that I cannot override them is annoying.  Nevertheless, let's fix them both

USB SUSPEND

After the initial enumeration, there is no activity on the bus, so the device must enter SUSPEND state (after 3 ms without a SYNC).  How can I tell whether the ST USB middleware is entering SUSPEND?  When I search for "SUSPEND" in the CubeMX generated code, I found a function USB_LL_Suspend() that informs the USB library that the core is entering the suspend mode.  This function is called right before the STM32 is put into stop mode, as you can see here:

void HAL_PCD_SuspendCallback(PCD_HandleTypeDef *hpcd)
{
  /* Inform USB library that core enters in suspend Mode */
  USBD_LL_Suspend(hpcd->pData);
  /*Enter in STOP mode */
  /* USER CODE BEGIN 2 */
  if (hpcd->Init.low_power_enable)
  {
    /* Set SLEEPDEEP bit and SleepOnExit of Cortex System Control Register */
    SCB->SCR |= (uint32_t)((uint32_t)(SCB_SCR_SLEEPDEEP_Msk | SCB_SCR_SLEEPONEXIT_Msk));
  }
  /* USER CODE END 2 */
}

Unfortunately, CubeMX hard codes low_power_enable to 0 in USBD_LL_Init() as you can see here:

  hpcd_USB_FS.Init.low_power_enable = DISABLE;

So I will have to  a modified version of usbd_conf.c USBD_LL_Init() if I want to put the uC to deep sleep, but note that the CubeMX code does NOT turn off the power domains before going to sleep, as shown in the HAL_PWR_EnterSTOPMode(PWR_LOWPOWERREGULATOR_ON, PWR_STOPENTRY_WFI) I currently use:

  tmpreg = PWR->CR;
 
  /* Clear PDDS and LPDS bits */
  tmpreg &= (uint32_t)~(PWR_CR_PDDS | PWR_CR_LPDS);

  /* Set LPDS bit according to Regulator value */
  tmpreg |= Regulator;

But I have a suspicion that the suspend callback is not even getting called.

Windows app to test drive the bus-powered custom HID device

In USB Complete, 4th Edition, I did not understand why Jan Alexson talked about the managed and unmanaged functions--until I realized that MSFT does not offer .NET Windows USB API.  Of course, someone would have already made a .NET library to access the Windows HID API and I can just use it.  I unzipped csharp-usb-hid-driver (GPL3 license), to find the project USBHIDDRIVER in its own folder.  I created my own C# console application project (named WinCustomHID, which also creates a solution WinCustomHID.sln) and added USBHIDDRIVER as another project in my own solution (I was forced to upgrade the USBHIDDRIVER project).  My own project will need a reference to the USBHIDDRIVER project (right click on the project --> Add --> Reference --> Projects --> USBHIDDRIVER).

The usage is simple: create an interface to the device, connect, and start writing, as shown in this example:

using USBHIDDRIVER;
namespace WinCustomHID
{
    class Program
    {
        static void Main(string[] args)
        {
            const string vid = "vid_0483", pid = "pid_5750";
            USBInterface usb = new USBInterface(vid, pid);
            if (!usb.Connect()) {
                Console.WriteLine("Could not connect to {}/{}", vid, pid);
            }
            byte[] command = { 0x42 };
            usb.write(command);
            Console.WriteLine("Done");
            // Have to comment out the usbThread in the lib to avoid crash!
            usb.Disconnect();
        }
    }
}

The documentation says that write() can handle up to 64 bytes, but I found that strange, given that my HID report descriptor explicitly stated that it can only handle 1 byte.  When I browsed the code, I found that "64 byte" was hard-coded--maybe because that is the maximum size of a HID report.

Writing to a custom HID on Android

The device, running the SAME FW, is NOT enumerating on Android.  But MCP2210 does enumerate to Android, so the problem must be in either my device or the FW...

Waking up the host

In USB Complete, 4th Edition, the only clue on how to remote wakeup the host from a device I could find was in Resuming Communications section:

The device ... indicates ... in ... bmAttributes.  The host enables remote wakeup by sending a Set Port Feature(DEVICE_REMOTE_WAKEUP) request to the hub port that is the device's link partner.  A suspended device with remote wakeup enabled can request to resume communications by driving the upstream bus in the Resume state for 1-15 ms.

To "drive the upstream bus" (HW action), I need to follow the  STM32F0 reference manual's USB chapter:

Resume sequence can be started by setting the RESUME bit in the USB_CNTR register to ‘1 and resetting it to 0 after an interval between 1ms and 15ms (this interval can be timed using ESOF interrupts, occurring with a 1ms period when the system clock is running at nominal frequency).

The FW will need to set the USB_CNTR_RESUME bit in hpcd->Instance->CNTR = wInterrupt_Mask after waking up from EXTI.