WARNING: WIP; incomplete
Modern audio devices have a lot of intellectual properties in them. I want to investigate creating an algorithmic device (an embedded device that runs non-trivial algorithms) that will interact with a mobile phone. I discovered that creating a Lightning device is onerous than making a USB device to connect to an Android phone, so that is the route I will take in this blog entry.The big micro-controller vendors like ST all have USB capable product lines, and provide some examples and library to ease the steep learning curve for USB. Since joining Jawbone, I have been learning about STM32 uCs, so I will stick to the ARM Cortex M4 line of STM32 offerings. I believe that all new algorithmic projects should start with a floating point capable platform. I especially like the STM32L4 because it is still a low-power uC. I downloaded the STM32Cube FW examples and SDK for the L4 line, and found a few USB device application FW examples for the STM32L476G eval board--such as the HID_Standalone/ project.
I already have a nucleo-l476rg board with STM32L476RG uC, which supports USB FS (full-speed: 12 Mbps). I propose to turn this board into a USB stereo input and output audio device. The USB FS bandwidth limitation limits the sample rate to 48 kHz (as a comparison, CD rate is 44.1 kHz, and high quality audio is 96 kHz or 192 kHz). The USB FS bandwidth is considerably greater than low power wireless radio bandwidth (e.g. ~1 Mbps for BLE, and 596 Kbps for NFMI).
HW to turn nucleo-l476 eval board into a USB device
Unlike its more expensive cousin STM32L476G-eval board, the nucleo board does not have a USB socket, or the nice EMI/ESD filter, which is shunted on the DP/DM the STM32L476G-eval board. But at least all the USB functions are exposed, as shown below.
- D+ and D- are on PA12 and PA11.
- When in device mode, the VBUS (PA9) can be connected to the Vbus (5 V) to detect the Vbus (with an on-chip voltage comparator), or a battery charge detector (how?). This may be useful for a battery powered device that is USB charged, but I am only trying to make a USB device that is entirely powered by (the mobile phone's) USB. PA9 will be deasserted when the transceiver is active, so one might be able to drive some indication of the USB activity.
- The uC can assert SOF pin (PA8) at the start of a frame--for synchronizing with an external chip (in a high performance application such as audio).
- U5V pin is the 5 V USB bus voltage that supplies the power to the whole board when it is connected to the USB host, as you can see in the schematic of the STM32L476G-eval board, showing a direct connection from the USB bus VCC to the U5V pin.
VDDUSB = 3.0 to 3.6 V: external independent power supply for USB transceivers. The VDDUSB voltage level is independent from the VDD voltage. After reset, the USB features supplied by VDDUSB are logically and electrically isolated and therefore are not available. The isolation must be removed before using the USB OTG peripheral, by setting the USV bit in the PWR_CR2 register, once the VDDUSB supply is present.To repeat, VDDUSB is a dedicated pin on the uC, expected to be ~3.3V (NOT the 5V VBUS straight from the USB port!). On the Nucleo64 boards (Nucleo boards for the LQFN64 packages), VDDUSD comes shorted to VDD over the solder bridge SB31, as shown below
BUT IF I want to power the device from USB, I have to generate VDD from VBUS, as Synopsis recommends in the STM32L4 reference manual, USB on-the-go full-speed (OTG_FS) chapter.
USB audio example generated by STM32CubeMX
ST's CubeMX is a graphical code generator for the ST uC. In it, I assigned pin functions and auto-configured the clocks for the desired feature sets.
- UART2
- Word length should be 8, but set to 7 currently.
- Baud rate was NOT selectable in CubeMX, as you can see here
- USB
- Endpoint 0 max packet size: 64 B, but CubeMX didn't offer this option, as you can see here.
- VBUS sensing: I had it disabled (see above), but report says "Enabled"
- "Enable internal IP DMA" turned off, but CubeMX didn't offer this option
- Low power, link power management: disabled
- USB audio device class middleware
- USBD_LPM_ENABLED: true; conflicts with above choice for the device
- USBD_AUDIO_FREQ:: 22100. I need it to be at least 44100 Hz!
- VID will be 1155
- PID will be 22336
- PRODUCT_STRING will be STM32_Audio_Class
- CONFIGURATION_STRING: AUDIO Config
- INTERFACE_STRING: AUDIO Interface
- NVIC
- Interrupt for USART2 disabled
- USB OTG FS global interrupt same priority as the systick, because I left the priorities unassigned in Configuration --> NVIC, as you can see here.
Importing the generated source into the GNUARMeclipse "Hello world" plugin
The example uses the ST HAL library, which may be more "help" than I would typically want, but I'll just follow along for now. The example comes in several project flavors: IAR EWARM, Keil uVision, Atollic TrueStudio, and SW4STM3. I have a free copy of uVision, and the project builds cleanly; furthermore, since the .text + .data is only about 20 KB (< 32 KB), I will be able to debug it on the target! The only problem is that uVision's code browse feature is horrible (when compared to Eclipse CDT). Since I prefer Eclipse CDT and gnu toolchain, I decided to create a new gnuarm Eclipse C project in another folder and just import the generated files. Since gnuarmeclipse plugin does NOT support STM32L4, I fell back to the Cortex-M project template, as shown below:
MEMORY
{
FLASH (rx) : ORIGIN = 0x08000000, LENGTH = 1024K
RAM (xrw) : ORIGIN = 0x20000000, LENGTH = 96K
So far, the Eclipse project is missing the files specific to the pins I selected in CubeMX. To overlay the CubeMX generated project on top of my Eclipse project, headers from the <CubeMX gen>/Inc/ folder to the include/ folder. When drag-and-dropping into the Eclipse Project Explorer, I got an option to link the files instead of copying, as you can see here.
- Inc/* --> include/
- Src/* --> src/
- Drivers/STM32L4xx_HAL_Driver/Inc/* --> system/include/stm32l4xx/
- Drivers/STM32L4xx_HAL_Driver/Src/* --> system/src/stm32l4xx/
- Drivers/CMSIS/Device/ST/STM32L4xx/Include/* --> system/include/cmsis/. Even though there are identically named files in there, I choose to use the CMSIS files that ST has tested against.
- Drivers/CMSIS/Device/ST/STM32L4xx/Source/Templates/system_stm32l4xx.c --> system/src/cmsis/
- Drivers/CMSIS/Device/ST/STM32L4xx/Source/Templates/gcc/startup_stm32l4xx.s --> system/src/cmsis/startup_stm32l4xx.S. The extension change is required because Eclipse CDT expects assembly file extension to be ".S".
- Middlewares/ST/STM32_USB_Device_Library/Core/Inc/* --> include
- Middlewares/ST/STM32_USB_Device_Library/Core/Src/* --> src
- Middlewares/ST/STM32_USB_Device_Library/Class/AUDIO/Inc/* --> include
- Middlewares/ST/STM32_USB_Device_Library/Class/AUDIO/Src/* --> src
The project needs to define the chip definition to the generated source, in the project properties --> C/C++ Build --> Settings, as you can see below. I supplied STM32L476xx for the gas and gcc preprocessors (for both the Debug and Release configurations), but didn't do it for the C++ preprocessor (because I don't use C++ in this project!).
Understanding the generated USB example; first pass: browsing the source
Right after HAL_Init() and SystemClock_Config(), the generated main() calls MX_USB_DEVICE_INIT().- USBD_Init(handle, &descriptor, DEVICE_FS)
- USBD_RegisterClass(handle, &USBD_AUDIO)
- USBD_AUDIO_RegisterInterface(handle, USBD_AUDIO_fileops)
- if BCD (battery charging detection) is NOT used, start the device(handle)
After the init, the FW just spins in a busy loop, so EVERYTHING else must happen in interrupts in the vector table--also generated by the CubeMX (startup_stm32l476xx.s I copied above). I found the OTG_FS_IRQHandler in the vector table; the IRQ number is 67 (OTG_FS_IRQn in IRQn_Type in stm32l476xx.h). CubeMX generated the ISR in stm32l4xx_it.c. In the project thus far, only 2 interrupts are necessary: SysTick (used by the HAL) and USB. The USB ISR simply dispatches into the HAL handler, as you can see here:
void OTG_FS_IRQHandler(void)
{
/* USER CODE BEGIN OTG_FS_IRQn 0 */
{
/* USER CODE BEGIN OTG_FS_IRQn 0 */
/* USER CODE END OTG_FS_IRQn 0 */
HAL_PCD_IRQHandler(&hpcd_USB_OTG_FS);
/* USER CODE BEGIN OTG_FS_IRQn 1 */
HAL_PCD_IRQHandler(&hpcd_USB_OTG_FS);
/* USER CODE BEGIN OTG_FS_IRQn 1 */
/* USER CODE END OTG_FS_IRQn 1 */
}
}
I will be on a lookout for where HAL_NVIC_EnableIRQ(OTG_FS_IRQn) is called in the generated code. Also, in the STM32L4x6 reference manual, I saw that EXTI (used to wake up the uC) line 17 is connected to the OTG_FS wakeup event. The different OTG interrupts are listed in the OTG_GINTMSK.
- In device mode, there is an "all endpoint" interrupt mask register (OTG_DAINTMSK)
- But each endpoints have a dedicated interrupt mask register (OTG_DIEPMSK for IN and OTG_DOEPMSK).
- There is also a FIFO empty interrupt (OTG_DIEPEMPMSK)
- In host mode, there is an "all channels" interrupt mask register (OTG_HAINTMSK) and per-channel interrupt mask (OTG_HCINTMSKx)
I will come back to interrupt handling after looking at the init code.
USBD_Handle
As you can see, the USB device handle is the "object" responsible for all USB device operations. It keeps many USB details and up to 15 endpoints in addition to the EP0, which is the control endpoint required by the USB spec.
USBD_Init(handle, &FS_Desc, DEVICE_FS)
The USB descriptors passed to USBD_Init is actually a virtual interface to not just the device descriptor, but also many other descriptors, including:- language ID
- manufacturer string
- product string
- serial string
- config string
- interface string
- user BOS (?)
#define USBD_MAX_NUM_INTERFACES 1
#define USBD_MAX_NUM_CONFIGURATION 1
#define USBD_MAX_STR_DESC_SIZ 512
#define USBD_SUPPORT_USER_STRING 0
#define USBD_DEBUG_LEVEL 0
#define USBD_LPM_ENABLED 1
#define USBD_SELF_POWERED 1
#define USBD_AUDIO_FREQ 22100
#define DEVICE_FS 0 /* #define for FS and HS identification */
USBD_Init() calls USBD_LL_Init(), which (strangely) configures some PCD (peripheral controller something?) properties contrarily to the above auto-generated code.
hpcd_USB_OTG_FS.Init.dev_endpoints = 7; //Q: why 7?
hpcd_USB_OTG_FS.Init.speed = PCD_SPEED_FULL;
hpcd_USB_OTG_FS.Init.ep0_mps = DEP0CTL_MPS_64;
hpcd_USB_OTG_FS.Init.phy_itface = PCD_PHY_EMBEDDED;
hpcd_USB_OTG_FS.Init.Sof_enable = DISABLE;
hpcd_USB_OTG_FS.Init.low_power_enable = DISABLE;
hpcd_USB_OTG_FS.Init.lpm_enable = DISABLE;
hpcd_USB_OTG_FS.Init.battery_charging_enable = DISABLE;
hpcd_USB_OTG_FS.Init.vbus_sensing_enable = ENABLE;
Recall that this structure is the same one that is passed to the PCD interrupt handler from the OTG ISR. It is also passed into HAL_PCD_Init(), which:
- HAL_PCD_MspInit() if PCD state is RESET
- Configures PA11/12 for alternate function (see the board pinout at the very beginning of the entry)
- __USB_OTG_FS_CLK_ENABLE(). Recall that the 48 MHz USB clock is isolated from the rest of the system, and can even be generated from the 32 KHz oscillator.
- HAL_PWREx_EnableVddUSB(), which just sets the PWR_CR2.USV bit to power up the USB transceivers according to the STM32L4x6 reference manual, Power control (PWR) chapter, Independent USB transceivers supply.
- Enable OTG_FS_IRQn (priority specified in CubeMX)
- __HAL_PCD_DISABLE(): disable all USB interrupts (Q: what reenables it?)
- USB_CoreInit()
- GUSBCFG |= USB_OTG_GUSBCFG_PHYSEL unnecessary since ref man says this bit is always 1?
- USB_CoreReset(): required after PHY select
- GCCFG = USB_OTG_GCCFG_PWRDWN: release transceiver power down
- GUSBCFG |= USB_OTG_GUSBCFG_SRPCAP: enable allow SRP
- USB_SetCurrentMode(..., USB_OTG_DEVICE_MODE)
- GUSBCFG |= USB_OTG_GUSBCFG_FDMOD: force device mode (ignore OTG ID pin)
- Delay 50 ms
- Initializes IN and OUT endpoints (to control type)
- USB_DevInit()
- GCCFG |= USB_OTG_GCCFG_VBDEN: Vbus detect enable
- USBx_DEVICE->DCFG |= DCFG_FRAME_INTERVAL_80: interrupt at 80 % of the frame interval (e.g. to check if all isochronous traffic for that frame is complete)
- Set PHY to full speed (vs. low speed)
- Flush all FIFOs (TX and RX)
- Reset all IN and OUT endpoint registers
- Clear all USB device interrupts
- DIEPMSK &= ~(USB_OTG_DIEPMSK_TXFURM)
- Configure DMA threshold
- Disable and clear all interrupts
- Enable interrupts for device mode only:
- USBSUSPM, USBRST, ENUMDNEM, IEPINT, OEPINT, PXFRM_IISOIXFRM, PXFRM_IISOOXFRM, WUIM
- SOFM if SOF enabled
- SRQIM | OTGINT if Vbus sensing enabled
- PCD state = READY
- HAL_PCDEx_ActiveLPM() if LPM enabled
- HAL_PCDEx_ActivateBCD(0 if battery charging enabled
- USB_DevDisconnect(): asserts DCTL.SDIS bin to "soft disconnect" the USB
HAL_PCD_SetRxFiFo(&hpcd_USB_OTG_FS, 0x80);
HAL_PCD_SetTxFiFo(&hpcd_USB_OTG_FS, 0, 0x40);
HAL_PCD_SetTxFiFo(&hpcd_USB_OTG_FS, 1, 0x80);
While each TX endpoints get its own FIFO, there is only 1 RX FIFO--which makes sense since we have to demux the incoming packet.
USBD_RegisterClass(handle, &USB_AUDIO)
As a protocol defined class, the audio class has the following implementations to respond to the USB stack:
USBD_ClassTypeDef USBD_AUDIO =
{
USBD_AUDIO_Init,
USBD_AUDIO_DeInit,
USBD_AUDIO_Setup,
USBD_AUDIO_EP0_TxReady,
USBD_AUDIO_EP0_RxReady,
USBD_AUDIO_DataIn,
USBD_AUDIO_DataOut,
USBD_AUDIO_SOF,
USBD_AUDIO_IsoINIncomplete,
USBD_AUDIO_IsoOutIncomplete,
USBD_AUDIO_GetCfgDesc,
USBD_AUDIO_GetCfgDesc,
USBD_AUDIO_GetCfgDesc,
USBD_AUDIO_GetDeviceQualifierDesc,
};
{
USBD_AUDIO_Init,
USBD_AUDIO_DeInit,
USBD_AUDIO_Setup,
USBD_AUDIO_EP0_TxReady,
USBD_AUDIO_EP0_RxReady,
USBD_AUDIO_DataIn,
USBD_AUDIO_DataOut,
USBD_AUDIO_SOF,
USBD_AUDIO_IsoINIncomplete,
USBD_AUDIO_IsoOutIncomplete,
USBD_AUDIO_GetCfgDesc,
USBD_AUDIO_GetCfgDesc,
USBD_AUDIO_GetCfgDesc,
USBD_AUDIO_GetDeviceQualifierDesc,
};
USBD_AUDIO_RegisterInterface(handle, &USBD_AUDIO_fops_FS)
The USB spec requires only the control endpoint for all audio class devices, and the USBD_AUDIO_fops_FS supplies that implementation:USBD_AUDIO_ItfTypeDef USBD_AUDIO_fops_FS =
{
AUDIO_Init_FS,
AUDIO_DeInit_FS,
AUDIO_AudioCmd_FS,
AUDIO_VolumeCtl_FS,
AUDIO_MuteCtl_FS,
AUDIO_PeriodicTC_FS,
AUDIO_GetState_FS,
};
The USB device keeps the pointer to the above structure in its pUserData field.
USBD_Start(handle)
Starting the USB device consists of enabling the pull-up/down by deasserting the OTG_DCTRL.SDIS (soft disconnect) bit, and THEN waiting for 3 ms (with HAL_Delay--which assumes 1000 Hz systick rate).Since I disabled battery charging (from USB 5V) for this device, MX_USB_DEVICE_Init() will call USBD_Start().
HAL_PCD_IRQHandler(&hpcd_USB_OTG_FS)
Apparently, it is possible for OTG HW to fire an interrupt that I did not enable (race condition in disabling interrupt, or protecting against mistake in the higher layer when disabling interrupt with interrupt already pended?). So the handler checks the current interrupt status against the enabled interrupt mask in __HAL_PCD_IS_INVALID_INTERRUPT(hpcd).The OTG can only be in host or device mode, and if there is a mixup, the MMIS (mode mismatch) interrupt will fire.
if(__HAL_PCD_GET_FLAG(hpcd, USB_OTG_GINTSTS_MMIS))
__HAL_PCD_CLEAR_FLAG(hpcd, USB_OTG_GINTSTS_MMIS);
Hiding this problem from the application and simply quenching the interrupt looks problematic...
A logical way to study this code might be the USBRST interrupt.
If the USB_OTG_GINTSTS_OEPINT bit is set, then the upper 16-bits (bottom 16-bit are for the IN endpoints) of USBx_DEVICE->DAINT (all endpoint interrupt register) is read. EP0 is treated specially (because it is a control endpoint), but the interrupt status is still at the OTG_DOEPINTx register.
If the programmed transfer is complete, the USB_OTG_DOEPINT_XFRC bit is set. To clear a bit, simply write '1' for that bit (other bits are apparently unaffected). The 0x8000 bit in the DOEPINTx register apparently means "packet received", If so, and DMA is enabled, the EP's xfer_count and xfer_buff is incremented. On setup phase done interrupt (STUP), the handler calls HAL_PCD_SetupStageCallback()--which the application should implement if it wants to handle the setup complete event. Strangely, the CubeMX generated handler only handles EP0 OUT interrupt. The IN endpoint handling is equally mysterious.
Trying out the example
TODOAndroid app to interact with the USB audio device
For Android Studio to control my phone as a development target, adb is necessary. Running adb over USB is straight-forward if USB debugging is enabled on the phone. But to write an application that uses USB, I would have to run adb over TCP--first by connecting to the phone over USB and running the adb command like this example:C:\Users\Henry\AppData\Local\Android\sdk\platform-tools\adb tcpip 5555
* daemon not running. starting it now on port 5037 *
* daemon started successfully *
restarting in TCP mode port: 5555
THEN with the phone disconnected, connect over TCP:
C:\Users\Henry\AppData\Local\Android\sdk\platform-tools\adb connect 192.168.43.1
connected to 192.168.43.1:5555
The phone's address is variable of course. Above address is my phone's IP with the Mobile Hotspot enabled.
Appendix: Play around with the Plantronics headset
I had to look for my headset. I used to use USBDeview to view all enumerated USB devices, but that doesn't seem to work on Windows 10 any more, so I just sucked it up and installed Windows SDK--just to pick up the Windows Debugging Tools package containing usbview.exe. I found the program in this folder after the install: C:\Program Files (x86)\Windows Kits\10\Debuggers\x64.This is the device information from usbview:
bLength: 0x12
bDescriptorType: 0x01
bcdUSB: 0x0200
bDeviceClass: 0x00 -> This is an Interface Class Defined Device
bDeviceSubClass: 0x00
bDeviceProtocol: 0x00
bMaxPacketSize0: 0x40 = (64) Bytes
idVendor: 0x047F = Plantronics, Inc.
idProduct: 0xC008
bcdDevice: 0x0132
iManufacturer: 0x01
English (United States) "Plantronics"
iProduct: 0x02
English (United States) "Plantronics .Audio 655 DSP"
iSerialNumber: 0x00
bNumConfigurations: 0x01
When I connected the headset to my phone (via an adapter), Spotify app played my playlist on it right away.
Appendix: Android documentation
The methods, parameter types and units exposed by the Equalizer implementation are directly mapping those defined by the OpenSL ES 1.0.1 Specification (http://www.khronos.org/opensles/) for the SLEqualizerItf interface. Please refer to this specification for more details. To attach the Equalizer to a particular AudioTrack or MediaPlayer, specify the audio session ID of this AudioTrack or MediaPlayer when constructing the Equalizer.NOTE: attaching an Equalizer to the global audio output mix by use of session 0 is deprecated.
Creating a Visualizer on the output mix (audio session 0) requires permission
MODIFY_AUDIO_SETTINGSYou can play back the audio data only to the standard output device. Currently, that is the mobile device speaker or a Bluetooth headset. You cannot play sound files in the conversation audio during a call.
You can query the
AudioManager to determine if the audio is currently being routed to the device speaker, wired headset, or attached Bluetooth device.A2DP (advanced audio distribution profile), AKA Bluetooth Audio Streaming
An Equalizer is used to alter the frequency response of a particular music source or of the main output mix.
Android audio terminology page.
AudioEffect is the base class for controlling audio effects provided by the android audio framework. Applications should not use the AudioEffect class directly but one of its derived classes to control specific effects:
Android USB host stack is just the mainline Linux kernel USB host stack--but only as a USB OTG host (the same as USB embedded host?), which supports only the TPL (targeted peripheral list). The stack is different.
libusbhost is a thinner version of the Linux libusb adopted to Android.
Intent to discover USB device: Intent.ACTION_USB_DEVICE_ATTACHED/DETACHED
USB audio transport framework register to ALSA (advanced Linux sound architecture), the Linux sound card driver.
Audio output device code a slave USB device (Android is the host): AudioManager.DEVICE_OUT_USB_DEVICE = AudioSystem.DEVICE_OUT_USB_DEVICE. Discover with Intent.ACTION_USB_AUDIO_DEVICE_PLUG.
USB_AUDIO_DEVICE_PLUG intent is defined but not generated in Jelly Bean.
If you can broadcast the USB_AUDIO_DEVICE_PLUG intent with the sound card details to the Android audio framwork, Android will play audio over the USB headset.
Appendix: notes taken while reading Unboxing Android USB
Android USB device (vs. host) stackAndroid USB host stack is just the mainline Linux kernel USB host stack--but only as a USB OTG host (the same as USB embedded host?), which supports only the TPL (targeted peripheral list). The stack is different.
android.hardware.usb package
android.hardware.usb.host package
- UsbDevice class: device connected to Android that is in USB host mode.
- UsbManager: state information. Provide permission to the USB device and state information intent.
- UsbDeviceConnection: send/receive data to USB device
- UsbInterface: expose device functions
- UsbEndpoint: no support for isochronous endpoint?
- UsbRequest: USB packet to read/write to device
Intent to discover USB device: Intent.ACTION_USB_DEVICE_ATTACHED/DETACHED
AOA (Android open accessory) protocol
An accessory like a docking station can power the phone and play the accessory role.
USB audio (USB-IF audio device class, release 1.0)
USB host audio (which can use USB headset) uses usb-core Linux kernel module.
USB device audio: receive audio stream from Android and play back to user.
An audio device exposes its functionality to a host through its interfaces (interfaceAudio streaming interface, MIDI streaming interface), as shown here:
Audio function consists of units and input/output terminals.
- MU, mixer unit
- SU, selector unit
- FU, feature unit
- PU, processing unit
- XU, extension unit
Audio output device code a slave USB device (Android is the host): AudioManager.DEVICE_OUT_USB_DEVICE = AudioSystem.DEVICE_OUT_USB_DEVICE. Discover with Intent.ACTION_USB_AUDIO_DEVICE_PLUG.
USB_AUDIO_DEVICE_PLUG intent is defined but not generated in Jelly Bean.
If you can broadcast the USB_AUDIO_DEVICE_PLUG intent with the sound card details to the Android audio framwork, Android will play audio over the USB headset.
- Connect a USB headset. Check with lsusb and "cat /proc/asound/cards"
- Broadcast "am broadcast -a android.intent.action.uSB_AUDIO_DEVICE_PLUG --ei state 1 --ei card 2 --ei device 0"
Debugging Android on OTG
- Connect the Android-powered device via USB to your computer.
- From your SDK platform-tools/ directory, enter adb tcpip 5555 at the command prompt.
- Enter adb connect 10.2.100.14:5555 You should now be connected to the Android- powered device and can issue the usual adb commands like adb logcat.
- To set your device to listen on USB again, enter adb usb.
