Binary message protocol for custom BLE service

Just can't stop staring at those LED strips

I have yet to meet anyone who doesn't think LEDs are not cool.  I am normally too busy with more serious problems, but I've been dreaming about decorating my home office with LED ceiling lamp that displays pleasant patterns.  The original LED strip that most hobbyists have surely seen by now is driven by the original WS1xxx LED + controller chip, which became a megahit partly because of the SPI-like daisy chained controllability.  When I discovered a follow-on product APA10x, which addresses one of the critical shortcomings of the predecessor--the timing difficulty due to the signal being only SPI-like rather than the real-time--I treated myself to a couple of APA102 LED strips.

I am not sure how I want to arrange the > 100 LED pixels, but I am sure I don't want to do it manually, which means I need a micro-controller that can drive out SPI through DMA.  The uC will be hidden away in my room, so I decided to use my phone as the UI--which means I need a bi-directional BLE (Bluetooth Low Energy--AKA Bluetooth Smart) communication between my uC and phone, as shown in the introduction for the Nordic nRF51DK.

I hate writing Spaghetti code often found in low level FW, and I don't want to run a full-fledged real-time OS on a resource constrained chip like the nRF51, so I ported my favorite real-time FW framework--QPC--to the nRF51DK and wrote about the process here.

NUS: custom BLE service that can be easily modified to my liking

As I wrote in the above Google doc, I did not found any Bluetooth SIG defined GATT service for my LED array ceiling lamp project.  Many BLE projects wind up creating a custom service, and the Nordic's UART service example is a great project to copy from: it offers an IN and an OUT characteristics under a service called NUS.  It keeps things simple: no security provision, encryption or bonding.  The MTU size is the Nordic stack default (23 bytes), which means that the usable maximum message size is 20 bytes.  The example will loop back a UTF-8 string from a phone to the nRF51DK and then back.  My experience of going through the NUS tutorial, and marrying the example with a simple state machine application is also in the above Google doc.

Self-describing binary message protocol

To get even a moderate bandwidth, a binary message protocol is a must.  The QS message protocol is part of the QP framework I like so much, and is adequately described in Miro Samek's book.  Briefly, I find its following features attractive for this project:

1. Binary
2. Self-describing message fields.  That is, even for the same message ID (say "COMMAND"), both the number and order of the arguments are changeable and does not have to be hard coded.
3. Light-weight: the whole library consists of a few functions, and the internal message buffer is loaned to the library by the application--which can decide the buffer size.
1. The light-weight is possible in part because the message protocol is NOT reliable or secure, which works because the BLE GATT gives you reliability and security.
4. A message can be fragmented for transports with small MTU size.  Precisely the situation here!

Miro is a stickler for efficient code, so QS has rich filtering capability (conditional evaluation of a few integer comparisons).  This is necessary for a SW tracing library, but not necessarily for a thin message protocol.  Because Miro focuses on C/C++, QS has not yet been ported to Java/Android (or to iOS for that matter).  Ripping out the filtering support from QS C source is straight-forward, so most of the work was in the Java port and integration with the Nordic NUS example Android app.
My code is in my GitHub realtime repository (https://github.com/henrychoi/realtime) Lyle folder. 
For now, I support only basic message types, and the peek/poke memory to the target:

public interface MsgType {
    public static final byte

            EMPTY = 0, /*!< MSG record for cleanly starting a session */

            PEEK_MEM = 1, // Reading target memory is very handy

            POKE_MEM = 2, // Writing target memory is very handy

            PEEK_RES = 3, // Answer to the peek

            APP_SPECIFIC = 4, // Begin application specific messages

            WPAR_S = APP_SPECIFIC, // Just 1 string arg

            WPAR_0 = 10, // Message with no arg

            WPAR_8 = 20, // Message with just 1 byte arg

            WPAR_16 = 30, // Message with just 2 byte arg

            STATE = WPAR_16,
            WPAR_32 = 40; // Message with just 4 byte arg}

You can add your own message IDs (and I plan to add lots more).  To use it, the Nordic's existing UartService class uses the new Msg class:

byte[] Msg_buf = new byte[1024];
Msg msg = new Msg(Msg_buf, (short) Msg_buf.length, this);

The UartService instance owning the message is passed as the last argument to the Msg, because it implements the Msgable interface (I'll introduce in the receive path) to handle message flush and received message from the target.

Msg send path (Android to the target)

Android sender

A message is bracketed with a header that includes the message type.  Between the beginning and the end of a message, supported primitive types can be encoded, as in the example below.

void write(short b) {
    msg.BEGIN(MsgType.STATE);
        msg.I16(b);//Direct the query to all SMs    msg.END();
    msg.FLUSH();//Send the request right away?}

Here, I chose to flush the message out right away, but you can just queue the message and have another thread drain the TX message queue (and I plan to), to increase throughput (at the cost of latency).

The writeOUT() was already provided by the Nordic's example code; I just renamed it to add the characteristic name for clarity.  One unique feature of the QS protocol is detection of message stream restart or corrupted message (and automatic tossing of the interrupted/corrupted message).  The EMPTY message in the MsgType you saw earlier makes the stream reset possible.  The Msg constructor encodes an EMPTY message:

public Msg(byte[] sto, short stoSize, Msgable ifc) {
    priv_.buf = sto;
    priv_.end = stoSize;
    this.ifc = ifc;

    /* produce an empty record to "flush" the Msg trace buffer */    beginRec(MsgType.EMPTY);
    endRec();
}

When the message stream is flushed, the flushTX() interface method of the Msgable is called.

public interface Msgable {
    void flushTX();
    void onTargetMsg(TargetMsg m);
}

An implementation is responsible for draining the TX queue and transferring to the appropriate transport, as in the UartService class example.

static final short NUS_PAYLOAD = 23 - 3;
public void flushTX() {
    while(true) {
        Pair<Short, Short> block = msg.getBlock(NUS_PAYLOAD);
        if (block == null) break;
        byte[] blk = Arrays.copyOfRange(Msg_buf //copy to tail+n-1
                , block.first, block.first + block.second);
        writeOUT(blk);
    }
}

The contract between the message protocol and the transport is the raw byte hand-off to the transport.

Target receiver

The Nordic UART example already supplies the nus_data_handler() callback, which I modified to plumb the received bytes into the message parser (after logging a trace message):

static void nus_data_handler(ble_nus_t * p_nus, uint8_t * p_data, uint16_t length)
{
    QS_BEGIN(TRACE_NUS_DATA, &l_softdevice)
        QS_MEM(p_data, length);
    QS_END()
    MSG_parse(p_data, length);
}

My parser creates a new event and publishes it to all interested (subscribed, in publish-subscribe parlance) active objects.

static void QSpyRecord_processUser(QSpyRecord * const me) {
    uint8_t fmt;
    uint32_t u32;
    NUSEvt* pe = Q_NEW(NUSEvt, NUS_SIG);
    pe->type = me->rec; // NUS message type
    while (me->len > 0) {
        fmt = (uint8_t)QSpyRecord_getUint32(me, 1);  /* get the format byte */
        switch (fmt) {
...
          default:
           QS_BEGIN(TRACE_MSG_ERROR, (void *)0)
            QS_U8(0, MSG_ERROR_UNEXPECTED);
            QS_U8(0, me->rec);
            QS_U8(0, fmt);
           QS_END()
           me->len = -1;
           break;
        }
    }
    QF_PUBLISH(&pe->super, me);
}

My particular handling is of course intimately tied to my FW infrastructure (QPC), but you are free to handle the message in your own way.

Target --> Android path

Target side

In this example, let's say the Android side wants to query the current state of all the active objects in the FW (this is how I like to build up the SW/FW interface).  My active object can then respond to the NUS_SIG sent by the low level code like this example.

 switch(e->sig) {
 case NUS_SIG: {
  const NUSEvt* pe = (const NUSEvt*)e;
  switch(pe->type) {
  case MSG_STATE: // state query
   MSG_BEGIN(MSG_STATE);
   MSG_I16(TABLE_STATE_ACTIVE | AO_TABLE);
   MSG_END();
   break;
  default: break;
  }
 } return Q_HANDLED();

The above code merely stuffs a message into the target's TX queue.  Unlike the Android side, it does not flush the queue right away, but rather waits for an idle time for that to happen.

void QV_onIdle(void) { /* called with interrupts disabled, see NOTE01 */
...
    if (m_nus.conn_handle != BLE_CONN_HANDLE_INVALID
      && m_nus.is_notification_enabled) {
        uint16_t n = BLE_NUS_MAX_DATA_LEN;
        uint8_t* msg_buf = (uint8_t*)MSG_getBlock(&n);
        if (msg_buf) {
            QS_BEGIN(TRACE_MSG_OUT, (void *)0)
               QS_MEM(msg_buf, n);
            QS_END()
            uint32_t err_code = ble_nus_string_send(&m_nus, msg_buf, n);
            Q_ASSERT(err_code == NRF_SUCCESS);
        }
    }

Android receiver

Nordic UART example already provides a couple of callback methods to handle the IN (BLE) characteristic reception.  I just have to hand it off to my own handler.

@Overridepublic void onCharacteristicRead(BluetoothGatt gatt,
                                 BluetoothGattCharacteristic characteristic,
                                 int status) {
    if (status == BluetoothGatt.GATT_SUCCESS

            && IN_CHAR_UUID.equals(characteristic.getUuid())) {
        handleChar(characteristic);
    }
}

@Overridepublic void onCharacteristicChanged(BluetoothGatt gatt,
                                    BluetoothGattCharacteristic characteristic) {
    if (IN_CHAR_UUID.equals(characteristic.getUuid())) {
        handleChar(characteristic);
    }
}

Like the target side, my handler will just delegate to the message parser.

//@brief Handle IN Characteristic notification of NUS service

void handleChar(final BluetoothGattCharacteristic characteristic) {
    byte[] data = characteristic.getValue();
    Log.d(TAG, String.format("Received IN characteristic, %d bytes", data.length));
    msg.parse(data); //Parse may find 0 or more TargetMsg (see onTargetMsg)}

The parser is a simple state machine that moves through the sequential states of the message, until the final FRAME (complete) byte is received, at which time a completed message is handled off to an intermediate level handler:

void process() {
    switch(rec) {
        case MsgType.EMPTY:
        case MsgType.PEEK_MEM:
        case MsgType.POKE_MEM:
            break; //silently ignore        case MsgType.PEEK_RES:
            TargetMsg m = new TargetMsg();
            m.data[0] = getInt();
            m.data[1] = getInt();
            ifc.onTargetMsg(m);
            break;
        default:
            processUser(); break;
    }
}

You've seen this movie before: processUser() method is the last leg of the message infrastructure before it is handed off to the application's own handler.

void processUser() {
    TargetMsg m = new TargetMsg();
    m.typ = rec;

    while (len > 0) {
        byte fmt = getByte();
        switch (fmt) {

...

                default:
                    Log.e(TAG, String.format("********** %d: Unknown format %d",
                            rec, fmt));
                    len = -1;
                    break;
            }
        }
        ifc.onTargetMsg(m);
    }
}

The "ifc" above is the Msgable interface, which UartService implements.

public void onTargetMsg(TargetMsg m) {
    final Intent intent = new Intent(ACTION_PERIPHERAL_MSG);
    intent.putExtra(PHERIPHERAL_MSG, m);
    LocalBroadcastManager.getInstance(this).sendBroadcast(intent);
}

You can see that I now hand the message to the Android Intent/Broadcast infrastructure.  The only missing piece is to conform the TargetMsg a Parcelable interface, so Android can serialize it and ship it off to different Activity/Service/processes.

public class TargetMsg implements Parcelable {
    public byte typ; //See MsgType    public int[] data = new int[2];

    public static final Parcelable.Creator<TargetMsg> CREATOR =

 new Parcelable.Creator<TargetMsg>() {
        @Override        public TargetMsg createFromParcel(Parcel parcel) {
            TargetMsg m = new TargetMsg();
            m.typ = parcel.readByte();
            m.data[0] = parcel.readInt();
            m.data[1] = parcel.readInt();
            return m;
        }

        @Override        public TargetMsg[] newArray(int size) {
            return new TargetMsg[size];
        }
    };

    @Override    public int describeContents() { return 0; }

    @Override    public void writeToParcel(Parcel parcel, int flags) {
        parcel.writeByte(typ);
        parcel.writeInt(data[0]);
        parcel.writeInt(data[1]);
    }

    @Override    public boolean equals(Object o) {
        if (this == o) return true;
        if (o == null || getClass() != o.getClass()) return false;
        TargetMsg that = (TargetMsg)o;
        return typ == that.typ                && data[0] == that.data[0]
                && data[1] == that.data[1]
                ;
    }

}

Finally, here is an example of what my main activity does with the received message:

...

} else if (action.equals(UartService.ACTION_PERIPHERAL_MSG)) {
    final TargetMsg m = intent.getParcelableExtra(UartService.PHERIPHERAL_MSG);

    runOnUiThread(new Runnable() {
        public void run() {
            try {
                String currentDateTimeString = DateFormat.getTimeInstance().format(new Date());
                String text = Byte.toString(m.typ); //new String(inValue, "UTF-8");
...

One shortcoming of my solution is that the TargetMsg has to handle all possible payloads.  It may be preferable to have different classes specialized for different message IDs.

Licensing

The NUS example derived files fall under the Nordic SDK license you will find in the Nordic SDK v 11.0, while the QS derived sources probably inherit the QPC's dual licensing model.  I myself do not claim any copyright for my contribution.