Missing incoming Bluetooth packets?

[espruino-discuss]

Posted at 2019-10-12 by user103949

Using nRF Connect to read data from my Bluetooth sensor, I write 'g' to the RX characteristic and then read 1099 packets of data via notifications (23 Byte MTU size, including 3 bytes header). See screenshot:

1. packet: 04-e1-4B-...: 2 bytes ID and then 4 bytes each for number of packets (55) and number of data bytes (1099)
2. packet: 41-A5-43-...: Sensor ID
3. packet: 00-00-D6-...: Sensor configuration

• following packets

Using the standard MDBT42Q 2V04 firmware and the application below, however, the packets are not correct. I get the first packet, but the 2nd packet seems to be missing, and the 3 and 4th packet appear to contain identical data (which they don't).

It seems like the event data passed into the characteristicsvaluechanged() callback is not updated properly on each callback, or is my application code incorrect?

Thanks,
-- Terrence

var gatt;
var service;
var cntdata;
var cntpackets;

function getData() {
  console.log("getData()");
  console.log("searching for devices ...");

  NRF.requestDevice({ timeout: 20000, filters: [{ name: 'device name'}], active:true })
  .then(function(device){
      console.log("found - connecting ...");
      return device.gatt.connect();
  })
  .then(function(gattp){
      console.log("get primary service");
      gatt = gattp;
      return gatt.getPrimaryService("6e400001-b5a3-f393-e0a9-e50e24dcca9e");
  })
  .then(function(servicep){
      console.log("get TX characteristic");
      service = servicep;
      return service.getCharacteristic("6e400003-b5a3-f393-e0a9-e50e24dcca9e");
  })
  .then(function(characteristicp){
      cntdata = 0;
      cntpackets = -1;
      console.log("start notfifications");
      characteristicp.on('characteristicvaluechanged', function(event) {
        var dataview = event.target.value;
        cntpackets++;
        if (cntpackets == 0) {
          console.log("total data: " + dataview.getInt32(2, true) + ", total packets: " + dataview.getInt32(6, true));
        } else {
          cntdata += dataview.byteLength;
        }
        console.log("packet: " + cntpackets + ", data: " + cntdata);
        if (cntpackets < 4) {
          console.log(dataview);
        }
      });
      return characteristicp.startNotifications();
  })
  .then(function(){
      console.log("get RX characteristic");
      return service.getCharacteristic("6e400002-b5a3-f393-e0a9-e50e24dcca9e");
  })
  .then(function(characteristicp){
      console.log("send 'g' command");
      characteristicp.writeValue('g');
  })
  .then(function() {
    return new Promise(function(resolve) {
      setTimeout(resolve, 10000);
    });
  })
  .then(function() {
    console.log("end delay - disconnecting ...");
    gatt.disconnect();
    console.log("getData(): done");
  })
  .catch(function(err) {
    console.log("ERROR", err);
  });
}

getData();

---

Attachments:

• Screen Shot 2019-10-12 at 10.46.19.png
  
• Screenshot_20191012-103459_nRF Toolbox.jpg

[espruino-discuss]

Posted at 2019-10-12 by AkosLukacs

IIRC, Espruino does only support 20 byte MTU size.
There is some work done to increase the MTU size on the increased_mtu github branch, but don't know how stable is that.

[espruino-discuss]

Posted at 2019-10-12 by user103949

Sorry, I should have clarified: 23 bytes includes headers, which means 20 bytes for application payload, which is what Espruino should support. However, in my understanding, even if there is a MTU mismatch I don't think that explains why packets are missing or being received twice.

[espruino-discuss]

Posted at 2019-10-12 by AkosLukacs

Ah, ok, that explains why I couldn't count 23 bytes on the screenshot :) My guess was the too large MTU somehow confused the Espruino.

But one more "maybe": remove the console.log statements from the receiving, and print the first 4 packets after receiving is done. Sending that 20 byte DataView in that "nice" format takes several 7.5ms connection intervals. Maybe that interferes. But it's just a guess...

[espruino-discuss]

Posted at 2019-10-12 by user103949

Yes, I considered that too - that the log statement might interfere with the timing/buffering (although the system should be able to handle that and not just silently mess up the data). For testing, I just changed the code to not print anything but just check the first byte of the 2nd packet. No luck, still 0x00 instead of 0x41.

[gfwilliams]

Posted at 2019-10-14 by @gfwilliams

remove the console.log statements from the receiving

In your example code I'd have said that was almost certainly the problem.

Can you run E.getErrorFlags() and see if it says anything? If data has been arriving faster than Espruino can process it, there should be a FIFO_FULL flag.

Could you maybe try using the extra arguments to http://www.espruino.com/Reference#l_BluetoothRemoteGATTServer_connect to force the device to use a lower connection interval, just in case that's the problem?

[espruino-discuss]

Posted at 2019-10-14 by user103949

Hi Gordon,

E.getErrorFlags() returns an empty array, both at the point when I have received the 2nd packet with the wrong data as well as at the end of the application after all packets have been received.

I tried {minInterval:50, maxInterval:500} and {minInterval:10, maxInterval:100} in gatt.connect() but that also didn't change the behavior.

What makes me suspicious is that the application does receive all 59 packet notifications -- so none are lost -- it's just that the DataView passed into the callback consistently contains the wrong data starting with the 2nd packet (EDIT). That seems to indicate a corner case in the native code that passes an object with the wrong buffer reference into the JS callback.

Thanks

[gfwilliams]

Posted at 2019-10-14 by @gfwilliams

Good news about no getErrorFlags - that means Espruino is pretty sure it's relayed all data it gets from the Bluetooth stack to the application.

the application does receive all 59 packet notifications

Not sure I understand here... By application you mean Espruino? So you do get the correct number of callbacks, just not with the right data in them?

Is it just packets 2/3/4 that appear to be wrong, or are all the rest of them wrong?

Do you know exactly what data you should be receiving or if there are certain bytes that definitely don't change in each packet? It might be we can see if the packets are getting combined somehow.

If you have an NRF DK then you should be able to put it into a wireshark mode where you can see what data is being sent over the air.

The way Espruino works internally is it shoves all the bluetooth events into a queue. So the event sits in that queue with its data, and then events are processed in sequence. I've just checked through the code again and it's extremely hard to see how they could get muddled up .

Do you have any thoughts about how I might be able to reproduce it here?

[espruino-discuss]

Posted at 2019-10-14 by user103949

(editing as my Internet connection dropped while I was replying)

Not sure I understand here... By application you mean Espruino? So you do get the correct number of callbacks, just not with the right data in them?

Yes, the Espruino JavaScript application. I get 59 callbacks, but not the right data, e.g. original packet 2 is missing but I get packet 3 twice.

Is it just packets 2/3/4 that appear to be wrong, or are all the rest of them wrong?

I have the structure of the data I should be receiving. Some fields are fixed while some are dynamically filled with data. The first couple of packets allow me to interpret the rest of the data. Packet 2 received by the application appears correct but it should have been received as packet 3 - the original packet 2 is missing. And packet 3 received by the application should not be a repeat of the previous packet. I haven't checked the following packets.

Do you know exactly what data you should be receiving or if there are certain bytes that definitely don't change in each packet? It might be we can see if the packets are getting combined somehow.

See above. I don't see any indication that packets are being combined. Packet 2 appears to be fully correct, it's just that it should have been received as packet 3.

If you have an NRF DK then you should be able to put it into a wireshark mode where you can see what data is being sent over the air.

I'm afraid I don't (yet).

Do you have any thoughts about how I might be able to reproduce it here?

Seems like the only way would be to send you one of these sensors, but I need to check with the vendor first.

Thanks

[espruino-discuss]

Posted at 2019-10-16 by user103949

Gordon,

The sensor vendor will supply me with another testing unit that I can forward to you. I will contact you separately to arrange delivery.

[gfwilliams]

Posted at 2020-04-28 by @gfwilliams

Sorry for the delay in this - better late than never.

If you update to the latest firmware I'm pretty sure this will now be fixed!