serial.pipe

[espruino-discuss2]

Posted at 2016-02-26 by countxerox

Hi, I'm writing a module to interface with a radio control receiver, the receiver has a serial interface which is like the Futaba SBUS protocol.

I'm piping the serial data, which comes in frames of 25 bytes, to an object with a write method so I've set the chunksize option to 25 but when I console.log the data being written to the object I see randomly sized chucks of 2-8 bytes?

So the module is kind of working now but I want to make some further refinements, I tried using the 'end' and 'complete' options but nothing seems to happen, is it the 'end' of the chunk or something else? I've tried reducing the chunksize. What indicates when the pipe activity is complete or the when source has finished? Are the end/complete functions written like 'function end() {...}' or should they be functions of the object, how would I go about that?

[espruino-discuss2]

Posted at 2016-02-26 by DrAzzy

Is there a reason you're using serial.pipe instead of using the serial.on("data",...) handler?

That's what I've always used for serial interfacing...

[espruino-discuss2]

Posted at 2016-02-26 by countxerox

No DrAzzy, no particular reason. I've already done version of the module using serial.on and serial.read, I'm learning JavaScript and just trying out the alternatives.

[gfwilliams]

Posted at 2016-02-26 by @gfwilliams

The serial port doesn't really have any concept of when it is finished - so you won't get any 'end' or 'complete' event produced by it.

Also, the chunk size used by pipe is more of a maximum chunk size in Espruino - it may not do the same as on node.js, but if it is idle and there is data available, it'll just push what it has.

To be honest in this case, using the Serial.on('data' will probably work best for you. pipe was really added for files and network sockets, but was extended to other things to make it more complete rather than because it's always the best way of doing things.

I don't know anything about SBUS, but you might get a framing error if the serial line is brought low after each transmission. Maybe try:

Serial1.on('framing', function() {
    console.log("Frame?");
  });

If that's the case, you can use the serial data handler to store the data, and can then parse it all when you get the framing error from the serial port - it'd be a nice way of working without worrying about possibly being out of sync.

[espruino-discuss2]

Posted at 2016-02-26 by countxerox

Good suggestion, thanks Gordon, I tried that and got this:

Serial1.setup(100000, {rx:B7,parity:E,stopbits:2});
Serial1.on('framing', function() {
  console.log("frame");
  print(E.toUint8Array(Serial1.read(25)));
});

new Uint8Array([15, 174, 112, 133])
frame
new Uint8Array([43])
frame
new Uint8Array([92])
frame
new Uint8Array([225])
frame
new Uint8Array([10, 87])
frame
new Uint8Array([184, 194, 21, 174, 112, 133])
frame
new Uint8Array([43])
frame
new Uint8Array([92])
frame
new Uint8Array([225])
frame
new Uint8Array([10, 87])
frame
new Uint8Array(0)
frame
new Uint8Array(0)
frame

[espruino-discuss2]

Posted at 2016-02-26 by countxerox

Here's my code using pipe:

/* Module for interfacing with FrSky SBUS radio control receivers

Serial2.setup(100000, {rx:A3,parity:E,stopbits:2});
mysbus = require('SBUS').connect(Serial2, function(data){
  console.log(data);
});

*/

function decode(frame, callback) {
  arr = E.toUint8Array(frame);
  callback({
    ch1 :  ((arr[0]     | arr[1]<<8)                & 0x07FF),
    ch2 :  ((arr[1]>>3  | arr[2]<<5)                & 0x07FF),
    ch3 :  ((arr[2]>>6  | arr[3]<<2  | arr[4]<<10)  & 0x07FF),
    ch4 :  ((arr[4]>>1  | arr[5]<<7)                & 0x07FF),
    ch5 :  ((arr[5]>>4  | arr[6]<<4)                & 0x07FF),
    ch6 :  ((arr[6]>>7  | arr[7]<<1  | arr[8]<<9)   & 0x07FF),
    ch7 :  ((arr[8]>>2  | arr[9]<<6)                & 0x07FF),
    ch8 :  ((arr[9]>>5  | arr[10]<<3)               & 0x07FF),
    ch9 :  ((arr[11]    | arr[12]<<8)               & 0x07FF),
    ch10 : ((arr[12]>>3 | arr[13]<<5)               & 0x07FF),
    ch11 : ((arr[13]>>6 | arr[14]<<2 | arr[15]<<10) & 0x07FF),
    ch12 : ((arr[15]>>1 | arr[16]<<7)               & 0x07FF),
    ch13 : ((arr[16]>>4 | arr[17]<<4)               & 0x07FF),
    ch14 : ((arr[17]>>7 | arr[18]<<1 | arr[19]<<9)  & 0x07FF),
    ch15 : ((arr[19]>>2 | arr[20]<<6)               & 0x07FF),
    ch16 : ((arr[20]>>5 | arr[21]<<3)               & 0x07FF),
    ch18 : (arr[22] & 0x0001) ? 2047:0,
    ch19 : ((arr[22] >> 1) & 0x0001) ? 2047:0,
    framelost : ((arr[22] >> 2) & 0x0001) ? true:false,
    failsafe  : ((arr[22] >> 3) & 0x0001) ? true:false,
  });
}

exports.connect = function (serial, callback) {
  dest = {
    state: 'sync',
    last: '',
    write : function(data) {
      switch(this.state){
        case 'sync':
          buff = data;
          if (this.last === "\x00" && buff.substr(0,1) == "\x0F"){
            this.state = 'read';
          }
          break;
        case 'read':
          buff += data;
          if (buff.length > 24) {
            if (buff.substr(24,1) === "\x00") {
              this.state = 'sync';
              decode(buff.substr(1,23), callback); 
            }else{
              buff = buff.substr(buff.length - buff.indexOf("\x0F",1));
            }
          }
          break;
      }
      this.last = data.substr(-1,1);
    }
  };
  serial.pipe(dest,{chunkSize : 25});
};

[gfwilliams]

Posted at 2016-02-29 by @gfwilliams

That's odd - getting that many framing errors points to something being wrong though.

Your parity:E, isn't going to do what you want though. I'm pretty surprised that doesn't error. I think what you actually want is parity:'e'

With that, you might find you get more sensible framing errors.

Also, I'm not sure you want Serial.read(25) - you'd be better off just doing Serial.read() - because otherwise if something changes and actually 26 characters get sent between framing errors, you're going to end up buffering a whole load of data.

[espruino-discuss2]

Posted at 2016-02-29 by countxerox

Yep, with parity:'e' all the framing errors are gone.

Serial1.setup(100000, {rx:B7,parity:'e',stopbits:2});
Serial1.on('framing', function() {
  console.log("frame");
  });
Serial1.on('parity', function() {
  console.log("parity");
  });
Serial1.on('data', function(data) {
  console.log(E.toUint8Array(data));
});
Output...
new Uint8Array([15])
new Uint8Array([172, 96, 5, 43])
new Uint8Array([88, 193, 10, 86, 176])
new Uint8Array([130, 21, 172, 96, 5])
new Uint8Array([43, 88, 193, 10, 86, 176])
new Uint8Array([130, 21, 0, 0])

0x0F is the startbyte, the following 22 bytes are the payload, then the next byte is for flags which show if the receiver has dropped frames or if the signal has been lost and then the last byte is the endbyte 0x00. The framerate is about 7ms but it wibbles at bit so I can't think of anyway to sync using timing.

The startbyte and endbyte could potentially appear in the payload so I'm watching for 0x00 followed by 0x0F to mark the start of a frame, then when the 25 bytes have been received I check the last byte is 0x00 for validation.

Hers's my code using Serial.on('data'...

/* Module for interfacing with FrSky SBUS radio control receivers

Serial1.setup(100000, {rx:B7,parity:'e',stopbits:2});
mysbus = require('SBUSON').connect(Serial1, function(data){
  console.log(data);
});

*/

function decode(frame, callback) {
  arr = E.toUint8Array(frame);
  callback({
    ch1 :  ((arr[0]     | arr[1]<<8)                & 0x07FF),
    ch2 :  ((arr[1]>>3  | arr[2]<<5)                & 0x07FF),
    ch3 :  ((arr[2]>>6  | arr[3]<<2  | arr[4]<<10)  & 0x07FF),
    ch4 :  ((arr[4]>>1  | arr[5]<<7)                & 0x07FF),
    ch5 :  ((arr[5]>>4  | arr[6]<<4)                & 0x07FF),
    ch6 :  ((arr[6]>>7  | arr[7]<<1  | arr[8]<<9)   & 0x07FF),
    ch7 :  ((arr[8]>>2  | arr[9]<<6)                & 0x07FF),
    ch8 :  ((arr[9]>>5  | arr[10]<<3)               & 0x07FF),
    ch9 :  ((arr[11]    | arr[12]<<8)               & 0x07FF),
    ch10 : ((arr[12]>>3 | arr[13]<<5)               & 0x07FF),
    ch11 : ((arr[13]>>6 | arr[14]<<2 | arr[15]<<10) & 0x07FF),
    ch12 : ((arr[15]>>1 | arr[16]<<7)               & 0x07FF),
    ch13 : ((arr[16]>>4 | arr[17]<<4)               & 0x07FF),
    ch14 : ((arr[17]>>7 | arr[18]<<1 | arr[19]<<9)  & 0x07FF),
    ch15 : ((arr[19]>>2 | arr[20]<<6)               & 0x07FF),
    ch16 : ((arr[20]>>5 | arr[21]<<3)               & 0x07FF),
    ch18 : (arr[22] & 0x0001) ? 2047:0,
    ch19 : ((arr[22] >> 1) & 0x0001) ? 2047:0,
    framelost : ((arr[22] >> 2) & 0x0001) ? true:false,
    failsafe  : ((arr[22] >> 3) & 0x0001) ? true:false,
  });
}

exports.connect = function (serial, callback) {
  dest = {
    state: 'sync',
    last: '',
    write : function(data) {
      switch(this.state){
        case 'sync':
          buff = data;
          if (this.last === "\x00" && buff.substr(0,1) == "\x0F"){
            this.state = 'read';
          }
          break;
        case 'read':
          buff += data;
          if (buff.length > 24) {
            if (buff.substr(24,1) === "\x00") {
              this.state = 'sync';
              decode(buff.substr(1,23), callback); 
            }else{
              buff = buff.substr(buff.length - buff.indexOf("\x0F",1));
            }
          }
          break;
      }
      this.last = data.substr(-1,1);
    }
  };
Serial1.on('data', function(data) {
  dest.write(data);
});
};

I seem to have bytes going missing sometimes when the callback's called

[gfwilliams]

Posted at 2016-03-01 by @gfwilliams

Depending on the datarate and what else you do when you get the decoded data it could be that the data can't be processed fast enough and the buffer overflows?

Maybe check E.getErrorFlags() to see if that's the case? If so it's finding a way to decode the data a bit faster - maybe something like:

Serial1.on('data', function(data) {
  buf += data;
  var i = buf.indexOf("\x00\xFF");
  while (i>=15) {
    decode(buf.substr(i-15,16));
    buf = buf.substr(i+1); 
    i = buf.indexOf("\x00\xFF");
  }
});

Not tested, but might work - of course the way I've done it above you're always one frame behind.

Also... which board are you using? different boards have different serial buffer sizes

[espruino-discuss2]

Posted at 2016-03-07 by countxerox

I wish I'd thought of that, I'm not seeing any E.getErrorFlags() but I'm getting better results with that method, thanks for the help. I'm using the pico board which seems to have a 512 byte buffer.