Trying to connect the Puck.js to Raspberry pi to Firebase DB

[espruino-discuss2]

Posted at 2020-07-27 by redfox

Hi all,

Im following https://www.espruino.com/Smart+Meter

TL;DR
How do you access data in an ArrayBuffer that is sent over a BLE advert to node-red.

And my end-goal is to get data from my Electricity Meter, to the Puck.js to Raspberry pi, to Firebase DB.

So far:
I've got the data to the puck.
I've got the data to the raspberry pi via the bluetooth advertising and node-red/EspruinoHub.

Now im trying to figure out the advertising data, how to read it, and then once I have that, I can focus on the connection to Firebase.

My current issue is that the data over MQTT is coming out as the current count from the Puck.js, seems all ok, but then it changes from a number (00000001) to something else (000004f9) which looks binary or bytes or something else.

I need help figuring out how the ArrayBuffer plays with the DataView and everything else.
I kind of get that you dont access data directly in the ArrayBuffer, you use the DataView for read and write, but in this case, its writing to it, and then it goes across the Bluetooth highway, into node-red as the unreadable string, and then.... well thats where im stuck.

How do I make sense of:

7/27/2020, 6:23:10 PMnode: 85a205c2.26d3f8
/ble/advertise/mac:address/manufacturer/0590 : msg.payload : string[10]
""0000050f""

And get it into something like Firebase (I'm thinking RealTime db seems best, normal db would go crazy if it was updated every 600ms / out of free credits). Im not tied to Firebase, just seemed like a good option.

[gfwilliams]

Posted at 2020-07-28 by @gfwilliams

Hi - that sounds like a good plan. I think what happens is that once you supply data to setAdvertising on the Puck, it's transmitted as a series of bytes regardless of how it was defined in JS.

Looks like it's being interpreted as a series of hexadecimal digits. If you've got "0000050f" then you should just be able to do msg.payload = parseInt(JSON.parse(msg.payload),16) in a 'function block' in node-red to get the number you want.

[espruino-discuss2]

Posted at 2020-07-28 by redfox

Thanks Gordon. Slight tweak needed, as in your smart meter tutorial, you set it as a Uint32, so changing the 16 to 32 gave me a number that certainly looks like what I need (But its the total overall count I think, not for the day, so a little more work needed).

Would it be possible for you to explain, maybe in pseudo code or high level story, what you do in your example code from https://www.espruino.com/Smart+Meter?

I'm trying to figure out what it does at each step, so that I can find the best way to build historical data from it.

Looking at the Historical data part with the Counter, I was trying to see what you are doing with the years, months, days etc, but it does not make sense to me. Im totally new to storing data in byte format, so thats probably why it looks alien to me.

I get that I can just strip out all the code, and in fact just use the basic example code at the start of your tutorial, and just reset it every day, but I would like to understand how and why you took your approach.

I'm thinking btw that in node-red I would need to track the current minute and last value, and then when the value changes, note the time that happened, and then subtract the counter value from the last time it changed, to get the amount of counts for a given minute. And then store that in Firebase to build up a per minute usage. Or maybe thats overkill and it needs to be every 10 minutes. Does that sound logical to you? Again, totally new to IoT, and im a web developer, so thats where my thought process only comes from.

[gfwilliams]

Posted at 2020-07-28 by @gfwilliams

Slight tweak needed

You mean changing parseInt(...,16) to parseInt(...,32)? If so that's definitely not what you want :) The 16 refers to hexadecimal, not to it being Uint16 :)

There are 8 characters, so 8 chars*4 bits in hexadecimal = 32 for the Uint32.

Would it be possible for you to explain...

You mean the history usage specifically?

So every time there's a pulse of light counter.inc gets fired, which updates the history.

It's got two parts - there's a rolling history which is just an array. Every time inc is called the last element is incremented - however if an hour has passed then it shifts the array up one and moves on to the next hour. That sounds like what you actually want for firebase - not the array, but just store a count and upload that every hour.

The second part is just a super simple historical graph. There's an array for each month of the year, one for day of the week, etc - and they're just incremented for the current day each time inc is called. In means after a few weeks you should start to see some decent patterns (like the difference between weekday and weekend, or the time of day).

[espruino-discuss2]

Posted at 2020-07-28 by redfox

Ahh, ok yes thats what I meant. ok I'll have to take a second look at that. I must have done something wrong, as using 16 got me no or weird results (cant remember) and 32 got me a big number. Checking again now, 32 gets me 69702, and 16 gets me 9254. So now its giving me a value. I'll go with 16 then, as you know more about it than me, who knows what typo I made before.

ok, I 'think' I follow what you're saying there for the storage. If thats the case, then yes maybe I do want to follow the same logic in the firebase data structure.

I was starting along the lines of this: https://jsfiddle.net/redfox05/7Lpmvbkz/27/

var min = d.getMinutes();
switch ( true ) {
    case min <= 15: console.log('segment1', min); break;
    case min <= 30: console.log('segment2', min); break;
    case min <= 45: console.log('segment3', min); break;
    default: console.log('segment4', min);
}

Trying to break each hour into 4 segments of 15 minutes, as perhaps doing it per minute would not be accurate enough / lead to errors given its tighter points.
From what I understand, each pulse flash is 1000th of a kWh, so 1000 counts in 1 kWh.

So ideally, I would like to get to a point where we know how many kWh or fraction thereof are used for every 15 min block.
The datastore needs to be able to be queried per 15, per hour, per day, month etc, so I'll keep thinking on that one.

Be really cool if I could expand this to then do prediction stuff. Im pretty sure Google/AWS has some clever tools for this.

[espruino-discuss2]

Posted at 2020-07-28 by redfox

Once this works, would be cool to work with you perhaps to have a Part 2 to your guide based on what I do, to deal with 'The Other Side' of the bluetooth transmission of the energy data. Unless you dont want the examples to be too 'complete' as it might not encourage people to think for themselves.

[gfwilliams]

Posted at 2020-07-28 by @gfwilliams

Yeah, using 32 would get you a big number, but you'd find that (if starting from 0) you'd get:

0,1,2,3....13,14,15,32,33,34...

when you wanted:

0,1,2,3....13,14,15,16,17,18...

I can't remember how you'd do it but I think really in node-red you'd just want some kind of global variable.

So basically:

• When you get advertising, store that number using the parseInt stuff
• Every 15 minutes, save the difference between your last advertising value and lastValue in your database, then make lastValue equal to the current advertising value.

But yeah, I'm definitely open to doing more on cloud connection. I'm actually working on a way to make this much easier as well - having the ability to use an old Android phone, ESP32, or Pi (or a combination) to get the data online.

[espruino-discuss2]

Posted at 2020-07-28 by redfox

Oh yeah, I didnt want a big number per se. It was more that it was a number that seemed right, but obviously was not.

And your suggestion of a var, that sounds much much simpler. I'll give that a go.
From the reading I did, its either context, flow, or global variables. In this case, im thinking a flow one, as it only needs to work within the given flow.