Internationalization for GadgetBridge Notification

[Weiming-Hu]

This is a issue branching from the discussion here. Since it probably would involve some dedicated discussion, I opened a separate ticket to track this issue specifically.

What is the issue

Non-English texts, Chinese in my case, are always displayed as '???' which is annoying.

Solution

Notification should be able to rendered as images to facilitate internationalization.

Tasks on board

• Change sender's API to support sending image data
• Change receiver's API to properly parse the notification based on the type, either text or image. Doc on sending images can be found here.

[Weiming-Hu]

I have located the GB code on sending notification here.

I plan to read more about the code and figure out:

1. how to convert text based notifications to images
2. how to structure images for sending
3. how to make the notification by image an option so that users can choose from sending as texts or images. This could be useful for backward compatibility.

[gfwilliams]

That sounds great, thanks! Just to add that for a customer I just added E.decodeUTF8 to the cutting edge Espruino builds recently: https://github.com/espruino/Espruino/blob/6571399db45f9d89de17ad0eb2f281e95309dbb5/src/jswrap_espruino.c#L1992-L1999

So that is another option (especially if I add the ability to include images inside text strings). However, I think realistically what you're attempting here is still the best solution, as it'll 'just work' - without requiring any 'language pack' to be installed on the Bangle

[Weiming-Hu]

Hi Gordon, I have pushed some changes to GadgetBridge. Please see my changes here.

I have done just minimal changes. Basically, I added a function to convert string to bitmap and then send the bitmap to Bangle.JS. Of course, the watch crushes as it is right now. I'm wondering whether this is along the lines that you are thinking of?

I'm actually just learning Java. So things have been very new to me. Please let me know if you have any suggestions on how to improve, e.g. API, code, and many more. Thank you.

[gfwilliams]

Hi - yes, that's along the lines of what we're after, but I think it needs to be a bit more complicated....

Maybe instead of setting body/etc, you could keep the original body/title/etc, and then set bodyImg as well? That way it'll be backwards compatible

I'm pretty sure convertStringToBitmap doesn't actually work right now. I think what you need to do is create a 240x240 bitmap, then render the text to it, then get the bitmap as base64.

• So to render text: https://stackoverflow.com/a/31166409/1215872 and maybe use Bitmap.Config.RGB_565
• Then convert that bitmap into something that can be used with Espruino - simplest would be one bit per pixel. Something like this should work, but not tested.

byte bmp[] = new byte[240*80 + 3];
int n=0;
bmp[n++] = 240; // width
bmp[n++] = 240; // height
bmp[n++] = 1; // bpp
for (int y=0;y<240;y++)
  for (int x=0;x<240;x+=8) {
   int c = 0;
   for (int i=0;i<8;i++)
    c = (c<<1) | ((bitmap.getPixel(x+i,y)&255)>128);
    bmp[n++] = c;   
  }

• Now convert that bmp array to base64

Having said all that, you actually need to be a little smarter about the size - for instance you don't want to send a 240x240px image for a simple message that says 'Hello'. I guess titles want to be just the height of one line of text, and the body just the height of the text that is in the message

[Weiming-Hu]

Hi - yes, that's along the lines of what we're after, but I think it needs to be a bit more complicated....

Maybe instead of setting body/etc, you could keep the original body/title/etc, and then set bodyImg as well? That way it'll be backwards compatible

That makes sense. I will simply add another field in the message called bodyImg which contains the data for the bitmap.

I'm pretty sure convertStringToBitmap doesn't actually work right now. I think what you need to do is create a 240x240 bitmap, then render the text to it, then get the bitmap as base64.

* So to render text: https://stackoverflow.com/a/31166409/1215872 and maybe use `Bitmap.Config.RGB_565`

* Then convert that bitmap into something that can be used with Espruino - simplest would be one bit per pixel. Something like this should work, but not tested.

byte bmp[] = new byte[240*80 + 3];
int n=0;
bmp[n++] = 240; // width
bmp[n++] = 240; // height
bmp[n++] = 1; // bpp
for (int y=0;y<240;y++)
  for (int x=0;x<240;x+=8) {
   int c = 0;
   for (int i=0;i<8;i++)
    c = (c<<1) | ((bitmap.getPixel(x+i,y)&255)>128);
    bmp[n++] = c;   
  }

* Now convert that `bmp` array to base64

Having said all that, you actually need to be a little smarter about the size - for instance you don't want to send a 240x240px image for a simple message that says 'Hello'. I guess titles want to be just the height of one line of text, and the body just the height of the text that is in the message

OK. I see. So I need to somehow arrange all text information into a single bitmap. The width will be fixed at 240 pixels because that is just how wide the watch screen is. The height should be dynamic because some message will be longer than others, but not higher than 240 pixels. Does that sound good?

And do you have suggestions on how to debug this? Because I suppose as it is right now, even if I modified the code, I won't see any changes from the watch side. I'm not really sure how I can debug it before I commit my changes.

Thank you for your pointers.

[gfwilliams]

The width will be fixed at 240 pixels because that is just how wide the watch screen is. The height should be dynamic because some message will be longer than others, but not higher than 240 pixels.

That sounds spot on.

And do you have suggestions on how to debug this?

I can't remember for sure, but it might be that the Bangle.js class in Gadgetbridge writes the data it sends as a log message? If not it should be pretty easy to add that functionality.

Once that's done, you should be able to copy the base64 code you're generating and paste it into g.drawImage(atob("base64...."), 0,0) and the data will be drawn on the Bangle.js screen (you can even use the emulator instead).

So that should really help with the debugging?

I guess initially you could just make your own super-simple app that displayed a Bitmap on the phone's screen, and then you can use that to try and develop the code that draws the text nicely onto a 240px bitmap?

[Weiming-Hu]

Hi @gfwilliams , I have committed some changes here.

Could you check whether this is on the right track? There are no errors on both the Android and the watch. Could you use whether you can draw the bodyImg?

Another issue is that, I seem to get FIFO_* errors, probably due to memory exhaustion, when I set the height to more than 10. Do you think we will be able to show at least some information within that limited space?

Thanks.

[gfwilliams]

This looks promising, but the bitmap format isn't right at all. Did you try just using the code I posted above?

As far as I can tell what you've got is:

• bb.putInt puts in 4 bytes of data for each element, when you only want one
• bb.put is for each pixel, so you're using one byte per pixel, not one bit.

So the image won't get interpreted correctly, but even if it was it's using 8x more data than it needs to, which probably explains the FIFO error (although usually that shouldn't happen if you have a watch face that doesn't take too long to draw).

I guess Gadgetbridge could actually implement some flow control for sending data - it's just that to date that hasn't been required

[Weiming-Hu]

Hi Gordan, thank you for your suggestions. Previously, I didn't quite understand your code. After your explanation, it makes sense to me now.

I have pushed a now commit here. I have tested the data string with the emulator and it is showing up correctly! Well, for now, it just simply print the body text. But I think the string is properly formatted now.

If you have some time, please change the gbridge code on JS so that it draws the bodyImg if it exsist. And then, I can design how to put the title, content, and other information in a prettier layout.

Thanks.

[gfwilliams]

Great - thanks! I'm a bit bust right now but I'll see about adding something as soon as I can

[ponderingm]

Hi.
I got a banglejs2 on a kickstarter.
I would like to display the notification in Japanese.

Please tell me how to use this solution.

[gfwilliams]

Hi - right now this is not something that'll work as-is, however we'll be doing an updated version of Gadgetbridge and I plan to include this in that.

We already have a lot of features (adding images inside text strings, gadgetbridge flow control) that will make this much easier to get going now.

[thyttan]

Candidate for being closed - leave a comment if you object!

Rationale:
I'm not sure but I suspect the changes to gadgetbridge mentioned above will be done with this PR to gadgetbridge?

I know there is the option "Text as Bitmaps" in gadgetbridge - but I don't think this is the same as sending and drawing the whole message as one bitmap image as discussed in the comments above.

Conclusively I believe the issue should remain open - but if I misinterpret the "Text as Bitmaps" option it should maybe be closed after the PR to gadgetbridge gets merged.

[gfwilliams]

Thanks - yes, this can be closed now - Text as Bitmaps solves this and has been merged for months now.

While we don't render the whole image, we render any words that contain unrenderable characters - which saves a bit of space and also allows us to wrap text correctly and stuff like that.

There may still be an issue with right-to-left text rendering which I guess might need solving by just rendering text in chunks of ~160px but nobody has complained about that yet

just a note on flow control: it's not really needed for this to work - it'll just make things more reliable especially if there are messages with lots of unknown text and a slow clock face