Graphics.getPixel(x,y) on ILI9341 Module for ILI9341 LCD CONTROLLER

[espruino-discuss2]

Posted at 2014-10-22 by @allObjects

I tried to use the .getPixel(x,y) to save areas of the display to later restore...

Reading from any location I always get 0 - even for locations I just set, and can actually see the just set pixel.

I know that the boards are connected properly (Espruino outputs and inputs to breakout board inputs and outputs). I also know that the boards communicate properly because can write and read from the board (for example, .setColor() and .getColor() work). I use the breakout board from http://www.adafruit.com/products/1770

Why do I get just 0 back with .getPixel()?

PS: 'looked around' and found some low level reading... about Highly optimized ILI9341 (320x240 TFT color display) at http://forum.pjrc.com/threads/26305 I'm sure this would all hide behind the Graphics.readPixel(x,y) JavaScript implementation. Since a single pixel is usually not of real use, the read of a rectangle and related (re-)write into and from, for example, a uInt16 like buffer (streaming in to and out of an SD card for larger areas), is the desired function. To make it w/ 16 bits, two colors have to be expanded/collapsed by a bit to make that happen, and from what I can read from the controller's datasheet, it may even be supported by the controller...). I'm not ready yet to take on such hardware close calls... but it could happen sooner with some help from any Mr or Mrs Geek G. Shoehorn.... ;-)

[espruino-discuss2]

Posted at 2014-10-23 by DrAzzy

The module doesn't buffer the screen (it wouldn't fit in memory) - so it has to ask the display for the color of a pixel. But no code to do that is present in the module.

See the module - it has fillRect and setPixel (which I figure must be called in turn by other methods of Graphics), but no getPixel. http://www.espruino.com/modules/ILI9341.js

setColor and getColor don't involve any communication with the display.

[gfwilliams]

Posted at 2014-10-23 by @gfwilliams

As @drazzy says, all drawing is done by Espruino (hence getColor) but is written direct to the screen.

As far as I know that module set up so it doesn't even need the MISO input. You could definitely add it though - shouldn't be too painful.

It sounds like you might want to make your own version that reads whole blocks anyway.

I'm not sure what you're trying to do, but if you want to save stuff for later use I'd consider creating an ArrayBuffer graphics and rendering it on Espruino in the first place. It'll be a lot faster.

[espruino-discuss2]

Posted at 2014-10-23 by JumJum

Some time ago I played around the ILI9341 and added a lot of functions.
GetPixel is done by this, and returns an array with r/g/b values.

getPixel = function(x,y){
    var c,z = 0xff;
    setRect(x,y,x+1,y+1);
    writeCMD(0x2E);
    dc.write(1);
    spi.send(z);
    var zzz = new Uint8Array([0,0,0]);
    c = spi.send(zzz);
    ce.write(1);
    return c;
  }

To read colors of a rect would be very timeconsuming. Translating the RGB-Array to a color used for ILI9341 takes a lot of time. Some lines in assembler speeded it up, but its still slow. I don't have a tutorial for that, could only send sources, if somebody is interested.

[espruino-discuss2]

Posted at 2014-10-23 by @allObjects

@drazzy,

exactly what I expected, but the ILI9341 controller chip has the memory and the ability to return information from it.

When inspecting the return object from var lcd = require("ILI9341").connect() (just typing lcd in command pane), I see iSetPixel and iFillRect functions listed... and yes, I would have seen also something like iGetPixel (and/or what I would like even more: iGetRect). I though wonder where all the other methods are that I can use with lcd object... looked at the modules code, and see the Graphics.createCallback(...). So I inspect global Graphics (Singleton) object, and get single function with [native code] back... so I conclude that there is something hybrid/tricky in the interpreter which is invisible/uninspectable from the outside..., which then let's the lcd object (as the returned object from Graphics.createCallback()) understand all the Graphics methods... last but not least also the getWidth() and getHeight(), which return just the LCD_WIDTH and LCD_HEIGHT constants passed to Graphics.createCallback() method in the first place. I guess that the literally passed 16 in that very same method tells the chip to use 16 bit color input mode... (glancing over http://www.newhavendisplay.com/app_notes/ILI9341.pdf).

@gordon,

Regarding creating my own version (or an extension) is the way to go for reading whole blocks.

Regarding what I'm trying to do is save and restore the areas overridden by the (about) 20x20 px calibration marks after using them in the calibration module - one at the time, or some (simple) drop down menu.

For (re)calibration I need save-and-restore only when I have drawn the screen already. If I (re)calibrate before the app draws the screen, I do not need it. Also, I think, I may not need re-calibration after all, because the touch detection accuracy is sufficient for what I want to do in the application: +-15..20 px when using the finger tip, +-5..10px with stylo, or in terms of drawn button sizes: 30..40 px minimal and 10..20 px, respective.

For a simple drop down menu, I need of course to save and restore a larger area.

The idea of an memory rendered element is not flying, because the area to backup and restore is changing along the application and a redrawing becomes practically impossible... and if made possible, the extra memory (to keep states) and hold the code to support that would become a problem as well - and with that the speed would not be faster either, because drawing just a section - a display section unrelated to the application section - is impossible, and therefore, the whole display would have to be redrawn (think of a PacMan game board state after some time after the start of the game...).

No matter what, save and restore would have to stream to a processor-external storage - SD card or (serial) RAM chip (does not need persistence), (default) Espruino's (48K) memory is just too small after all...

'We' really need the option of a board with a chip with larger/largest memory... because I feel the cpu power of the chip and amount of memory are not balanced: memory is not at par with what amount of JavaScript application code can be crunched - a VM w/ garbage collection needs by definition just way more memory than a native straight forward programming with C-like language as Arduino and most of the other micro controllers (may be you have some good news there) - yes, price is an item... but any other additional chip to alleviate the issue adds cost AND hassle beyond what the larger chip costs, especially when you want to do some serious projects with it, an not just educational pet/bricollage/t(h)inkering projects (no insult here). Look at the bare BASIC Stamp... and its price... I would rather make learners use JavaScript than BASIC (or C) - Web technologies are omnipresent, after all (would like to say: 'before' all). Btw, what is the lot size you (have to) order from a manufacturer? I know that 'socketing' pulls it right away 'out of the socks' price-wise - if even possible, and hand-soldering for an average dude like me is not really an option... (the times where I needed only my bare eyes to actually see and do things the size the processor legs are have passed a while ago...). I messed around with various Arduino's, but when I came across Espruino, the 5-times price was not a question at all, because of JavaScript and the processing power (and of course, what you made out of the idea of combining a powerful micro controller and JavaScript... you see the 'softy' in me...). Also: Espruino - not any Arduino - got me really going again with my micro controller t(h)inkering projects after my initial playing around with ubicom.com / Parallax 's SX a while ago.

@jumjum, thank you for the code snippet. It will get me going... ;-) ...btw, what is setRect() function looing like and doing?

[gfwilliams]

Posted at 2014-10-23 by @gfwilliams

Ahh - you haven't included setRect in that - but it might be in another forum post :)

You should also be able to use a larger 'setRect', and then a bigger array for zzz, and then you can load in a whole load of pixels at once.

[espruino-discuss2]

Posted at 2014-10-23 by JumJum

I have a ILI9341_extended module, including a tiny window implementation.
setRect defines the Rect, which is used for reading and writing.

int(int,int,int)
 [0xBFE1,0x08C0,0x0889,0x08D2,0x02C0,0xEA4F,0x1141,0x4408,
  0x4410,0x4601,0xBFE1,0x0200,0x0A09,0x4408,0xF64F,0x71FF,
  0xEA00,0x0001,0x4770]

To get it running, you need attached module and the assembler part loaded using the name ILIcalc. Please see http://www.espruino.com/Assembler for more information.

Attachments:

• ILI9341_extended.js

[espruino-discuss2]

Posted at 2014-10-23 by @allObjects

@jumjum, parsed through your posts - which not necessarily expose ILI9341 relationship in title... nice 'Goldgrube'. The examples - and the provided extension - give me the opportunity to stand on giant's shoulders. Especially intriguing is the windowing aspect... last but not least to make sure a 'display' does not get messed up (...and did you have scrolling in mind?).

[espruino-discuss2]

Posted at 2014-10-23 by JumJum

There are some items which sound like scrolling,in ILI9341-commands, but I did not get them running.
Some other commands sound like overlay, but same problem, did not get it running.
If you find anything like examples for other boards, please let me know.

[espruino-discuss2]

Posted at 2014-10-25 by @allObjects

I ventured into the ILI9341 datasheet and - of course - some coding. First I did an analysis of the existing module... I hope I got the comments right.. I added also getPixel(), getRect()angle, setRect() - for saving and restoring on overlay, and a helper method to convert from 3 byte 6 bit <<2 pixel data from reading a pixel w/ getPixel() to writing using setPixel()*. Code - ready to be loaded into the editor, a shot and a clip are attached to the post.

The attached clip shows you the behavior:

1. LCD initializes, a border 1px border is drawn, and Hello Espruino! is printed.
2. Top left a green, 20x20 px rectangular is drawn (filled).
3. A 20x20 px 'save' is taken from the area between Hellow and Espruino
4. A red , 20x20 px rectangular is drawn, just overriding the saved area
5. Task and time is printed (save: 495 [ms])
6. Red rectangular overridden area is restored
7. Task and time is printed (restore: 1 [ms])

Attached is the inline file that can be loaded right into the editor.

// ILI9341E_DOCD_inline.js

/* Copyright (c) 2013 Juergen Marsch and Gordon Williams, Pur3 Ltd. See the file LICENSE for copying permission. */
/*
Module Enhanced for the ILI9341 LCD controller

Just:

SPI1.setup({sck:B3, miso:B4, mosi:B5, baud: 1000000});
var g = require("ILI9341").connect(SPI1, B6, B8, B7, function() {
  g.clear();
  g.drawString("Hello",0,0);
  g.setFontVector(20);
  g.setColor(0,0.5,1);
  g.drawString("Espruino",0,10);
  // --- "Enhacements"
  g.getPixel(x,y) -> [R,G,B] each 6 bits << 2 (white = 252,252,252)
  g.pxToInt([R,G,B] (from getPixel()) -> int color for setPixel(x,y,color)
  g.getRect(x1,y1,x2,y2); --> Uint16Array of [colors,...] 
  g.setRect(x1,y1,x2,y2,rect); // rect from getRect() 
});

*/
var LCD_WIDTH = 240;
var LCD_HEIGHT = 320;

var ILI9341 = {connect: function(spi, dc, ce, rst, callback) { // inline for next line
// exports.connect = function(spi, dc, ce, rst, callback) { 
  
  // about SPI (see http://www.espruino.com/Reference#SPI):  
  // d is data1, p is data1 or pin, w/ data1 is int|string|array|{data:e, count:#} and 
  // e is data2, w/ data2 is int|string|array
  // - spi.write(d,...,p); // sends LSBytes of all d, and if last arg is pin, pin held 0  
  // - spi.send(e,p);      // sends LSBytes of e or all in e, if p present, pin p held 0
  
  function writeCMD(c){  // --- write command (cmd) only
    ce.write(0);         // chip enable
    spi.write(c,dc);     // spi write cmd w/ dc data/cmd pin held cmd mode (0)
  }

  function writeCD(c,d){ // --- write command w/ data/parms
    ce.write(0);         // chip enable
    spi.write(c,dc);     // spi write cmd w/ dc data/cmd pin held cmd mode (0)
    spi.send(d);         // spi send data/parms
    ce.write(1);         // chip disable
  }

  var LCD = Graphics.createCallback(LCD_WIDTH, LCD_HEIGHT, 16, {

    // set pixel is like fill rectangle w/ [x1/y1=x,y inclusive and x2/y2=x+1,y+1) exclusive
    setPixel:function(x,y,c){   // x, y, c(olor)/image data for 1px
      ce.reset();               // chip enable
      spi.write(0x2A,dc);       // cmd 0x2A - x/col addr - w/ dc for cmd (0) 
      spi.write(x>>8,x,x>>8,x); // x [Start Point MSByte, LSByte, then End Point)
      spi.write(0x2B,dc);       // cmd 0x2B - y/row/page addr - w/ dc for cmd (0)
      spi.write(y>>8,y,y>>8,y); // x [Start Point MSByte, LSByte, then End Point) 
      spi.write(0x2C,dc);       // cmd 0x2C - memory write - w/ dc for cmd (0)
      spi.write(c>>8,c);        // 65K 2 bytes color (LSByte 1st, MSB 2nd)
      ce.set();
    },

    fillRect:function(x1,y1,x2,y2,c){ // x1,y1,y1,y2, c(olor)/image data for 1px
      ce.reset();                     // chip enable
      spi.write(0x2A,dc);             // cmd 0x2A - x/col addr - w/ dc for cmd (0)
      spi.write(x1>>8,x1,x2>>8,x2);   // x [Start Point MSByte, LSByte, then End Point)
      spi.write(0x2B,dc);             // cmd 0x2B - y/row/page addr - w/ dc for cmd (0)
      spi.write(y1>>8,y1,y2>>8,y2);   // y [Start Point MSByte, LSByte, then End Point)
      spi.write(0x2C,dc);             // md 0x2C - memory write - w/ dc for cmd (0)
      spi.write({data:String.fromCharCode(c>>8)+String.fromCharCode(c), count:(x2-x1+1)*(y2-y1+1)});
      ce.set();                       // chip disable
    }
  });

  ce.write(1);               // chip disable
  dc.write(1);               // data mode 
  rst.write(0);              // reset
  setTimeout(function(){     // time reset pin low 1[ms] using timeout
    rst.write(1);            // end reset 
    setTimeout(function(){   // time after reset pin back to high 5[ms]
      writeCMD(0x01);        // cmd 01 software Reset
      setTimeout(function(){ // time for software Reset 5[ms]
        writeCMD(0x28);      // cmd 28 Display OFF
        ce.write(1);               // chip disable
        writeCD(0xCF,[0x00,0x83,0x30]);      // CF Power control B 
        writeCD(0xED,[0x64,0x03,0x12,0x81]); // ED Power on sequence control
        writeCD(0xE8,[0x85,0x01,0x79]);      // E8 Driver timing control A
        writeCD(0xCB,[0x39,0x2C,0x00,0x34,0x02]); // CB Power control B
        writeCD(0xF7,0x20);        // F7 Pump ratio control
        writeCD(0xEA,[0x00,0x00]); // EA Driver timing control B
        writeCD(0xC0,0x26);        // C0 Power Control 1
        writeCD(0xC1,0x11);        // C1 Power Control 2
        writeCD(0xC5,[0x35,0x3E]); // C5 VCOM Control 1
        writeCD(0xC7,0xBE);        // C7 VCOM Control 2
        writeCD(0x36,0x48);        // 36 Memory Access Control: row col exch, BGR order
        writeCD(0x3A,0x55);        // 3A COLMOD: Pixel Format Set 65K 16bit R(5) G(6) B(5)
        writeCD(0xB1,[0x00,0x1B]); // B1 Frame Rate Control (in Normal Mode/Full Colors)
        writeCD(0xF2,0x08);        // F2 Enable 3G
        writeCD(0x26,0x01);        // 26 Gamma Set // E0 Positive Gamma Correction // E1 Negative...
        writeCD(0xE0,[0x1F,0x1A,0x18,0x0A,0x0F,0x06,0x45,0x87,0x32,0x0A,0x07,0x02,0x07,0x05,0x00]);
        writeCD(0xE1,[0x00,0x25,0x27,0x05,0x10,0x09,0x3A,0x78,0x4D,0x05,0x18,0x0D,0x38,0x3A,0x1F]);
        writeCD(0xB7,0x07);                  // B7 Entry Mode Set
        writeCD(0xB6,[0x0A,0x82,0x27,0x00]); // B6 Display Function Control
        writeCMD(0x11);                      // 11 Sleep Out
        ce.write(1);               // chip disable 
        setTimeout(function(){        // time for Sleep Out 100[ms]
          writeCMD(0x29);ce.write(1); // 29 Display ON
          setTimeout(function(){      // time for DISPLAY ON 100[ms]
            if (callback!==undefined) callback(); // application callback
          },100); // 100[ms] for switching DISPLAY ON
        },100);   // 100[ms] for Sleep Out
      },5);       //   5[ms] for Software Reset
    },5);         //   5[ms] time after reset pin back to high
  },1);
  
  LCD.getPixel = function(x,y) {
    ce.reset();               // chip enable
    spi.write(0x2A,dc);       // cmd 0x2A - x/col addr - w/ dc for cmd (0) 
    spi.write(x>>8,x,x>>8,x); // x [Start Point MSByte, LSByte, then End Point)
    spi.write(0x2B,dc);       // cmd 0x2B - y/row/page addr - w/ dc for cmd (0)
    spi.write(y>>8,y,y>>8,y); // x [Start Point MSByte, LSByte, then End Point) 
    spi.write(0x2E,dc);       // cmd 0x2E - memory red - w/ dc for cmd (0)
    spi.write(0x00,0x00,0x00);
    var p = spi.send([0x00,0x00,0x00]);
    ce.set();
    return p;
  };
  
  LCD.pxToInt = function(p) { return ((p[0]>>3)<<3)+(p[1]>>5)*256+(p[1]<<3)%256+(p[2]>>3); };

  LCD.getRect = function(x1,y1,x2,y2) {
    var m = 0, n = (x2-x1+1)*(y2-y1+1)*2, r = new Uint8Array(n);
    var xfr = function(p) {
      r[m++] = ((p[0]>>3)<<3)+(p[1]>>5); r[m++] = (p[1]<<3)%256+(p[2]>>3); };
    ce.reset();                   // chip enable
    spi.write(0x2A,dc);           // cmd 0x2A - x/col addr - w/ dc for cmd (0) 
    spi.write(x1>>8,x1,x2>>8,x2); // x [Start/Ent Pts MSByte, LSByte)
    spi.write(0x2B,dc);           // cmd 0x2B - y/row/page addr - w/ dc for cmd (0)
    spi.write(y1>>8,y1,y2>>8,y2); // x [Start/End PTs MSByte, LSByte) 
    spi.write(0x2E,dc);           // cmd 0x2E - memory read - w/ dc for cmd (0)
    spi.write(0x00,0x00,0x00);    // dmy reads
    while (m < n) { xfr(spi.send([0x00,0x00,0x00])); } // a px at a time incl conv to Uint16
    ce.set();
    return r;
  };
   
  LCD.setRect = function(x1,y1,x2,y2,r) {
    ce.reset();                   // chip enable
    spi.write(0x2A,dc);           // cmd 0x2A - x/col addr - w/ dc for cmd (0) 
    spi.write(x1>>8,x1,x2>>8,x2); // x [Start Point MSByte, LSByte, then End Point)
    spi.write(0x2B,dc);           // cmd 0x2B - y/row/page addr - w/ dc for cmd (0)
    spi.write(y1>>8,y1,y2>>8,y2); // x [Start Point MSByte, LSByte, then End Point) 
    spi.write(0x2C,dc);           // cmd 0x2C - memory write - w/ dc for cmd (0)
    spi.write(r);
    ce.set();
  };  
  return LCD;
 } // inline
};


SPI1.setup({sck:B3, miso:B4, mosi:B5, baud: 1000000});
var lcd = ILI9341.connect(SPI1, B6, B8, B7, function() { // inline for next line
// var lcd = require("ILI9341").connect(SPI1, B6, B8, B7, function() {
  B2.set();
  lcd.clear();
  lcd.setColor(0.0,1.0,1.0);
  lcd.drawRect(0,0,lcd.getWidth()-1,lcd.getHeight()-1);
  lcd.setColor(1.0,1.0,1.0);
  lcd.setFontVector(20);
  lcd.drawString("Hello Espruino!",35,5);
  lcd.setFontVector(15);
  var r,t;
  lcd.setColor(0.0,1.0,0.0);
  lcd.fillRect(1,1,20,20);
  t = getTime();
  r = lcd.getRect( 85,  0,104, 19);   // get/save rectangle for later restor
  console.log(t = getTime() - t,"[s]  bytes:",r.length," memory",process.memory().usage);
  lcd.setColor(1.0,1.0,1.0);
  lcd.drawString("save 20x20 rectangle:",5,35);
  lcd.setColor(1.0,1.0,0.0);
  lcd.drawString(Math.round(t*1000) + " [ms] 1200-->800 bytes",5,55);
  setTimeout(function(){
    lcd.setColor(1.0,0.0,0.0); 
    lcd.fillRect( 85,  0,104, 19);    // override saved rectangle area
    setTimeout(function(){
      t = getTime();
      lcd.setRect( 85,  0,104, 19,r); // restore rectangle area
      console.log(t = getTime() - t,"[s]  bytes:",r.length," memory",process.memory().usage);
      lcd.setColor(1.0,1.0,1.0);
      lcd.drawString("restore 20x20 rectangle:",5,75);
      lcd.setColor(1.0,1.0,0.0);
      lcd.drawString(Math.round(t*1000) + " [ms] <--800 bytes",5,95);
    },1000);
  },1000);
});

Most difficult part in getRect() and setRect() for save and restore (or copy) of an area was the different pixel format: it is read with 3 6bit bytes and a left shift of <<2 (White is 252,252,252). Writing back requires two Uint8 that cover 5 bits read, 6 bits green (straddling the bytes), and 5 bits blue. After a while I got the hang of it.

You can use the color values to set the color and then call setPixel with no color argument... but that's super strolly...

var px = lcd.getPixel(x,y); // --> array [R,G,B], 6 bit each with left shift of <<2
lcd.setColor(px[0]/252,px[1]/252,px[2]/252);
lcd.setPixel(x,y);

The method .pixToInt(px) in line 126 helps to get this a bit easier (and is borrowed from getRect() method (line 133) - a slow version of ILICalc in assembler from @jumjum ;), figured that out in hindsight.

Initially I just read all the bytes in one swoosh by creating and sending a 0x00 initialized Uint8Array --- and wasted lots of memory... 4 or more bytes per pixel... and could read only 10x10 before running out of memory. Also, I kept the read format until restoring and did above mentioned conversion from 3 to 2 bytes when restoring: Another tie up of additional 50% memory. So I came up with the one px at a time (which works quite well - form me - but could be optimized by some bulking algorithm), and do the conversion right when reading.. Now, only the bare minimum on memory is used. This seems to work for my purposes. But I'm looking int adding some serial RAM (flash is an option too, but sucks in power... on the other hand it's on the board...) I was also thinking of using the controller's 'porches' outside of the visual area - if there is memory left; but I did not go into detail thoughts yet).

The approach of reading just a pixel and do the conversion turned out not much slower than reading and keeping all data memory efficiently, but the write is now a very quick one:

1. time 566 [ms] is the read time for 20x20 area... 1200 Bytes packed into 800 to be keept until restore.
2. time of 14 [ms] is the restore / write time.

 1v70 Copyright 2014 G.Williams
>echo(0);
=undefined
0.56678676605 [s]  bytes: 800  memory 1315
0.01441287994 [s]  bytes: 800  memory 1312
>

I hoped to get more out of it when making it a module and using minification: Speed was insignificant faster... 12% (last but not least removed the inline/infunction comments), but memory looks great:

 1v70 Copyright 2014 G.Williams
>echo(0);
=undefined
0.49496555328 [s]  bytes: 800  memory 678
0.01411247253 [s]  bytes: 800  memory 682
> 

I had also some difficulties with reading from the controller via SPI, because for each byte to read, a byte has to be sent (?) with SPI.send(). Unfortunately, SPI.send() is not supporting the object {data:data, count#} parameter as the write does... and the SPI.write - which supports it - does not return anything... So my suggestion is that the send() should support the object parameter option... the count would have made it really easy: repeatedly sending 0x00 (send is always a 0x00)

Small changes to the existing module are possible: when defining the rectangle (column and page address), the rectangle can have an extend of 0: no need to add 1. And as said, the spi.send could use the object parameter option... (and a native map.reduce function...).

@jumjum, I did not venture into assembler yet...

---

Attachments:

• ILI9341_DOCD_inline.js
• saveRestore20x20rectangle.mp4
  
• saveRestore20x20rectangle.png