size limit with GetRequestContent

[philippeminerve]

I'm looking for parsing big json request and find a limit in GetRequestContent().

Here is the code used :

def _httpHandlerLEDPost(httpClient, httpResponse):
print("we pass here")
print("content type:",httpClient.GetRequestContentType())
print("Size:",httpClient.GetRequestContentLength())
content = httpClient.ReadRequestContent() # Read JSON color data
print("Size content:",str(len(content)))

....

Here is the trace :

we pass here
content type: application/json
Size:: 10151
Size content: 4308

I have tried ReadRequestContentAsJSON() but unfortunatly no way to get my data according to the use of the same methode ReadRequestContent.

[jczic]

Do you use threading mode ?
May be a ram limitation by the thread for that.

[philippeminerve]

No I do not use Threading mode.

[jczic]

Ok.
Can you try to open your JSON as a file with (u)json.loads or load and check if an error occurs ?

[philippeminerve]

You are right, I try this code and the memory problem occurs :

aLongString= '{"matrice":[{"L":0,"R":0,"V":0,"B":0},{"L":31,"R":0,"V":0,"B":0},{"L":32,"R":0,"V":0,"B":0},{"L":63,"R":0,"V":0,"B":0},{"L":64,"R":0,"V":0,"B":0},{"L":95,"R":0,"V":0,"B":0},{"L":96,"R":0,"V":0,"B":0},{"L":127,"R":0,"V":0,"B":0},{"L":128,"R":0,"V":0,"B":0},{"L":159,"R":0,"V":0,"B":0},{"L":160,"R":0,"V":0,"B":0},{"L":191,"R":0,"V":0,"B":0},{"L":192,"R":0,"V":0,"B":0},{"L":223,"R":0,"V":0,"B":0},{"L":224,"R":0,"V":0,"B":0},{"L":255,"R":0,"V":0,"B":0},{"L":1,"R":0,"V":0,"B":0},{"L":30,"R":0,"V":0,"B":0},{"L":33,"R":0,"V":0,"B":0},{"L":62,"R":0,"V":0,"B":0},{"L":65,"R":255,"V":255,"B":0},{"L":94,"R":0,"V":0,"B":0},{"L":97,"R":0,"V":0,"B":0},{"L":126,"R":0,"V":0,"B":0},{"L":129,"R":0,"V":0,"B":0},{"L":158,"R":0,"V":0,"B":0},{"L":161,"R":0,"V":0,"B":0},{"L":190,"R":0,"V":0,"B":0},{"L":193,"R":0,"V":0,"B":0},{"L":222,"R":0,"V":0,"B":0},{"L":225,"R":0,"V":0,"B":0},{"L":254,"R":0,"V":0,"B":0},{"L":2,"R":0,"V":0,"B":0},{"L":29,"R":0,"V":0,"B":0},{"L":34,"R":0,"V":0,"B":0},{"L":61,"R":0,"V":0,"B":0},{"L":66,"R":0,"V":0,"B":0},{"L":93,"R":0,"V":0,"B":0},{"L":98,"R":0,"V":0,"B":0},{"L":125,"R":0,"V":0,"B":0},{"L":130,"R":0,"V":0,"B":0},{"L":157,"R":0,"V":0,"B":0},{"L":162,"R":0,"V":0,"B":0},{"L":189,"R":0,"V":0,"B":0},{"L":194,"R":0,"V":0,"B":0},{"L":221,"R":0,"V":0,"B":0},{"L":226,"R":0,"V":0,"B":0},{"L":253,"R":0,"V":0,"B":0},{"L":3,"R":0,"V":0,"B":0},{"L":28,"R":0,"V":0,"B":0},{"L":35,"R":0,"V":0,"B":0},{"L":60,"R":0,"V":0,"B":0},{"L":67,"R":0,"V":0,"B":0},{"L":92,"R":0,"V":0,"B":0},{"L":99,"R":0,"V":0,"B":0},{"L":124,"R":0,"V":0,"B":0},{"L":131,"R":0,"V":0,"B":0},{"L":156,"R":0,"V":0,"B":0},{"L":163,"R":0,"V":0,"B":0},{"L":188,"R":0,"V":0,"B":0},{"L":195,"R":0,"V":0,"B":0},{"L":220,"R":0,"V":0,"B":0},{"L":227,"R":0,"V":0,"B":0},{"L":252,"R":0,"V":0,"B":0},{"L":4,"R":0,"V":0,"B":0},{"L":27,"R":0,"V":0,"B":0},{"L":36,"R":0,"V":0,"B":0},{"L":59,"R":0,"V":0,"B":0},{"L":68,"R":0,"V":0,"B":0},{"L":91,"R":0,"V":0,"B":0},{"L":100,"R":0,"V":0,"B":0},{"L":123,"R":255,"V":255,"B":0},{"L":132,"R":0,"V":0,"B":0},{"L":155,"R":0,"V":0,"B":0},{"L":164,"R":0,"V":0,"B":0},{"L":187,"R":0,"V":0,"B":0},{"L":196,"R":0,"V":0,"B":0},{"L":219,"R":0,"V":0,"B":0},{"L":228,"R":0,"V":0,"B":0},{"L":251,"R":0,"V":0,"B":0},{"L":5,"R":0,"V":0,"B":0},{"L":26,"R":0,"V":0,"B":0},{"L":37,"R":0,"V":0,"B":0},{"L":58,"R":0,"V":0,"B":0},{"L":69,"R":0,"V":0,"B":0},{"L":90,"R":0,"V":0,"B":0},{"L":101,"R":0,"V":0,"B":0},{"L":122,"R":0,"V":0,"B":0},{"L":133,"R":0,"V":0,"B":0},{"L":154,"R":0,"V":0,"B":0},{"L":165,"R":0,"V":0,"B":0},{"L":186,"R":0,"V":0,"B":0},{"L":197,"R":0,"V":0,"B":0},{"L":218,"R":0,"V":0,"B":0},{"L":229,"R":0,"V":0,"B":0},{"L":250,"R":0,"V":0,"B":0},{"L":6,"R":0,"V":0,"B":0},{"L":25,"R":0,"V":0,"B":0},{"L":38,"R":0,"V":0,"B":0},{"L":57,"R":0,"V":0,"B":0},{"L":70,"R":0,"V":0,"B":0},{"L":89,"R":0,"V":0,"B":0},{"L":102,"R":0,"V":0,"B":0},{"L":121,"R":0,"V":0,"B":0},{"L":134,"R":0,"V":0,"B":0},{"L":153,"R":0,"V":0,"B":0},{"L":166,"R":0,"V":0,"B":0},{"L":185,"R":0,"V":0,"B":0},{"L":198,"R":0,"V":0,"B":0},{"L":217,"R":0,"V":0,"B":0},{"L":230,"R":0,"V":0,"B":0},{"L":249,"R":0,"V":0,"B":0},{"L":7,"R":0,"V":0,"B":0},{"L":24,"R":0,"V":0,"B":0},{"L":39,"R":0,"V":0,"B":0},{"L":56,"R":0,"V":0,"B":0},{"L":71,"R":0,"V":0,"B":0},{"L":88,"R":0,"V":0,"B":0},{"L":103,"R":0,"V":0,"B":0},{"L":120,"R":0,"V":0,"B":0},{"L":135,"R":0,"V":0,"B":0},{"L":152,"R":0,"V":0,"B":0},{"L":167,"R":0,"V":0,"B":0},{"L":184,"R":0,"V":0,"B":0},{"L":199,"R":0,"V":0,"B":0},{"L":216,"R":0,"V":0,"B":0},{"L":231,"R":0,"V":0,"B":0},{"L":248,"R":0,"V":0,"B":0},{"L":8,"R":0,"V":0,"B":0},{"L":23,"R":0,"V":0,"B":0},{"L":40,"R":0,"V":0,"B":0},{"L":55,"R":0,"V":0,"B":0},{"L":72,"R":0,"V":0,"B":0},{"L":87,"R":0,"V":0,"B":0},{"L":104,"R":0,"V":0,"B":0},{"L":119,"R":0,"V":0,"B":0},{"L":136,"R":255,"V":255,"B":0},{"L":151,"R":0,"V":0,"B":0},{"L":168,"R":0,"V":0,"B":0},{"L":183,"R":0,"V":0,"B":0},{"L":200,"R":0,"V":0,"B":0},{"L":215,"R":0,"V":0,"B":0},{"L":232,"R":0,"V":0,"B":0},{"L":247,"R":0,"V":0,"B":0},{"L":9,"R":0,"V":0,"B":0},{"L":22,"R":0,"V":0,"B":0},{"L":41,"R":0,"V":0,"B":0},{"L":54,"R":0,"V":0,"B":0},{"L":73,"R":0,"V":0,"B":0},{"L":86,"R":0,"V":0,"B":0},{"L":105,"R":0,"V":0,"B":0},{"L":118,"R":0,"V":0,"B":0},{"L":137,"R":0,"V":0,"B":0},{"L":150,"R":0,"V":0,"B":0},{"L":169,"R":0,"V":0,"B":0},{"L":182,"R":0,"V":0,"B":0},{"L":201,"R":0,"V":0,"B":0},{"L":214,"R":0,"V":0,"B":0},{"L":233,"R":0,"V":0,"B":0},{"L":246,"R":0,"V":0,"B":0},{"L":10,"R":0,"V":0,"B":0},{"L":21,"R":0,"V":0,"B":0},{"L":42,"R":0,"V":0,"B":0},{"L":53,"R":0,"V":0,"B":0},{"L":74,"R":0,"V":0,"B":0},{"L":85,"R":0,"V":0,"B":0},{"L":106,"R":0,"V":0,"B":0},{"L":117,"R":0,"V":0,"B":0},{"L":138,"R":0,"V":0,"B":0},{"L":149,"R":0,"V":0,"B":0},{"L":170,"R":0,"V":0,"B":0},{"L":181,"R":0,"V":0,"B":0},{"L":202,"R":0,"V":0,"B":0},{"L":213,"R":0,"V":0,"B":0},{"L":234,"R":0,"V":0,"B":0},{"L":245,"R":0,"V":0,"B":0},{"L":11,"R":0,"V":0,"B":0},{"L":20,"R":0,"V":0,"B":0},{"L":43,"R":0,"V":0,"B":0},{"L":52,"R":0,"V":0,"B":0},{"L":75,"R":0,"V":0,"B":0},{"L":84,"R":0,"V":0,"B":0},{"L":107,"R":0,"V":0,"B":0},{"L":116,"R":0,"V":0,"B":0},{"L":139,"R":0,"V":0,"B":0},{"L":148,"R":0,"V":0,"B":0},{"L":171,"R":0,"V":0,"B":0},{"L":180,"R":0,"V":0,"B":0},{"L":203,"R":0,"V":0,"B":0},{"L":212,"R":0,"V":0,"B":0},{"L":235,"R":0,"V":0,"B":0},{"L":244,"R":0,"V":0,"B":0},{"L":12,"R":0,"V":0,"B":0},{"L":19,"R":0,"V":0,"B":0},{"L":44,"R":0,"V":0,"B":0},{"L":51,"R":0,"V":0,"B":0},{"L":76,"R":0,"V":0,"B":0},{"L":83,"R":0,"V":0,"B":0},{"L":108,"R":0,"V":0,"B":0},{"L":115,"R":0,"V":0,"B":0},{"L":140,"R":0,"V":0,"B":0},{"L":147,"R":0,"V":0,"B":0},{"L":172,"R":0,"V":0,"B":0},{"L":179,"R":0,"V":0,"B":0},{"L":204,"R":0,"V":0,"B":0},{"L":211,"R":0,"V":0,"B":0},{"L":236,"R":0,"V":0,"B":0},{"L":243,"R":0,"V":0,"B":0},{"L":13,"R":0,"V":0,"B":0},{"L":18,"R":0,"V":0,"B":0},{"L":45,"R":0,"V":0,"B":0},{"L":50,"R":0,"V":0,"B":0},{"L":77,"R":0,"V":0,"B":0},{"L":82,"R":0,"V":0,"B":0},{"L":109,"R":0,"V":0,"B":0},{"L":114,"R":0,"V":0,"B":0},{"L":141,"R":0,"V":0,"B":0},{"L":146,"R":0,"V":0,"B":0},{"L":173,"R":0,"V":0,"B":0},{"L":178,"R":0,"V":0,"B":0},{"L":205,"R":0,"V":0,"B":0},{"L":210,"R":0,"V":0,"B":0},{"L":237,"R":0,"V":0,"B":0},{"L":242,"R":0,"V":0,"B":0},{"L":14,"R":0,"V":0,"B":0},{"L":17,"R":0,"V":0,"B":0},{"L":46,"R":0,"V":0,"B":0},{"L":49,"R":0,"V":0,"B":0},{"L":78,"R":0,"V":0,"B":0},{"L":81,"R":0,"V":0,"B":0},{"L":110,"R":0,"V":0,"B":0},{"L":113,"R":0,"V":0,"B":0},{"L":142,"R":0,"V":0,"B":0},{"L":145,"R":0,"V":0,"B":0},{"L":174,"R":0,"V":0,"B":0},{"L":177,"R":0,"V":0,"B":0},{"L":206,"R":0,"V":0,"B":0},{"L":209,"R":0,"V":0,"B":0},{"L":238,"R":0,"V":0,"B":0},{"L":241,"R":0,"V":0,"B":0},{"L":15,"R":0,"V":0,"B":0},{"L":16,"R":0,"V":0,"B":0},{"L":47,"R":0,"V":0,"B":0},{"L":48,"R":0,"V":0,"B":0},{"L":79,"R":0,"V":0,"B":0},{"L":80,"R":0,"V":0,"B":0},{"L":111,"R":0,"V":0,"B":0},{"L":112,"R":0,"V":0,"B":0},{"L":143,"R":0,"V":0,"B":0},{"L":144,"R":0,"V":0,"B":0},{"L":175,"R":0,"V":0,"B":0},{"L":176,"R":0,"V":0,"B":0},{"L":207,"R":0,"V":0,"B":0},{"L":208,"R":0,"V":0,"B":0},{"L":239,"R":0,"V":0,"B":0},{"L":240,"R":0,"V":0,"B":0}]}'
print("a Long String:",len(aLongString))
aLongJsonLong = ujson.loads(aLongString)

Result :

a Long String: 7083
Traceback (most recent call last):
File "", line 35, in
MemoryError:

It seems that converting a sting into Json takes more memory than possible. I will look that way. Thanks for your precious help

[jczic]

Feel free to keep me informed 👍

[philippeminerve]

To conclude, I change all my javascript code to reduce size of data but unfortunately JSON is really a problem with large amount of data above 4000 caracteres (4Mo ESP32 is not enough and memory crash).
To be able to go ahead with my 256 LEDs, I choose to use POST with content type: text/plain instead of content type: application/json.

My data are, now : 000,000000000&031,000255000&032,000000000
First 3 caracteres for LED number, "," as field separator and 9 caracteres to produce Red, Green and Blue colors.

Now my data have reach 3583 lenght and everything is done by parsing the string file without any JSON function.

I close the issue and really thanks you for reactive help !
Thanks to share and to be so kind !

[jczic]

Indeed.... 4000 is however low but it is not so much the number of JSON characters but rather the number of nodes.
However, with 4MB it seems crazy to me.
Perhaps the processes are each limited in SDRAM...
I will test this on my OEM Pycom board and come back to give you feedback:)
Good luck with the rest:)

[philippeminerve]

The 4000 limit is really the number of json characters cf.JSON post in my previous comment. My projet use 256 LED and each one was only one node in the previous JSON.
My project is now working and I can change each LED one by one via a javascript grid in a webapp - HAPPY.

Feel free to share your feedback after your Python board test, i'm still astonish about this crazy limit ^^

[jczic]

Hello, I released a fully new version (v2.0) of my web server here : github.com/jczic/MicroWebSrv2. Open source MIT, fully asynchronous, more robust, more fast and more efficient! It is delivered with a good documentation. Thank you for your support and feedback. ☺️ Jean-Christophe Bos Le mar. 5 févr. 2019 à 23:01, philippeminerve <[email protected]> a écrit :

…

The 4000 limite is really the number of json characters. My projet use 256 LED and each one was only one node in the previous JSON. My project is now working and I can change each LED one by one via a javascript grid in a webapp - HAPPY. Feel free to share your feedback after your Python board test, i'm still astonish about this crazy limit ^^ — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#51 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAegLL9oHSgDy7DlO8ONntwIH7Z8Sob-ks5vKf9TgaJpZM4ajEzd> .

-- *Sincèrement,*