Reading the complete content of an HTTP GET request

[espruino-discuss]

Posted at 2015-10-26 by Kolban

In this puzzle, my goal is to issue an HTTP GET request and when the complete response has arrived, process it.

Looking at the APIs, I see:

http.get(options, function(response) {
   // handle response here
});

In the response handler, I am given an object of type httpCRs (I believe). However, if I try and print:

response.available()

I am returned a value of 0. I figured that was ok, because at this point, in execution, data had not yet arrived and there was indeed none available.

So I figured that I could code up the following:

response.on("close", function() {
   print("Available = " + response.available());
});

However something seems to be wrong here. The "close" event function is not being called. By sheer luck, I found that if I coded:

response.on("data", function() {
   print("We have received some data");
});
response.on("close", function() {
   print("Available = " + response.available());
});

I then find that the code event callback is being invoked ... but of course, I have consumed my data.

So it seems that if I code a response.on("close"...) without a response.on("data"...) the close callback is not being invoked.

Am I missing a concept here?

[espruino-discuss]

Posted at 2015-10-26 by tve

The client does not close an HTTP connection after sending the request. Specially not with HTTP 1.1. You have to check whether there's a Content-Length header and if there is, read that number of bytes from the socket as payload. (If the encoding is chunked you have to do the equivalent using the chunk length prefixes.)
If the there is no Content-Length header or the value is zero then you've reached the end of the request when you see \r\n\r\n.

[gfwilliams]

Posted at 2015-10-26 by @gfwilliams

Espruino's HTTP GET does close the socket right after requesting a page (it tests via content length)... Adding the ability to leave the socket open to request more stuff just seemed like overkill.

Can you not just do:

var data = "";
response.on("data", function(d) {
   data+=d;
});
response.on("close", function() {
   print("Available = " + data);
});

That's what is shown in all the examples...

But you're saying that if you don't have a data handler, you don't get close called? That could be a bug.

Thing is, even if it worked, doing what you're doing would be a really bad idea - so as not to fill up Espruino's memory if you're not reading data, serial/sockets/etc only buffer ~500 bytes worth of data, and chuck away the rest. You're best off having a data handler, and manually saving away the data as it arrives.

[espruino-discuss]

Posted at 2015-10-26 by Kolban

What I'm trying to do is "play" end user. So I go to the API docs for httpSRq and found the read function.

http://www.espruino.com/Reference#l_httpSRq_read

The few words here say that when read is called it returns the characters that have been received but not yet consumed.

Then I pair that with the close function ... which is a callback event called when the partner has closed the connection. My assumption was that the data received but not yet consumed could be consumed at that time.

I am sensing that this is not a valid assumption ... and if that is the case, what I'd like to do is document that consideration so that future travelers won't be led down this path.

[gfwilliams]

Posted at 2015-10-26 by @gfwilliams

Yes, sounds like a good plan.

However if you look at the docs for http.get there's a code example right there. There are also a bunch of examples linked that do the right thing, there's also the 'Internet' page, ESP8266 page, and pretty much any tutorial that does an HTTP GET.

It's presumably just if you look at the Reference page itself for HttpCRs (in isolation) while ignoring the code under http.get that you get the wrong impression? Perhaps the docs for HttpCRs could just link back to get and request?