optimize publish()

[hmueller01]

Is it needed that we copy the payload to the buffer?

pubsubclient3/src/PubSubClient.cpp

Lines 439 to 442 in 989de7f

	// Add payload
	for (size_t i = 0; i < plength; i++) {
	this->buffer[length++] = payload[i];
	}

instead we could just use

write(payload, plength);

and write the header / topic stuff before ... (e.g. by using the beginPublish() function).

This way we could send longer messages without the need to increase the buffer size.

[TD-er]

I do have something similar in my version of this library in ESPEasy. See: https://github.com/letscontrolit/ESPEasy/blob/60a0ce320a37eed05dbc360e22b73f497a75319d/lib/pubsubclient/src/PubSubClient.h#L172-L183

I have added a separate function to stream data. This works fine with messages upto 30-ish kB. (Larger messages seemed to perform less efficient compared to multiple smaller ones, so it was not like I hit the max. message size)

So my suggestion is to add that later in a different PR, as using a stream is making more sense than adding it here.
You still need a relative large buffer for receiving messages, so it is not like we can reduce the buffer size to just header + topic.

Also some platforms require special functions for pointers directing to different memory areas.
For example Flash Strings on ESP8266.
Not sure if all client class implementations are made aware of those requirements.

[hmueller01]

Wow, you already did lots of things I also had in mind. 👍

[TD-er]

I browsed through the GitBlame to find the commits I made for writing stuff to the client...

See: letscontrolit/ESPEasy@541f382

Added extra functions named appendBuffer to send data to the client between beginPublish and endPublish.

[hmueller01]

@TD-er
True, we need a receiving buffer for the payload and can not reduce the puffer to only header+topic.
But this was not the intention. It was just to broadcast (publish) big payloads without the need to have these buffers twice (in the application and in PubSubClient). So letting the client handle the sending of the original buffer seem to me the better solution.

I havn't found it in WiFiClient, but in WiFiClientSecure it is done like this
https://github.com/esp8266/Arduino/blob/587435110f03a3ad53a0cc7144387a353f478521/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp#L297C1-L336C1

The issue here was only about publish(), not publish_P(). In the later we need a better solution. See #36. Or didn't I get your point yet?

[TD-er]

The point of my previous post here was about giving the user some options to just generate any payload data and send it immediately without having to allocate huge buffers.

The use case I made it for is to 'dump' prerecorded measurements in large MQTT messages whenever the ESP device returns back to some location where it has WiFi.
So I know in advance exactly how much data I will be sending in that message and then send it in a time-efficient way.
First I sent it in a publish message per sample. After switching to sending it in bulk, about 30k per message, it was roughly 1000x faster to upload the data.
However as you can imagine, allocating 30k (or more...) is not really an option. Especially not on ESP8266.

So I just have to call beginPublish, then appendBuffer to directly send data to the broker and then endPublish.

This is also how I create my web pages in ESPEasy. I have a buffer object which streams the data in chunks and all code to generate web pages is sending to this buffer.
This way I don't need to allocate anything other than some chunk sized buffer and pages load extremely fast.

[hmueller01]

Looking again in your posted commit, I think I understand now. You are using e.g.

beginPublish("topic", plen1 + plen2 + 1, false);
write(buf1, plen1);
write(buf2, plen2);
write('a');
endPublish();

And this is faster using your appendBuffer() change, as it writes to the broker only if the send buffer is full.
Ok. This is a good point and why not using the anyway reserved memory buffer. But I think it's out of scope of PR #31.

Do you plan to use PubSubClient3 in your letscontrolit/ESPEasy in the future?
If so, it would be good if you open a PR with the change.

[TD-er]

Do you plan to use PubSubClient3 in your letscontrolit/ESPEasy in the future?
If so, it would be good if you open a PR with the change.

Yep, that's why I accepted your request to cooperate on this fork :)

I will later make a PR for those changes to allow making arbitrary large publish calls.
I think it also requires some test code to explain how it is intended to be used.

[hmueller01]

I'd like to merge all the open PRs an do based on that further changes (like #34). So it'll be good it you approve the PRs if ok from your side. Then you also have a better base to make your PR ...