feat: Call provide endpoint in batches.

[ajnavarro]

This will avoid huge provide payloads.

It closes #55

Signed-off-by: Antonio Navarro Perez antnavper@gmail.com

[lidel]

Quick nits:

1. batch size based on item count is not enough, we may need second one based on byte size (see comment below why)
2. server side is does not enforce batch size limits, which is a DoS vector
3. limits should be documented in specs (ipfs/specs#337)

[lidel]

nit (1): this comes with various assumptions about average size of a single item, which will quickly get outdated when we have WebTransport enabled by default (/webtransport multiaddrs with two /certhash segments will baloon the size of the batch beyond initial estimate), or add more transports in the future.

In other places, such as UnixFS autosharding, we've moved away from ballpark counting items assuming they are of some arbitrary average size, and switched to calculating the total size of the final block.

Thoughts on swithcing to byte size, or having two limits? ProvideBatchSize (current one) and ProvideBatchByteSize (new one)

[ajnavarro]

nit (1): this comes with various assumptions about average size of a single item, which will quickly get outdated when we have WebTransport enabled by default (/webtransport multiaddrs with two /certhash segments will baloon the size of the batch beyond initial estimate), or add more transports in the future.

Then we need to add a specific limit on the amount of multiaddrs allowed. But that is not the problem right now.

Checking the raw byte size of the payload will make the code hard to read and understand. In my opinion, is not worth it (have a payload of ~2Mb instead of ~900Kb because we added more multiaddresses)

[ischasny]

LGTM. Thanks for implementing.

[ajnavarro]

@lidel:

1. batch size based on item count is not enough, we may need second one based on byte size (see comment below why)

See my comment about why I think check byte size is not worth-it in that case.

2. server side is does not enforce batch size limits, which is a DoS vector

Is up to the implementation to limit it.

3. limits should be documented in specs

They are here: https://github.com/ipfs/specs/blob/main/reframe/REFRAME_KNOWN_METHODS.md#provide