Add SinceLastSend replication mode

[cBournhonesque]

Lightyear had two modes for replication:

• SinceLastAck: send components that changed since the last time we received an ACK from the receiver that the update was received. I believe this is what replicon is doing now
• SinceLastSend: send components that changed since the last time we sent a replication message. This works because in most conditions users have <1% packet loss, so sending a packet is almost equivalent to receiving an ack for it. We then need to be able to detect NACKs, in which case we do re-send updates since the last ack

This serves as a bandwidth optimization to avoid cases where you update a component once and you send multiple updates:

• Send A1
• Send A1 since we didn't receive an ACK
• Send A1 since we didn't receive an ACK
• ACK A1: stop sending

The necessary building blocs for this feature is probably providing an API where the message layer can provide ACK/NACK for individual messages.

[Shatur]

I believe this is what replicon is doing now

Correct!

Yes, I think we should support this. I even think it should be the default way 🤔

[Shatur]

I'd even suggest to make "SinceLastSend" the only mode. I think it's superior and supporting both will needlessly complicate things.

@cBournhonesque if you're interesting to picking it up, I'd welcome a PR. If not - I'll do this myself eventually. This shouldn't be a hard change.

[cBournhonesque]

I can pick it up; I don't think it's that obvious to implement correctly. I don't remember the details, but I think you need NACK support to be able to handle this. (so that if we have a NACK for a given message, we switch to sending from the last known ack instead of the last known send)

Also i'm not sure, but I suspect that some of the assumptions of ServerMutateTicks / Diffs might only be valid under SinceLastAck mode? Maybe not, if ack-handling is done separately.

I'll try to add a bandwidth test or benchmark where we have a component that is updated frequently, to see how much difference this makes.

[Shatur]

think you need NACK support to be able to handle this

Yep! Right now we only send ACKs:

bevy_replicon/src/client.rs

Lines 252 to 260 in 60e3c5e

	if acks_size != 0 {
	let mut acks = Vec::with_capacity(acks_size);
	for message in messages.receive(ServerChannel::Mutations) {
	if let Err(e) = buffer_mutate_message(params, buffered_mutations, message, &mut acks) {
	error!("unable to buffer mutate message: {e}");
	}
	}
	messages.send(ClientChannel::MutationAcks, acks);
	}

I suspect that some of the assumptions of ServerMutateTicks / Diffs might only be valid under SinceLastAck mode?

Hm... ServerMutateTicks is client-only and it just reports which ticks were received. I'd expect this change only affects what server sends. But sending the diffs might be affected, yeah. For diffs the receive buffer is also might be affected since diffs could have gaps.