Add LazyNotFound Store Optimization, Support for fast_slow_store (S3, GCS slow_store targets)

[mkeen]

Description

As requested, this implements an optimization that skips .has() checks on streamed syncs from fast to supported slow stores (S3Store or GCSStore for now) in FastSlowStore.

Fixes #1999

Type of change

• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to
  not work as expected)

How Has This Been Tested?

Some light testing locally. But feedback is requested. I was thinking of writing a new mock/set of tests in nativelink-store/tests/fast_slow_store_test.rs. Let me know if that sounds ok or if there is a better place.

I don't believe a documentation change is needed for this optimization, but correct me if you disagree.

Checklist

• Updated documentation if needed
• Tests added/amended
• bazel test //... passes locally
• PR is contained in a single commit, using git amend see some docs

---

This change is 

[MarcusSorealheis]

would like to see tests

[MarcusSorealheis]

Sorry for the delay here. I'd been squeezing in work the past few days.

Overall, this looks good. I do have concerns that I have mentioned. I know the original issue may have used LazyNotFound but can we be even more explicit in referring to it.

I'm thinking LazyExistenceOnWrite or something that explains what exactly we are lazing about. Let me know your thoughts.

[MarcusSorealheis]

nit: I wonder about LazyNotFound but curious on your thoughts.

[mkeen]

I think that your point is true. Building on your thought, I think the intent is only half-captured via the current name since what we are doing foregoes finding out anything about the file's existence until we attempt to start reading from it in a fast to slow store streamed sync.

I like where you're going with LazyExistenceOnWrite, but the actual optimization happens on reads, and more pedantically, outgoing streams triggered by a caller. Should we go with LazyExistenceOnSync? Or maybe SpeculativeRead. I'm good with any of these including yours, or something else if there are other ideas.

[MarcusSorealheis]

this is overall an awesome PR!